2024-03-16T09:56:04.256 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-16T09:56:04.256 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} --name pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --verbose --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362/orig.config.yaml 2024-03-16T09:56:04.308 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T09:56:04.399 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362 branch: wip-batrick-testing-20240314.183243 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7602362' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ2FZxf5/LJXZGsV8pmiuXF3Hjd41v7mum45mz3oKFyNtE6oX42CH5CTeJSv3SOumW7IPJxQmor8Zu3lbPhAr+o= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAzFybMcil7BL7NFNbBAVHUUnrirtRtw+GwH9Tp8nJ1BcaKKkr6HkNgN0NTbwykXMLYWg03eBExHTXYoqm/Mk9Q= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHFlL6gCLB5Xa1zmqlMg5/mtIGvkcW8EeXZc+KA/vmSb70RX6OxRF1/PDtnZVk0mTww4mAdtjekKyUzNmXgndPg= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-15_13:00:41 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-16T09:56:04.400 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa; will attempt to use it 2024-03-16T09:56:04.400 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa/tasks 2024-03-16T09:56:04.400 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-16T09:56:04.402 INFO:teuthology.task.internal:Checking packages... 2024-03-16T09:56:04.423 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-16T09:56:04.423 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-16T09:56:04.423 INFO:teuthology.packaging:ref: None 2024-03-16T09:56:04.424 INFO:teuthology.packaging:tag: None 2024-03-16T09:56:04.424 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-16T09:56:04.424 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T09:56:04.424 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-16T09:56:04.646 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-16T09:56:04.648 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-16T09:56:04.654 INFO:teuthology.task.internal:no buildpackages task found 2024-03-16T09:56:04.654 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-16T09:56:04.669 INFO:teuthology.task.internal:Saving configuration 2024-03-16T09:56:04.688 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-16T09:56:04.694 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T09:56:04.716 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 09:51:16.111295', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ2FZxf5/LJXZGsV8pmiuXF3Hjd41v7mum45mz3oKFyNtE6oX42CH5CTeJSv3SOumW7IPJxQmor8Zu3lbPhAr+o='} 2024-03-16T09:56:04.737 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 09:51:16.112595', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAzFybMcil7BL7NFNbBAVHUUnrirtRtw+GwH9Tp8nJ1BcaKKkr6HkNgN0NTbwykXMLYWg03eBExHTXYoqm/Mk9Q='} 2024-03-16T09:56:04.760 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 09:51:16.113766', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHFlL6gCLB5Xa1zmqlMg5/mtIGvkcW8EeXZc+KA/vmSb70RX6OxRF1/PDtnZVk0mTww4mAdtjekKyUzNmXgndPg='} 2024-03-16T09:56:04.760 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-16T09:56:04.767 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-16T09:56:04.767 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-16T09:56:04.767 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-16T09:56:04.767 INFO:teuthology.run_tasks:Running task console_log... 2024-03-16T09:56:04.920 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f17bfd5e160>, signals=[15]) 2024-03-16T09:56:04.921 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-16T09:56:04.927 INFO:teuthology.task.internal:Opening connections... 2024-03-16T09:56:04.927 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T09:56:04.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:56:05.017 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-03-16T09:56:05.018 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:56:05.103 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-03-16T09:56:05.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:56:05.188 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-16T09:56:05.195 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-03-16T09:56:05.213 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-03-16T09:56:05.213 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-03-16T09:56:05.270 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="9" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="9" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el9" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:LOGO="fedora-logo-icon" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T09:56:05.271 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T09:56:05.272 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-03-16T09:56:05.295 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-03-16T09:56:05.312 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-03-16T09:56:05.313 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-03-16T09:56:05.370 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-03-16T09:56:05.370 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2024-03-16T09:56:05.371 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-03-16T09:56:05.371 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-03-16T09:56:05.371 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2024-03-16T09:56:05.371 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2024-03-16T09:56:05.371 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T09:56:05.371 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-03-16T09:56:05.371 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2024-03-16T09:56:05.371 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T09:56:05.371 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-03-16T09:56:05.371 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T09:56:05.371 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T09:56:05.371 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T09:56:05.372 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-03-16T09:56:05.394 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-03-16T09:56:05.412 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-03-16T09:56:05.412 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-03-16T09:56:05.469 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-03-16T09:56:05.469 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="9" 2024-03-16T09:56:05.469 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-03-16T09:56:05.469 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-03-16T09:56:05.469 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="9" 2024-03-16T09:56:05.469 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el9" 2024-03-16T09:56:05.469 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T09:56:05.469 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-03-16T09:56:05.469 INFO:teuthology.orchestra.run.smithi193.stdout:LOGO="fedora-logo-icon" 2024-03-16T09:56:05.469 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T09:56:05.469 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-03-16T09:56:05.469 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T09:56:05.470 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T09:56:05.470 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T09:56:05.470 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-03-16T09:56:05.494 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-16T09:56:05.505 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-16T09:56:05.511 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-16T09:56:05.512 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-03-16T09:56:05.514 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-03-16T09:56:05.517 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-03-16T09:56:05.533 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-16T09:56:05.539 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-16T09:56:05.539 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-03-16T09:56:05.571 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-03-16T09:56:05.574 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-03-16T09:56:05.679 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-16T09:56:05.753 INFO:teuthology.run_tasks:Running task kernel... 2024-03-16T09:56:05.770 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-16T09:56:05.770 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-16T09:56:05.770 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-16T09:56:05.771 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T09:56:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:56:05.787 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-03-16T09:56:05.844 INFO:teuthology.orchestra.run.smithi077.stdout:5.14.0-361.el9.x86_64 2024-03-16T09:56:05.845 INFO:teuthology.task.kernel:Running kernel on smithi077: 5.14.0-361.el9.x86_64 2024-03-16T09:56:05.845 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2024-03-16T09:56:07.995 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-16T09:56:09.370 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - BaseOS 8.3 MB/s | 2.0 MB 00:00 2024-03-16T09:56:10.144 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.0 MB 00:00 2024-03-16T09:56:27.436 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-03-16T09:56:31.624 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.1 MB 00:00 2024-03-16T09:56:33.096 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 15 kB 00:00 2024-03-16T09:56:34.533 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-16T09:56:39.564 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-16T09:56:41.754 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-16T09:56:41.754 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-16T09:56:41.911 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-03-16T09:56:41.922 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-03-16T09:56:41.922 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-03-16T09:56:41.922 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-03-16T09:56:41.922 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-03-16T09:56:41.922 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-16T09:56:41.922 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-03-16T09:56:41.922 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-16T09:56:41.922 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-16T09:56:41.923 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-16T09:56:41.923 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:56:41.923 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-03-16T09:56:41.923 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-03-16T09:56:41.923 INFO:teuthology.orchestra.run.smithi077.stdout:Install 4 Packages 2024-03-16T09:56:41.923 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:56:41.923 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 100 M 2024-03-16T09:56:41.924 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 123 M 2024-03-16T09:56:41.924 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-03-16T09:56:42.682 INFO:teuthology.orchestra.run.smithi077.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 12 MB/s | 6.0 MB 00:00 2024-03-16T09:56:43.116 INFO:teuthology.orchestra.run.smithi077.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 23 MB/s | 21 MB 00:00 2024-03-16T09:56:43.941 INFO:teuthology.orchestra.run.smithi077.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 27 MB/s | 33 MB 00:01 2024-03-16T09:56:45.450 INFO:teuthology.orchestra.run.smithi077.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 12 MB/s | 39 MB 00:03 2024-03-16T09:56:45.450 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-03-16T09:56:45.451 INFO:teuthology.orchestra.run.smithi077.stdout:Total 28 MB/s | 100 MB 00:03 2024-03-16T09:56:46.334 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-03-16T09:56:46.712 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-03-16T09:56:46.712 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-03-16T09:56:48.063 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-03-16T09:56:48.064 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-03-16T09:56:50.728 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-03-16T09:56:51.386 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-16T09:56:51.501 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T09:56:54.295 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T09:56:54.471 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T09:56:59.395 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T09:56:59.581 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:57:04.383 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:57:33.615 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:57:33.615 INFO:teuthology.orchestra.run.smithi077.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T09:57:33.616 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:57:33.700 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:57:34.093 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:57:34.093 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-16T09:57:34.093 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T09:57:34.093 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T09:57:34.558 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T09:57:34.558 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:57:34.558 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-03-16T09:57:34.558 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-16T09:57:34.558 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-16T09:57:34.558 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-16T09:57:34.558 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-16T09:57:34.559 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:57:34.559 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-03-16T09:57:35.032 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T09:57:35.187 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T09:57:35.187 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T09:57:35.187 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-16T09:57:35.188 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T09:57:35.188 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T09:57:35.188 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-16T09:57:35.188 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T09:57:35.188 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T09:57:35.308 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T09:57:35.308 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T09:57:35.308 DEBUG:teuthology.orchestra.run.smithi077:> sudo rpm -qi grub2-tools 2024-03-16T09:57:35.342 INFO:teuthology.orchestra.run.smithi077.stdout:Name : grub2-tools 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Epoch : 1 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Version : 2.06 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Release : 68.el9 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: x86_64 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Group : Unspecified 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Size : 8264254 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:License : GPLv3+ 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Packager : builder@centos.org 2024-03-16T09:57:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:Vendor : CentOS 2024-03-16T09:57:35.344 INFO:teuthology.orchestra.run.smithi077.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T09:57:35.344 INFO:teuthology.orchestra.run.smithi077.stdout:Summary : Support tools for GRUB. 2024-03-16T09:57:35.344 INFO:teuthology.orchestra.run.smithi077.stdout:Description : 2024-03-16T09:57:35.344 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:57:35.344 INFO:teuthology.orchestra.run.smithi077.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T09:57:35.344 INFO:teuthology.orchestra.run.smithi077.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T09:57:35.344 INFO:teuthology.orchestra.run.smithi077.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T09:57:35.344 INFO:teuthology.orchestra.run.smithi077.stdout:hardware devices. 2024-03-16T09:57:35.344 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T09:57:35.344 INFO:teuthology.orchestra.run.smithi077.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T09:57:35.345 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T09:57:35.345 INFO:teuthology.task.kernel:Updating grub on smithi077 to boot 5.14.0-427.el9.x86_64 2024-03-16T09:57:35.345 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T09:57:35.988 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2024-03-16T09:57:37.337 INFO:teuthology.orchestra.run.smithi077.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T09:57:37.370 INFO:teuthology.orchestra.run.smithi077.stderr:done 2024-03-16T09:57:37.372 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-03-16T09:57:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.WDrcamSvx2 2024-03-16T09:57:37.388 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WDrcamSvx2 2024-03-16T09:57:37.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.WDrcamSvx2 2024-03-16T09:57:37.591 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.WDrcamSvx2 is 6KB 2024-03-16T09:57:37.639 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.WDrcamSvx2 2024-03-16T09:57:37.653 DEBUG:teuthology.orchestra.run.smithi077:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T09:57:37.727 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T09:57:37.727 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T09:57:37.727 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T09:57:37.727 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-16T09:57:37.729 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-16T09:57:37.881 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2024-03-16T09:58:07.884 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-16T09:58:07.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T09:58:07.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:58:26.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-03-16T09:58:35.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T09:58:35.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:58:38.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-03-16T09:58:50.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T09:58:50.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:59:08.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-03-16T09:59:23.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T09:59:23.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T09:59:24.256 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-03-16T09:59:24.676 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T09:59:24.676 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-16T09:59:24.676 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-03-16T09:59:24.690 INFO:teuthology.orchestra.run.smithi077.stdout:5.14.0-427.el9.x86_64 2024-03-16T09:59:24.690 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-16T09:59:24.691 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T09:59:24.691 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T09:59:25.691 DEBUG:teuthology.parallel:result is None 2024-03-16T09:59:25.692 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-16T09:59:25.699 INFO:teuthology.task.internal:Creating test directory... 2024-03-16T09:59:25.699 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T09:59:25.701 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T09:59:25.704 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T09:59:25.721 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-16T09:59:25.728 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-16T09:59:25.735 INFO:teuthology.task.internal:Creating archive directory... 2024-03-16T09:59:25.735 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T09:59:25.761 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T09:59:25.763 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T09:59:25.852 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-16T09:59:25.859 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-16T09:59:25.859 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T09:59:25.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:59:25.875 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T09:59:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:59:25.892 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T09:59:25.894 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T09:59:25.919 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T09:59:25.963 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T09:59:25.967 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T09:59:25.982 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T09:59:25.989 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T09:59:26.007 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T09:59:26.024 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T09:59:26.025 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-16T09:59:26.033 INFO:teuthology.task.internal:Configuring sudo... 2024-03-16T09:59:26.033 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T09:59:26.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T09:59:26.038 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T09:59:26.067 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-16T09:59:26.076 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-16T09:59:26.076 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T09:59:26.108 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T09:59:26.110 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T09:59:26.126 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T09:59:26.203 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T09:59:26.254 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T09:59:26.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T09:59:26.412 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T09:59:26.412 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T09:59:26.483 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T09:59:26.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T09:59:26.582 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T09:59:26.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T09:59:26.710 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-16T09:59:26.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T09:59:26.779 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T09:59:26.824 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T09:59:26.889 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T09:59:26.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T09:59:26.976 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-16T09:59:26.976 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T09:59:27.044 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-03-16T09:59:27.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-03-16T09:59:27.049 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-03-16T09:59:27.084 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T09:59:27.113 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T09:59:27.121 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T09:59:27.617 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-16T09:59:27.626 INFO:teuthology.task.internal:Starting timer... 2024-03-16T09:59:27.626 INFO:teuthology.run_tasks:Running task pcp... 2024-03-16T09:59:27.636 INFO:teuthology.run_tasks:Running task selinux... 2024-03-16T09:59:27.644 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-16T09:59:27.645 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-03-16T09:59:27.718 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-03-16T09:59:27.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-03-16T09:59:27.783 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-03-16T09:59:27.785 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-03-16T09:59:27.834 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-03-16T09:59:27.836 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-16T09:59:27.836 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-03-16T09:59:27.853 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-03-16T09:59:27.854 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-03-16T09:59:27.876 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-03-16T09:59:27.877 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-03-16T09:59:27.894 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-03-16T09:59:27.894 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-03-16T09:59:27.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T09:59:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:59:27.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T09:59:27.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:59:27.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T09:59:27.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T09:59:27.984 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-16T09:59:27.984 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-03-16T09:59:28.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-03-16T09:59:28.062 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-03-16T09:59:28.095 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-16T09:59:28.105 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-16T09:59:28.119 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-16T09:59:28.122 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi192.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-16T10:04:06.865 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-03-16T10:04:06.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T10:04:06.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:04:06.960 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-03-16T10:04:07.036 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-03-16T10:04:07.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-16T10:04:07.038 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:04:07.125 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-03-16T10:04:07.197 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-16T10:04:07.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-16T10:04:07.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T10:04:07.287 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-03-16T10:04:07.358 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-16T10:04:07.359 INFO:teuthology.run_tasks:Running task clock... 2024-03-16T10:04:07.376 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-16T10:04:07.376 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T10:04:07.377 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T10:04:07.379 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T10:04:07.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T10:04:07.382 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T10:04:07.382 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T10:04:07.407 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T10:04:07.412 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T10:04:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T10:04:07.430 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T10:04:07.433 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T10:04:07.450 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T10:04:07.452 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-03-16T10:04:07.460 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-03-16T10:04:07.465 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-03-16T10:04:07.474 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-03-16T10:04:07.481 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-03-16T10:04:07.482 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T10:04:07.491 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T10:04:07.495 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-03-16T10:04:07.498 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T10:04:07.507 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T10:04:07.512 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T10:04:07.528 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T10:04:07.547 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-03-16T10:04:07.550 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T10:04:07.551 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-03-16T10:04:07.551 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:04:07.551 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:04:07.551 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:04:07.551 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:04:07.552 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-03-16T10:04:07.554 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T10:04:07.554 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-03-16T10:04:07.554 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:04:07.555 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:04:07.555 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:04:07.555 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:04:07.572 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-03-16T10:04:07.574 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T10:04:07.575 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-16T10:04:07.575 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:04:07.575 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:04:07.575 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:04:07.575 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:04:07.576 INFO:teuthology.run_tasks:Running task install... 2024-03-16T10:04:07.584 DEBUG:teuthology.task.install:project ceph 2024-03-16T10:04:07.584 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-16T10:04:07.584 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-16T10:04:07.584 INFO:teuthology.task.install:Using flavor: default 2024-03-16T10:04:07.592 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-16T10:04:07.593 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-16T10:04:07.593 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T10:04:07.593 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:04:07.595 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T10:04:07.596 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:04:07.597 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T10:04:07.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:04:07.740 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T10:04:07.740 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T10:04:07.744 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T10:04:07.744 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T10:04:07.813 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T10:04:07.813 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T10:04:07.853 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T10:04:07.854 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-16T10:04:07.854 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T10:04:07.881 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T10:04:07.881 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T10:04:07.909 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T10:04:07.909 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-16T10:04:07.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T10:04:07.920 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T10:04:07.920 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:04:07.920 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T10:04:07.938 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T10:04:07.938 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T10:04:07.948 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T10:04:07.949 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T10:04:07.949 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T10:04:08.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T10:04:08.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T10:04:08.027 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T10:04:08.086 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T10:04:08.093 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-03-16T10:04:08.094 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-03-16T10:04:08.097 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T10:04:08.151 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-03-16T10:04:08.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-03-16T10:04:08.164 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2024-03-16T10:04:08.166 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2024-03-16T10:04:08.328 INFO:teuthology.orchestra.run.smithi193.stdout:74 files removed 2024-03-16T10:04:08.355 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T10:04:08.386 INFO:teuthology.orchestra.run.smithi192.stdout:74 files removed 2024-03-16T10:04:08.401 INFO:teuthology.orchestra.run.smithi077.stdout:74 files removed 2024-03-16T10:04:08.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T10:04:08.427 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T10:04:09.094 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 238 kB/s | 83 kB 00:00 2024-03-16T10:04:09.120 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 174 kB/s | 83 kB 00:00 2024-03-16T10:04:09.154 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 185 kB/s | 83 kB 00:00 2024-03-16T10:04:09.436 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 34 kB/s | 10 kB 00:00 2024-03-16T10:04:09.529 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-16T10:04:09.537 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-16T10:04:09.787 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-16T10:04:09.929 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-16T10:04:09.946 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-16T10:04:10.190 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-16T10:04:10.550 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-16T10:04:10.573 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-16T10:04:11.681 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-16T10:04:11.966 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-16T10:04:12.215 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-16T10:04:12.556 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-16T10:04:13.298 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-16T10:04:13.307 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 7.9 MB/s | 8.0 MB 00:01 2024-03-16T10:04:15.349 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-16T10:04:15.369 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-16T10:04:16.051 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 7.7 MB/s | 19 MB 00:02 2024-03-16T10:04:19.668 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.1 MB 00:00 2024-03-16T10:04:20.227 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.1 MB 00:00 2024-03-16T10:04:21.208 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 15 kB 00:00 2024-03-16T10:04:22.511 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 15 kB 00:01 2024-03-16T10:04:22.929 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-16T10:04:23.864 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-16T10:04:26.892 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 778 kB/s | 6.1 MB 00:08 2024-03-16T10:04:27.985 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-16T10:04:28.343 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 15 kB 00:00 2024-03-16T10:04:28.704 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-16T10:04:29.320 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-16T10:04:30.449 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T10:04:30.450 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T10:04:30.464 INFO:teuthology.orchestra.run.smithi193.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T10:04:30.465 INFO:teuthology.orchestra.run.smithi193.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T10:04:30.467 INFO:teuthology.orchestra.run.smithi193.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T10:04:30.468 INFO:teuthology.orchestra.run.smithi193.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T10:04:30.470 INFO:teuthology.orchestra.run.smithi193.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T10:04:30.473 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T10:04:30.474 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T10:04:30.555 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-16T10:04:30.570 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T10:04:30.571 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T10:04:30.572 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T10:04:30.573 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T10:04:30.574 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T10:04:30.575 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T10:04:30.576 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T10:04:30.577 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T10:04:30.578 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T10:04:30.579 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T10:04:30.580 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T10:04:30.581 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T10:04:30.582 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout:Installing weak dependencies: 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout:Install 150 Packages 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 24 Packages 2024-03-16T10:04:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:04:30.584 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 224 M 2024-03-16T10:04:30.584 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-16T10:04:31.298 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T10:04:31.299 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T10:04:31.310 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T10:04:31.311 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T10:04:31.312 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T10:04:31.313 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T10:04:31.315 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T10:04:31.317 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T10:04:31.318 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T10:04:31.397 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-03-16T10:04:31.410 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-03-16T10:04:31.410 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-03-16T10:04:31.410 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-03-16T10:04:31.410 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-03-16T10:04:31.410 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T10:04:31.411 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T10:04:31.412 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T10:04:31.413 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T10:04:31.414 INFO:teuthology.orchestra.run.smithi077.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T10:04:31.415 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T10:04:31.416 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T10:04:31.417 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T10:04:31.418 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T10:04:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T10:04:31.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T10:04:31.421 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T10:04:31.422 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T10:04:31.423 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:04:31.424 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-03-16T10:04:31.424 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-03-16T10:04:31.424 INFO:teuthology.orchestra.run.smithi077.stdout:Install 150 Packages 2024-03-16T10:04:31.424 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 24 Packages 2024-03-16T10:04:31.424 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:04:31.424 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 224 M 2024-03-16T10:04:31.424 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-03-16T10:04:31.503 INFO:teuthology.orchestra.run.smithi193.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2024-03-16T10:04:31.703 INFO:teuthology.orchestra.run.smithi193.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 4.4 MB/s | 890 kB 00:00 2024-03-16T10:04:31.745 INFO:teuthology.orchestra.run.smithi193.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 3.4 MB/s | 145 kB 00:00 2024-03-16T10:04:31.871 INFO:teuthology.orchestra.run.smithi077.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-03-16T10:04:32.053 INFO:teuthology.orchestra.run.smithi193.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 7.7 MB/s | 2.4 MB 00:00 2024-03-16T10:04:32.122 INFO:teuthology.orchestra.run.smithi077.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 3.5 MB/s | 890 kB 00:00 2024-03-16T10:04:32.189 INFO:teuthology.orchestra.run.smithi077.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 2.1 MB/s | 145 kB 00:00 2024-03-16T10:04:32.262 INFO:teuthology.orchestra.run.smithi193.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 7.2 MB/s | 1.5 MB 00:00 2024-03-16T10:04:32.781 INFO:teuthology.orchestra.run.smithi077.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 4.0 MB/s | 2.4 MB 00:00 2024-03-16T10:04:32.795 INFO:teuthology.orchestra.run.smithi193.stdout:(6/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 3.7 MB/s | 5.4 MB 00:01 2024-03-16T10:04:32.946 INFO:teuthology.orchestra.run.smithi193.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 6.9 MB/s | 4.7 MB 00:00 2024-03-16T10:04:33.181 INFO:teuthology.orchestra.run.smithi077.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 3.7 MB/s | 1.5 MB 00:00 2024-03-16T10:04:33.523 INFO:teuthology.orchestra.run.smithi077.stdout:(6/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 2.9 MB/s | 5.4 MB 00:01 2024-03-16T10:04:34.286 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-16T10:04:34.746 INFO:teuthology.orchestra.run.smithi193.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 5.5 MB/s | 9.8 MB 00:01 2024-03-16T10:04:34.829 INFO:teuthology.orchestra.run.smithi193.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 303 kB/s | 25 kB 00:00 2024-03-16T10:04:35.046 INFO:teuthology.orchestra.run.smithi193.stdout:(10/174): ceph-common-19.0.0-2162.g3d78f5ba.el9 5.6 MB/s | 21 MB 00:03 2024-03-16T10:04:35.146 INFO:teuthology.orchestra.run.smithi193.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.2 MB/s | 221 kB 00:00 2024-03-16T10:04:35.230 INFO:teuthology.orchestra.run.smithi193.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 325 kB/s | 27 kB 00:00 2024-03-16T10:04:35.305 INFO:teuthology.orchestra.run.smithi193.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 423 kB/s | 32 kB 00:00 2024-03-16T10:04:35.480 INFO:teuthology.orchestra.run.smithi193.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 4.2 MB/s | 747 kB 00:00 2024-03-16T10:04:35.514 INFO:teuthology.orchestra.run.smithi077.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 2.0 MB/s | 4.7 MB 00:02 2024-03-16T10:04:35.572 INFO:teuthology.orchestra.run.smithi193.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.7 MB/s | 163 kB 00:00 2024-03-16T10:04:35.681 INFO:teuthology.orchestra.run.smithi193.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.1 MB/s | 127 kB 00:00 2024-03-16T10:04:35.773 INFO:teuthology.orchestra.run.smithi193.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 5.3 MB/s | 501 kB 00:00 2024-03-16T10:04:35.790 INFO:teuthology.orchestra.run.smithi077.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 5.0 MB/s | 21 MB 00:04 2024-03-16T10:04:35.873 INFO:teuthology.orchestra.run.smithi077.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 303 kB/s | 25 kB 00:00 2024-03-16T10:04:36.198 INFO:teuthology.orchestra.run.smithi193.stdout:(18/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 4.9 MB/s | 17 MB 00:03 2024-03-16T10:04:36.298 INFO:teuthology.orchestra.run.smithi193.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 451 kB/s | 45 kB 00:00 2024-03-16T10:04:36.407 INFO:teuthology.orchestra.run.smithi193.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.2 MB/s | 131 kB 00:00 2024-03-16T10:04:36.482 INFO:teuthology.orchestra.run.smithi193.stdout:(21/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 7.2 MB/s | 5.1 MB 00:00 2024-03-16T10:04:36.508 INFO:teuthology.orchestra.run.smithi193.stdout:(22/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.6 MB/s | 162 kB 00:00 2024-03-16T10:04:36.566 INFO:teuthology.orchestra.run.smithi193.stdout:(23/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.7 MB/s | 320 kB 00:00 2024-03-16T10:04:36.592 INFO:teuthology.orchestra.run.smithi193.stdout:(24/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.5 MB/s | 298 kB 00:00 2024-03-16T10:04:36.634 INFO:teuthology.orchestra.run.smithi193.stdout:(25/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.4 MB/s | 99 kB 00:00 2024-03-16T10:04:36.640 INFO:teuthology.orchestra.run.smithi077.stdout:(10/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 5.4 MB/s | 17 MB 00:03 2024-03-16T10:04:36.667 INFO:teuthology.orchestra.run.smithi193.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-16T10:04:36.741 INFO:teuthology.orchestra.run.smithi077.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.2 MB/s | 221 kB 00:00 2024-03-16T10:04:36.776 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T10:04:36.777 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T10:04:36.794 INFO:teuthology.orchestra.run.smithi193.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.3 MB/s | 170 kB 00:00 2024-03-16T10:04:36.794 INFO:teuthology.orchestra.run.smithi192.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T10:04:36.794 INFO:teuthology.orchestra.run.smithi192.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T10:04:36.795 INFO:teuthology.orchestra.run.smithi192.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T10:04:36.795 INFO:teuthology.orchestra.run.smithi192.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T10:04:36.796 INFO:teuthology.orchestra.run.smithi192.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T10:04:36.799 INFO:teuthology.orchestra.run.smithi192.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T10:04:36.800 INFO:teuthology.orchestra.run.smithi192.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T10:04:36.824 INFO:teuthology.orchestra.run.smithi077.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 325 kB/s | 27 kB 00:00 2024-03-16T10:04:36.879 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-16T10:04:36.893 INFO:teuthology.orchestra.run.smithi193.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 339 kB/s | 34 kB 00:00 2024-03-16T10:04:36.906 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-16T10:04:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-16T10:04:36.906 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-16T10:04:36.906 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-16T10:04:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T10:04:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T10:04:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T10:04:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T10:04:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T10:04:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T10:04:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T10:04:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T10:04:36.913 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T10:04:36.914 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T10:04:36.915 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T10:04:36.916 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T10:04:36.917 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T10:04:36.918 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T10:04:36.919 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T10:04:36.919 INFO:teuthology.orchestra.run.smithi192.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T10:04:36.919 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T10:04:36.919 INFO:teuthology.orchestra.run.smithi192.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T10:04:36.919 INFO:teuthology.orchestra.run.smithi192.stdout:Installing weak dependencies: 2024-03-16T10:04:36.919 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T10:04:36.919 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:04:36.919 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-16T10:04:36.919 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-16T10:04:36.919 INFO:teuthology.orchestra.run.smithi192.stdout:Install 150 Packages 2024-03-16T10:04:36.919 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 24 Packages 2024-03-16T10:04:36.919 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:04:36.920 INFO:teuthology.orchestra.run.smithi077.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 346 kB/s | 32 kB 00:00 2024-03-16T10:04:36.920 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 224 M 2024-03-16T10:04:36.920 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-16T10:04:37.001 INFO:teuthology.orchestra.run.smithi193.stdout:(29/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 8.4 MB/s | 3.1 MB 00:00 2024-03-16T10:04:37.027 INFO:teuthology.orchestra.run.smithi193.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.0 MB/s | 139 kB 00:00 2024-03-16T10:04:37.066 INFO:teuthology.orchestra.run.smithi077.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 4.9 MB/s | 747 kB 00:00 2024-03-16T10:04:37.167 INFO:teuthology.orchestra.run.smithi077.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.6 MB/s | 163 kB 00:00 2024-03-16T10:04:37.258 INFO:teuthology.orchestra.run.smithi077.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.4 MB/s | 127 kB 00:00 2024-03-16T10:04:37.260 INFO:teuthology.orchestra.run.smithi193.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 6.9 MB/s | 1.8 MB 00:00 2024-03-16T10:04:37.369 INFO:teuthology.orchestra.run.smithi193.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.2 MB/s | 248 kB 00:00 2024-03-16T10:04:37.400 INFO:teuthology.orchestra.run.smithi077.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 3.5 MB/s | 501 kB 00:00 2024-03-16T10:04:37.444 INFO:teuthology.orchestra.run.smithi193.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 658 kB/s | 49 kB 00:00 2024-03-16T10:04:37.544 INFO:teuthology.orchestra.run.smithi193.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 164 kB/s | 16 kB 00:00 2024-03-16T10:04:37.653 INFO:teuthology.orchestra.run.smithi193.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 2.4 MB/s | 264 kB 00:00 2024-03-16T10:04:37.837 INFO:teuthology.orchestra.run.smithi193.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.0 MB/s | 748 kB 00:00 2024-03-16T10:04:37.912 INFO:teuthology.orchestra.run.smithi193.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 244 kB/s | 18 kB 00:00 2024-03-16T10:04:38.054 INFO:teuthology.orchestra.run.smithi193.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-16T10:04:38.112 INFO:teuthology.orchestra.run.smithi193.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-16T10:04:38.146 INFO:teuthology.orchestra.run.smithi193.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-16T10:04:38.188 INFO:teuthology.orchestra.run.smithi193.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-16T10:04:38.272 INFO:teuthology.orchestra.run.smithi193.stdout:(42/174): ceph-mgr-diskprediction-local-19.0.0- 6.0 MB/s | 7.4 MB 00:01 2024-03-16T10:04:38.289 INFO:teuthology.orchestra.run.smithi193.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 315 kB/s | 32 kB 00:00 2024-03-16T10:04:38.305 INFO:teuthology.orchestra.run.smithi192.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-03-16T10:04:38.317 INFO:teuthology.orchestra.run.smithi077.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 5.6 MB/s | 5.1 MB 00:00 2024-03-16T10:04:38.364 INFO:teuthology.orchestra.run.smithi193.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-16T10:04:38.376 INFO:teuthology.orchestra.run.smithi077.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 777 kB/s | 45 kB 00:00 2024-03-16T10:04:38.380 INFO:teuthology.orchestra.run.smithi193.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 188 kB/s | 17 kB 00:00 2024-03-16T10:04:38.398 INFO:teuthology.orchestra.run.smithi193.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 299 kB/s | 10 kB 00:00 2024-03-16T10:04:38.423 INFO:teuthology.orchestra.run.smithi193.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-16T10:04:38.440 INFO:teuthology.orchestra.run.smithi193.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-16T10:04:38.442 INFO:teuthology.orchestra.run.smithi077.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.9 MB/s | 131 kB 00:00 2024-03-16T10:04:38.474 INFO:teuthology.orchestra.run.smithi193.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-16T10:04:38.507 INFO:teuthology.orchestra.run.smithi193.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-16T10:04:38.524 INFO:teuthology.orchestra.run.smithi193.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 940 kB/s | 46 kB 00:00 2024-03-16T10:04:38.526 INFO:teuthology.orchestra.run.smithi077.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 162 kB 00:00 2024-03-16T10:04:38.550 INFO:teuthology.orchestra.run.smithi193.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-16T10:04:38.575 INFO:teuthology.orchestra.run.smithi193.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-16T10:04:38.600 INFO:teuthology.orchestra.run.smithi193.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-16T10:04:38.613 INFO:teuthology.orchestra.run.smithi192.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 2.8 MB/s | 890 kB 00:00 2024-03-16T10:04:38.617 INFO:teuthology.orchestra.run.smithi193.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 818 kB/s | 34 kB 00:00 2024-03-16T10:04:38.618 INFO:teuthology.orchestra.run.smithi077.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.4 MB/s | 320 kB 00:00 2024-03-16T10:04:38.663 INFO:teuthology.orchestra.run.smithi192.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 2.8 MB/s | 145 kB 00:00 2024-03-16T10:04:38.710 INFO:teuthology.orchestra.run.smithi077.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.2 MB/s | 298 kB 00:00 2024-03-16T10:04:38.734 INFO:teuthology.orchestra.run.smithi193.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 35 MB/s | 4.7 MB 00:00 2024-03-16T10:04:38.776 INFO:teuthology.orchestra.run.smithi193.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 9.6 MB/s | 1.5 MB 00:00 2024-03-16T10:04:38.785 INFO:teuthology.orchestra.run.smithi077.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.3 MB/s | 99 kB 00:00 2024-03-16T10:04:38.810 INFO:teuthology.orchestra.run.smithi193.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-16T10:04:38.827 INFO:teuthology.orchestra.run.smithi193.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.7 MB/s | 134 kB 00:00 2024-03-16T10:04:38.894 INFO:teuthology.orchestra.run.smithi077.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 788 kB/s | 85 kB 00:00 2024-03-16T10:04:38.952 INFO:teuthology.orchestra.run.smithi193.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-16T10:04:38.986 INFO:teuthology.orchestra.run.smithi193.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-16T10:04:39.011 INFO:teuthology.orchestra.run.smithi193.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-16T10:04:39.019 INFO:teuthology.orchestra.run.smithi077.stdout:(26/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 2.8 MB/s | 9.8 MB 00:03 2024-03-16T10:04:39.120 INFO:teuthology.orchestra.run.smithi193.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 43 MB/s | 5.8 MB 00:00 2024-03-16T10:04:39.145 INFO:teuthology.orchestra.run.smithi193.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-03-16T10:04:39.169 INFO:teuthology.orchestra.run.smithi077.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.1 MB/s | 170 kB 00:00 2024-03-16T10:04:39.170 INFO:teuthology.orchestra.run.smithi193.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-16T10:04:39.195 INFO:teuthology.orchestra.run.smithi193.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-16T10:04:39.197 INFO:teuthology.orchestra.run.smithi192.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 4.4 MB/s | 2.4 MB 00:00 2024-03-16T10:04:39.220 INFO:teuthology.orchestra.run.smithi077.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 682 kB/s | 34 kB 00:00 2024-03-16T10:04:39.220 INFO:teuthology.orchestra.run.smithi193.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 632 kB/s | 32 kB 00:00 2024-03-16T10:04:39.338 INFO:teuthology.orchestra.run.smithi193.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-16T10:04:39.362 INFO:teuthology.orchestra.run.smithi077.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 980 kB/s | 139 kB 00:00 2024-03-16T10:04:39.363 INFO:teuthology.orchestra.run.smithi193.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-16T10:04:39.388 INFO:teuthology.orchestra.run.smithi193.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-16T10:04:39.414 INFO:teuthology.orchestra.run.smithi193.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-16T10:04:39.430 INFO:teuthology.orchestra.run.smithi193.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 989 kB/s | 41 kB 00:00 2024-03-16T10:04:39.456 INFO:teuthology.orchestra.run.smithi193.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-16T10:04:39.481 INFO:teuthology.orchestra.run.smithi193.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-16T10:04:39.504 INFO:teuthology.orchestra.run.smithi193.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.3 MB/s | 211 kB 00:00 2024-03-16T10:04:39.529 INFO:teuthology.orchestra.run.smithi193.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 997 kB/s | 47 kB 00:00 2024-03-16T10:04:39.554 INFO:teuthology.orchestra.run.smithi193.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 853 kB/s | 43 kB 00:00 2024-03-16T10:04:39.564 INFO:teuthology.orchestra.run.smithi192.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 4.1 MB/s | 1.5 MB 00:00 2024-03-16T10:04:39.662 INFO:teuthology.orchestra.run.smithi077.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 4.0 MB/s | 3.1 MB 00:00 2024-03-16T10:04:39.697 INFO:teuthology.orchestra.run.smithi193.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 276 kB/s | 39 kB 00:00 2024-03-16T10:04:39.847 INFO:teuthology.orchestra.run.smithi193.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-03-16T10:04:39.872 INFO:teuthology.orchestra.run.smithi193.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 237 kB/s | 41 kB 00:00 2024-03-16T10:04:39.897 INFO:teuthology.orchestra.run.smithi193.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-16T10:04:39.923 INFO:teuthology.orchestra.run.smithi193.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-16T10:04:39.948 INFO:teuthology.orchestra.run.smithi193.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-16T10:04:39.973 INFO:teuthology.orchestra.run.smithi193.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2024-03-16T10:04:39.998 INFO:teuthology.orchestra.run.smithi193.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-16T10:04:40.024 INFO:teuthology.orchestra.run.smithi193.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-16T10:04:40.056 INFO:teuthology.orchestra.run.smithi192.stdout:(6/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 2.5 MB/s | 5.4 MB 00:02 2024-03-16T10:04:40.057 INFO:teuthology.orchestra.run.smithi193.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-16T10:04:40.074 INFO:teuthology.orchestra.run.smithi193.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-03-16T10:04:40.100 INFO:teuthology.orchestra.run.smithi193.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-16T10:04:40.125 INFO:teuthology.orchestra.run.smithi193.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 187 kB 00:00 2024-03-16T10:04:40.250 INFO:teuthology.orchestra.run.smithi193.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-16T10:04:40.301 INFO:teuthology.orchestra.run.smithi193.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-16T10:04:40.326 INFO:teuthology.orchestra.run.smithi193.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-03-16T10:04:40.343 INFO:teuthology.orchestra.run.smithi193.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-03-16T10:04:40.412 INFO:teuthology.orchestra.run.smithi077.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 1.7 MB/s | 1.8 MB 00:01 2024-03-16T10:04:40.573 INFO:teuthology.orchestra.run.smithi192.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 4.7 MB/s | 4.7 MB 00:01 2024-03-16T10:04:40.596 INFO:teuthology.orchestra.run.smithi077.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 1.3 MB/s | 248 kB 00:00 2024-03-16T10:04:40.671 INFO:teuthology.orchestra.run.smithi077.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 660 kB/s | 49 kB 00:00 2024-03-16T10:04:40.755 INFO:teuthology.orchestra.run.smithi077.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 197 kB/s | 16 kB 00:00 2024-03-16T10:04:40.902 INFO:teuthology.orchestra.run.smithi193.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 37 kB/s | 21 kB 00:00 2024-03-16T10:04:40.951 INFO:teuthology.orchestra.run.smithi193.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 42 kB/s | 25 kB 00:00 2024-03-16T10:04:40.971 INFO:teuthology.orchestra.run.smithi077.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 1.2 MB/s | 264 kB 00:00 2024-03-16T10:04:41.297 INFO:teuthology.orchestra.run.smithi077.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 4.5 MB/s | 7.4 MB 00:01 2024-03-16T10:04:41.372 INFO:teuthology.orchestra.run.smithi077.stdout:(37/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 1.8 MB/s | 748 kB 00:00 2024-03-16T10:04:41.397 INFO:teuthology.orchestra.run.smithi077.stdout:(38/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 182 kB/s | 18 kB 00:00 2024-03-16T10:04:41.406 INFO:teuthology.orchestra.run.smithi192.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 5.8 MB/s | 21 MB 00:03 2024-03-16T10:04:41.456 INFO:teuthology.orchestra.run.smithi192.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 505 kB/s | 25 kB 00:00 2024-03-16T10:04:41.640 INFO:teuthology.orchestra.run.smithi077.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-16T10:04:41.673 INFO:teuthology.orchestra.run.smithi077.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 3.3 MB/s | 920 kB 00:00 2024-03-16T10:04:41.698 INFO:teuthology.orchestra.run.smithi077.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-16T10:04:41.723 INFO:teuthology.orchestra.run.smithi077.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-16T10:04:41.749 INFO:teuthology.orchestra.run.smithi077.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 639 kB/s | 32 kB 00:00 2024-03-16T10:04:41.841 INFO:teuthology.orchestra.run.smithi077.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-16T10:04:41.866 INFO:teuthology.orchestra.run.smithi077.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 147 kB/s | 17 kB 00:00 2024-03-16T10:04:41.891 INFO:teuthology.orchestra.run.smithi077.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-16T10:04:41.917 INFO:teuthology.orchestra.run.smithi077.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-16T10:04:41.942 INFO:teuthology.orchestra.run.smithi077.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 890 kB/s | 44 kB 00:00 2024-03-16T10:04:41.967 INFO:teuthology.orchestra.run.smithi077.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-16T10:04:42.003 INFO:teuthology.orchestra.run.smithi193.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 234 kB/s | 245 kB 00:01 2024-03-16T10:04:42.009 INFO:teuthology.orchestra.run.smithi077.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 578 kB 00:00 2024-03-16T10:04:42.034 INFO:teuthology.orchestra.run.smithi077.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 694 kB/s | 46 kB 00:00 2024-03-16T10:04:42.068 INFO:teuthology.orchestra.run.smithi077.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-16T10:04:42.093 INFO:teuthology.orchestra.run.smithi077.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-16T10:04:42.127 INFO:teuthology.orchestra.run.smithi077.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-16T10:04:42.160 INFO:teuthology.orchestra.run.smithi077.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 510 kB/s | 34 kB 00:00 2024-03-16T10:04:42.273 INFO:teuthology.orchestra.run.smithi192.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 5.8 MB/s | 9.8 MB 00:01 2024-03-16T10:04:42.349 INFO:teuthology.orchestra.run.smithi192.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.9 MB/s | 221 kB 00:00 2024-03-16T10:04:42.360 INFO:teuthology.orchestra.run.smithi077.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-03-16T10:04:42.402 INFO:teuthology.orchestra.run.smithi077.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 6.3 MB/s | 1.5 MB 00:00 2024-03-16T10:04:42.415 INFO:teuthology.orchestra.run.smithi192.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 405 kB/s | 27 kB 00:00 2024-03-16T10:04:42.428 INFO:teuthology.orchestra.run.smithi077.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-16T10:04:42.453 INFO:teuthology.orchestra.run.smithi077.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.7 MB/s | 134 kB 00:00 2024-03-16T10:04:42.474 INFO:teuthology.orchestra.run.smithi192.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 543 kB/s | 32 kB 00:00 2024-03-16T10:04:42.570 INFO:teuthology.orchestra.run.smithi077.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-16T10:04:42.604 INFO:teuthology.orchestra.run.smithi077.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.8 MB/s | 1.0 MB 00:00 2024-03-16T10:04:42.616 INFO:teuthology.orchestra.run.smithi192.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 5.1 MB/s | 747 kB 00:00 2024-03-16T10:04:42.629 INFO:teuthology.orchestra.run.smithi077.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-16T10:04:42.692 INFO:teuthology.orchestra.run.smithi192.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.1 MB/s | 163 kB 00:00 2024-03-16T10:04:42.775 INFO:teuthology.orchestra.run.smithi192.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.5 MB/s | 127 kB 00:00 2024-03-16T10:04:42.779 INFO:teuthology.orchestra.run.smithi077.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 5.8 MB 00:00 2024-03-16T10:04:42.796 INFO:teuthology.orchestra.run.smithi077.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 148 kB/s | 24 kB 00:00 2024-03-16T10:04:42.813 INFO:teuthology.orchestra.run.smithi077.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 543 kB/s | 18 kB 00:00 2024-03-16T10:04:42.838 INFO:teuthology.orchestra.run.smithi077.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-16T10:04:42.855 INFO:teuthology.orchestra.run.smithi077.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 761 kB/s | 32 kB 00:00 2024-03-16T10:04:42.901 INFO:teuthology.orchestra.run.smithi192.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 3.9 MB/s | 501 kB 00:00 2024-03-16T10:04:42.989 INFO:teuthology.orchestra.run.smithi077.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-16T10:04:43.014 INFO:teuthology.orchestra.run.smithi077.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-16T10:04:43.040 INFO:teuthology.orchestra.run.smithi077.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-16T10:04:43.065 INFO:teuthology.orchestra.run.smithi077.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-16T10:04:43.090 INFO:teuthology.orchestra.run.smithi077.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 821 kB/s | 41 kB 00:00 2024-03-16T10:04:43.115 INFO:teuthology.orchestra.run.smithi077.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-16T10:04:43.149 INFO:teuthology.orchestra.run.smithi077.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-16T10:04:43.174 INFO:teuthology.orchestra.run.smithi077.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-16T10:04:43.225 INFO:teuthology.orchestra.run.smithi077.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 631 kB/s | 47 kB 00:00 2024-03-16T10:04:43.250 INFO:teuthology.orchestra.run.smithi077.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2024-03-16T10:04:43.319 INFO:teuthology.orchestra.run.smithi193.stdout:(98/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 5.6 MB/s | 48 MB 00:08 2024-03-16T10:04:43.344 INFO:teuthology.orchestra.run.smithi193.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 185 kB/s | 248 kB 00:01 2024-03-16T10:04:43.508 INFO:teuthology.orchestra.run.smithi077.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 149 kB/s | 39 kB 00:00 2024-03-16T10:04:43.618 INFO:teuthology.orchestra.run.smithi192.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 7.2 MB/s | 5.1 MB 00:00 2024-03-16T10:04:43.676 INFO:teuthology.orchestra.run.smithi192.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 774 kB/s | 45 kB 00:00 2024-03-16T10:04:43.717 INFO:teuthology.orchestra.run.smithi077.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-16T10:04:43.742 INFO:teuthology.orchestra.run.smithi077.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 176 kB/s | 41 kB 00:00 2024-03-16T10:04:43.743 INFO:teuthology.orchestra.run.smithi192.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.9 MB/s | 131 kB 00:00 2024-03-16T10:04:43.768 INFO:teuthology.orchestra.run.smithi077.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-16T10:04:43.793 INFO:teuthology.orchestra.run.smithi077.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-16T10:04:43.810 INFO:teuthology.orchestra.run.smithi192.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 2.4 MB/s | 162 kB 00:00 2024-03-16T10:04:43.818 INFO:teuthology.orchestra.run.smithi077.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-16T10:04:43.843 INFO:teuthology.orchestra.run.smithi077.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-16T10:04:43.869 INFO:teuthology.orchestra.run.smithi077.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-16T10:04:43.885 INFO:teuthology.orchestra.run.smithi192.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 4.2 MB/s | 320 kB 00:00 2024-03-16T10:04:43.894 INFO:teuthology.orchestra.run.smithi077.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-16T10:04:43.952 INFO:teuthology.orchestra.run.smithi192.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 4.4 MB/s | 298 kB 00:00 2024-03-16T10:04:43.997 INFO:teuthology.orchestra.run.smithi077.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 9.0 MB/s | 1.1 MB 00:00 2024-03-16T10:04:44.002 INFO:teuthology.orchestra.run.smithi193.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 53 kB/s | 35 kB 00:00 2024-03-16T10:04:44.011 INFO:teuthology.orchestra.run.smithi192.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.7 MB/s | 99 kB 00:00 2024-03-16T10:04:44.078 INFO:teuthology.orchestra.run.smithi192.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-16T10:04:44.095 INFO:teuthology.orchestra.run.smithi077.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 517 kB/s | 103 kB 00:00 2024-03-16T10:04:44.120 INFO:teuthology.orchestra.run.smithi077.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 1.0 MB/s | 124 kB 00:00 2024-03-16T10:04:44.145 INFO:teuthology.orchestra.run.smithi077.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-16T10:04:44.270 INFO:teuthology.orchestra.run.smithi193.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 228 kB/s | 216 kB 00:00 2024-03-16T10:04:44.270 INFO:teuthology.orchestra.run.smithi077.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-16T10:04:44.329 INFO:teuthology.orchestra.run.smithi077.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-03-16T10:04:44.354 INFO:teuthology.orchestra.run.smithi077.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-16T10:04:44.379 INFO:teuthology.orchestra.run.smithi077.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-16T10:04:44.447 INFO:teuthology.orchestra.run.smithi077.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 234 kB/s | 21 kB 00:00 2024-03-16T10:04:44.472 INFO:teuthology.orchestra.run.smithi077.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 277 kB/s | 25 kB 00:00 2024-03-16T10:04:44.520 INFO:teuthology.orchestra.run.smithi192.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 7.0 MB/s | 3.1 MB 00:00 2024-03-16T10:04:44.579 INFO:teuthology.orchestra.run.smithi192.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.9 MB/s | 170 kB 00:00 2024-03-16T10:04:44.612 INFO:teuthology.orchestra.run.smithi193.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 73 kB/s | 44 kB 00:00 2024-03-16T10:04:44.614 INFO:teuthology.orchestra.run.smithi077.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-16T10:04:44.637 INFO:teuthology.orchestra.run.smithi192.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 583 kB/s | 34 kB 00:00 2024-03-16T10:04:44.687 INFO:teuthology.orchestra.run.smithi193.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 298 kB/s | 22 kB 00:00 2024-03-16T10:04:44.696 INFO:teuthology.orchestra.run.smithi192.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 2.3 MB/s | 139 kB 00:00 2024-03-16T10:04:44.697 INFO:teuthology.orchestra.run.smithi077.stdout:(98/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 16 MB/s | 4.1 MB 00:00 2024-03-16T10:04:44.804 INFO:teuthology.orchestra.run.smithi193.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 7.2 MB/s | 863 kB 00:00 2024-03-16T10:04:44.938 INFO:teuthology.orchestra.run.smithi192.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 7.4 MB/s | 1.8 MB 00:00 2024-03-16T10:04:45.012 INFO:teuthology.orchestra.run.smithi193.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 126 kB/s | 94 kB 00:00 2024-03-16T10:04:45.054 INFO:teuthology.orchestra.run.smithi193.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.1 MB/s | 548 kB 00:00 2024-03-16T10:04:45.106 INFO:teuthology.orchestra.run.smithi077.stdout:(99/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 5.2 MB/s | 48 MB 00:09 2024-03-16T10:04:45.129 INFO:teuthology.orchestra.run.smithi193.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-16T10:04:45.140 INFO:teuthology.orchestra.run.smithi077.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 471 kB/s | 248 kB 00:00 2024-03-16T10:04:45.173 INFO:teuthology.orchestra.run.smithi077.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 455 kB/s | 216 kB 00:00 2024-03-16T10:04:45.190 INFO:teuthology.orchestra.run.smithi077.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-16T10:04:45.209 INFO:teuthology.orchestra.run.smithi077.stdout:(103/174): qatlib-service-23.11.0-1.el9.x86_64. 347 kB/s | 35 kB 00:00 2024-03-16T10:04:45.232 INFO:teuthology.orchestra.run.smithi077.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-16T10:04:45.391 INFO:teuthology.orchestra.run.smithi077.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-03-16T10:04:45.583 INFO:teuthology.orchestra.run.smithi077.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-03-16T10:04:45.605 INFO:teuthology.orchestra.run.smithi192.stdout:(31/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 11 MB/s | 48 MB 00:04 2024-03-16T10:04:45.617 INFO:teuthology.orchestra.run.smithi077.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-03-16T10:04:45.642 INFO:teuthology.orchestra.run.smithi077.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 335 kB/s | 19 kB 00:00 2024-03-16T10:04:45.676 INFO:teuthology.orchestra.run.smithi077.stdout:(109/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-03-16T10:04:45.730 INFO:teuthology.orchestra.run.smithi193.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2024-03-16T10:04:45.788 INFO:teuthology.orchestra.run.smithi193.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 447 kB/s | 26 kB 00:00 2024-03-16T10:04:45.792 INFO:teuthology.orchestra.run.smithi077.stdout:(110/174): libarrow-9.0.0-10.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-03-16T10:04:45.797 INFO:teuthology.orchestra.run.smithi192.stdout:(32/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 2.9 MB/s | 17 MB 00:05 2024-03-16T10:04:45.818 INFO:teuthology.orchestra.run.smithi077.stdout:(111/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 148 kB/s | 26 kB 00:00 2024-03-16T10:04:45.822 INFO:teuthology.orchestra.run.smithi192.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2162.g3d 1.1 MB/s | 248 kB 00:00 2024-03-16T10:04:45.843 INFO:teuthology.orchestra.run.smithi077.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 294 kB/s | 49 kB 00:00 2024-03-16T10:04:45.847 INFO:teuthology.orchestra.run.smithi193.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-03-16T10:04:45.868 INFO:teuthology.orchestra.run.smithi077.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 899 kB/s | 67 kB 00:00 2024-03-16T10:04:45.893 INFO:teuthology.orchestra.run.smithi077.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-03-16T10:04:45.897 INFO:teuthology.orchestra.run.smithi192.stdout:(34/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 493 kB/s | 49 kB 00:00 2024-03-16T10:04:45.905 INFO:teuthology.orchestra.run.smithi193.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-16T10:04:45.923 INFO:teuthology.orchestra.run.smithi192.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2162.g3 164 kB/s | 16 kB 00:00 2024-03-16T10:04:45.935 INFO:teuthology.orchestra.run.smithi077.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-03-16T10:04:45.961 INFO:teuthology.orchestra.run.smithi077.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-03-16T10:04:45.972 INFO:teuthology.orchestra.run.smithi193.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-16T10:04:45.985 INFO:teuthology.orchestra.run.smithi077.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 481 kB/s | 43 kB 00:00 2024-03-16T10:04:46.006 INFO:teuthology.orchestra.run.smithi192.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 6.9 MB/s | 7.4 MB 00:01 2024-03-16T10:04:46.011 INFO:teuthology.orchestra.run.smithi077.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-03-16T10:04:46.032 INFO:teuthology.orchestra.run.smithi192.stdout:(37/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 1.9 MB/s | 264 kB 00:00 2024-03-16T10:04:46.036 INFO:teuthology.orchestra.run.smithi077.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-03-16T10:04:46.057 INFO:teuthology.orchestra.run.smithi192.stdout:(38/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 364 kB/s | 18 kB 00:00 2024-03-16T10:04:46.061 INFO:teuthology.orchestra.run.smithi077.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-16T10:04:46.091 INFO:teuthology.orchestra.run.smithi192.stdout:(39/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.4 MB/s | 748 kB 00:00 2024-03-16T10:04:46.095 INFO:teuthology.orchestra.run.smithi077.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-16T10:04:46.120 INFO:teuthology.orchestra.run.smithi077.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-03-16T10:04:46.145 INFO:teuthology.orchestra.run.smithi077.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 425 kB/s | 35 kB 00:00 2024-03-16T10:04:46.156 INFO:teuthology.orchestra.run.smithi193.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-03-16T10:04:46.171 INFO:teuthology.orchestra.run.smithi077.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-16T10:04:46.229 INFO:teuthology.orchestra.run.smithi077.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 25 MB/s | 2.0 MB 00:00 2024-03-16T10:04:46.249 INFO:teuthology.orchestra.run.smithi192.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 988 kB/s | 106 kB 00:00 2024-03-16T10:04:46.254 INFO:teuthology.orchestra.run.smithi077.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-03-16T10:04:46.275 INFO:teuthology.orchestra.run.smithi192.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 662 kB 00:00 2024-03-16T10:04:46.280 INFO:teuthology.orchestra.run.smithi077.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-03-16T10:04:46.281 INFO:teuthology.orchestra.run.smithi193.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-03-16T10:04:46.305 INFO:teuthology.orchestra.run.smithi077.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 236 kB/s | 18 kB 00:00 2024-03-16T10:04:46.308 INFO:teuthology.orchestra.run.smithi192.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 3.6 MB/s | 920 kB 00:00 2024-03-16T10:04:46.330 INFO:teuthology.orchestra.run.smithi077.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 464 kB/s | 23 kB 00:00 2024-03-16T10:04:46.334 INFO:teuthology.orchestra.run.smithi192.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 307 kB 00:00 2024-03-16T10:04:46.340 INFO:teuthology.orchestra.run.smithi193.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 748 kB/s | 43 kB 00:00 2024-03-16T10:04:46.355 INFO:teuthology.orchestra.run.smithi077.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-03-16T10:04:46.359 INFO:teuthology.orchestra.run.smithi192.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 379 kB/s | 32 kB 00:00 2024-03-16T10:04:46.380 INFO:teuthology.orchestra.run.smithi077.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 262 kB/s | 20 kB 00:00 2024-03-16T10:04:46.385 INFO:teuthology.orchestra.run.smithi192.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 341 kB/s | 17 kB 00:00 2024-03-16T10:04:46.399 INFO:teuthology.orchestra.run.smithi193.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2024-03-16T10:04:46.414 INFO:teuthology.orchestra.run.smithi077.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 18 MB/s | 1.0 MB 00:00 2024-03-16T10:04:46.441 INFO:teuthology.orchestra.run.smithi077.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 552 kB/s | 46 kB 00:00 2024-03-16T10:04:46.457 INFO:teuthology.orchestra.run.smithi193.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 257 kB/s | 15 kB 00:00 2024-03-16T10:04:46.460 INFO:teuthology.orchestra.run.smithi192.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-16T10:04:46.465 INFO:teuthology.orchestra.run.smithi077.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 945 kB/s | 79 kB 00:00 2024-03-16T10:04:46.477 INFO:teuthology.orchestra.run.smithi192.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 86 kB/s | 10 kB 00:00 2024-03-16T10:04:46.490 INFO:teuthology.orchestra.run.smithi077.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 764 kB/s | 58 kB 00:00 2024-03-16T10:04:46.502 INFO:teuthology.orchestra.run.smithi192.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2024-03-16T10:04:46.515 INFO:teuthology.orchestra.run.smithi077.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-16T10:04:46.515 INFO:teuthology.orchestra.run.smithi193.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-16T10:04:46.518 INFO:teuthology.orchestra.run.smithi192.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 756 kB/s | 44 kB 00:00 2024-03-16T10:04:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 219 kB/s | 16 kB 00:00 2024-03-16T10:04:46.552 INFO:teuthology.orchestra.run.smithi192.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-16T10:04:46.565 INFO:teuthology.orchestra.run.smithi077.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-16T10:04:46.586 INFO:teuthology.orchestra.run.smithi192.stdout:(51/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-16T10:04:46.591 INFO:teuthology.orchestra.run.smithi077.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 565 kB/s | 28 kB 00:00 2024-03-16T10:04:46.603 INFO:teuthology.orchestra.run.smithi192.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 553 kB/s | 46 kB 00:00 2024-03-16T10:04:46.616 INFO:teuthology.orchestra.run.smithi077.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 410 kB/s | 31 kB 00:00 2024-03-16T10:04:46.616 INFO:teuthology.orchestra.run.smithi193.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-16T10:04:46.641 INFO:teuthology.orchestra.run.smithi077.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-03-16T10:04:46.667 INFO:teuthology.orchestra.run.smithi077.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-16T10:04:46.708 INFO:teuthology.orchestra.run.smithi193.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-16T10:04:46.708 INFO:teuthology.orchestra.run.smithi077.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 391 kB/s | 36 kB 00:00 2024-03-16T10:04:46.721 INFO:teuthology.orchestra.run.smithi192.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 282 kB 00:00 2024-03-16T10:04:46.733 INFO:teuthology.orchestra.run.smithi077.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 556 kB/s | 51 kB 00:00 2024-03-16T10:04:46.759 INFO:teuthology.orchestra.run.smithi077.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-16T10:04:46.766 INFO:teuthology.orchestra.run.smithi193.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 607 kB/s | 35 kB 00:00 2024-03-16T10:04:46.770 INFO:teuthology.orchestra.run.smithi192.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-16T10:04:46.784 INFO:teuthology.orchestra.run.smithi077.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-16T10:04:46.796 INFO:teuthology.orchestra.run.smithi192.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-16T10:04:46.809 INFO:teuthology.orchestra.run.smithi077.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-03-16T10:04:46.821 INFO:teuthology.orchestra.run.smithi192.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 346 kB/s | 34 kB 00:00 2024-03-16T10:04:46.833 INFO:teuthology.orchestra.run.smithi193.stdout:(122/174): python3-google-auth-2.28.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-16T10:04:46.834 INFO:teuthology.orchestra.run.smithi077.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 405 kB/s | 20 kB 00:00 2024-03-16T10:04:46.860 INFO:teuthology.orchestra.run.smithi077.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-16T10:04:46.871 INFO:teuthology.orchestra.run.smithi192.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2024-03-16T10:04:46.893 INFO:teuthology.orchestra.run.smithi077.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-16T10:04:46.955 INFO:teuthology.orchestra.run.smithi192.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-16T10:04:46.997 INFO:teuthology.orchestra.run.smithi192.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 4.5 MB/s | 811 kB 00:00 2024-03-16T10:04:47.022 INFO:teuthology.orchestra.run.smithi192.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 907 kB/s | 134 kB 00:00 2024-03-16T10:04:47.052 INFO:teuthology.orchestra.run.smithi077.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.0 MB/s | 321 kB 00:00 2024-03-16T10:04:47.122 INFO:teuthology.orchestra.run.smithi192.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-16T10:04:47.125 INFO:teuthology.orchestra.run.smithi193.stdout:(123/174): openssl-devel-3.0.7-27.el9.x86_64.rp 680 kB/s | 4.1 MB 00:06 2024-03-16T10:04:47.148 INFO:teuthology.orchestra.run.smithi192.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 634 kB/s | 78 kB 00:00 2024-03-16T10:04:47.181 INFO:teuthology.orchestra.run.smithi192.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.5 MB/s | 1.0 MB 00:00 2024-03-16T10:04:47.235 INFO:teuthology.orchestra.run.smithi077.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 13 MB/s | 2.3 MB 00:00 2024-03-16T10:04:47.282 INFO:teuthology.orchestra.run.smithi192.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 37 MB/s | 5.8 MB 00:00 2024-03-16T10:04:47.301 INFO:teuthology.orchestra.run.smithi193.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.4 MB/s | 2.0 MB 00:00 2024-03-16T10:04:47.302 INFO:teuthology.orchestra.run.smithi077.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 27 MB/s | 1.8 MB 00:00 2024-03-16T10:04:47.307 INFO:teuthology.orchestra.run.smithi192.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 156 kB/s | 24 kB 00:00 2024-03-16T10:04:47.332 INFO:teuthology.orchestra.run.smithi192.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 122 kB/s | 18 kB 00:00 2024-03-16T10:04:47.344 INFO:teuthology.orchestra.run.smithi077.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-16T10:04:47.351 INFO:teuthology.orchestra.run.smithi193.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-03-16T10:04:47.357 INFO:teuthology.orchestra.run.smithi192.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-16T10:04:47.382 INFO:teuthology.orchestra.run.smithi192.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 633 kB/s | 32 kB 00:00 2024-03-16T10:04:47.394 INFO:teuthology.orchestra.run.smithi077.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 4.9 MB/s | 248 kB 00:00 2024-03-16T10:04:47.411 INFO:teuthology.orchestra.run.smithi193.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 293 kB/s | 18 kB 00:00 2024-03-16T10:04:47.453 INFO:teuthology.orchestra.run.smithi077.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-16T10:04:47.470 INFO:teuthology.orchestra.run.smithi193.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 398 kB/s | 23 kB 00:00 2024-03-16T10:04:47.489 INFO:teuthology.orchestra.run.smithi192.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 31 MB/s | 4.8 MB 00:00 2024-03-16T10:04:47.503 INFO:teuthology.orchestra.run.smithi077.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 730 kB/s | 36 kB 00:00 2024-03-16T10:04:47.514 INFO:teuthology.orchestra.run.smithi192.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-16T10:04:47.528 INFO:teuthology.orchestra.run.smithi193.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 332 kB/s | 19 kB 00:00 2024-03-16T10:04:47.548 INFO:teuthology.orchestra.run.smithi192.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-16T10:04:47.562 INFO:teuthology.orchestra.run.smithi077.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-16T10:04:47.573 INFO:teuthology.orchestra.run.smithi192.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 827 kB/s | 69 kB 00:00 2024-03-16T10:04:47.587 INFO:teuthology.orchestra.run.smithi193.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 338 kB/s | 20 kB 00:00 2024-03-16T10:04:47.598 INFO:teuthology.orchestra.run.smithi192.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-16T10:04:47.604 INFO:teuthology.orchestra.run.smithi077.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 256 kB 00:00 2024-03-16T10:04:47.633 INFO:teuthology.orchestra.run.smithi192.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-16T10:04:47.646 INFO:teuthology.orchestra.run.smithi077.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 553 kB/s | 23 kB 00:00 2024-03-16T10:04:47.658 INFO:teuthology.orchestra.run.smithi192.stdout:(75/174): python3-prettytable-0.7.2-27.el9.noar 375 kB/s | 41 kB 00:00 2024-03-16T10:04:47.684 INFO:teuthology.orchestra.run.smithi192.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.4 MB/s | 211 kB 00:00 2024-03-16T10:04:47.688 INFO:teuthology.orchestra.run.smithi077.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 108 kB 00:00 2024-03-16T10:04:47.708 INFO:teuthology.orchestra.run.smithi192.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 632 kB/s | 47 kB 00:00 2024-03-16T10:04:47.730 INFO:teuthology.orchestra.run.smithi077.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 155 kB 00:00 2024-03-16T10:04:47.733 INFO:teuthology.orchestra.run.smithi192.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2024-03-16T10:04:47.780 INFO:teuthology.orchestra.run.smithi077.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 2.7 MB/s | 136 kB 00:00 2024-03-16T10:04:47.839 INFO:teuthology.orchestra.run.smithi077.stdout:(164/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 3.2 MB/s | 3.1 MB 00:00 2024-03-16T10:04:47.855 INFO:teuthology.orchestra.run.smithi077.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 849 kB/s | 64 kB 00:00 2024-03-16T10:04:47.909 INFO:teuthology.orchestra.run.smithi192.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 195 kB/s | 39 kB 00:00 2024-03-16T10:04:47.921 INFO:teuthology.orchestra.run.smithi193.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 3.1 MB/s | 1.0 MB 00:00 2024-03-16T10:04:47.956 INFO:teuthology.orchestra.run.smithi077.stdout:(166/174): openssl-3.0.7-27.el9.x86_64.rpm 12 MB/s | 1.2 MB 00:00 2024-03-16T10:04:47.979 INFO:teuthology.orchestra.run.smithi193.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 795 kB/s | 46 kB 00:00 2024-03-16T10:04:47.993 INFO:teuthology.orchestra.run.smithi192.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 160 kB/s | 41 kB 00:00 2024-03-16T10:04:47.998 INFO:teuthology.orchestra.run.smithi077.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 180 kB/s | 29 kB 00:00 2024-03-16T10:04:48.046 INFO:teuthology.orchestra.run.smithi193.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-16T10:04:48.073 INFO:teuthology.orchestra.run.smithi077.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 18 MB/s | 2.2 MB 00:00 2024-03-16T10:04:48.104 INFO:teuthology.orchestra.run.smithi193.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 993 kB/s | 58 kB 00:00 2024-03-16T10:04:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 9.1 MB/s | 465 kB 00:00 2024-03-16T10:04:48.152 INFO:teuthology.orchestra.run.smithi192.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-16T10:04:48.165 INFO:teuthology.orchestra.run.smithi077.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 749 kB/s | 31 kB 00:00 2024-03-16T10:04:48.176 INFO:teuthology.orchestra.run.smithi192.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 300 kB 00:00 2024-03-16T10:04:48.202 INFO:teuthology.orchestra.run.smithi192.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 272 kB/s | 56 kB 00:00 2024-03-16T10:04:48.224 INFO:teuthology.orchestra.run.smithi077.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 10 MB/s | 2.3 MB 00:00 2024-03-16T10:04:48.230 INFO:teuthology.orchestra.run.smithi193.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-16T10:04:48.235 INFO:teuthology.orchestra.run.smithi192.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-16T10:04:48.261 INFO:teuthology.orchestra.run.smithi192.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 528 kB/s | 44 kB 00:00 2024-03-16T10:04:48.274 INFO:teuthology.orchestra.run.smithi077.stdout:(172/174): librados2-19.0.0-2162.g3d78f5ba.el9. 2.4 MB/s | 3.4 MB 00:01 2024-03-16T10:04:48.286 INFO:teuthology.orchestra.run.smithi192.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 936 kB/s | 79 kB 00:00 2024-03-16T10:04:48.288 INFO:teuthology.orchestra.run.smithi193.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 282 kB/s | 16 kB 00:00 2024-03-16T10:04:48.308 INFO:teuthology.orchestra.run.smithi077.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.6 MB/s | 1.1 MB 00:00 2024-03-16T10:04:48.319 INFO:teuthology.orchestra.run.smithi192.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-16T10:04:48.349 INFO:teuthology.orchestra.run.smithi077.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 594 kB/s | 74 kB 00:00 2024-03-16T10:04:48.353 INFO:teuthology.orchestra.run.smithi192.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-16T10:04:48.355 INFO:teuthology.orchestra.run.smithi193.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-16T10:04:48.356 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-03-16T10:04:48.356 INFO:teuthology.orchestra.run.smithi077.stdout:Total 13 MB/s | 224 MB 00:16 2024-03-16T10:04:48.378 INFO:teuthology.orchestra.run.smithi192.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-16T10:04:48.404 INFO:teuthology.orchestra.run.smithi192.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-16T10:04:48.414 INFO:teuthology.orchestra.run.smithi193.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 486 kB/s | 28 kB 00:00 2024-03-16T10:04:48.429 INFO:teuthology.orchestra.run.smithi192.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 187 kB 00:00 2024-03-16T10:04:48.473 INFO:teuthology.orchestra.run.smithi193.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 526 kB/s | 31 kB 00:00 2024-03-16T10:04:48.556 INFO:teuthology.orchestra.run.smithi193.stdout:(139/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-16T10:04:48.615 INFO:teuthology.orchestra.run.smithi193.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-16T10:04:48.629 INFO:teuthology.orchestra.run.smithi192.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 747 kB/s | 187 kB 00:00 2024-03-16T10:04:48.682 INFO:teuthology.orchestra.run.smithi193.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 535 kB/s | 36 kB 00:00 2024-03-16T10:04:48.688 INFO:teuthology.orchestra.run.smithi192.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-16T10:04:48.713 INFO:teuthology.orchestra.run.smithi192.stdout:(94/174): libquadmath-11.4.1-3.el9.x86_64.rpm 688 kB/s | 195 kB 00:00 2024-03-16T10:04:48.740 INFO:teuthology.orchestra.run.smithi193.stdout:(142/174): python3-typing-extensions-4.4.0-2.el 875 kB/s | 51 kB 00:00 2024-03-16T10:04:48.755 INFO:teuthology.orchestra.run.smithi192.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-03-16T10:04:48.781 INFO:teuthology.orchestra.run.smithi192.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 233 kB/s | 21 kB 00:00 2024-03-16T10:04:48.832 INFO:teuthology.orchestra.run.smithi193.stdout:(143/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-16T10:04:48.837 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-03-16T10:04:48.848 INFO:teuthology.orchestra.run.smithi192.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 190 kB/s | 25 kB 00:00 2024-03-16T10:04:48.899 INFO:teuthology.orchestra.run.smithi193.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-03-16T10:04:48.923 INFO:teuthology.orchestra.run.smithi192.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-16T10:04:49.015 INFO:teuthology.orchestra.run.smithi192.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-03-16T10:04:49.040 INFO:teuthology.orchestra.run.smithi192.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.8 MB/s | 216 kB 00:00 2024-03-16T10:04:49.058 INFO:teuthology.orchestra.run.smithi193.stdout:(145/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-03-16T10:04:49.065 INFO:teuthology.orchestra.run.smithi192.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 696 kB/s | 35 kB 00:00 2024-03-16T10:04:49.091 INFO:teuthology.orchestra.run.smithi192.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 889 kB/s | 44 kB 00:00 2024-03-16T10:04:49.116 INFO:teuthology.orchestra.run.smithi192.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-16T10:04:49.117 INFO:teuthology.orchestra.run.smithi193.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 342 kB/s | 20 kB 00:00 2024-03-16T10:04:49.147 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-03-16T10:04:49.147 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-03-16T10:04:49.183 INFO:teuthology.orchestra.run.smithi192.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 9.7 MB/s | 4.1 MB 00:00 2024-03-16T10:04:49.275 INFO:teuthology.orchestra.run.smithi192.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 121 kB/s | 22 kB 00:00 2024-03-16T10:04:49.333 INFO:teuthology.orchestra.run.smithi193.stdout:(147/174): re2-20211101-3.el9.x86_64.rpm 885 kB/s | 192 kB 00:00 2024-03-16T10:04:49.434 INFO:teuthology.orchestra.run.smithi192.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-03-16T10:04:49.467 INFO:teuthology.orchestra.run.smithi192.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-03-16T10:04:49.534 INFO:teuthology.orchestra.run.smithi192.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-03-16T10:04:49.559 INFO:teuthology.orchestra.run.smithi192.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-16T10:04:49.593 INFO:teuthology.orchestra.run.smithi192.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-03-16T10:04:49.668 INFO:teuthology.orchestra.run.smithi192.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2024-03-16T10:04:49.694 INFO:teuthology.orchestra.run.smithi192.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-16T10:04:49.761 INFO:teuthology.orchestra.run.smithi192.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-16T10:04:49.786 INFO:teuthology.orchestra.run.smithi192.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-16T10:04:49.828 INFO:teuthology.orchestra.run.smithi192.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-03-16T10:04:49.862 INFO:teuthology.orchestra.run.smithi192.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-16T10:04:49.887 INFO:teuthology.orchestra.run.smithi192.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 432 kB/s | 43 kB 00:00 2024-03-16T10:04:49.913 INFO:teuthology.orchestra.run.smithi192.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 382 kB/s | 32 kB 00:00 2024-03-16T10:04:49.930 INFO:teuthology.orchestra.run.smithi192.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 222 kB/s | 15 kB 00:00 2024-03-16T10:04:49.963 INFO:teuthology.orchestra.run.smithi192.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-16T10:04:49.989 INFO:teuthology.orchestra.run.smithi192.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-03-16T10:04:50.014 INFO:teuthology.orchestra.run.smithi192.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-03-16T10:04:50.043 INFO:teuthology.orchestra.run.smithi192.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 444 kB/s | 35 kB 00:00 2024-03-16T10:04:50.068 INFO:teuthology.orchestra.run.smithi192.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.5 MB/s | 203 kB 00:00 2024-03-16T10:04:50.118 INFO:teuthology.orchestra.run.smithi192.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-16T10:04:50.143 INFO:teuthology.orchestra.run.smithi192.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-16T10:04:50.194 INFO:teuthology.orchestra.run.smithi192.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-03-16T10:04:50.219 INFO:teuthology.orchestra.run.smithi192.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-03-16T10:04:50.244 INFO:teuthology.orchestra.run.smithi192.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-03-16T10:04:50.270 INFO:teuthology.orchestra.run.smithi192.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-16T10:04:50.295 INFO:teuthology.orchestra.run.smithi192.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 395 kB/s | 20 kB 00:00 2024-03-16T10:04:50.329 INFO:teuthology.orchestra.run.smithi192.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 792 kB/s | 46 kB 00:00 2024-03-16T10:04:50.354 INFO:teuthology.orchestra.run.smithi192.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-16T10:04:50.387 INFO:teuthology.orchestra.run.smithi192.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-03-16T10:04:50.409 INFO:teuthology.orchestra.run.smithi193.stdout:(148/174): thrift-0.15.0-2.el9.x86_64.rpm 1.5 MB/s | 1.6 MB 00:01 2024-03-16T10:04:50.412 INFO:teuthology.orchestra.run.smithi192.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 688 kB/s | 58 kB 00:00 2024-03-16T10:04:50.438 INFO:teuthology.orchestra.run.smithi192.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-16T10:04:50.463 INFO:teuthology.orchestra.run.smithi192.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-03-16T10:04:50.475 INFO:teuthology.orchestra.run.smithi193.stdout:(149/174): python3-grpcio-tools-1.46.7-10.el9.x 43 kB/s | 144 kB 00:03 2024-03-16T10:04:50.497 INFO:teuthology.orchestra.run.smithi192.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-03-16T10:04:50.522 INFO:teuthology.orchestra.run.smithi192.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-03-16T10:04:50.549 INFO:teuthology.orchestra.run.smithi192.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-03-16T10:04:50.572 INFO:teuthology.orchestra.run.smithi192.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-16T10:04:50.606 INFO:teuthology.orchestra.run.smithi192.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-16T10:04:50.631 INFO:teuthology.orchestra.run.smithi192.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 428 kB/s | 36 kB 00:00 2024-03-16T10:04:50.656 INFO:teuthology.orchestra.run.smithi192.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-03-16T10:04:50.682 INFO:teuthology.orchestra.run.smithi192.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-16T10:04:50.715 INFO:teuthology.orchestra.run.smithi192.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-03-16T10:04:50.749 INFO:teuthology.orchestra.run.smithi192.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 297 kB/s | 20 kB 00:00 2024-03-16T10:04:50.775 INFO:teuthology.orchestra.run.smithi192.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-03-16T10:04:50.800 INFO:teuthology.orchestra.run.smithi192.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-16T10:04:50.850 INFO:teuthology.orchestra.run.smithi192.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-16T10:04:51.067 INFO:teuthology.orchestra.run.smithi192.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.4 MB/s | 321 kB 00:00 2024-03-16T10:04:51.084 INFO:teuthology.orchestra.run.smithi193.stdout:(150/174): librados2-19.0.0-2162.g3d78f5ba.el9. 5.0 MB/s | 3.4 MB 00:00 2024-03-16T10:04:51.126 INFO:teuthology.orchestra.run.smithi193.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-16T10:04:51.127 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-03-16T10:04:51.127 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-03-16T10:04:51.309 INFO:teuthology.orchestra.run.smithi193.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-03-16T10:04:51.359 INFO:teuthology.orchestra.run.smithi192.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 7.9 MB/s | 2.3 MB 00:00 2024-03-16T10:04:51.385 INFO:teuthology.orchestra.run.smithi193.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-03-16T10:04:51.427 INFO:teuthology.orchestra.run.smithi193.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-16T10:04:51.451 INFO:teuthology.orchestra.run.smithi192.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 20 MB/s | 1.8 MB 00:00 2024-03-16T10:04:51.485 INFO:teuthology.orchestra.run.smithi193.stdout:(155/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 3.1 MB/s | 3.1 MB 00:01 2024-03-16T10:04:51.510 INFO:teuthology.orchestra.run.smithi193.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-16T10:04:51.526 INFO:teuthology.orchestra.run.smithi192.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 565 kB/s | 42 kB 00:00 2024-03-16T10:04:51.553 INFO:teuthology.orchestra.run.smithi193.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 871 kB/s | 36 kB 00:00 2024-03-16T10:04:51.578 INFO:teuthology.orchestra.run.smithi193.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 772 kB/s | 71 kB 00:00 2024-03-16T10:04:51.593 INFO:teuthology.orchestra.run.smithi192.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 3.6 MB/s | 248 kB 00:00 2024-03-16T10:04:51.603 INFO:teuthology.orchestra.run.smithi193.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 192 kB 00:00 2024-03-16T10:04:51.637 INFO:teuthology.orchestra.run.smithi193.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 692 kB/s | 23 kB 00:00 2024-03-16T10:04:51.652 INFO:teuthology.orchestra.run.smithi192.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-16T10:04:51.662 INFO:teuthology.orchestra.run.smithi193.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.0 MB/s | 256 kB 00:00 2024-03-16T10:04:51.687 INFO:teuthology.orchestra.run.smithi193.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-16T10:04:51.710 INFO:teuthology.orchestra.run.smithi192.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 623 kB/s | 36 kB 00:00 2024-03-16T10:04:51.712 INFO:teuthology.orchestra.run.smithi193.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-16T10:04:51.738 INFO:teuthology.orchestra.run.smithi193.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-16T10:04:51.763 INFO:teuthology.orchestra.run.smithi193.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-16T10:04:51.788 INFO:teuthology.orchestra.run.smithi193.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 571 kB/s | 29 kB 00:00 2024-03-16T10:04:51.847 INFO:teuthology.orchestra.run.smithi193.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 15 MB/s | 1.2 MB 00:00 2024-03-16T10:04:51.889 INFO:teuthology.orchestra.run.smithi193.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 22 MB/s | 2.2 MB 00:00 2024-03-16T10:04:51.928 INFO:teuthology.orchestra.run.smithi192.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 887 kB/s | 192 kB 00:00 2024-03-16T10:04:51.939 INFO:teuthology.orchestra.run.smithi193.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 9.1 MB/s | 465 kB 00:00 2024-03-16T10:04:51.981 INFO:teuthology.orchestra.run.smithi193.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 17 MB/s | 2.3 MB 00:00 2024-03-16T10:04:52.003 INFO:teuthology.orchestra.run.smithi192.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.3 MB/s | 256 kB 00:00 2024-03-16T10:04:52.006 INFO:teuthology.orchestra.run.smithi193.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 465 kB/s | 31 kB 00:00 2024-03-16T10:04:52.040 INFO:teuthology.orchestra.run.smithi193.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-16T10:04:52.078 INFO:teuthology.orchestra.run.smithi192.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 307 kB/s | 23 kB 00:00 2024-03-16T10:04:52.153 INFO:teuthology.orchestra.run.smithi192.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-16T10:04:52.204 INFO:teuthology.orchestra.run.smithi192.stdout:(162/174): librados2-19.0.0-2162.g3d78f5ba.el9. 2.4 MB/s | 3.4 MB 00:01 2024-03-16T10:04:52.254 INFO:teuthology.orchestra.run.smithi192.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-16T10:04:52.348 INFO:teuthology.orchestra.run.smithi193.stdout:(173/174): gperftools-libs-2.9.1-2.el9.x86_64.r 42 kB/s | 309 kB 00:07 2024-03-16T10:04:52.362 INFO:teuthology.orchestra.run.smithi192.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 761 kB/s | 64 kB 00:00 2024-03-16T10:04:52.429 INFO:teuthology.orchestra.run.smithi192.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 431 kB/s | 29 kB 00:00 2024-03-16T10:04:52.454 INFO:teuthology.orchestra.run.smithi192.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 541 kB/s | 136 kB 00:00 2024-03-16T10:04:52.521 INFO:teuthology.orchestra.run.smithi192.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 13 MB/s | 1.2 MB 00:00 2024-03-16T10:04:52.655 INFO:teuthology.orchestra.run.smithi192.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 17 MB/s | 2.3 MB 00:00 2024-03-16T10:04:52.772 INFO:teuthology.orchestra.run.smithi192.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.9 MB/s | 465 kB 00:00 2024-03-16T10:04:52.807 INFO:teuthology.orchestra.run.smithi193.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 93 kB/s | 74 kB 00:00 2024-03-16T10:04:52.811 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-16T10:04:52.811 INFO:teuthology.orchestra.run.smithi193.stdout:Total 10 MB/s | 224 MB 00:22 2024-03-16T10:04:52.897 INFO:teuthology.orchestra.run.smithi192.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 249 kB/s | 31 kB 00:00 2024-03-16T10:04:52.939 INFO:teuthology.orchestra.run.smithi192.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.4 MB/s | 2.2 MB 00:00 2024-03-16T10:04:53.015 INFO:teuthology.orchestra.run.smithi192.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-03-16T10:04:53.107 INFO:teuthology.orchestra.run.smithi192.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 446 kB/s | 74 kB 00:00 2024-03-16T10:04:53.297 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-16T10:04:53.356 INFO:teuthology.orchestra.run.smithi192.stdout:(174/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 1.2 MB/s | 3.1 MB 00:02 2024-03-16T10:04:53.362 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-16T10:04:53.362 INFO:teuthology.orchestra.run.smithi192.stdout:Total 14 MB/s | 224 MB 00:16 2024-03-16T10:04:53.569 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-16T10:04:53.570 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-16T10:04:53.816 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-16T10:04:54.088 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-16T10:04:54.088 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-16T10:04:55.286 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-16T10:04:55.286 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-16T10:04:55.793 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-16T10:04:55.794 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-16T10:04:58.081 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-03-16T10:04:58.199 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T10:04:58.239 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T10:04:58.285 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T10:04:58.295 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T10:04:58.335 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T10:04:58.388 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T10:04:58.421 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T10:04:58.447 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T10:04:58.643 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T10:04:59.127 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T10:04:59.190 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:04:59.319 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:04:59.349 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:04:59.449 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:04:59.488 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T10:04:59.539 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T10:04:59.723 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T10:04:59.767 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T10:04:59.796 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:04:59.902 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:04:59.930 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T10:04:59.995 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T10:05:00.025 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T10:05:00.056 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T10:05:00.072 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T10:05:00.241 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T10:05:00.289 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T10:05:00.328 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T10:05:00.518 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T10:05:00.647 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T10:05:00.707 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T10:05:00.820 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:05:00.883 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:05:01.088 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:05:01.198 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T10:05:01.299 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T10:05:01.329 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T10:05:01.356 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T10:05:01.391 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T10:05:01.465 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T10:05:01.519 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T10:05:01.557 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T10:05:01.621 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T10:05:01.656 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T10:05:01.700 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T10:05:01.733 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T10:05:01.768 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T10:05:01.855 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T10:05:01.890 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T10:05:01.940 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T10:05:01.970 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T10:05:02.042 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T10:05:02.093 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T10:05:02.141 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-16T10:05:02.167 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T10:05:02.252 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T10:05:02.303 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T10:05:02.354 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T10:05:02.434 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T10:05:02.525 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T10:05:02.537 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T10:05:02.576 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T10:05:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T10:05:02.671 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T10:05:02.705 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T10:05:02.718 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-16T10:05:02.748 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T10:05:02.819 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T10:05:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T10:05:02.872 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T10:05:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T10:05:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T10:05:02.950 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T10:05:02.957 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T10:05:03.003 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T10:05:03.037 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T10:05:03.047 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T10:05:03.063 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T10:05:03.069 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T10:05:03.102 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T10:05:03.105 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T10:05:03.110 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T10:05:03.113 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:05:03.151 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T10:05:03.199 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T10:05:03.217 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:05:03.237 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T10:05:03.257 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:05:03.268 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T10:05:03.347 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:05:03.377 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T10:05:03.414 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T10:05:03.429 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T10:05:03.481 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:05:03.577 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T10:05:03.594 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T10:05:03.610 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:05:03.640 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:05:03.643 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T10:05:03.672 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:05:03.714 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:05:03.744 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T10:05:03.767 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T10:05:03.775 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:05:03.796 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T10:05:03.798 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T10:05:03.860 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T10:05:03.862 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T10:05:03.902 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T10:05:03.914 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T10:05:03.924 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T10:05:03.940 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T10:05:03.968 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T10:05:04.017 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T10:05:04.045 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:05:04.051 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T10:05:04.091 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T10:05:04.107 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T10:05:04.131 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T10:05:04.148 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:05:04.162 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T10:05:04.162 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T10:05:04.171 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T10:05:04.234 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T10:05:04.235 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T10:05:04.274 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T10:05:04.275 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T10:05:04.283 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T10:05:04.311 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T10:05:04.314 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T10:05:04.329 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T10:05:04.360 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T10:05:04.396 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T10:05:04.398 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T10:05:04.439 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T10:05:04.459 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T10:05:04.481 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T10:05:04.515 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T10:05:04.545 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T10:05:04.577 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T10:05:04.615 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T10:05:04.653 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T10:05:04.681 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T10:05:04.682 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T10:05:04.725 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T10:05:04.743 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:05:04.771 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T10:05:04.818 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T10:05:04.831 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T10:05:04.858 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T10:05:04.898 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T10:05:04.903 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:05:04.923 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T10:05:04.933 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:05:04.948 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T10:05:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:05:05.029 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T10:05:05.156 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:05:05.156 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T10:05:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T10:05:05.264 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T10:05:05.266 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T10:05:05.293 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T10:05:05.314 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T10:05:05.362 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T10:05:05.362 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T10:05:05.384 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T10:05:05.399 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T10:05:05.419 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T10:05:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T10:05:05.426 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T10:05:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T10:05:05.470 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T10:05:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T10:05:05.544 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T10:05:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T10:05:05.598 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T10:05:05.610 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T10:05:05.619 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T10:05:05.645 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T10:05:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T10:05:05.660 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T10:05:05.710 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T10:05:05.722 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T10:05:05.743 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T10:05:05.765 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T10:05:05.787 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T10:05:05.788 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T10:05:05.818 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T10:05:05.819 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T10:05:05.833 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T10:05:05.849 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T10:05:05.856 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T10:05:05.865 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T10:05:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T10:05:05.895 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T10:05:05.924 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T10:05:05.926 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T10:05:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T10:05:05.958 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T10:05:05.961 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T10:05:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T10:05:05.997 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T10:05:06.005 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T10:05:06.027 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T10:05:06.028 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T10:05:06.033 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T10:05:06.055 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T10:05:06.058 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T10:05:06.102 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T10:05:06.125 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T10:05:06.134 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T10:05:06.156 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T10:05:06.168 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T10:05:06.169 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T10:05:06.202 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T10:05:06.230 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T10:05:06.243 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T10:05:06.251 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T10:05:06.278 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T10:05:06.363 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T10:05:06.367 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T10:05:06.391 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T10:05:06.416 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T10:05:06.446 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T10:05:06.459 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T10:05:06.501 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T10:05:06.532 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T10:05:06.578 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T10:05:06.677 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T10:05:06.725 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T10:05:06.942 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T10:05:06.962 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T10:05:07.021 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T10:05:07.042 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T10:05:07.117 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T10:05:07.128 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T10:05:07.158 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T10:05:07.166 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T10:05:07.178 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T10:05:07.187 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T10:05:07.197 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T10:05:07.218 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T10:05:07.247 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T10:05:07.267 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T10:05:07.285 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T10:05:07.305 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T10:05:07.322 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T10:05:07.335 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T10:05:07.448 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T10:05:07.489 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T10:05:07.522 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T10:05:07.523 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T10:05:07.562 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T10:05:07.571 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T10:05:07.659 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T10:05:07.702 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T10:05:07.740 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T10:05:07.741 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T10:05:07.792 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T10:05:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T10:05:07.836 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T10:05:07.864 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T10:05:07.917 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T10:05:07.946 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T10:05:07.967 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T10:05:08.011 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T10:05:08.023 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T10:05:08.023 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T10:05:08.076 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T10:05:08.077 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T10:05:08.130 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T10:05:08.141 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T10:05:08.182 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T10:05:08.218 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T10:05:08.265 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T10:05:08.301 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T10:05:08.344 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T10:05:08.387 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T10:05:08.441 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T10:05:08.477 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T10:05:08.521 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T10:05:08.539 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T10:05:08.559 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T10:05:08.586 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T10:05:08.589 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T10:05:08.630 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T10:05:08.647 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T10:05:08.670 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T10:05:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T10:05:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T10:05:08.738 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T10:05:08.778 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T10:05:08.791 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T10:05:08.818 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T10:05:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T10:05:08.858 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T10:05:08.869 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T10:05:08.912 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T10:05:08.932 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T10:05:08.942 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T10:05:08.983 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T10:05:09.061 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T10:05:09.077 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T10:05:09.117 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T10:05:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T10:05:09.159 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T10:05:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T10:05:09.203 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T10:05:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T10:05:09.251 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T10:05:09.279 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T10:05:09.306 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T10:05:09.400 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T10:05:09.490 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T10:05:09.506 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T10:05:09.540 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T10:05:09.549 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T10:05:09.597 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T10:05:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T10:05:09.669 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T10:05:09.705 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T10:05:09.737 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T10:05:09.768 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T10:05:09.796 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T10:05:09.830 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T10:05:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T10:05:09.870 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T10:05:09.901 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T10:05:09.903 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T10:05:09.931 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T10:05:10.007 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T10:05:10.023 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T10:05:10.031 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T10:05:10.042 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T10:05:10.067 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T10:05:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T10:05:10.099 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T10:05:10.103 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T10:05:10.116 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T10:05:10.134 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T10:05:10.139 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T10:05:10.155 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T10:05:10.165 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T10:05:10.175 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T10:05:10.192 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T10:05:10.210 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T10:05:10.240 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T10:05:10.249 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T10:05:10.280 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T10:05:10.310 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T10:05:10.338 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T10:05:10.382 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T10:05:10.386 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T10:05:10.414 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T10:05:10.421 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T10:05:10.454 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T10:05:10.461 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T10:05:10.495 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T10:05:10.503 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T10:05:10.531 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T10:05:10.538 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T10:05:10.558 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T10:05:10.607 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T10:05:10.621 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T10:05:10.635 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T10:05:10.652 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T10:05:10.689 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T10:05:10.726 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T10:05:10.786 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T10:05:10.817 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T10:05:10.865 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T10:05:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T10:05:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T10:05:11.152 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T10:05:11.184 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T10:05:11.238 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T10:05:11.268 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T10:05:11.340 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T10:05:11.396 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T10:05:11.430 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T10:05:11.478 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T10:05:11.557 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T10:05:11.589 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T10:05:11.753 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T10:05:11.766 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T10:05:11.787 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T10:05:11.799 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T10:05:11.833 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T10:05:11.865 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T10:05:11.881 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T10:05:11.977 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T10:05:12.017 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T10:05:12.194 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T10:05:12.240 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T10:05:12.317 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T10:05:12.503 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T10:05:12.611 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T10:05:12.957 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T10:05:13.072 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T10:05:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:05:13.556 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:05:13.833 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T10:05:13.914 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T10:05:13.945 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T10:05:13.985 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T10:05:14.012 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T10:05:14.097 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:05:14.135 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T10:05:14.216 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T10:05:14.217 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T10:05:14.217 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T10:05:14.217 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:14.253 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T10:05:14.289 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T10:05:14.324 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T10:05:14.351 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T10:05:14.382 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T10:05:14.413 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T10:05:14.441 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T10:05:14.444 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T10:05:14.471 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T10:05:14.472 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T10:05:14.508 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T10:05:14.786 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T10:05:14.818 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T10:05:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T10:05:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T10:05:14.967 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T10:05:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T10:05:15.004 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T10:05:15.075 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T10:05:15.097 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T10:05:15.443 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T10:05:15.613 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T10:05:15.684 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T10:05:15.706 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T10:05:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:05:17.369 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:05:17.908 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:05:17.948 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T10:05:18.020 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:05:18.029 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T10:05:18.029 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T10:05:18.029 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T10:05:18.030 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:18.057 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T10:05:18.061 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:05:18.602 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T10:05:18.640 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T10:05:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T10:05:18.721 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T10:05:18.721 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T10:05:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:18.749 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T10:05:26.016 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T10:05:26.016 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2024-03-16T10:05:26.016 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2024-03-16T10:05:26.016 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2024-03-16T10:05:26.016 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2024-03-16T10:05:26.016 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2024-03-16T10:05:26.016 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /root 2024-03-16T10:05:26.016 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2024-03-16T10:05:26.016 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:26.182 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T10:05:26.197 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T10:05:26.198 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:26.198 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T10:05:26.198 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:27.686 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T10:05:27.710 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T10:05:27.711 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:27.711 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T10:05:27.711 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T10:05:27.711 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T10:05:27.711 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:27.936 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T10:05:27.958 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T10:05:27.958 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:27.959 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T10:05:27.959 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T10:05:27.959 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T10:05:27.959 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:28.395 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T10:05:28.418 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T10:05:28.418 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:28.418 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T10:05:28.418 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T10:05:28.418 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T10:05:28.418 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:28.513 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T10:05:28.847 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T10:05:28.878 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T10:05:28.910 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T10:05:28.970 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T10:05:28.993 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T10:05:29.228 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T10:05:29.260 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T10:05:29.692 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T10:05:29.693 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-03-16T10:05:29.693 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-03-16T10:05:29.693 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-03-16T10:05:29.693 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-03-16T10:05:29.693 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-03-16T10:05:29.693 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /root 2024-03-16T10:05:29.693 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-03-16T10:05:29.693 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T10:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T10:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T10:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T10:05:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T10:05:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T10:05:30.393 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T10:05:30.393 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-03-16T10:05:30.393 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-03-16T10:05:30.393 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-03-16T10:05:30.393 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-03-16T10:05:30.393 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-03-16T10:05:30.393 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /root 2024-03-16T10:05:30.393 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-03-16T10:05:30.393 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T10:05:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T10:05:30.555 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T10:05:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T10:05:30.561 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:30.561 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T10:05:30.561 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T10:05:30.561 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T10:05:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:30.570 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T10:05:30.570 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:30.570 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T10:05:30.570 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T10:05:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T10:05:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T10:05:31.335 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T10:05:31.361 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T10:05:31.361 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:31.362 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T10:05:31.362 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T10:05:31.362 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T10:05:31.362 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:31.539 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T10:05:31.565 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T10:05:31.586 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T10:05:31.589 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T10:05:31.589 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:31.589 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T10:05:31.589 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T10:05:31.589 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T10:05:31.589 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:31.609 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T10:05:31.609 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:31.609 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T10:05:31.609 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T10:05:31.609 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T10:05:31.609 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:31.641 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T10:05:31.663 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T10:05:31.663 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:31.664 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T10:05:31.664 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:31.708 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:05:31.731 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:05:31.731 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:31.731 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T10:05:31.731 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T10:05:31.731 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T10:05:31.731 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:32.034 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T10:05:32.045 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T10:05:32.049 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T10:05:32.057 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T10:05:32.057 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:32.057 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T10:05:32.057 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T10:05:32.057 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T10:05:32.057 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:32.068 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T10:05:32.068 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:32.068 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T10:05:32.068 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T10:05:32.068 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T10:05:32.068 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:32.075 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T10:05:32.075 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:32.075 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T10:05:32.075 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T10:05:32.075 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T10:05:32.075 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:32.172 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T10:05:32.317 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T10:05:32.338 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T10:05:32.339 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:32.339 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T10:05:32.339 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T10:05:32.339 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T10:05:32.339 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:32.623 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T10:05:32.654 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T10:05:32.686 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T10:05:32.745 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T10:05:32.784 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T10:05:32.785 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T10:05:32.806 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T10:05:32.807 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:32.807 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T10:05:32.807 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T10:05:32.807 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T10:05:32.807 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:32.894 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T10:05:33.018 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T10:05:33.045 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T10:05:33.261 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T10:05:33.293 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T10:05:33.324 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T10:05:33.384 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T10:05:33.407 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T10:05:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T10:05:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T10:05:34.014 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T10:05:34.045 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T10:05:34.147 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T10:05:34.317 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T10:05:34.348 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T10:05:34.371 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T10:05:34.371 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:34.371 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T10:05:34.371 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T10:05:34.372 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T10:05:34.372 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:34.415 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T10:05:34.426 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T10:05:34.471 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T10:05:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T10:05:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T10:05:34.776 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T10:05:34.940 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T10:05:34.986 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T10:05:35.009 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T10:05:35.010 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:35.010 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T10:05:35.010 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T10:05:35.010 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T10:05:35.010 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:35.053 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T10:05:35.064 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T10:05:35.109 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T10:05:35.343 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T10:05:35.367 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T10:05:35.391 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T10:05:35.391 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:35.391 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T10:05:35.391 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T10:05:35.391 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T10:05:35.391 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:35.435 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T10:05:35.457 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T10:05:35.457 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:35.457 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T10:05:35.457 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:35.502 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:05:35.525 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:05:35.525 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:35.525 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T10:05:35.525 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T10:05:35.525 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T10:05:35.525 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:35.836 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T10:05:35.858 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T10:05:35.859 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:35.859 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T10:05:35.859 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T10:05:35.859 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T10:05:35.859 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:35.993 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T10:05:36.031 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T10:05:36.053 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T10:05:36.053 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:36.053 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T10:05:36.053 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T10:05:36.054 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T10:05:36.054 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:36.098 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T10:05:36.120 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T10:05:36.120 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:36.120 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T10:05:36.120 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:36.165 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:05:36.187 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:05:36.187 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:36.187 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T10:05:36.187 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T10:05:36.188 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T10:05:36.188 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:36.480 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T10:05:36.509 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T10:05:36.521 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T10:05:36.530 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T10:05:36.530 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T10:05:36.530 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T10:05:36.530 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T10:05:36.530 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T10:05:36.531 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:36.551 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T10:05:36.570 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T10:05:36.751 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T10:05:36.861 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T10:05:36.910 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T10:05:36.968 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T10:05:37.026 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T10:05:37.062 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T10:05:37.094 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T10:05:37.127 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T10:05:37.156 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T10:05:37.235 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T10:05:37.309 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T10:05:37.346 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T10:05:37.966 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:05:38.002 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:05:38.022 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:05:38.830 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T10:05:38.922 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T10:05:38.963 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T10:05:39.052 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T10:05:39.084 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T10:05:39.109 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T10:05:39.134 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T10:05:39.160 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T10:05:39.160 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T10:05:39.257 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T10:05:39.286 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T10:05:39.286 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T10:05:39.371 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T10:05:39.397 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T10:05:39.430 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T10:05:39.456 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T10:05:39.481 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T10:05:39.521 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T10:05:39.562 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T10:05:39.594 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T10:05:39.644 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T10:05:39.661 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T10:05:39.686 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T10:05:39.711 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T10:05:39.718 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:05:39.718 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:05:39.906 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:05:39.906 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T10:05:39.989 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T10:05:40.014 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T10:05:40.038 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T10:05:40.071 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T10:05:40.098 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T10:05:40.164 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T10:05:40.301 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T10:05:40.342 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T10:05:40.372 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T10:05:40.399 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T10:05:40.584 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T10:05:40.702 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T10:05:40.749 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T10:05:40.788 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T10:05:40.832 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T10:05:40.875 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T10:05:40.908 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T10:05:40.940 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T10:05:40.969 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T10:05:41.020 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T10:05:41.024 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T10:05:41.070 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T10:05:41.086 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T10:05:41.100 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T10:05:41.118 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T10:05:41.127 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T10:05:41.313 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T10:05:41.437 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T10:05:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T10:05:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T10:05:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T10:05:41.610 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T10:05:41.643 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T10:05:41.673 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:05:41.675 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T10:05:41.709 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:05:41.713 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T10:05:41.729 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:05:41.773 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T10:05:41.829 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T10:05:41.870 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T10:05:42.449 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:05:42.484 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:05:42.504 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T10:05:42.557 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T10:05:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T10:05:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T10:05:42.760 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T10:05:42.808 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T10:05:42.833 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T10:05:42.859 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T10:05:42.885 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T10:05:42.885 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T10:05:42.973 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T10:05:43.012 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T10:05:43.012 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T10:05:43.096 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T10:05:43.122 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T10:05:43.155 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T10:05:43.181 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T10:05:43.205 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T10:05:43.238 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T10:05:43.279 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T10:05:43.311 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T10:05:43.333 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T10:05:43.336 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T10:05:43.361 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T10:05:43.387 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T10:05:43.412 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T10:05:43.419 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:05:43.419 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:05:43.421 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T10:05:43.470 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T10:05:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T10:05:43.598 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:05:43.598 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T10:05:43.600 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T10:05:43.625 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T10:05:43.650 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T10:05:43.677 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T10:05:43.677 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T10:05:43.690 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T10:05:43.714 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T10:05:43.740 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T10:05:43.764 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T10:05:43.773 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T10:05:43.794 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T10:05:43.794 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T10:05:43.799 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T10:05:43.863 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T10:05:43.887 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T10:05:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T10:05:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T10:05:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T10:05:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T10:05:44.047 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T10:05:44.095 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T10:05:44.135 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T10:05:44.160 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T10:05:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T10:05:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T10:05:44.236 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T10:05:44.243 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:05:44.243 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:05:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T10:05:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T10:05:44.506 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T10:05:44.530 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T10:05:44.556 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T10:05:44.588 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T10:05:44.615 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T10:05:44.679 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T10:05:45.152 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T10:05:45.152 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T10:05:45.153 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T10:05:45.153 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T10:05:45.153 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T10:05:45.153 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T10:05:45.153 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T10:05:45.153 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T10:05:45.153 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T10:05:45.153 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T10:05:45.153 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:05:45.153 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:05:45.153 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T10:05:45.153 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T10:05:45.154 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T10:05:45.155 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T10:05:45.155 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T10:05:45.170 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T10:05:45.170 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T10:05:45.170 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T10:05:45.171 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T10:05:45.172 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T10:05:45.173 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T10:05:45.174 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T10:05:45.175 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T10:05:45.176 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T10:05:45.177 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T10:05:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T10:05:45.179 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T10:05:45.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T10:05:45.181 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T10:05:45.182 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T10:05:45.182 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T10:05:45.182 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T10:05:45.182 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T10:05:45.182 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T10:05:45.182 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T10:05:45.182 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T10:05:45.182 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T10:05:45.182 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T10:05:45.182 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T10:05:48.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T10:05:48.306 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:48.306 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-03-16T10:05:48.306 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T10:05:48.306 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T10:05:48.306 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T10:05:48.306 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T10:05:48.306 INFO:teuthology.orchestra.run.smithi077.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T10:05:48.306 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T10:05:48.307 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.308 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:48.309 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T10:05:48.310 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: jq-1.6-16.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.311 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T10:05:48.312 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.313 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T10:05:48.314 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T10:05:48.315 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T10:05:48.316 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T10:05:48.317 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T10:05:48.318 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T10:05:48.319 INFO:teuthology.orchestra.run.smithi077.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T10:05:48.319 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T10:05:48.319 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T10:05:48.319 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T10:05:48.319 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:05:48.319 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-03-16T10:05:48.631 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T10:05:48.632 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T10:05:48.633 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T10:05:48.652 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T10:05:48.652 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T10:05:48.652 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T10:05:48.652 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T10:05:48.652 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T10:05:48.652 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T10:05:48.652 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T10:05:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T10:05:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T10:05:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T10:05:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T10:05:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T10:05:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T10:05:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T10:05:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T10:05:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T10:05:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T10:05:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T10:05:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T10:05:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T10:05:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T10:05:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T10:05:48.654 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T10:05:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T10:05:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T10:05:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T10:05:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T10:05:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T10:05:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T10:05:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T10:05:48.655 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T10:05:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T10:05:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T10:05:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T10:05:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T10:05:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T10:05:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T10:05:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T10:05:48.656 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T10:05:48.657 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T10:05:48.658 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T10:05:48.659 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T10:05:48.660 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T10:05:48.661 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T10:05:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T10:05:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T10:05:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T10:05:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T10:05:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T10:05:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T10:05:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T10:05:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T10:05:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T10:05:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T10:05:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T10:05:48.662 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T10:05:48.663 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T10:05:48.664 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T10:05:48.665 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T10:05:48.666 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T10:05:48.666 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T10:05:48.666 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T10:05:48.666 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T10:05:48.666 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T10:05:48.666 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T10:05:48.666 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T10:05:48.666 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T10:05:48.798 DEBUG:teuthology.parallel:result is None 2024-03-16T10:05:49.489 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T10:05:49.489 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T10:05:49.489 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T10:05:49.490 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T10:05:49.491 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T10:05:49.492 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T10:05:49.493 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T10:05:49.494 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T10:05:49.494 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T10:05:49.494 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T10:05:49.494 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T10:05:49.494 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T10:05:49.494 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T10:05:49.494 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T10:05:49.494 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T10:05:49.494 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T10:05:49.494 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T10:05:49.494 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T10:05:49.494 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T10:05:49.495 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T10:05:49.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T10:05:49.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T10:05:49.498 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T10:05:49.499 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T10:05:49.500 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T10:05:49.501 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T10:05:49.502 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T10:05:49.503 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T10:05:49.503 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T10:05:49.503 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T10:05:49.503 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T10:05:51.778 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T10:05:51.779 INFO:teuthology.orchestra.run.smithi193.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T10:05:51.780 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:51.781 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.782 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: jq-1.6-16.el9.x86_64 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T10:05:51.783 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T10:05:51.784 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T10:05:51.785 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T10:05:51.786 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T10:05:51.787 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T10:05:51.788 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T10:05:51.789 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.790 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:51.791 INFO:teuthology.orchestra.run.smithi193.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T10:05:51.791 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T10:05:51.791 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T10:05:51.791 INFO:teuthology.orchestra.run.smithi193.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T10:05:51.791 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T10:05:51.791 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T10:05:51.791 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T10:05:51.791 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:05:51.791 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-16T10:05:52.238 DEBUG:teuthology.parallel:result is None 2024-03-16T10:05:52.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T10:05:52.509 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:52.509 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T10:05:52.510 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T10:05:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:52.512 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T10:05:52.513 INFO:teuthology.orchestra.run.smithi192.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: jq-1.6-16.el9.x86_64 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.514 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T10:05:52.515 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T10:05:52.516 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T10:05:52.517 INFO:teuthology.orchestra.run.smithi192.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T10:05:52.518 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T10:05:52.519 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T10:05:52.520 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.521 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.522 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T10:05:52.522 INFO:teuthology.orchestra.run.smithi192.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T10:05:52.522 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T10:05:52.522 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T10:05:52.522 INFO:teuthology.orchestra.run.smithi192.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T10:05:52.522 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T10:05:52.522 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T10:05:52.522 INFO:teuthology.orchestra.run.smithi192.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T10:05:52.522 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-16T10:05:52.523 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-16T10:05:52.972 DEBUG:teuthology.parallel:result is None 2024-03-16T10:05:52.973 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:05:53.196 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T10:05:53.221 INFO:teuthology.orchestra.run.smithi077.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T10:05:53.222 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T10:05:53.222 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T10:05:53.224 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:05:53.437 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T10:05:53.462 INFO:teuthology.orchestra.run.smithi192.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T10:05:53.463 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T10:05:53.463 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T10:05:53.464 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:05:53.680 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T10:05:53.705 INFO:teuthology.orchestra.run.smithi193.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T10:05:53.705 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T10:05:53.705 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T10:05:53.707 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-16T10:05:53.707 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:05:53.707 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T10:05:53.741 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-16T10:05:53.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T10:05:53.776 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-16T10:05:53.777 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T10:05:53.811 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-16T10:05:53.812 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:05:53.812 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T10:05:53.841 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T10:05:53.911 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-16T10:05:53.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T10:05:53.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T10:05:54.014 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-16T10:05:54.014 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T10:05:54.045 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T10:05:54.114 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-16T10:05:54.115 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:05:54.115 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T10:05:54.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T10:05:54.212 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-16T10:05:54.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T10:05:54.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T10:05:54.313 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-16T10:05:54.313 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T10:05:54.344 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T10:05:54.414 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-16T10:05:54.414 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:05:54.414 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T10:05:54.443 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T10:05:54.511 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-16T10:05:54.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T10:05:54.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T10:05:54.612 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-16T10:05:54.612 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T10:05:54.643 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T10:05:54.713 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-16T10:05:54.802 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'subvols': {'create': 2}} 2024-03-16T10:05:54.802 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:05:54.802 INFO:tasks.cephadm:Cluster fsid is c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:05:54.803 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-16T10:05:54.803 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.77', 'mon.b': '172.21.15.192', 'mon.c': '172.21.15.193'} 2024-03-16T10:05:54.803 INFO:tasks.cephadm:First mon is mon.a on smithi077 2024-03-16T10:05:54.803 INFO:tasks.cephadm:First mgr is x 2024-03-16T10:05:54.803 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-16T10:05:54.803 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname $(hostname -s) 2024-03-16T10:05:54.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2024-03-16T10:05:54.863 DEBUG:teuthology.orchestra.run.smithi193:> sudo hostname $(hostname -s) 2024-03-16T10:05:54.894 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-16T10:05:54.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:05:55.044 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77658/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2162-g3d78f5ba', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2162.g3d78f5ba'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 01:33:04.210510', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240314.183243', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-16T10:05:55.263 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-batrick-testing-20240314.183243, sha1 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:05:55.264 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm 2024-03-16T10:05:55.264 DEBUG:teuthology.orchestra.run.smithi077:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T10:05:55.831 INFO:teuthology.orchestra.run.smithi077.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 10:05 /home/ubuntu/cephtest/cephadm 2024-03-16T10:05:55.832 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T10:05:56.261 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 10:05 /home/ubuntu/cephtest/cephadm 2024-03-16T10:05:56.261 DEBUG:teuthology.orchestra.run.smithi193:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T10:05:56.703 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 10:05 /home/ubuntu/cephtest/cephadm 2024-03-16T10:05:56.704 DEBUG:teuthology.orchestra.run.smithi077:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T10:05:56.723 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T10:05:56.741 DEBUG:teuthology.orchestra.run.smithi193:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T10:05:56.776 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af on all hosts... 2024-03-16T10:05:56.776 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T10:05:56.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T10:05:56.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T10:05:57.632 INFO:teuthology.orchestra.run.smithi193.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T10:05:57.633 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T10:05:57.744 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T10:06:27.531 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-16T10:06:27.531 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T10:06:27.531 INFO:teuthology.orchestra.run.smithi193.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T10:06:27.532 INFO:teuthology.orchestra.run.smithi193.stdout: "repo_digests": [ 2024-03-16T10:06:27.532 INFO:teuthology.orchestra.run.smithi193.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T10:06:27.532 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-16T10:06:27.532 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-16T10:06:48.485 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-16T10:06:48.486 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T10:06:48.486 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T10:06:48.486 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2024-03-16T10:06:48.486 INFO:teuthology.orchestra.run.smithi192.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T10:06:48.486 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-16T10:06:48.486 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-16T10:06:49.101 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:06:49.101 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T10:06:49.101 INFO:teuthology.orchestra.run.smithi077.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T10:06:49.102 INFO:teuthology.orchestra.run.smithi077.stdout: "repo_digests": [ 2024-03-16T10:06:49.102 INFO:teuthology.orchestra.run.smithi077.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T10:06:49.102 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T10:06:49.102 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:06:49.125 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2024-03-16T10:06:49.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2024-03-16T10:06:49.196 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph 2024-03-16T10:06:49.232 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /etc/ceph 2024-03-16T10:06:49.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2024-03-16T10:06:49.290 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /etc/ceph 2024-03-16T10:06:49.321 INFO:tasks.cephadm:Writing seed config... 2024-03-16T10:06:49.322 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-16T10:06:49.322 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-16T10:06:49.322 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-16T10:06:49.322 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-16T10:06:49.322 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-16T10:06:49.323 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-16T10:06:49.324 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-16T10:06:49.325 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-16T10:06:49.325 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-16T10:06:49.325 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-16T10:06:49.325 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-16T10:06:49.325 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-16T10:06:49.325 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-16T10:06:49.325 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-16T10:06:49.325 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-16T10:06:49.325 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-16T10:06:49.325 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-16T10:06:49.325 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-16T10:06:49.326 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:06:49.326 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-16T10:06:49.341 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c6ee6c60-e37c-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-16T10:06:49.341 DEBUG:teuthology.orchestra.run.smithi077:mon.a> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.a.service 2024-03-16T10:06:49.383 DEBUG:teuthology.orchestra.run.smithi077:mgr.x> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.x.service 2024-03-16T10:06:49.426 INFO:tasks.cephadm:Bootstrapping... 2024-03-16T10:06:49.426 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -v bootstrap --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.77 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-16T10:06:49.614 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-03-16T10:06:49.614 INFO:teuthology.orchestra.run.smithi077.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', '-v', 'bootstrap', '--fsid', 'c6ee6c60-e37c-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.77', '--skip-admin-label'] 2024-03-16T10:06:49.639 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T10:06:49.639 INFO:teuthology.orchestra.run.smithi077.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-16T10:06:49.640 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying podman|docker is present... 2024-03-16T10:06:49.662 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T10:06:49.662 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying lvm2 is present... 2024-03-16T10:06:49.662 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying time synchronization is in place... 2024-03-16T10:06:49.670 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T10:06:49.670 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T10:06:49.676 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T10:06:49.676 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-03-16T10:06:49.682 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout enabled 2024-03-16T10:06:49.688 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout active 2024-03-16T10:06:49.688 INFO:teuthology.orchestra.run.smithi077.stdout:Unit chronyd.service is enabled and running 2024-03-16T10:06:49.688 INFO:teuthology.orchestra.run.smithi077.stdout:Repeating the final host check... 2024-03-16T10:06:49.711 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T10:06:49.712 INFO:teuthology.orchestra.run.smithi077.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-16T10:06:49.712 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl is present 2024-03-16T10:06:49.712 INFO:teuthology.orchestra.run.smithi077.stdout:lvcreate is present 2024-03-16T10:06:49.718 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T10:06:49.718 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T10:06:49.724 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T10:06:49.724 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-03-16T10:06:49.730 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout enabled 2024-03-16T10:06:49.736 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout active 2024-03-16T10:06:49.736 INFO:teuthology.orchestra.run.smithi077.stdout:Unit chronyd.service is enabled and running 2024-03-16T10:06:49.736 INFO:teuthology.orchestra.run.smithi077.stdout:Host looks OK 2024-03-16T10:06:49.736 INFO:teuthology.orchestra.run.smithi077.stdout:Cluster fsid: c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:06:49.736 INFO:teuthology.orchestra.run.smithi077.stdout:Acquiring lock 139802470288832 on /run/cephadm/c6ee6c60-e37c-11ee-95c9-87774f69a715.lock 2024-03-16T10:06:49.736 INFO:teuthology.orchestra.run.smithi077.stdout:Lock 139802470288832 acquired on /run/cephadm/c6ee6c60-e37c-11ee-95c9-87774f69a715.lock 2024-03-16T10:06:49.736 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying IP 172.21.15.77 port 3300 ... 2024-03-16T10:06:49.737 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying IP 172.21.15.77 port 6789 ... 2024-03-16T10:06:49.737 INFO:teuthology.orchestra.run.smithi077.stdout:Base mon IP(s) is [172.21.15.77:3300, 172.21.15.77:6789], mon addrv is [v2:172.21.15.77:3300,v1:172.21.15.77:6789] 2024-03-16T10:06:49.740 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.77 metric 100 2024-03-16T10:06:49.740 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.77 metric 100 2024-03-16T10:06:49.743 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-16T10:06:49.743 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-16T10:06:49.743 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1726sec hoplimit 64 pref medium 2024-03-16T10:06:49.745 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-16T10:06:49.745 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-16T10:06:49.745 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T10:06:49.745 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-16T10:06:49.745 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-16T10:06:49.745 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T10:06:49.746 INFO:teuthology.orchestra.run.smithi077.stdout:Mon IP `172.21.15.77` is in CIDR network `172.21.0.0/20` 2024-03-16T10:06:49.746 INFO:teuthology.orchestra.run.smithi077.stdout:Mon IP `172.21.15.77` is in CIDR network `172.21.0.0/20` 2024-03-16T10:06:49.746 INFO:teuthology.orchestra.run.smithi077.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-16T10:06:49.746 INFO:teuthology.orchestra.run.smithi077.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-16T10:06:49.747 INFO:teuthology.orchestra.run.smithi077.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T10:06:50.170 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stdout be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-16T10:06:50.170 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T10:06:50.170 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Getting image source signatures 2024-03-16T10:06:50.170 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Copying blob sha256:66487374cb611c95435850c17b53ebb64c22ed9555f72cd43d678b60262a12b1 2024-03-16T10:06:50.170 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-16T10:06:50.170 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Copying config sha256:be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-16T10:06:50.170 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-16T10:06:51.125 INFO:teuthology.orchestra.run.smithi077.stdout:ceph: stdout ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-16T10:06:51.125 INFO:teuthology.orchestra.run.smithi077.stdout:Ceph version: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-16T10:06:51.125 INFO:teuthology.orchestra.run.smithi077.stdout:Extracting ceph user uid/gid from container image... 2024-03-16T10:06:51.980 INFO:teuthology.orchestra.run.smithi077.stdout:stat: stdout 167 167 2024-03-16T10:06:51.980 INFO:teuthology.orchestra.run.smithi077.stdout:Creating initial keys... 2024-03-16T10:06:52.869 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph-authtool: stdout AQA8b/VlsNWzERAAikrBJWCj6Ri7UClU4oQo+w== 2024-03-16T10:06:53.749 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph-authtool: stdout AQA9b/VlaH2gCxAA5lPKgLz5z0qrbxr8YdJFtA== 2024-03-16T10:06:54.647 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph-authtool: stdout AQA+b/VlsTgCBRAAUF2f9QO3HLuKaazjzgw+hw== 2024-03-16T10:06:54.647 INFO:teuthology.orchestra.run.smithi077.stdout:Creating initial monmap... 2024-03-16T10:06:55.587 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T10:06:55.587 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-16T10:06:55.587 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:06:55.587 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T10:06:55.587 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool for a [v2:172.21.15.77:3300,v1:172.21.15.77:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T10:06:55.587 INFO:teuthology.orchestra.run.smithi077.stdout:setting min_mon_release = quincy 2024-03-16T10:06:55.587 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: set fsid to c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:06:55.588 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T10:06:55.588 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:06:55.588 INFO:teuthology.orchestra.run.smithi077.stdout:Creating mon... 2024-03-16T10:06:56.559 INFO:teuthology.orchestra.run.smithi077.stdout:create mon.a on 2024-03-16T10:06:56.774 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-16T10:06:56.929 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-16T10:06:57.107 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715.target. 2024-03-16T10:06:57.108 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715.target. 2024-03-16T10:06:57.356 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.a 2024-03-16T10:06:57.356 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-16T10:06:57.520 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715.target.wants/ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@.service. 2024-03-16T10:06:58.024 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T10:06:58.024 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout disabled 2024-03-16T10:06:58.031 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T10:06:58.032 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-03-16T10:06:58.032 INFO:teuthology.orchestra.run.smithi077.stdout:firewalld.service is not enabled 2024-03-16T10:06:58.032 INFO:teuthology.orchestra.run.smithi077.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T10:06:58.032 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mon to start... 2024-03-16T10:06:58.032 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mon... 2024-03-16T10:06:59.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout cluster: 2024-03-16T10:06:59.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout id: c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:06:59.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-16T10:06:59.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:06:59.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout services: 2024-03-16T10:06:59.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.821035s) 2024-03-16T10:06:59.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-16T10:06:59.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-16T10:06:59.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:06:59.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout data: 2024-03-16T10:06:59.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-16T10:06:59.353 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-16T10:06:59.353 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-16T10:06:59.353 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout pgs: 2024-03-16T10:06:59.353 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:06:59.353 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.760+0000 7f9406def700 1 Processor -- start 2024-03-16T10:06:59.353 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.760+0000 7f9406def700 1 -- start start 2024-03-16T10:06:59.353 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.760+0000 7f9406def700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 0x7f940010abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:06:59.353 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.760+0000 7f9406def700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f940007bdf0 con 0x7f94001087c0 2024-03-16T10:06:59.353 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.761+0000 7f9404b8b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 0x7f940010abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:06:59.353 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.761+0000 7f9404b8b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 0x7f940010abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41016/0 (socket says 172.21.15.77:41016) 2024-03-16T10:06:59.354 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.761+0000 7f9404b8b700 1 -- 172.21.15.77:0/3260791836 learned_addr learned my addr 172.21.15.77:0/3260791836 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:06:59.354 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.761+0000 7f9404b8b700 1 -- 172.21.15.77:0/3260791836 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f940010b0f0 con 0x7f94001087c0 2024-03-16T10:06:59.354 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.761+0000 7f9404b8b700 1 --2- 172.21.15.77:0/3260791836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 0x7f940010abb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f93f40098a0 tx=0x7f93f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c3d58f78e7a20c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:06:59.354 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.761+0000 7f93ff7fe700 1 -- 172.21.15.77:0/3260791836 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93f4004160 con 0x7f94001087c0 2024-03-16T10:06:59.354 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.762+0000 7f93ff7fe700 1 -- 172.21.15.77:0/3260791836 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f4004300 con 0x7f94001087c0 2024-03-16T10:06:59.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.762+0000 7f93ff7fe700 1 -- 172.21.15.77:0/3260791836 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93f40036a0 con 0x7f94001087c0 2024-03-16T10:06:59.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.762+0000 7f9406def700 1 -- 172.21.15.77:0/3260791836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 msgr2=0x7f940010abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:06:59.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.762+0000 7f9406def700 1 --2- 172.21.15.77:0/3260791836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 0x7f940010abb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f93f40098a0 tx=0x7f93f4009bb0 comp rx=0 tx=0).stop 2024-03-16T10:06:59.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.762+0000 7f9406def700 1 -- 172.21.15.77:0/3260791836 shutdown_connections 2024-03-16T10:06:59.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.762+0000 7f9406def700 1 --2- 172.21.15.77:0/3260791836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 0x7f940010abb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:06:59.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.762+0000 7f9406def700 1 -- 172.21.15.77:0/3260791836 >> 172.21.15.77:0/3260791836 conn(0x7f9400102110 msgr2=0x7f9400104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:06:59.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.762+0000 7f9406def700 1 -- 172.21.15.77:0/3260791836 shutdown_connections 2024-03-16T10:06:59.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.762+0000 7f9406def700 1 -- 172.21.15.77:0/3260791836 wait complete. 2024-03-16T10:06:59.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.763+0000 7f9406def700 1 Processor -- start 2024-03-16T10:06:59.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.763+0000 7f9406def700 1 -- start start 2024-03-16T10:06:59.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.763+0000 7f9406def700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 0x7f940019b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:06:59.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.763+0000 7f9406def700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f940010b500 con 0x7f94001087c0 2024-03-16T10:06:59.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.763+0000 7f9404b8b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 0x7f940019b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:06:59.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.763+0000 7f9404b8b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 0x7f940019b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41028/0 (socket says 172.21.15.77:41028) 2024-03-16T10:06:59.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.763+0000 7f9404b8b700 1 -- 172.21.15.77:0/4214348201 learned_addr learned my addr 172.21.15.77:0/4214348201 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:06:59.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.763+0000 7f9404b8b700 1 -- 172.21.15.77:0/4214348201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f940019bbd0 con 0x7f94001087c0 2024-03-16T10:06:59.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.763+0000 7f9404b8b700 1 --2- 172.21.15.77:0/4214348201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 0x7f940019b690 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f93f4000c00 tx=0x7f93f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:06:59.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.763+0000 7f93fdffb700 1 -- 172.21.15.77:0/4214348201 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93f4003db0 con 0x7f94001087c0 2024-03-16T10:06:59.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.764+0000 7f93fdffb700 1 -- 172.21.15.77:0/4214348201 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f4011630 con 0x7f94001087c0 2024-03-16T10:06:59.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.764+0000 7f93fdffb700 1 -- 172.21.15.77:0/4214348201 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93f4011950 con 0x7f94001087c0 2024-03-16T10:06:59.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.764+0000 7f9406def700 1 -- 172.21.15.77:0/4214348201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f940019be60 con 0x7f94001087c0 2024-03-16T10:06:59.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.764+0000 7f9406def700 1 -- 172.21.15.77:0/4214348201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f940019eb00 con 0x7f94001087c0 2024-03-16T10:06:59.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.764+0000 7f93fdffb700 1 -- 172.21.15.77:0/4214348201 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f93f4011af0 con 0x7f94001087c0 2024-03-16T10:06:59.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.764+0000 7f93fdffb700 1 -- 172.21.15.77:0/4214348201 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f93f401c450 con 0x7f94001087c0 2024-03-16T10:06:59.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.765+0000 7f9406def700 1 -- 172.21.15.77:0/4214348201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9400063900 con 0x7f94001087c0 2024-03-16T10:06:59.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.767+0000 7f93fdffb700 1 -- 172.21.15.77:0/4214348201 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f93f4004260 con 0x7f94001087c0 2024-03-16T10:06:59.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.836+0000 7f9406def700 1 -- 172.21.15.77:0/4214348201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f940002db70 con 0x7f94001087c0 2024-03-16T10:06:59.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.836+0000 7f93fdffb700 1 -- 172.21.15.77:0/4214348201 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f93f401b450 con 0x7f94001087c0 2024-03-16T10:06:59.359 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.837+0000 7f9406def700 1 -- 172.21.15.77:0/4214348201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 msgr2=0x7f940019b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:06:59.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.837+0000 7f9406def700 1 --2- 172.21.15.77:0/4214348201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 0x7f940019b690 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f93f4000c00 tx=0x7f93f4004750 comp rx=0 tx=0).stop 2024-03-16T10:06:59.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.837+0000 7f9406def700 1 -- 172.21.15.77:0/4214348201 shutdown_connections 2024-03-16T10:06:59.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.837+0000 7f9406def700 1 --2- 172.21.15.77:0/4214348201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94001087c0 0x7f940019b690 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:06:59.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.837+0000 7f9406def700 1 -- 172.21.15.77:0/4214348201 >> 172.21.15.77:0/4214348201 conn(0x7f9400102110 msgr2=0x7f9400103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:06:59.360 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.837+0000 7f9406def700 1 -- 172.21.15.77:0/4214348201 shutdown_connections 2024-03-16T10:06:59.361 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:58.837+0000 7f9406def700 1 -- 172.21.15.77:0/4214348201 wait complete. 2024-03-16T10:06:59.361 INFO:teuthology.orchestra.run.smithi077.stdout:mon is available 2024-03-16T10:06:59.361 INFO:teuthology.orchestra.run.smithi077.stdout:Assimilating anything we can from ceph.conf... 2024-03-16T10:07:00.373 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:00 smithi077 ceph-mon[32024]: from='client.? 172.21.15.77:0/3483982718' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout fsid = c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.77:3300,v1:172.21.15.77:6789] 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:07:00.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.732+0000 7f4380052700 1 Processor -- start 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.732+0000 7f4380052700 1 -- start start 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.732+0000 7f4380052700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 0x7f437807c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.732+0000 7f4380052700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f437807e000 con 0x7f437807c320 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.733+0000 7f437ddee700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 0x7f437807c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.733+0000 7f437ddee700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 0x7f437807c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41030/0 (socket says 172.21.15.77:41030) 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.733+0000 7f437ddee700 1 -- 172.21.15.77:0/2032599464 learned_addr learned my addr 172.21.15.77:0/2032599464 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.733+0000 7f437ddee700 1 -- 172.21.15.77:0/2032599464 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437807cc80 con 0x7f437807c320 2024-03-16T10:07:00.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.733+0000 7f437ddee700 1 --2- 172.21.15.77:0/2032599464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 0x7f437807c740 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f43740098a0 tx=0x7f4374009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa84be2857e6dc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.733+0000 7f437cdec700 1 -- 172.21.15.77:0/2032599464 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4374004160 con 0x7f437807c320 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.733+0000 7f437cdec700 1 -- 172.21.15.77:0/2032599464 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4374004300 con 0x7f437807c320 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.733+0000 7f437cdec700 1 -- 172.21.15.77:0/2032599464 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43740036a0 con 0x7f437807c320 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.734+0000 7f4380052700 1 -- 172.21.15.77:0/2032599464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 msgr2=0x7f437807c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.734+0000 7f4380052700 1 --2- 172.21.15.77:0/2032599464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 0x7f437807c740 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f43740098a0 tx=0x7f4374009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.734+0000 7f4380052700 1 -- 172.21.15.77:0/2032599464 shutdown_connections 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.734+0000 7f4380052700 1 --2- 172.21.15.77:0/2032599464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 0x7f437807c740 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.734+0000 7f4380052700 1 -- 172.21.15.77:0/2032599464 >> 172.21.15.77:0/2032599464 conn(0x7f4378103320 msgr2=0x7f4378105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.734+0000 7f4380052700 1 -- 172.21.15.77:0/2032599464 shutdown_connections 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.734+0000 7f4380052700 1 -- 172.21.15.77:0/2032599464 wait complete. 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.735+0000 7f4380052700 1 Processor -- start 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.735+0000 7f4380052700 1 -- start start 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.735+0000 7f4380052700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 0x7f437819d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.735+0000 7f4380052700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f437807d1b0 con 0x7f437807c320 2024-03-16T10:07:00.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.735+0000 7f437ddee700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 0x7f437819d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.735+0000 7f437ddee700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 0x7f437819d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41032/0 (socket says 172.21.15.77:41032) 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.735+0000 7f437ddee700 1 -- 172.21.15.77:0/3483982718 learned_addr learned my addr 172.21.15.77:0/3483982718 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.735+0000 7f437ddee700 1 -- 172.21.15.77:0/3483982718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437819db50 con 0x7f437807c320 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.735+0000 7f437ddee700 1 --2- 172.21.15.77:0/3483982718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 0x7f437819d610 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4374000c00 tx=0x7f4374004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.735+0000 7f436effd700 1 -- 172.21.15.77:0/3483982718 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4374003db0 con 0x7f437807c320 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.736+0000 7f436effd700 1 -- 172.21.15.77:0/3483982718 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4374011630 con 0x7f437807c320 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.736+0000 7f4380052700 1 -- 172.21.15.77:0/3483982718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f437819dde0 con 0x7f437807c320 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.736+0000 7f436effd700 1 -- 172.21.15.77:0/3483982718 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4374011950 con 0x7f437807c320 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.736+0000 7f4380052700 1 -- 172.21.15.77:0/3483982718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43781a0ad0 con 0x7f437807c320 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.736+0000 7f436effd700 1 -- 172.21.15.77:0/3483982718 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4374011af0 con 0x7f437807c320 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.736+0000 7f436effd700 1 -- 172.21.15.77:0/3483982718 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f437401c450 con 0x7f437807c320 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.737+0000 7f4380052700 1 -- 172.21.15.77:0/3483982718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4378063900 con 0x7f437807c320 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.739+0000 7f436effd700 1 -- 172.21.15.77:0/3483982718 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4374032020 con 0x7f437807c320 2024-03-16T10:07:00.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.804+0000 7f4380052700 1 -- 172.21.15.77:0/3483982718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f437802dc10 con 0x7f437807c320 2024-03-16T10:07:00.386 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.808+0000 7f436effd700 1 -- 172.21.15.77:0/3483982718 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f4374004260 con 0x7f437807c320 2024-03-16T10:07:00.386 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.808+0000 7f436effd700 1 -- 172.21.15.77:0/3483982718 <== mon.0 v2:172.21.15.77:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f437401cb60 con 0x7f437807c320 2024-03-16T10:07:00.386 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.808+0000 7f4380052700 1 -- 172.21.15.77:0/3483982718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 msgr2=0x7f437819d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:00.386 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.808+0000 7f4380052700 1 --2- 172.21.15.77:0/3483982718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 0x7f437819d610 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4374000c00 tx=0x7f4374004750 comp rx=0 tx=0).stop 2024-03-16T10:07:00.386 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.808+0000 7f4380052700 1 -- 172.21.15.77:0/3483982718 shutdown_connections 2024-03-16T10:07:00.386 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.808+0000 7f4380052700 1 --2- 172.21.15.77:0/3483982718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f437807c320 0x7f437819d610 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:00.386 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.808+0000 7f4380052700 1 -- 172.21.15.77:0/3483982718 >> 172.21.15.77:0/3483982718 conn(0x7f4378103320 msgr2=0x7f43781041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:00.386 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.809+0000 7f4380052700 1 -- 172.21.15.77:0/3483982718 shutdown_connections 2024-03-16T10:07:00.386 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:06:59.809+0000 7f4380052700 1 -- 172.21.15.77:0/3483982718 wait complete. 2024-03-16T10:07:00.386 INFO:teuthology.orchestra.run.smithi077.stdout:Generating new minimal ceph.conf... 2024-03-16T10:07:01.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:01 smithi077 ceph-mon[32024]: from='client.? 172.21.15.77:0/991775895' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:07:01.503 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.807+0000 7f06dfabb700 1 Processor -- start 2024-03-16T10:07:01.503 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.807+0000 7f06dfabb700 1 -- start start 2024-03-16T10:07:01.503 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.808+0000 7f06dfabb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 0x7f06d8106f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:01.503 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.808+0000 7f06dfabb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f06d8107540 con 0x7f06d8106b50 2024-03-16T10:07:01.503 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.808+0000 7f06dd857700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 0x7f06d8106f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:01.503 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.808+0000 7f06dd857700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 0x7f06d8106f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41036/0 (socket says 172.21.15.77:41036) 2024-03-16T10:07:01.503 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.808+0000 7f06dd857700 1 -- 172.21.15.77:0/4057220710 learned_addr learned my addr 172.21.15.77:0/4057220710 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:01.503 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.808+0000 7f06dd857700 1 -- 172.21.15.77:0/4057220710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d8107d50 con 0x7f06d8106b50 2024-03-16T10:07:01.503 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.808+0000 7f06dd857700 1 --2- 172.21.15.77:0/4057220710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 0x7f06d8106f70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f06c80098a0 tx=0x7f06c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f834509a86a8cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:01.503 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.808+0000 7f06dc855700 1 -- 172.21.15.77:0/4057220710 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06c8004160 con 0x7f06d8106b50 2024-03-16T10:07:01.503 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.809+0000 7f06dc855700 1 -- 172.21.15.77:0/4057220710 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f06c8004300 con 0x7f06d8106b50 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.809+0000 7f06dc855700 1 -- 172.21.15.77:0/4057220710 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06c8011b50 con 0x7f06d8106b50 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.809+0000 7f06dfabb700 1 -- 172.21.15.77:0/4057220710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 msgr2=0x7f06d8106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.809+0000 7f06dfabb700 1 --2- 172.21.15.77:0/4057220710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 0x7f06d8106f70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f06c80098a0 tx=0x7f06c8009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.809+0000 7f06dfabb700 1 -- 172.21.15.77:0/4057220710 shutdown_connections 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.809+0000 7f06dfabb700 1 --2- 172.21.15.77:0/4057220710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 0x7f06d8106f70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.809+0000 7f06dfabb700 1 -- 172.21.15.77:0/4057220710 >> 172.21.15.77:0/4057220710 conn(0x7f06d8102130 msgr2=0x7f06d8104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.809+0000 7f06dfabb700 1 -- 172.21.15.77:0/4057220710 shutdown_connections 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.809+0000 7f06dfabb700 1 -- 172.21.15.77:0/4057220710 wait complete. 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.810+0000 7f06dfabb700 1 Processor -- start 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.810+0000 7f06dfabb700 1 -- start start 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.810+0000 7f06dfabb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 0x7f06d819bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.810+0000 7f06dfabb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f06d810b100 con 0x7f06d8106b50 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.810+0000 7f06dd857700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 0x7f06d819bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:01.504 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.810+0000 7f06dd857700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 0x7f06d819bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41040/0 (socket says 172.21.15.77:41040) 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.810+0000 7f06dd857700 1 -- 172.21.15.77:0/991775895 learned_addr learned my addr 172.21.15.77:0/991775895 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.810+0000 7f06dd857700 1 -- 172.21.15.77:0/991775895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d819c160 con 0x7f06d8106b50 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.810+0000 7f06dd857700 1 --2- 172.21.15.77:0/991775895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 0x7f06d819bc20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f06c8011670 tx=0x7f06c8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.811+0000 7f06ceffd700 1 -- 172.21.15.77:0/991775895 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06c80039c0 con 0x7f06d8106b50 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.811+0000 7f06ceffd700 1 -- 172.21.15.77:0/991775895 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f06c8011e50 con 0x7f06d8106b50 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.811+0000 7f06ceffd700 1 -- 172.21.15.77:0/991775895 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06c801a5c0 con 0x7f06d8106b50 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.811+0000 7f06dfabb700 1 -- 172.21.15.77:0/991775895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d819c3f0 con 0x7f06d8106b50 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.811+0000 7f06dfabb700 1 -- 172.21.15.77:0/991775895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d819f0e0 con 0x7f06d8106b50 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.811+0000 7f06ceffd700 1 -- 172.21.15.77:0/991775895 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f06c801ac00 con 0x7f06d8106b50 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.811+0000 7f06ceffd700 1 -- 172.21.15.77:0/991775895 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f06c8019700 con 0x7f06d8106b50 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.812+0000 7f06dfabb700 1 -- 172.21.15.77:0/991775895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d8063900 con 0x7f06d8106b50 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.814+0000 7f06ceffd700 1 -- 172.21.15.77:0/991775895 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f06c8003b60 con 0x7f06d8106b50 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.880+0000 7f06dfabb700 1 -- 172.21.15.77:0/991775895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f06d819f2b0 con 0x7f06d8106b50 2024-03-16T10:07:01.505 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.881+0000 7f06ceffd700 1 -- 172.21.15.77:0/991775895 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f06c80199a0 con 0x7f06d8106b50 2024-03-16T10:07:01.506 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.881+0000 7f06dfabb700 1 -- 172.21.15.77:0/991775895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 msgr2=0x7f06d819bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:01.506 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.881+0000 7f06dfabb700 1 --2- 172.21.15.77:0/991775895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 0x7f06d819bc20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f06c8011670 tx=0x7f06c8011b60 comp rx=0 tx=0).stop 2024-03-16T10:07:01.506 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.881+0000 7f06dfabb700 1 -- 172.21.15.77:0/991775895 shutdown_connections 2024-03-16T10:07:01.506 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.881+0000 7f06dfabb700 1 --2- 172.21.15.77:0/991775895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06d8106b50 0x7f06d819bc20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:01.506 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.881+0000 7f06dfabb700 1 -- 172.21.15.77:0/991775895 >> 172.21.15.77:0/991775895 conn(0x7f06d8102130 msgr2=0x7f06d8102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:01.506 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.881+0000 7f06dfabb700 1 -- 172.21.15.77:0/991775895 shutdown_connections 2024-03-16T10:07:01.506 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:00.882+0000 7f06dfabb700 1 -- 172.21.15.77:0/991775895 wait complete. 2024-03-16T10:07:01.506 INFO:teuthology.orchestra.run.smithi077.stdout:Restarting the monitor... 2024-03-16T10:07:01.749 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:01 smithi077 systemd[1]: Stopping Ceph mon.a for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:07:02.207 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:01 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a[32020]: 2024-03-16T10:07:01.748+0000 7fbe07dac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:07:02.207 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:01 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a[32020]: 2024-03-16T10:07:01.748+0000 7fbe07dac700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-16T10:07:02.207 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:01 smithi077 podman[32272]: 2024-03-16 10:07:01.912359422 +0000 UTC m=+0.251010911 container died 9844b00272d4585632af8e9af8d44dfa500e57234c91a73a3fc1b65b454b3315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:07:02.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:02 smithi077 podman[32272]: 2024-03-16 10:07:02.298111838 +0000 UTC m=+0.636763335 container cleanup 9844b00272d4585632af8e9af8d44dfa500e57234c91a73a3fc1b65b454b3315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-16T10:07:02.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:02 smithi077 bash[32272]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a 2024-03-16T10:07:02.852 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:02 smithi077 podman[32286]: 2024-03-16 10:07:02.515575628 +0000 UTC m=+0.595274167 container remove 9844b00272d4585632af8e9af8d44dfa500e57234c91a73a3fc1b65b454b3315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-16T10:07:02.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:02 smithi077 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-16T10:07:02.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:02 smithi077 systemd[1]: Stopped Ceph mon.a for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:07:02.853 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:02 smithi077 systemd[1]: Starting Ceph mon.a for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:07:03.103 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:02 smithi077 podman[32381]: 2024-03-16 10:07:02.755945829 +0000 UTC m=+0.018088202 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:07:03.103 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:02 smithi077 podman[32381]: 2024-03-16 10:07:02.858118905 +0000 UTC m=+0.120261269 container create e223b065e454dab2fb1a8ac03938cd8ff6e5bf2079ab742e294be811e58ba028 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:07:03.103 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 podman[32381]: 2024-03-16 10:07:03.077084322 +0000 UTC m=+0.339226688 container init e223b065e454dab2fb1a8ac03938cd8ff6e5bf2079ab742e294be811e58ba028 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-16T10:07:03.103 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 podman[32381]: 2024-03-16 10:07:03.079894243 +0000 UTC m=+0.342036608 container start e223b065e454dab2fb1a8ac03938cd8ff6e5bf2079ab742e294be811e58ba028 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-03-16T10:07:03.174 INFO:teuthology.orchestra.run.smithi077.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-16T10:07:03.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T10:07:03.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-16T10:07:03.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: pidfile_write: ignore empty --pid-file 2024-03-16T10:07:03.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: load: jerasure load: lrc 2024-03-16T10:07:03.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: RocksDB version: 7.9.2 2024-03-16T10:07:03.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Git sha 0 2024-03-16T10:07:03.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-16T10:07:03.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: DB SUMMARY 2024-03-16T10:07:03.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: DB Session ID: QS70U02EU0Z7Y7P922X5 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: CURRENT file: CURRENT 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104892 ; 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.error_if_exists: 0 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.create_if_missing: 0 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.paranoid_checks: 1 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.env: 0x55a83eb77c60 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T10:07:03.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.info_log: 0x55a8412e3ac0 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.statistics: (nil) 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.use_fsync: 0 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_log_file_size: 0 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.allow_fallocate: 1 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.use_direct_reads: 0 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T10:07:03.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.db_log_dir: 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.wal_dir: 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.write_buffer_manager: 0x55a8412b61e0 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T10:07:03.400 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.unordered_write: 0 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.row_cache: None 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.wal_filter: None 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.two_write_queues: 0 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.wal_compression: 0 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.atomic_flush: 0 2024-03-16T10:07:03.401 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T10:07:03.402 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T10:07:03.402 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T10:07:03.402 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.log_readahead_size: 0 2024-03-16T10:07:03.402 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T10:07:03.402 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T10:07:03.402 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T10:07:03.402 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T10:07:03.402 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T10:07:03.402 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T10:07:03.402 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T10:07:03.403 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_background_jobs: 2 2024-03-16T10:07:03.403 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_background_compactions: -1 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_subcompactions: 1 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_open_files: -1 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T10:07:03.404 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_background_flushes: -1 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Compression algorithms supported: 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: kZSTD supported: 0 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: kXpressCompression supported: 0 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: kZlibCompression supported: 1 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: kSnappyCompression supported: 1 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: kLZ4Compression supported: 1 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: kBZip2Compression supported: 0 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T10:07:03.405 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.merge_operator: 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_filter: None 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a8412e3bc0) 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout: cache_index_and_filter_blocks: 1 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout: pin_top_level_index_and_filter: 1 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout: index_type: 0 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout: data_block_index_type: 0 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout: index_shortening: 1 2024-03-16T10:07:03.406 INFO:journalctl@ceph.mon.a.smithi077.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: checksum: 4 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: no_block_cache: 0 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache: 0x55a8412e9090 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache_name: BinnedLRUCache 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache_options: 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: capacity : 536870912 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: num_shard_bits : 4 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: strict_capacity_limit : 0 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: high_pri_pool_ratio: 0.000 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache_compressed: (nil) 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: persistent_cache: (nil) 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_size: 4096 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_size_deviation: 10 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_restart_interval: 16 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: index_block_restart_interval: 1 2024-03-16T10:07:03.407 INFO:journalctl@ceph.mon.a.smithi077.stdout: metadata_block_size: 4096 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: partition_filters: 0 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_delta_encoding: 1 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: filter_policy: bloomfilter 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: whole_key_filtering: 1 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: verify_compression: 0 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: read_amp_bytes_per_bit: 0 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: format_version: 5 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: enable_index_compression: 1 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_align: 0 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: max_auto_readahead_size: 262144 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: prepopulate_block_cache: 0 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: initial_auto_readahead_size: 8192 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T10:07:03.408 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compression: NoCompression 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.num_levels: 7 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T10:07:03.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T10:07:03.410 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T10:07:03.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.table_properties_collectors: 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.inplace_update_support: 0 2024-03-16T10:07:03.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.bloom_locality: 0 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.max_successive_merges: 0 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.ttl: 2592000 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.enable_blob_files: false 2024-03-16T10:07:03.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.min_blob_size: 0 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8258e111-0248-4e9e-abd2-ad57a1957cf0 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710583623108925, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-16T10:07:03.414 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710583623109908, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100041, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97953, "index_size": 306, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14108, "raw_average_key_size": 49, "raw_value_size": 90279, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710583623, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8258e111-0248-4e9e-abd2-ad57a1957cf0", "db_session_id": "QS70U02EU0Z7Y7P922X5", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710583623109996, "job": 1, "event": "recovery_finished"} 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a8413dc000 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: DB pointer 0x55a8413c6000 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** DB Stats ** 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T10:07:03.415 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** Compaction Stats [default] ** 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: L0 2/0 99.53 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: Sum 2/0 99.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 153.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** Compaction Stats [default] ** 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 153.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-03-16T10:07:03.416 INFO:journalctl@ceph.mon.a.smithi077.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative compaction: 0.00 GB write, 32.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval compaction: 0.00 GB write, 32.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: Block cache BinnedLRUCache@0x55a8412e9090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: starting mon.a rank 0 at public addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] at bind addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:07:03.417 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: mon.a@-1(???) e1 preinit fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: mon.a@-1(???).mds e1 new map 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: mon.a@-1(???).mds e1 print_map 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout: e1 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout: legacy client fscid: -1 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout: No filesystems configured 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-16T10:07:03.418 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 bash[32381]: e223b065e454dab2fb1a8ac03938cd8ff6e5bf2079ab742e294be811e58ba028 2024-03-16T10:07:03.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 systemd[1]: Started Ceph mon.a for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:07:03.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T10:07:03.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: monmap epoch 1 2024-03-16T10:07:03.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:07:03.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: last_changed 2024-03-16T10:06:54.961708+0000 2024-03-16T10:07:03.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: created 2024-03-16T10:06:54.961708+0000 2024-03-16T10:07:03.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: min_mon_release 19 (squid) 2024-03-16T10:07:03.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: election_strategy: 1 2024-03-16T10:07:03.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T10:07:03.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: fsmap 2024-03-16T10:07:03.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: osdmap e1: 0 total, 0 up, 0 in 2024-03-16T10:07:03.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:03 smithi077 ceph-mon[32400]: mgrmap e1: no daemons active 2024-03-16T10:07:04.126 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.598+0000 7f7a4caca700 1 Processor -- start 2024-03-16T10:07:04.126 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.598+0000 7f7a4caca700 1 -- start start 2024-03-16T10:07:04.126 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.598+0000 7f7a4caca700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 0x7f7a4807c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:04.127 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.598+0000 7f7a4caca700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4807c910 con 0x7f7a4807dee0 2024-03-16T10:07:04.127 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.598+0000 7f7a4659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 0x7f7a4807c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:04.127 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.598+0000 7f7a4659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 0x7f7a4807c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60970/0 (socket says 172.21.15.77:60970) 2024-03-16T10:07:04.127 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.598+0000 7f7a4659c700 1 -- 172.21.15.77:0/2430899909 learned_addr learned my addr 172.21.15.77:0/2430899909 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:04.127 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.599+0000 7f7a4659c700 1 -- 172.21.15.77:0/2430899909 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4807ca90 con 0x7f7a4807dee0 2024-03-16T10:07:04.127 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.599+0000 7f7a4659c700 1 --2- 172.21.15.77:0/2430899909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 0x7f7a4807c340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7a380098a0 tx=0x7f7a38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d3d4195cc7cbcc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:04.127 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.599+0000 7f7a4559a700 1 -- 172.21.15.77:0/2430899909 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a38004160 con 0x7f7a4807dee0 2024-03-16T10:07:04.127 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.599+0000 7f7a4559a700 1 -- 172.21.15.77:0/2430899909 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7a38004300 con 0x7f7a4807dee0 2024-03-16T10:07:04.127 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.599+0000 7f7a4559a700 1 -- 172.21.15.77:0/2430899909 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a38011b50 con 0x7f7a4807dee0 2024-03-16T10:07:04.128 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.600+0000 7f7a4caca700 1 -- 172.21.15.77:0/2430899909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 msgr2=0x7f7a4807c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:04.128 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.600+0000 7f7a4caca700 1 --2- 172.21.15.77:0/2430899909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 0x7f7a4807c340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7a380098a0 tx=0x7f7a38009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:04.128 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.600+0000 7f7a4caca700 1 -- 172.21.15.77:0/2430899909 shutdown_connections 2024-03-16T10:07:04.128 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.600+0000 7f7a4caca700 1 --2- 172.21.15.77:0/2430899909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 0x7f7a4807c340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:04.128 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.600+0000 7f7a4caca700 1 -- 172.21.15.77:0/2430899909 >> 172.21.15.77:0/2430899909 conn(0x7f7a48103210 msgr2=0x7f7a48105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:04.128 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.600+0000 7f7a4caca700 1 -- 172.21.15.77:0/2430899909 shutdown_connections 2024-03-16T10:07:04.128 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.600+0000 7f7a4caca700 1 -- 172.21.15.77:0/2430899909 wait complete. 2024-03-16T10:07:04.128 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a4caca700 1 Processor -- start 2024-03-16T10:07:04.129 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a4caca700 1 -- start start 2024-03-16T10:07:04.129 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a4caca700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 0x7f7a4819e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:04.129 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a4caca700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4810e270 con 0x7f7a4807dee0 2024-03-16T10:07:04.129 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a4659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 0x7f7a4819e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:04.129 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a4659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 0x7f7a4819e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60976/0 (socket says 172.21.15.77:60976) 2024-03-16T10:07:04.129 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a4659c700 1 -- 172.21.15.77:0/3596143260 learned_addr learned my addr 172.21.15.77:0/3596143260 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:04.129 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a4659c700 1 -- 172.21.15.77:0/3596143260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4819e660 con 0x7f7a4807dee0 2024-03-16T10:07:04.129 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a4659c700 1 --2- 172.21.15.77:0/3596143260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 0x7f7a4819e120 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7a38011670 tx=0x7f7a38011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:04.129 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a377fe700 1 -- 172.21.15.77:0/3596143260 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a380039c0 con 0x7f7a4807dee0 2024-03-16T10:07:04.129 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a4caca700 1 -- 172.21.15.77:0/3596143260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a4819e8f0 con 0x7f7a4807dee0 2024-03-16T10:07:04.129 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a377fe700 1 -- 172.21.15.77:0/3596143260 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7a38011e50 con 0x7f7a4807dee0 2024-03-16T10:07:04.129 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.601+0000 7f7a377fe700 1 -- 172.21.15.77:0/3596143260 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a3801a5c0 con 0x7f7a4807dee0 2024-03-16T10:07:04.130 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.602+0000 7f7a4caca700 1 -- 172.21.15.77:0/3596143260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a481a15e0 con 0x7f7a4807dee0 2024-03-16T10:07:04.130 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.602+0000 7f7a377fe700 1 -- 172.21.15.77:0/3596143260 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7a3801ac50 con 0x7f7a4807dee0 2024-03-16T10:07:04.130 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.602+0000 7f7a377fe700 1 -- 172.21.15.77:0/3596143260 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7a38019700 con 0x7f7a4807dee0 2024-03-16T10:07:04.130 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.603+0000 7f7a4caca700 1 -- 172.21.15.77:0/3596143260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a48063900 con 0x7f7a4807dee0 2024-03-16T10:07:04.130 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.605+0000 7f7a377fe700 1 -- 172.21.15.77:0/3596143260 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7a38003b60 con 0x7f7a4807dee0 2024-03-16T10:07:04.130 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.671+0000 7f7a4caca700 1 -- 172.21.15.77:0/3596143260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f7a48067b10 con 0x7f7a4807dee0 2024-03-16T10:07:04.130 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.672+0000 7f7a377fe700 1 -- 172.21.15.77:0/3596143260 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f7a38019900 con 0x7f7a4807dee0 2024-03-16T10:07:04.130 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.673+0000 7f7a4caca700 1 -- 172.21.15.77:0/3596143260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 msgr2=0x7f7a4819e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:04.130 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.673+0000 7f7a4caca700 1 --2- 172.21.15.77:0/3596143260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 0x7f7a4819e120 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7a38011670 tx=0x7f7a38011b60 comp rx=0 tx=0).stop 2024-03-16T10:07:04.131 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.673+0000 7f7a4caca700 1 -- 172.21.15.77:0/3596143260 shutdown_connections 2024-03-16T10:07:04.131 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.673+0000 7f7a4caca700 1 --2- 172.21.15.77:0/3596143260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a4807dee0 0x7f7a4819e120 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:04.131 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.673+0000 7f7a4caca700 1 -- 172.21.15.77:0/3596143260 >> 172.21.15.77:0/3596143260 conn(0x7f7a48103210 msgr2=0x7f7a48103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:04.131 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.673+0000 7f7a4caca700 1 -- 172.21.15.77:0/3596143260 shutdown_connections 2024-03-16T10:07:04.131 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:03.673+0000 7f7a4caca700 1 -- 172.21.15.77:0/3596143260 wait complete. 2024-03-16T10:07:04.131 INFO:teuthology.orchestra.run.smithi077.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-16T10:07:04.143 INFO:teuthology.orchestra.run.smithi077.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-16T10:07:04.143 INFO:teuthology.orchestra.run.smithi077.stdout:Creating mgr... 2024-03-16T10:07:04.144 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-16T10:07:04.144 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-16T10:07:04.386 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.x 2024-03-16T10:07:04.386 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-16T10:07:04.539 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715.target.wants/ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@.service. 2024-03-16T10:07:04.879 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:04 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3596143260' entity='client.admin' 2024-03-16T10:07:05.079 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T10:07:05.079 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout disabled 2024-03-16T10:07:05.086 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T10:07:05.086 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-03-16T10:07:05.086 INFO:teuthology.orchestra.run.smithi077.stdout:firewalld.service is not enabled 2024-03-16T10:07:05.087 INFO:teuthology.orchestra.run.smithi077.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T10:07:05.095 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T10:07:05.095 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout disabled 2024-03-16T10:07:05.103 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T10:07:05.103 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-03-16T10:07:05.104 INFO:teuthology.orchestra.run.smithi077.stdout:firewalld.service is not enabled 2024-03-16T10:07:05.104 INFO:teuthology.orchestra.run.smithi077.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-16T10:07:05.104 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mgr to start... 2024-03-16T10:07:05.104 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mgr... 2024-03-16T10:07:05.903 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:05 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1933335119' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T10:07:06.134 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 0 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T10:07:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T10:07:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:07:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-16T10:07:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T10:07:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T10:07:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T10:07:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T10:07:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T10:07:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T10:07:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T10:07:06.144 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T10:07:06.144 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T10:07:06.144 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T10:07:06.144 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:06.144 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T10:06:58.029497+0000", 2024-03-16T10:07:06.146 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.491+0000 7f28bea45700 1 Processor -- start 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.491+0000 7f28bea45700 1 -- start start 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.491+0000 7f28bea45700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 0x7f28b810ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.491+0000 7f28bea45700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f28b807be70 con 0x7f28b8108710 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.491+0000 7f28b7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 0x7f28b810ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.491+0000 7f28b7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 0x7f28b810ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32768/0 (socket says 172.21.15.77:32768) 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.491+0000 7f28b7fff700 1 -- 172.21.15.77:0/438366185 learned_addr learned my addr 172.21.15.77:0/438366185 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.492+0000 7f28b7fff700 1 -- 172.21.15.77:0/438366185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b810b040 con 0x7f28b8108710 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.492+0000 7f28b7fff700 1 --2- 172.21.15.77:0/438366185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 0x7f28b810ab00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f28a80098a0 tx=0x7f28a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10073b26555c1eb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.492+0000 7f28b6ffd700 1 -- 172.21.15.77:0/438366185 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28a8004160 con 0x7f28b8108710 2024-03-16T10:07:06.147 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.492+0000 7f28b6ffd700 1 -- 172.21.15.77:0/438366185 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f28a8004300 con 0x7f28b8108710 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.492+0000 7f28b6ffd700 1 -- 172.21.15.77:0/438366185 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28a8011b50 con 0x7f28b8108710 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.493+0000 7f28bea45700 1 -- 172.21.15.77:0/438366185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 msgr2=0x7f28b810ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.493+0000 7f28bea45700 1 --2- 172.21.15.77:0/438366185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 0x7f28b810ab00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f28a80098a0 tx=0x7f28a8009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.493+0000 7f28bea45700 1 -- 172.21.15.77:0/438366185 shutdown_connections 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.493+0000 7f28bea45700 1 --2- 172.21.15.77:0/438366185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 0x7f28b810ab00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.493+0000 7f28bea45700 1 -- 172.21.15.77:0/438366185 >> 172.21.15.77:0/438366185 conn(0x7f28b81020e0 msgr2=0x7f28b8104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.493+0000 7f28bea45700 1 -- 172.21.15.77:0/438366185 shutdown_connections 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.493+0000 7f28bea45700 1 -- 172.21.15.77:0/438366185 wait complete. 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.494+0000 7f28bea45700 1 Processor -- start 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.494+0000 7f28bea45700 1 -- start start 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.494+0000 7f28bea45700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 0x7f28b8107f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.494+0000 7f28bea45700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f28b810c060 con 0x7f28b8108710 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.494+0000 7f28b7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 0x7f28b8107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.494+0000 7f28b7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 0x7f28b8107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32774/0 (socket says 172.21.15.77:32774) 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.494+0000 7f28b7fff700 1 -- 172.21.15.77:0/1933335119 learned_addr learned my addr 172.21.15.77:0/1933335119 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:06.148 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.495+0000 7f28b7fff700 1 -- 172.21.15.77:0/1933335119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b81084d0 con 0x7f28b8108710 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.495+0000 7f28b7fff700 1 --2- 172.21.15.77:0/1933335119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 0x7f28b8107f90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f28a8011670 tx=0x7f28a8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.495+0000 7f28b57fa700 1 -- 172.21.15.77:0/1933335119 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28a80039c0 con 0x7f28b8108710 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.495+0000 7f28bea45700 1 -- 172.21.15.77:0/1933335119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b81066c0 con 0x7f28b8108710 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.495+0000 7f28b57fa700 1 -- 172.21.15.77:0/1933335119 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f28a8011e50 con 0x7f28b8108710 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.495+0000 7f28b57fa700 1 -- 172.21.15.77:0/1933335119 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28a801a5c0 con 0x7f28b8108710 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.495+0000 7f28bea45700 1 -- 172.21.15.77:0/1933335119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b8106c00 con 0x7f28b8108710 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.496+0000 7f28b57fa700 1 -- 172.21.15.77:0/1933335119 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f28a801ac50 con 0x7f28b8108710 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.496+0000 7f28b57fa700 1 -- 172.21.15.77:0/1933335119 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f28a8019700 con 0x7f28b8108710 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.496+0000 7f28bea45700 1 -- 172.21.15.77:0/1933335119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b8063900 con 0x7f28b8108710 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.498+0000 7f28b57fa700 1 -- 172.21.15.77:0/1933335119 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f28a8003b60 con 0x7f28b8108710 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.570+0000 7f28bea45700 1 -- 172.21.15.77:0/1933335119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f28b8106f40 con 0x7f28b8108710 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.571+0000 7f28b57fa700 1 -- 172.21.15.77:0/1933335119 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f28a8019950 con 0x7f28b8108710 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.572+0000 7f28bea45700 1 -- 172.21.15.77:0/1933335119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 msgr2=0x7f28b8107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.572+0000 7f28bea45700 1 --2- 172.21.15.77:0/1933335119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 0x7f28b8107f90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f28a8011670 tx=0x7f28a8011b60 comp rx=0 tx=0).stop 2024-03-16T10:07:06.149 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.572+0000 7f28bea45700 1 -- 172.21.15.77:0/1933335119 shutdown_connections 2024-03-16T10:07:06.150 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.572+0000 7f28bea45700 1 --2- 172.21.15.77:0/1933335119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28b8108710 0x7f28b8107f90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:06.150 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.572+0000 7f28bea45700 1 -- 172.21.15.77:0/1933335119 >> 172.21.15.77:0/1933335119 conn(0x7f28b81020e0 msgr2=0x7f28b8102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:06.150 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.572+0000 7f28bea45700 1 -- 172.21.15.77:0/1933335119 shutdown_connections 2024-03-16T10:07:06.150 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:05.572+0000 7f28bea45700 1 -- 172.21.15.77:0/1933335119 wait complete. 2024-03-16T10:07:06.150 INFO:teuthology.orchestra.run.smithi077.stdout:mgr not available, waiting (1/15)... 2024-03-16T10:07:06.642 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:06 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:06.545+0000 7fd464ef6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T10:07:06.982 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:06 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:06.640+0000 7fd464ef6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T10:07:06.982 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:06 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:06.722+0000 7fd464ef6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T10:07:06.982 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:06 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:06.981+0000 7fd464ef6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T10:07:07.245 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:07.091+0000 7fd464ef6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T10:07:07.245 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:07.177+0000 7fd464ef6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T10:07:07.625 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:07.272+0000 7fd464ef6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T10:07:07.625 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:07.459+0000 7fd464ef6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T10:07:08.481 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:08.455+0000 7fd464ef6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T10:07:08.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:08 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1405678499' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T10:07:08.745 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:08.591+0000 7fd464ef6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T10:07:09.185 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:08.927+0000 7fd464ef6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T10:07:09.185 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:09.016+0000 7fd464ef6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T10:07:09.189 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:07:09.189 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 0 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-16T10:07:09.190 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T10:07:09.191 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:09.191 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T10:07:09.191 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T10:07:09.191 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:09.191 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T10:07:09.191 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:09.191 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T10:07:09.191 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T10:07:09.191 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T10:07:09.198 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T10:07:09.198 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T10:07:09.198 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T10:07:09.198 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:09.198 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T10:07:09.198 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T10:07:09.198 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T10:07:09.199 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T10:06:58.029497+0000", 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.504+0000 7f88c6cd0700 1 Processor -- start 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.504+0000 7f88c6cd0700 1 -- start start 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.504+0000 7f88c6cd0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 0x7f88c0106950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.504+0000 7f88c6cd0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f88c007be70 con 0x7f88c0106530 2024-03-16T10:07:09.200 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.504+0000 7f88c4a6c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 0x7f88c0106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.504+0000 7f88c4a6c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 0x7f88c0106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32790/0 (socket says 172.21.15.77:32790) 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.504+0000 7f88c4a6c700 1 -- 172.21.15.77:0/1224593877 learned_addr learned my addr 172.21.15.77:0/1224593877 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.504+0000 7f88c4a6c700 1 -- 172.21.15.77:0/1224593877 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88c0106e90 con 0x7f88c0106530 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.504+0000 7f88c4a6c700 1 --2- 172.21.15.77:0/1224593877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 0x7f88c0106950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f88b40098a0 tx=0x7f88b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcd0545dafa30c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.505+0000 7f88bf7fe700 1 -- 172.21.15.77:0/1224593877 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88b4004160 con 0x7f88c0106530 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.505+0000 7f88bf7fe700 1 -- 172.21.15.77:0/1224593877 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f88b4004300 con 0x7f88c0106530 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.505+0000 7f88bf7fe700 1 -- 172.21.15.77:0/1224593877 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88b4011b50 con 0x7f88c0106530 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.505+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1224593877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 msgr2=0x7f88c0106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.505+0000 7f88c6cd0700 1 --2- 172.21.15.77:0/1224593877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 0x7f88c0106950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f88b40098a0 tx=0x7f88b4009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.505+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1224593877 shutdown_connections 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.505+0000 7f88c6cd0700 1 --2- 172.21.15.77:0/1224593877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 0x7f88c0106950 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.505+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1224593877 >> 172.21.15.77:0/1224593877 conn(0x7f88c0102110 msgr2=0x7f88c0104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.505+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1224593877 shutdown_connections 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.505+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1224593877 wait complete. 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.506+0000 7f88c6cd0700 1 Processor -- start 2024-03-16T10:07:09.201 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.506+0000 7f88c6cd0700 1 -- start start 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.506+0000 7f88c6cd0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 0x7f88c019e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.506+0000 7f88c6cd0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f88c0107f20 con 0x7f88c0106530 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.506+0000 7f88c4a6c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 0x7f88c019e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.506+0000 7f88c4a6c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 0x7f88c019e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32794/0 (socket says 172.21.15.77:32794) 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.506+0000 7f88c4a6c700 1 -- 172.21.15.77:0/1405678499 learned_addr learned my addr 172.21.15.77:0/1405678499 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.507+0000 7f88c4a6c700 1 -- 172.21.15.77:0/1405678499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88c019e660 con 0x7f88c0106530 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.507+0000 7f88c4a6c700 1 --2- 172.21.15.77:0/1405678499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 0x7f88c019e120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f88b4011670 tx=0x7f88b4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.507+0000 7f88bdffb700 1 -- 172.21.15.77:0/1405678499 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88b40039c0 con 0x7f88c0106530 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.507+0000 7f88bdffb700 1 -- 172.21.15.77:0/1405678499 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f88b4011e50 con 0x7f88c0106530 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.507+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1405678499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88c019e8f0 con 0x7f88c0106530 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.507+0000 7f88bdffb700 1 -- 172.21.15.77:0/1405678499 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88b401a5c0 con 0x7f88c0106530 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.507+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1405678499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88c01a15e0 con 0x7f88c0106530 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.507+0000 7f88bdffb700 1 -- 172.21.15.77:0/1405678499 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f88b401ac00 con 0x7f88c0106530 2024-03-16T10:07:09.202 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.507+0000 7f88bdffb700 1 -- 172.21.15.77:0/1405678499 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f88b4019700 con 0x7f88c0106530 2024-03-16T10:07:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.508+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1405678499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88c0063900 con 0x7f88c0106530 2024-03-16T10:07:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.510+0000 7f88bdffb700 1 -- 172.21.15.77:0/1405678499 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f88b4003b60 con 0x7f88c0106530 2024-03-16T10:07:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.582+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1405678499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f88c002dba0 con 0x7f88c0106530 2024-03-16T10:07:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.583+0000 7f88bdffb700 1 -- 172.21.15.77:0/1405678499 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f88b40199a0 con 0x7f88c0106530 2024-03-16T10:07:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.584+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1405678499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 msgr2=0x7f88c019e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.584+0000 7f88c6cd0700 1 --2- 172.21.15.77:0/1405678499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 0x7f88c019e120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f88b4011670 tx=0x7f88b4011b60 comp rx=0 tx=0).stop 2024-03-16T10:07:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.584+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1405678499 shutdown_connections 2024-03-16T10:07:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.584+0000 7f88c6cd0700 1 --2- 172.21.15.77:0/1405678499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f88c0106530 0x7f88c019e120 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.584+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1405678499 >> 172.21.15.77:0/1405678499 conn(0x7f88c0102110 msgr2=0x7f88c0194a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.584+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1405678499 shutdown_connections 2024-03-16T10:07:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:08.584+0000 7f88c6cd0700 1 -- 172.21.15.77:0/1405678499 wait complete. 2024-03-16T10:07:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:mgr not available, waiting (2/15)... 2024-03-16T10:07:09.495 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:09.212+0000 7fd464ef6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:07:09.495 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:09.395+0000 7fd464ef6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T10:07:09.927 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:09.673+0000 7fd464ef6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:07:09.927 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:09.926+0000 7fd464ef6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T10:07:10.245 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:10 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:10.157+0000 7fd464ef6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T10:07:11.245 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:10 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:10.941+0000 7fd464ef6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T10:07:11.602 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:11 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:11.516+0000 7fd464ef6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T10:07:11.973 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:11 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1009939999' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T10:07:11.974 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:11 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:11.620+0000 7fd464ef6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T10:07:11.974 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:11 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:11.839+0000 7fd464ef6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T10:07:12.203 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 0 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:07:12.204 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-16T10:07:12.205 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T10:07:12.205 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:12.205 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T10:07:12.205 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T10:07:12.205 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:12.205 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T10:07:12.205 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:12.205 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T10:07:12.205 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T10:07:12.205 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T10:07:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T10:07:12.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T10:06:58.029497+0000", 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.600+0000 7f9ae71c5700 1 Processor -- start 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.600+0000 7f9ae71c5700 1 -- start start 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.600+0000 7f9ae71c5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 0x7f9ae010ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.600+0000 7f9ae71c5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae007be70 con 0x7f9ae0108740 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.601+0000 7f9ae4f61700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 0x7f9ae010ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:12.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.601+0000 7f9ae4f61700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 0x7f9ae010ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51878/0 (socket says 172.21.15.77:51878) 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.601+0000 7f9ae4f61700 1 -- 172.21.15.77:0/1651581007 learned_addr learned my addr 172.21.15.77:0/1651581007 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.601+0000 7f9ae4f61700 1 -- 172.21.15.77:0/1651581007 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae010b070 con 0x7f9ae0108740 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.601+0000 7f9ae4f61700 1 --2- 172.21.15.77:0/1651581007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 0x7f9ae010ab30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40098a0 tx=0x7f9ad4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8dc051fc63c4a2f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.601+0000 7f9adf7fe700 1 -- 172.21.15.77:0/1651581007 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ad4004160 con 0x7f9ae0108740 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.601+0000 7f9adf7fe700 1 -- 172.21.15.77:0/1651581007 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9ad4004300 con 0x7f9ae0108740 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.601+0000 7f9adf7fe700 1 -- 172.21.15.77:0/1651581007 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ad4011b50 con 0x7f9ae0108740 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.602+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1651581007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 msgr2=0x7f9ae010ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.602+0000 7f9ae71c5700 1 --2- 172.21.15.77:0/1651581007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 0x7f9ae010ab30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40098a0 tx=0x7f9ad4009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.602+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1651581007 shutdown_connections 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.602+0000 7f9ae71c5700 1 --2- 172.21.15.77:0/1651581007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 0x7f9ae010ab30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.602+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1651581007 >> 172.21.15.77:0/1651581007 conn(0x7f9ae0102110 msgr2=0x7f9ae0104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.602+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1651581007 shutdown_connections 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.602+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1651581007 wait complete. 2024-03-16T10:07:12.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.603+0000 7f9ae71c5700 1 Processor -- start 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.603+0000 7f9ae71c5700 1 -- start start 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.603+0000 7f9ae71c5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 0x7f9ae019bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.603+0000 7f9ae71c5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae010c090 con 0x7f9ae0108740 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.603+0000 7f9ae4f61700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 0x7f9ae019bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.603+0000 7f9ae4f61700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 0x7f9ae019bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51882/0 (socket says 172.21.15.77:51882) 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.603+0000 7f9ae4f61700 1 -- 172.21.15.77:0/1009939999 learned_addr learned my addr 172.21.15.77:0/1009939999 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.603+0000 7f9ae4f61700 1 -- 172.21.15.77:0/1009939999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae019c4a0 con 0x7f9ae0108740 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.603+0000 7f9ae4f61700 1 --2- 172.21.15.77:0/1009939999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 0x7f9ae019bf60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4011670 tx=0x7f9ad4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.604+0000 7f9addffb700 1 -- 172.21.15.77:0/1009939999 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ad40039c0 con 0x7f9ae0108740 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.604+0000 7f9addffb700 1 -- 172.21.15.77:0/1009939999 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9ad4011e50 con 0x7f9ae0108740 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.604+0000 7f9addffb700 1 -- 172.21.15.77:0/1009939999 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ad401a5c0 con 0x7f9ae0108740 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.604+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1009939999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae019c730 con 0x7f9ae0108740 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.604+0000 7f9addffb700 1 -- 172.21.15.77:0/1009939999 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9ad401ac00 con 0x7f9ae0108740 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.604+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1009939999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae019f3d0 con 0x7f9ae0108740 2024-03-16T10:07:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.604+0000 7f9addffb700 1 -- 172.21.15.77:0/1009939999 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9ad400b980 con 0x7f9ae0108740 2024-03-16T10:07:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.605+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1009939999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ae0063900 con 0x7f9ae0108740 2024-03-16T10:07:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.609+0000 7f9addffb700 1 -- 172.21.15.77:0/1009939999 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9ad4003b60 con 0x7f9ae0108740 2024-03-16T10:07:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.679+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1009939999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9ae002db70 con 0x7f9ae0108740 2024-03-16T10:07:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.680+0000 7f9addffb700 1 -- 172.21.15.77:0/1009939999 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9ad400bb80 con 0x7f9ae0108740 2024-03-16T10:07:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.681+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1009939999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 msgr2=0x7f9ae019bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.681+0000 7f9ae71c5700 1 --2- 172.21.15.77:0/1009939999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 0x7f9ae019bf60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4011670 tx=0x7f9ad4011b60 comp rx=0 tx=0).stop 2024-03-16T10:07:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.681+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1009939999 shutdown_connections 2024-03-16T10:07:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.681+0000 7f9ae71c5700 1 --2- 172.21.15.77:0/1009939999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae0108740 0x7f9ae019bf60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.681+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1009939999 >> 172.21.15.77:0/1009939999 conn(0x7f9ae0102110 msgr2=0x7f9ae0102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.681+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1009939999 shutdown_connections 2024-03-16T10:07:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:11.681+0000 7f9ae71c5700 1 -- 172.21.15.77:0/1009939999 wait complete. 2024-03-16T10:07:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:mgr not available, waiting (3/15)... 2024-03-16T10:07:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: Activating manager daemon x 2024-03-16T10:07:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: mgrmap e2: x(active, starting, since 0.00334285s) 2024-03-16T10:07:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: from='mgr.14100 172.21.15.77:0/3973189142' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T10:07:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: from='mgr.14100 172.21.15.77:0/3973189142' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T10:07:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: from='mgr.14100 172.21.15.77:0/3973189142' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T10:07:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: from='mgr.14100 172.21.15.77:0/3973189142' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:07:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: from='mgr.14100 172.21.15.77:0/3973189142' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T10:07:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: Manager daemon x is now available 2024-03-16T10:07:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: from='mgr.14100 172.21.15.77:0/3973189142' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T10:07:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: from='mgr.14100 172.21.15.77:0/3973189142' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T10:07:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: from='mgr.14100 172.21.15.77:0/3973189142' entity='mgr.x' 2024-03-16T10:07:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: from='mgr.14100 172.21.15.77:0/3973189142' entity='mgr.x' 2024-03-16T10:07:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:12 smithi077 ceph-mon[32400]: from='mgr.14100 172.21.15.77:0/3973189142' entity='mgr.x' 2024-03-16T10:07:14.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:13 smithi077 ceph-mon[32400]: mgrmap e3: x(active, since 1.00687s) 2024-03-16T10:07:16.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:15 smithi077 ceph-mon[32400]: mgrmap e4: x(active, since 3s) 2024-03-16T10:07:17.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:17 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/889063707' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T10:07:23.387 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 0 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-16T10:07:23.388 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T10:07:23.389 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:23.389 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T10:07:23.389 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T10:07:23.389 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:23.389 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T10:07:23.389 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:23.389 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T10:07:23.389 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T10:07:23.389 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T10:07:23.397 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T10:07:23.397 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T10:07:23.398 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T10:07:23.399 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T10:06:58.029497+0000", 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.373+0000 7efc5e49f700 1 Processor -- start 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.373+0000 7efc5e49f700 1 -- start start 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.373+0000 7efc5e49f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 0x7efc5810abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.373+0000 7efc5e49f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efc5807bdd0 con 0x7efc581087f0 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.373+0000 7efc57fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 0x7efc5810abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.373+0000 7efc57fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 0x7efc5810abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51952/0 (socket says 172.21.15.77:51952) 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.373+0000 7efc57fff700 1 -- 172.21.15.77:0/1600636126 learned_addr learned my addr 172.21.15.77:0/1600636126 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.374+0000 7efc57fff700 1 -- 172.21.15.77:0/1600636126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc5810b120 con 0x7efc581087f0 2024-03-16T10:07:23.400 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.374+0000 7efc57fff700 1 --2- 172.21.15.77:0/1600636126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 0x7efc5810abe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efc480098a0 tx=0x7efc48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba4974076fd8e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.374+0000 7efc56ffd700 1 -- 172.21.15.77:0/1600636126 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc48004160 con 0x7efc581087f0 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.374+0000 7efc56ffd700 1 -- 172.21.15.77:0/1600636126 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efc48004300 con 0x7efc581087f0 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.374+0000 7efc56ffd700 1 -- 172.21.15.77:0/1600636126 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc48011b50 con 0x7efc581087f0 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.374+0000 7efc5e49f700 1 -- 172.21.15.77:0/1600636126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 msgr2=0x7efc5810abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.374+0000 7efc5e49f700 1 --2- 172.21.15.77:0/1600636126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 0x7efc5810abe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efc480098a0 tx=0x7efc48009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.375+0000 7efc5e49f700 1 -- 172.21.15.77:0/1600636126 shutdown_connections 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.375+0000 7efc5e49f700 1 --2- 172.21.15.77:0/1600636126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 0x7efc5810abe0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.375+0000 7efc5e49f700 1 -- 172.21.15.77:0/1600636126 >> 172.21.15.77:0/1600636126 conn(0x7efc58102120 msgr2=0x7efc58104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.375+0000 7efc5e49f700 1 -- 172.21.15.77:0/1600636126 shutdown_connections 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.375+0000 7efc5e49f700 1 -- 172.21.15.77:0/1600636126 wait complete. 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.375+0000 7efc5e49f700 1 Processor -- start 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.375+0000 7efc5e49f700 1 -- start start 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.375+0000 7efc5e49f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 0x7efc5819bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:23.401 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.376+0000 7efc5e49f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efc5810c080 con 0x7efc581087f0 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.376+0000 7efc57fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 0x7efc5819bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.376+0000 7efc57fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 0x7efc5819bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51968/0 (socket says 172.21.15.77:51968) 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.376+0000 7efc57fff700 1 -- 172.21.15.77:0/889063707 learned_addr learned my addr 172.21.15.77:0/889063707 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.376+0000 7efc57fff700 1 -- 172.21.15.77:0/889063707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc5819c4b0 con 0x7efc581087f0 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.376+0000 7efc57fff700 1 --2- 172.21.15.77:0/889063707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 0x7efc5819bf70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efc48009fb0 tx=0x7efc48011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.376+0000 7efc557fa700 1 -- 172.21.15.77:0/889063707 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc480038b0 con 0x7efc581087f0 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.376+0000 7efc5e49f700 1 -- 172.21.15.77:0/889063707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc5819c740 con 0x7efc581087f0 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.376+0000 7efc557fa700 1 -- 172.21.15.77:0/889063707 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efc48011e50 con 0x7efc581087f0 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.376+0000 7efc557fa700 1 -- 172.21.15.77:0/889063707 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc48019420 con 0x7efc581087f0 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.376+0000 7efc5e49f700 1 -- 172.21.15.77:0/889063707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc5819f430 con 0x7efc581087f0 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.377+0000 7efc557fa700 1 -- 172.21.15.77:0/889063707 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7efc48019720 con 0x7efc581087f0 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.377+0000 7efc5e49f700 1 -- 172.21.15.77:0/889063707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc58063900 con 0x7efc581087f0 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.377+0000 7efc557fa700 1 --2- 172.21.15.77:0/889063707 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7efc40038e80 0x7efc4003b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:23.402 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.377+0000 7efc557fa700 1 -- 172.21.15.77:0/889063707 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efc4804d160 con 0x7efc581087f0 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.377+0000 7efc577fe700 1 --2- 172.21.15.77:0/889063707 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7efc40038e80 0x7efc4003b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.378+0000 7efc577fe700 1 --2- 172.21.15.77:0/889063707 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7efc40038e80 0x7efc4003b340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efc4c006fd0 tx=0x7efc4c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.382+0000 7efc557fa700 1 -- 172.21.15.77:0/889063707 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efc48003ac0 con 0x7efc581087f0 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.657+0000 7efc5e49f700 1 -- 172.21.15.77:0/889063707 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7efc5802db70 con 0x7efc581087f0 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.657+0000 7efc557fa700 1 -- 172.21.15.77:0/889063707 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7efc4802d7c0 con 0x7efc581087f0 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.659+0000 7efc5e49f700 1 -- 172.21.15.77:0/889063707 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7efc40038e80 msgr2=0x7efc4003b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.659+0000 7efc5e49f700 1 --2- 172.21.15.77:0/889063707 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7efc40038e80 0x7efc4003b340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efc4c006fd0 tx=0x7efc4c006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.659+0000 7efc5e49f700 1 -- 172.21.15.77:0/889063707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 msgr2=0x7efc5819bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.659+0000 7efc5e49f700 1 --2- 172.21.15.77:0/889063707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 0x7efc5819bf70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efc48009fb0 tx=0x7efc48011bf0 comp rx=0 tx=0).stop 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.659+0000 7efc5e49f700 1 -- 172.21.15.77:0/889063707 shutdown_connections 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.659+0000 7efc5e49f700 1 --2- 172.21.15.77:0/889063707 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7efc40038e80 0x7efc4003b340 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.659+0000 7efc5e49f700 1 --2- 172.21.15.77:0/889063707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc581087f0 0x7efc5819bf70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.659+0000 7efc5e49f700 1 -- 172.21.15.77:0/889063707 >> 172.21.15.77:0/889063707 conn(0x7efc58102120 msgr2=0x7efc58102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.659+0000 7efc5e49f700 1 -- 172.21.15.77:0/889063707 shutdown_connections 2024-03-16T10:07:23.403 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:17.659+0000 7efc5e49f700 1 -- 172.21.15.77:0/889063707 wait complete. 2024-03-16T10:07:23.404 INFO:teuthology.orchestra.run.smithi077.stdout:mgr is available 2024-03-16T10:07:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:24 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2105488139' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-16T10:07:25.453 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:07:25.453 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T10:07:25.453 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout fsid = c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:07:25.453 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T10:07:25.453 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T10:07:25.453 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T10:07:25.453 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.688+0000 7fe1e4300700 1 Processor -- start 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.688+0000 7fe1e4300700 1 -- start start 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.689+0000 7fe1e4300700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 0x7fe1dc10ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.689+0000 7fe1e4300700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc07be50 con 0x7fe1dc108810 2024-03-16T10:07:25.454 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.689+0000 7fe1e209c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 0x7fe1dc10ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.689+0000 7fe1e209c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 0x7fe1dc10ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52332/0 (socket says 172.21.15.77:52332) 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.689+0000 7fe1e209c700 1 -- 172.21.15.77:0/3832541119 learned_addr learned my addr 172.21.15.77:0/3832541119 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.689+0000 7fe1e209c700 1 -- 172.21.15.77:0/3832541119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1dc10b140 con 0x7fe1dc108810 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.689+0000 7fe1e209c700 1 --2- 172.21.15.77:0/3832541119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 0x7fe1dc10ac00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80098a0 tx=0x7fe1d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0392bf8b248338 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.690+0000 7fe1e109a700 1 -- 172.21.15.77:0/3832541119 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1d8004160 con 0x7fe1dc108810 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.690+0000 7fe1e109a700 1 -- 172.21.15.77:0/3832541119 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe1d8004300 con 0x7fe1dc108810 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.690+0000 7fe1e109a700 1 -- 172.21.15.77:0/3832541119 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1d8011b50 con 0x7fe1dc108810 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.690+0000 7fe1e4300700 1 -- 172.21.15.77:0/3832541119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 msgr2=0x7fe1dc10ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.690+0000 7fe1e4300700 1 --2- 172.21.15.77:0/3832541119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 0x7fe1dc10ac00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80098a0 tx=0x7fe1d8009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.690+0000 7fe1e4300700 1 -- 172.21.15.77:0/3832541119 shutdown_connections 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.690+0000 7fe1e4300700 1 --2- 172.21.15.77:0/3832541119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 0x7fe1dc10ac00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.690+0000 7fe1e4300700 1 -- 172.21.15.77:0/3832541119 >> 172.21.15.77:0/3832541119 conn(0x7fe1dc102220 msgr2=0x7fe1dc104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.690+0000 7fe1e4300700 1 -- 172.21.15.77:0/3832541119 shutdown_connections 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.690+0000 7fe1e4300700 1 -- 172.21.15.77:0/3832541119 wait complete. 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.691+0000 7fe1e4300700 1 Processor -- start 2024-03-16T10:07:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.691+0000 7fe1e4300700 1 -- start start 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.691+0000 7fe1e4300700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 0x7fe1dc19bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.691+0000 7fe1e4300700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc10c110 con 0x7fe1dc108810 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.691+0000 7fe1e209c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 0x7fe1dc19bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.691+0000 7fe1e209c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 0x7fe1dc19bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52342/0 (socket says 172.21.15.77:52342) 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.691+0000 7fe1e209c700 1 -- 172.21.15.77:0/2105488139 learned_addr learned my addr 172.21.15.77:0/2105488139 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.692+0000 7fe1e209c700 1 -- 172.21.15.77:0/2105488139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1dc19c500 con 0x7fe1dc108810 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.692+0000 7fe1e209c700 1 --2- 172.21.15.77:0/2105488139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 0x7fe1dc19bfc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8011670 tx=0x7fe1d8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.692+0000 7fe1d37fe700 1 -- 172.21.15.77:0/2105488139 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1d80039c0 con 0x7fe1dc108810 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.692+0000 7fe1d37fe700 1 -- 172.21.15.77:0/2105488139 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe1d8011e50 con 0x7fe1dc108810 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.692+0000 7fe1e4300700 1 -- 172.21.15.77:0/2105488139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1dc19c790 con 0x7fe1dc108810 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.692+0000 7fe1d37fe700 1 -- 172.21.15.77:0/2105488139 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1d801a5c0 con 0x7fe1dc108810 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.692+0000 7fe1e4300700 1 -- 172.21.15.77:0/2105488139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1dc19f480 con 0x7fe1dc108810 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.693+0000 7fe1d37fe700 1 -- 172.21.15.77:0/2105488139 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fe1d801a870 con 0x7fe1dc108810 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.693+0000 7fe1e4300700 1 -- 172.21.15.77:0/2105488139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1dc063900 con 0x7fe1dc108810 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.693+0000 7fe1d37fe700 1 --2- 172.21.15.77:0/2105488139 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7fe1c8038ed0 0x7fe1c803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:25.456 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.693+0000 7fe1d37fe700 1 -- 172.21.15.77:0/2105488139 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe1d804d240 con 0x7fe1dc108810 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.693+0000 7fe1e189b700 1 --2- 172.21.15.77:0/2105488139 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7fe1c8038ed0 0x7fe1c803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.693+0000 7fe1e189b700 1 --2- 172.21.15.77:0/2105488139 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7fe1c8038ed0 0x7fe1c803b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc006fd0 tx=0x7fe1cc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.698+0000 7fe1d37fe700 1 -- 172.21.15.77:0/2105488139 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe1d802da70 con 0x7fe1dc108810 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.889+0000 7fe1e4300700 1 -- 172.21.15.77:0/2105488139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fe1dc194c30 con 0x7fe1dc108810 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.889+0000 7fe1d37fe700 1 -- 172.21.15.77:0/2105488139 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fe1d8019c70 con 0x7fe1dc108810 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.891+0000 7fe1e4300700 1 -- 172.21.15.77:0/2105488139 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7fe1c8038ed0 msgr2=0x7fe1c803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.891+0000 7fe1e4300700 1 --2- 172.21.15.77:0/2105488139 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7fe1c8038ed0 0x7fe1c803b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc006fd0 tx=0x7fe1cc006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.891+0000 7fe1e4300700 1 -- 172.21.15.77:0/2105488139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 msgr2=0x7fe1dc19bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.891+0000 7fe1e4300700 1 --2- 172.21.15.77:0/2105488139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 0x7fe1dc19bfc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8011670 tx=0x7fe1d8011b60 comp rx=0 tx=0).stop 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.891+0000 7fe1e4300700 1 -- 172.21.15.77:0/2105488139 shutdown_connections 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.891+0000 7fe1e4300700 1 --2- 172.21.15.77:0/2105488139 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7fe1c8038ed0 0x7fe1c803b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.892+0000 7fe1e4300700 1 --2- 172.21.15.77:0/2105488139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1dc108810 0x7fe1dc19bfc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.892+0000 7fe1e4300700 1 -- 172.21.15.77:0/2105488139 >> 172.21.15.77:0/2105488139 conn(0x7fe1dc102220 msgr2=0x7fe1dc102eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.892+0000 7fe1e4300700 1 -- 172.21.15.77:0/2105488139 shutdown_connections 2024-03-16T10:07:25.457 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:24.892+0000 7fe1e4300700 1 -- 172.21.15.77:0/2105488139 wait complete. 2024-03-16T10:07:25.458 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling cephadm module... 2024-03-16T10:07:26.388 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:26 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1835651960' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-16T10:07:26.389 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:26 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: ignoring --setuser ceph since I am not root 2024-03-16T10:07:26.389 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:26 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: ignoring --setgroup ceph since I am not root 2024-03-16T10:07:26.936 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.809+0000 7f6737f75700 1 Processor -- start 2024-03-16T10:07:26.936 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:26 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:26.684+0000 7f6d2ee60200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T10:07:26.936 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:26 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:26.778+0000 7f6d2ee60200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T10:07:26.941 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.809+0000 7f6737f75700 1 -- start start 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.809+0000 7f6737f75700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 0x7f67301074c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.809+0000 7f6737f75700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6730107a90 con 0x7f67301070a0 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.810+0000 7f6735d11700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 0x7f67301074c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.810+0000 7f6735d11700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 0x7f67301074c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52354/0 (socket says 172.21.15.77:52354) 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.810+0000 7f6735d11700 1 -- 172.21.15.77:0/30065873 learned_addr learned my addr 172.21.15.77:0/30065873 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.810+0000 7f6735d11700 1 -- 172.21.15.77:0/30065873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6730107dc0 con 0x7f67301070a0 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.810+0000 7f6735d11700 1 --2- 172.21.15.77:0/30065873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 0x7f67301074c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f67200098a0 tx=0x7f6720009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c284fc084b24b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.810+0000 7f6734d0f700 1 -- 172.21.15.77:0/30065873 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6720004160 con 0x7f67301070a0 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.810+0000 7f6734d0f700 1 -- 172.21.15.77:0/30065873 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6720004300 con 0x7f67301070a0 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.810+0000 7f6734d0f700 1 -- 172.21.15.77:0/30065873 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6720011b50 con 0x7f67301070a0 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.811+0000 7f6737f75700 1 -- 172.21.15.77:0/30065873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 msgr2=0x7f67301074c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.811+0000 7f6737f75700 1 --2- 172.21.15.77:0/30065873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 0x7f67301074c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f67200098a0 tx=0x7f6720009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:26.942 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.811+0000 7f6737f75700 1 -- 172.21.15.77:0/30065873 shutdown_connections 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.811+0000 7f6737f75700 1 --2- 172.21.15.77:0/30065873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 0x7f67301074c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.811+0000 7f6737f75700 1 -- 172.21.15.77:0/30065873 >> 172.21.15.77:0/30065873 conn(0x7f67301018a0 msgr2=0x7f6730103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.811+0000 7f6737f75700 1 -- 172.21.15.77:0/30065873 shutdown_connections 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.811+0000 7f6737f75700 1 -- 172.21.15.77:0/30065873 wait complete. 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.811+0000 7f6737f75700 1 Processor -- start 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.811+0000 7f6737f75700 1 -- start start 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.811+0000 7f6737f75700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 0x7f67301066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.811+0000 7f6737f75700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f673010b170 con 0x7f67301070a0 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.812+0000 7f6735d11700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 0x7f67301066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.812+0000 7f6735d11700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 0x7f67301066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52370/0 (socket says 172.21.15.77:52370) 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.812+0000 7f6735d11700 1 -- 172.21.15.77:0/1835651960 learned_addr learned my addr 172.21.15.77:0/1835651960 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.812+0000 7f6735d11700 1 -- 172.21.15.77:0/1835651960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f673019ecb0 con 0x7f67301070a0 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.812+0000 7f6735d11700 1 --2- 172.21.15.77:0/1835651960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 0x7f67301066a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6720009fb0 tx=0x7f6720011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:26.943 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.812+0000 7f6726ffd700 1 -- 172.21.15.77:0/1835651960 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67200038b0 con 0x7f67301070a0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.812+0000 7f6726ffd700 1 -- 172.21.15.77:0/1835651960 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6720011e50 con 0x7f67301070a0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.812+0000 7f6737f75700 1 -- 172.21.15.77:0/1835651960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f673019ee80 con 0x7f67301070a0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.812+0000 7f6726ffd700 1 -- 172.21.15.77:0/1835651960 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6720019420 con 0x7f67301070a0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.812+0000 7f6737f75700 1 -- 172.21.15.77:0/1835651960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f673019f2a0 con 0x7f67301070a0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.813+0000 7f6726ffd700 1 -- 172.21.15.77:0/1835651960 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f672001a430 con 0x7f67301070a0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.813+0000 7f6737f75700 1 -- 172.21.15.77:0/1835651960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6730063900 con 0x7f67301070a0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.813+0000 7f6726ffd700 1 --2- 172.21.15.77:0/1835651960 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f671c038ed0 0x7f671c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.813+0000 7f6726ffd700 1 -- 172.21.15.77:0/1835651960 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f672004cfe0 con 0x7f67301070a0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.813+0000 7f6735510700 1 --2- 172.21.15.77:0/1835651960 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f671c038ed0 0x7f671c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.813+0000 7f6735510700 1 --2- 172.21.15.77:0/1835651960 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f671c038ed0 0x7f671c03b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f672c006fd0 tx=0x7f672c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:25.818+0000 7f6726ffd700 1 -- 172.21.15.77:0/1835651960 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6720029260 con 0x7f67301070a0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.068+0000 7f6737f75700 1 -- 172.21.15.77:0/1835651960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f673002dc10 con 0x7f67301070a0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.120+0000 7f6726ffd700 1 -- 172.21.15.77:0/1835651960 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f6720003b00 con 0x7f67301070a0 2024-03-16T10:07:26.944 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.121+0000 7f6726ffd700 1 -- 172.21.15.77:0/1835651960 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f672001a720 con 0x7f67301070a0 2024-03-16T10:07:26.945 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.123+0000 7f6737f75700 1 -- 172.21.15.77:0/1835651960 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f671c038ed0 msgr2=0x7f671c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:26.945 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.123+0000 7f6737f75700 1 --2- 172.21.15.77:0/1835651960 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f671c038ed0 0x7f671c03b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f672c006fd0 tx=0x7f672c006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:26.945 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.123+0000 7f6737f75700 1 -- 172.21.15.77:0/1835651960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 msgr2=0x7f67301066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:26.945 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.123+0000 7f6737f75700 1 --2- 172.21.15.77:0/1835651960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 0x7f67301066a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6720009fb0 tx=0x7f6720011bf0 comp rx=0 tx=0).stop 2024-03-16T10:07:26.945 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.123+0000 7f6737f75700 1 -- 172.21.15.77:0/1835651960 shutdown_connections 2024-03-16T10:07:26.945 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.123+0000 7f6737f75700 1 --2- 172.21.15.77:0/1835651960 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f671c038ed0 0x7f671c03b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:26.945 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.123+0000 7f6737f75700 1 --2- 172.21.15.77:0/1835651960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67301070a0 0x7f67301066a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:26.945 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.123+0000 7f6737f75700 1 -- 172.21.15.77:0/1835651960 >> 172.21.15.77:0/1835651960 conn(0x7f67301018a0 msgr2=0x7f6730102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:26.945 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.123+0000 7f6737f75700 1 -- 172.21.15.77:0/1835651960 shutdown_connections 2024-03-16T10:07:26.945 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:26.123+0000 7f6737f75700 1 -- 172.21.15.77:0/1835651960 wait complete. 2024-03-16T10:07:27.227 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:27 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1835651960' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-16T10:07:27.228 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:27 smithi077 ceph-mon[32400]: mgrmap e5: x(active, since 14s) 2024-03-16T10:07:27.495 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:27 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:27.438+0000 7f6d2ee60200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T10:07:27.856 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:27 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:27.532+0000 7f6d2ee60200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T10:07:27.856 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:27 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:27.677+0000 7f6d2ee60200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T10:07:27.856 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:27 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:27.770+0000 7f6d2ee60200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T10:07:27.856 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:27 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:27.854+0000 7f6d2ee60200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T10:07:28.089 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:07:28.089 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.326+0000 7f15fb7e2700 1 Processor -- start 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.326+0000 7f15fb7e2700 1 -- start start 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.326+0000 7f15fb7e2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 0x7f15f4106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.326+0000 7f15fb7e2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f15f4107580 con 0x7f15f4106b90 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.326+0000 7f15f957e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 0x7f15f4106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.326+0000 7f15f957e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 0x7f15f4106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52384/0 (socket says 172.21.15.77:52384) 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.326+0000 7f15f957e700 1 -- 172.21.15.77:0/2192637739 learned_addr learned my addr 172.21.15.77:0/2192637739 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.327+0000 7f15f957e700 1 -- 172.21.15.77:0/2192637739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f4107d90 con 0x7f15f4106b90 2024-03-16T10:07:28.090 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.327+0000 7f15f957e700 1 --2- 172.21.15.77:0/2192637739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 0x7f15f4106fb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15f00098a0 tx=0x7f15f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1462b72b68448ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.327+0000 7f15ebfff700 1 -- 172.21.15.77:0/2192637739 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15f0004160 con 0x7f15f4106b90 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.327+0000 7f15ebfff700 1 -- 172.21.15.77:0/2192637739 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f15f0004300 con 0x7f15f4106b90 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.327+0000 7f15ebfff700 1 -- 172.21.15.77:0/2192637739 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15f0011b50 con 0x7f15f4106b90 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.328+0000 7f15fb7e2700 1 -- 172.21.15.77:0/2192637739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 msgr2=0x7f15f4106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.328+0000 7f15fb7e2700 1 --2- 172.21.15.77:0/2192637739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 0x7f15f4106fb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15f00098a0 tx=0x7f15f0009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.328+0000 7f15fb7e2700 1 -- 172.21.15.77:0/2192637739 shutdown_connections 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.328+0000 7f15fb7e2700 1 --2- 172.21.15.77:0/2192637739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 0x7f15f4106fb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.328+0000 7f15fb7e2700 1 -- 172.21.15.77:0/2192637739 >> 172.21.15.77:0/2192637739 conn(0x7f15f4102110 msgr2=0x7f15f4104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.328+0000 7f15fb7e2700 1 -- 172.21.15.77:0/2192637739 shutdown_connections 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.328+0000 7f15fb7e2700 1 -- 172.21.15.77:0/2192637739 wait complete. 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.329+0000 7f15fb7e2700 1 Processor -- start 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.329+0000 7f15fb7e2700 1 -- start start 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.329+0000 7f15fb7e2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 0x7f15f407ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.329+0000 7f15fb7e2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f15f410b140 con 0x7f15f4106b90 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.329+0000 7f15f957e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 0x7f15f407ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:28.091 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.329+0000 7f15f957e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 0x7f15f407ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52388/0 (socket says 172.21.15.77:52388) 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.329+0000 7f15f957e700 1 -- 172.21.15.77:0/3996867296 learned_addr learned my addr 172.21.15.77:0/3996867296 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.329+0000 7f15f957e700 1 -- 172.21.15.77:0/3996867296 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f419ecb0 con 0x7f15f4106b90 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.329+0000 7f15f957e700 1 --2- 172.21.15.77:0/3996867296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 0x7f15f407ba50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f15f0011670 tx=0x7f15f0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.329+0000 7f15ea7fc700 1 -- 172.21.15.77:0/3996867296 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15f00039c0 con 0x7f15f4106b90 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.330+0000 7f15ea7fc700 1 -- 172.21.15.77:0/3996867296 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f15f0011e50 con 0x7f15f4106b90 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.330+0000 7f15fb7e2700 1 -- 172.21.15.77:0/3996867296 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f419ee80 con 0x7f15f4106b90 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.330+0000 7f15ea7fc700 1 -- 172.21.15.77:0/3996867296 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15f001a5c0 con 0x7f15f4106b90 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.330+0000 7f15fb7e2700 1 -- 172.21.15.77:0/3996867296 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f419f220 con 0x7f15f4106b90 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.330+0000 7f15fb7e2700 1 -- 172.21.15.77:0/3996867296 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15f4063900 con 0x7f15f4106b90 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.331+0000 7f15ea7fc700 1 -- 172.21.15.77:0/3996867296 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f15f001a8c0 con 0x7f15f4106b90 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.331+0000 7f15ea7fc700 1 --2- 172.21.15.77:0/3996867296 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f15e0038ed0 0x7f15e003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.331+0000 7f15ea7fc700 1 -- 172.21.15.77:0/3996867296 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f15f004d2b0 con 0x7f15f4106b90 2024-03-16T10:07:28.092 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.331+0000 7f15f8d7d700 1 -- 172.21.15.77:0/3996867296 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f15e0038ed0 msgr2=0x7f15e003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1093017736 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.331+0000 7f15f8d7d700 1 --2- 172.21.15.77:0/3996867296 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f15e0038ed0 0x7f15e003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.336+0000 7f15ea7fc700 1 -- 172.21.15.77:0/3996867296 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f15f0019c70 con 0x7f15f4106b90 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.531+0000 7f15f8d7d700 1 -- 172.21.15.77:0/3996867296 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f15e0038ed0 msgr2=0x7f15e003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1093017736 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.531+0000 7f15f8d7d700 1 --2- 172.21.15.77:0/3996867296 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f15e0038ed0 0x7f15e003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.593+0000 7f15fb7e2700 1 -- 172.21.15.77:0/3996867296 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f15f402dd60 con 0x7f15f4106b90 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.594+0000 7f15ea7fc700 1 -- 172.21.15.77:0/3996867296 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f15f0003b60 con 0x7f15f4106b90 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.596+0000 7f15fb7e2700 1 -- 172.21.15.77:0/3996867296 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f15e0038ed0 msgr2=0x7f15e003b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.596+0000 7f15fb7e2700 1 --2- 172.21.15.77:0/3996867296 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f15e0038ed0 0x7f15e003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.596+0000 7f15fb7e2700 1 -- 172.21.15.77:0/3996867296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 msgr2=0x7f15f407ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.596+0000 7f15fb7e2700 1 --2- 172.21.15.77:0/3996867296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 0x7f15f407ba50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f15f0011670 tx=0x7f15f0011b60 comp rx=0 tx=0).stop 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.596+0000 7f15fb7e2700 1 -- 172.21.15.77:0/3996867296 shutdown_connections 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.596+0000 7f15fb7e2700 1 --2- 172.21.15.77:0/3996867296 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f15e0038ed0 0x7f15e003b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.596+0000 7f15fb7e2700 1 --2- 172.21.15.77:0/3996867296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15f4106b90 0x7f15f407ba50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.596+0000 7f15fb7e2700 1 -- 172.21.15.77:0/3996867296 >> 172.21.15.77:0/3996867296 conn(0x7f15f4102110 msgr2=0x7f15f4102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.596+0000 7f15fb7e2700 1 -- 172.21.15.77:0/3996867296 shutdown_connections 2024-03-16T10:07:28.093 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:27.596+0000 7f15fb7e2700 1 -- 172.21.15.77:0/3996867296 wait complete. 2024-03-16T10:07:28.094 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for the mgr to restart... 2024-03-16T10:07:28.094 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mgr epoch 5... 2024-03-16T10:07:28.392 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:28 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3996867296' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T10:07:28.392 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:28 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:28.122+0000 7f6d2ee60200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T10:07:28.392 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:28 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:28.231+0000 7f6d2ee60200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T10:07:28.392 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:28 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:28.319+0000 7f6d2ee60200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T10:07:28.698 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:28 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:28.414+0000 7f6d2ee60200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T10:07:28.698 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:28 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:28.592+0000 7f6d2ee60200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T10:07:29.791 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:29 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:29.539+0000 7f6d2ee60200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T10:07:29.791 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:29 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:29.669+0000 7f6d2ee60200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T10:07:30.098 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:30 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:30.006+0000 7f6d2ee60200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T10:07:30.098 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:30 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:30.097+0000 7f6d2ee60200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T10:07:30.370 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:30 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:30.293+0000 7f6d2ee60200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:07:30.745 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:30 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:30.486+0000 7f6d2ee60200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T10:07:31.027 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:30 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:30.771+0000 7f6d2ee60200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:07:31.028 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:31 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:31.026+0000 7f6d2ee60200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T10:07:31.494 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:31 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:31.261+0000 7f6d2ee60200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T10:07:32.495 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:32.047+0000 7f6d2ee60200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T10:07:32.911 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:32.603+0000 7f6d2ee60200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T10:07:32.911 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:32.699+0000 7f6d2ee60200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T10:07:32.911 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:32.910+0000 7f6d2ee60200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T10:07:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: Active manager daemon x restarted 2024-03-16T10:07:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: Activating manager daemon x 2024-03-16T10:07:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: osdmap e2: 0 total, 0 up, 0 in 2024-03-16T10:07:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: mgrmap e6: x(active, starting, since 0.00341873s) 2024-03-16T10:07:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:07:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T10:07:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T10:07:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T10:07:33.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T10:07:33.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: Manager daemon x is now available 2024-03-16T10:07:33.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:33.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:33.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:07:33.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:32 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:07:34.237 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:33 smithi077 ceph-mon[32400]: Found migration_current of "None". Setting to last migration. 2024-03-16T10:07:34.237 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:33 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:07:34.237 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:33 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T10:07:34.237 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:33 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T10:07:34.237 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:33 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:34.237 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:33 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:34.237 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:33 smithi077 ceph-mon[32400]: mgrmap e7: x(active, since 1.00627s) 2024-03-16T10:07:34.614 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.491+0000 7f12b16af700 1 Processor -- start 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.491+0000 7f12b16af700 1 -- start start 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.491+0000 7f12b16af700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 0x7f12ac108500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.491+0000 7f12b16af700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac108ad0 con 0x7f12ac1080e0 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.491+0000 7f12aaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 0x7f12ac108500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.491+0000 7f12aaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 0x7f12ac108500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52402/0 (socket says 172.21.15.77:52402) 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.491+0000 7f12aaffd700 1 -- 172.21.15.77:0/852302780 learned_addr learned my addr 172.21.15.77:0/852302780 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.492+0000 7f12aaffd700 1 -- 172.21.15.77:0/852302780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ac106d40 con 0x7f12ac1080e0 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.492+0000 7f12aaffd700 1 --2- 172.21.15.77:0/852302780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 0x7f12ac108500 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f12a000bb40 tx=0x7f12a000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73dcca9c798265d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:34.615 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.492+0000 7f12a9ffb700 1 -- 172.21.15.77:0/852302780 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12a000dbe0 con 0x7f12ac1080e0 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.492+0000 7f12a9ffb700 1 -- 172.21.15.77:0/852302780 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f12a00044f0 con 0x7f12ac1080e0 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.492+0000 7f12a9ffb700 1 -- 172.21.15.77:0/852302780 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12a0010470 con 0x7f12ac1080e0 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.493+0000 7f12b16af700 1 -- 172.21.15.77:0/852302780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 msgr2=0x7f12ac108500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.493+0000 7f12b16af700 1 --2- 172.21.15.77:0/852302780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 0x7f12ac108500 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f12a000bb40 tx=0x7f12a000be50 comp rx=0 tx=0).stop 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.493+0000 7f12b16af700 1 -- 172.21.15.77:0/852302780 shutdown_connections 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.493+0000 7f12b16af700 1 --2- 172.21.15.77:0/852302780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 0x7f12ac108500 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.493+0000 7f12b16af700 1 -- 172.21.15.77:0/852302780 >> 172.21.15.77:0/852302780 conn(0x7f12ac1018d0 msgr2=0x7f12ac103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.493+0000 7f12b16af700 1 -- 172.21.15.77:0/852302780 shutdown_connections 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.493+0000 7f12b16af700 1 -- 172.21.15.77:0/852302780 wait complete. 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.494+0000 7f12b16af700 1 Processor -- start 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.494+0000 7f12b16af700 1 -- start start 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.494+0000 7f12b16af700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 0x7f12ac19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.494+0000 7f12b16af700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac105c30 con 0x7f12ac1080e0 2024-03-16T10:07:34.616 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.494+0000 7f12aaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 0x7f12ac19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.494+0000 7f12aaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 0x7f12ac19de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52414/0 (socket says 172.21.15.77:52414) 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.494+0000 7f12aaffd700 1 -- 172.21.15.77:0/383136598 learned_addr learned my addr 172.21.15.77:0/383136598 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.494+0000 7f12aaffd700 1 -- 172.21.15.77:0/383136598 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ac19e340 con 0x7f12ac1080e0 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.495+0000 7f12aaffd700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 0x7f12ac19de00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f12a000b340 tx=0x7f12a00040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.495+0000 7f1293fff700 1 -- 172.21.15.77:0/383136598 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12a0010720 con 0x7f12ac1080e0 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.495+0000 7f1293fff700 1 -- 172.21.15.77:0/383136598 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f12a001d070 con 0x7f12ac1080e0 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.495+0000 7f12b16af700 1 -- 172.21.15.77:0/383136598 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12ac19e5d0 con 0x7f12ac1080e0 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.495+0000 7f1293fff700 1 -- 172.21.15.77:0/383136598 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12a0021c70 con 0x7f12ac1080e0 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.495+0000 7f12b16af700 1 -- 172.21.15.77:0/383136598 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12ac1a12c0 con 0x7f12ac1080e0 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.496+0000 7f1293fff700 1 -- 172.21.15.77:0/383136598 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f12a001f030 con 0x7f12ac1080e0 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.496+0000 7f1293fff700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 0x7f129403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.496+0000 7f1293fff700 1 -- 172.21.15.77:0/383136598 --> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f129403baa0 con 0x7f1294038ed0 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.496+0000 7f1293fff700 1 -- 172.21.15.77:0/383136598 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f12a004cf00 con 0x7f12ac1080e0 2024-03-16T10:07:34.617 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.496+0000 7f12aa7fc700 1 -- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 msgr2=0x7f129403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1093017736 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.496+0000 7f12aa7fc700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 0x7f129403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.697+0000 7f12aa7fc700 1 -- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 msgr2=0x7f129403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1093017736 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:28.697+0000 7f12aa7fc700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 0x7f129403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:29.098+0000 7f12aa7fc700 1 -- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 msgr2=0x7f129403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1093017736 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:29.098+0000 7f12aa7fc700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 0x7f129403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:29.899+0000 7f12aa7fc700 1 -- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 msgr2=0x7f129403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1093017736 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:29.899+0000 7f12aa7fc700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 0x7f129403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:31.500+0000 7f12aa7fc700 1 -- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 msgr2=0x7f129403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1093017736 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:31.500+0000 7f12aa7fc700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 0x7f129403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:32.926+0000 7f1293fff700 1 -- 172.21.15.77:0/383136598 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7f12a0019910 con 0x7f12ac1080e0 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:32.926+0000 7f1293fff700 1 -- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 msgr2=0x7f129403b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:32.926+0000 7f1293fff700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 0x7f129403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.927+0000 7f1293fff700 1 -- 172.21.15.77:0/383136598 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7f12a004e3b0 con 0x7f12ac1080e0 2024-03-16T10:07:34.618 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.928+0000 7f1293fff700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f129403c7c0 0x7f129403ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.928+0000 7f1293fff700 1 -- 172.21.15.77:0/383136598 --> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f12a0011e50 con 0x7f129403c7c0 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.928+0000 7f12aa7fc700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f129403c7c0 0x7f129403ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.929+0000 7f12aa7fc700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f129403c7c0 0x7f129403ebb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f129c003a10 tx=0x7f129c009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.929+0000 7f1293fff700 1 -- 172.21.15.77:0/383136598 <== mgr.14120 v2:172.21.15.77:6800/1353424963 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f12a0011e50 con 0x7f129403c7c0 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.936+0000 7f12b16af700 1 -- 172.21.15.77:0/383136598 --> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f12ac063900 con 0x7f129403c7c0 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.936+0000 7f1293fff700 1 -- 172.21.15.77:0/383136598 <== mgr.14120 v2:172.21.15.77:6800/1353424963 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f12ac063900 con 0x7f129403c7c0 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.936+0000 7f12b16af700 1 -- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f129403c7c0 msgr2=0x7f129403ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.936+0000 7f12b16af700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f129403c7c0 0x7f129403ebb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f129c003a10 tx=0x7f129c009290 comp rx=0 tx=0).stop 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.936+0000 7f12b16af700 1 -- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 msgr2=0x7f12ac19de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.936+0000 7f12b16af700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 0x7f12ac19de00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f12a000b340 tx=0x7f12a00040b0 comp rx=0 tx=0).stop 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.936+0000 7f12b16af700 1 -- 172.21.15.77:0/383136598 shutdown_connections 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.936+0000 7f12b16af700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f129403c7c0 0x7f129403ebb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.936+0000 7f12b16af700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12ac1080e0 0x7f12ac19de00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.936+0000 7f12b16af700 1 --2- 172.21.15.77:0/383136598 >> [v2:172.21.15.77:6800/1093017736,v1:172.21.15.77:6801/1093017736] conn(0x7f1294038ed0 0x7f129403b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:34.619 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.936+0000 7f12b16af700 1 -- 172.21.15.77:0/383136598 >> 172.21.15.77:0/383136598 conn(0x7f12ac1018d0 msgr2=0x7f12ac194440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:34.620 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.937+0000 7f12b16af700 1 -- 172.21.15.77:0/383136598 shutdown_connections 2024-03-16T10:07:34.620 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:33.937+0000 7f12b16af700 1 -- 172.21.15.77:0/383136598 wait complete. 2024-03-16T10:07:34.620 INFO:teuthology.orchestra.run.smithi077.stdout:mgr epoch 5 is available 2024-03-16T10:07:34.620 INFO:teuthology.orchestra.run.smithi077.stdout:Setting orchestrator backend to cephadm... 2024-03-16T10:07:35.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:34 smithi077 ceph-mon[32400]: [16/Mar/2024:10:07:33] ENGINE Bus STARTING 2024-03-16T10:07:35.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:34 smithi077 ceph-mon[32400]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T10:07:35.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:34 smithi077 ceph-mon[32400]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T10:07:35.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:34 smithi077 ceph-mon[32400]: [16/Mar/2024:10:07:34] ENGINE Serving on http://172.21.15.77:8765 2024-03-16T10:07:35.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:34 smithi077 ceph-mon[32400]: [16/Mar/2024:10:07:34] ENGINE Serving on https://172.21.15.77:7150 2024-03-16T10:07:35.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:34 smithi077 ceph-mon[32400]: [16/Mar/2024:10:07:34] ENGINE Bus STARTED 2024-03-16T10:07:35.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:34 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:07:35.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.025+0000 7f2952eda700 1 Processor -- start 2024-03-16T10:07:35.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.025+0000 7f2952eda700 1 -- start start 2024-03-16T10:07:35.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.025+0000 7f2952eda700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 0x7f294c10aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:35.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.025+0000 7f2952eda700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f294c075a00 con 0x7f294c108700 2024-03-16T10:07:35.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.026+0000 7f2950c76700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 0x7f294c10aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:35.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.026+0000 7f2950c76700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 0x7f294c10aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50176/0 (socket says 172.21.15.77:50176) 2024-03-16T10:07:35.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.026+0000 7f2950c76700 1 -- 172.21.15.77:0/219505962 learned_addr learned my addr 172.21.15.77:0/219505962 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.026+0000 7f2950c76700 1 -- 172.21.15.77:0/219505962 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c10b030 con 0x7f294c108700 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.026+0000 7f2950c76700 1 --2- 172.21.15.77:0/219505962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 0x7f294c10aaf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f29400098a0 tx=0x7f2940009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17f41b98d6e4317 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.026+0000 7f294b7fe700 1 -- 172.21.15.77:0/219505962 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2940004160 con 0x7f294c108700 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.026+0000 7f294b7fe700 1 -- 172.21.15.77:0/219505962 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2940004300 con 0x7f294c108700 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.026+0000 7f294b7fe700 1 -- 172.21.15.77:0/219505962 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2940011b50 con 0x7f294c108700 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.027+0000 7f2952eda700 1 -- 172.21.15.77:0/219505962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 msgr2=0x7f294c10aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.027+0000 7f2952eda700 1 --2- 172.21.15.77:0/219505962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 0x7f294c10aaf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f29400098a0 tx=0x7f2940009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.027+0000 7f2952eda700 1 -- 172.21.15.77:0/219505962 shutdown_connections 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.027+0000 7f2952eda700 1 --2- 172.21.15.77:0/219505962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 0x7f294c10aaf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.027+0000 7f2952eda700 1 -- 172.21.15.77:0/219505962 >> 172.21.15.77:0/219505962 conn(0x7f294c102130 msgr2=0x7f294c104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.027+0000 7f2952eda700 1 -- 172.21.15.77:0/219505962 shutdown_connections 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.027+0000 7f2952eda700 1 -- 172.21.15.77:0/219505962 wait complete. 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.027+0000 7f2952eda700 1 Processor -- start 2024-03-16T10:07:35.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.027+0000 7f2952eda700 1 -- start start 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.028+0000 7f2952eda700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 0x7f294c19bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.028+0000 7f2952eda700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f294c10c050 con 0x7f294c108700 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.028+0000 7f2950c76700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 0x7f294c19bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.028+0000 7f2950c76700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 0x7f294c19bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50192/0 (socket says 172.21.15.77:50192) 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.028+0000 7f2950c76700 1 -- 172.21.15.77:0/1255606851 learned_addr learned my addr 172.21.15.77:0/1255606851 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.028+0000 7f2950c76700 1 -- 172.21.15.77:0/1255606851 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c19c120 con 0x7f294c108700 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.028+0000 7f2950c76700 1 --2- 172.21.15.77:0/1255606851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 0x7f294c19bbe0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2940011670 tx=0x7f2940011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.028+0000 7f2949ffb700 1 -- 172.21.15.77:0/1255606851 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29400039c0 con 0x7f294c108700 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.028+0000 7f2949ffb700 1 -- 172.21.15.77:0/1255606851 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2940011e50 con 0x7f294c108700 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.028+0000 7f2952eda700 1 -- 172.21.15.77:0/1255606851 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294c19c3b0 con 0x7f294c108700 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.028+0000 7f2949ffb700 1 -- 172.21.15.77:0/1255606851 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f294001a5c0 con 0x7f294c108700 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.029+0000 7f2952eda700 1 -- 172.21.15.77:0/1255606851 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294c19f0a0 con 0x7f294c108700 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.029+0000 7f2949ffb700 1 -- 172.21.15.77:0/1255606851 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f294001a870 con 0x7f294c108700 2024-03-16T10:07:35.762 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.029+0000 7f2949ffb700 1 --2- 172.21.15.77:0/1255606851 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f293c038f20 0x7f293c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:35.763 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.029+0000 7f2949ffb700 1 -- 172.21.15.77:0/1255606851 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f294004d3e0 con 0x7f294c108700 2024-03-16T10:07:35.763 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.030+0000 7f294bfff700 1 --2- 172.21.15.77:0/1255606851 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f293c038f20 0x7f293c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:35.763 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.030+0000 7f2952eda700 1 -- 172.21.15.77:0/1255606851 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f294c063900 con 0x7f294c108700 2024-03-16T10:07:35.763 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.030+0000 7f294bfff700 1 --2- 172.21.15.77:0/1255606851 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f293c038f20 0x7f293c03b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2938006fd0 tx=0x7f2938006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:35.763 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.035+0000 7f2949ffb700 1 -- 172.21.15.77:0/1255606851 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f294002cc00 con 0x7f294c108700 2024-03-16T10:07:35.763 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.248+0000 7f2952eda700 1 -- 172.21.15.77:0/1255606851 --> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f294c103330 con 0x7f293c038f20 2024-03-16T10:07:35.763 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.257+0000 7f2949ffb700 1 -- 172.21.15.77:0/1255606851 <== mgr.14120 v2:172.21.15.77:6800/1353424963 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f294c103330 con 0x7f293c038f20 2024-03-16T10:07:35.763 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.259+0000 7f2952eda700 1 -- 172.21.15.77:0/1255606851 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f293c038f20 msgr2=0x7f293c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:35.763 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.259+0000 7f2952eda700 1 --2- 172.21.15.77:0/1255606851 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f293c038f20 0x7f293c03b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2938006fd0 tx=0x7f2938006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:35.763 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.259+0000 7f2952eda700 1 -- 172.21.15.77:0/1255606851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 msgr2=0x7f294c19bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:35.763 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.259+0000 7f2952eda700 1 --2- 172.21.15.77:0/1255606851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 0x7f294c19bbe0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2940011670 tx=0x7f2940011b60 comp rx=0 tx=0).stop 2024-03-16T10:07:35.764 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.259+0000 7f2952eda700 1 -- 172.21.15.77:0/1255606851 shutdown_connections 2024-03-16T10:07:35.764 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.259+0000 7f2952eda700 1 --2- 172.21.15.77:0/1255606851 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f293c038f20 0x7f293c03b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:35.764 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.259+0000 7f2952eda700 1 --2- 172.21.15.77:0/1255606851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f294c108700 0x7f294c19bbe0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:35.764 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.259+0000 7f2952eda700 1 -- 172.21.15.77:0/1255606851 >> 172.21.15.77:0/1255606851 conn(0x7f294c102130 msgr2=0x7f294c102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:35.764 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.259+0000 7f2952eda700 1 -- 172.21.15.77:0/1255606851 shutdown_connections 2024-03-16T10:07:35.764 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:35.259+0000 7f2952eda700 1 -- 172.21.15.77:0/1255606851 wait complete. 2024-03-16T10:07:36.119 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:35 smithi077 ceph-mon[32400]: mgrmap e8: x(active, since 2s) 2024-03-16T10:07:36.119 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:35 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:36.119 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:35 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:07:36.858 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-16T10:07:36.858 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.118+0000 7f9ae7ba3700 1 Processor -- start 2024-03-16T10:07:36.858 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.118+0000 7f9ae7ba3700 1 -- start start 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.118+0000 7f9ae7ba3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 0x7f9ae007c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.118+0000 7f9ae7ba3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae007dfe0 con 0x7f9ae007c300 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.119+0000 7f9ae593f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 0x7f9ae007c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.119+0000 7f9ae593f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 0x7f9ae007c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50208/0 (socket says 172.21.15.77:50208) 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.119+0000 7f9ae593f700 1 -- 172.21.15.77:0/3762756102 learned_addr learned my addr 172.21.15.77:0/3762756102 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.119+0000 7f9ae593f700 1 -- 172.21.15.77:0/3762756102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae007cc60 con 0x7f9ae007c300 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.119+0000 7f9ae593f700 1 --2- 172.21.15.77:0/3762756102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 0x7f9ae007c720 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9adc0098a0 tx=0x7f9adc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59912f55af321355 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.119+0000 7f9ae493d700 1 -- 172.21.15.77:0/3762756102 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9adc004160 con 0x7f9ae007c300 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.119+0000 7f9ae493d700 1 -- 172.21.15.77:0/3762756102 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9adc004300 con 0x7f9ae007c300 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.119+0000 7f9ae493d700 1 -- 172.21.15.77:0/3762756102 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9adc011b50 con 0x7f9ae007c300 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.120+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/3762756102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 msgr2=0x7f9ae007c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.120+0000 7f9ae7ba3700 1 --2- 172.21.15.77:0/3762756102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 0x7f9ae007c720 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9adc0098a0 tx=0x7f9adc009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.120+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/3762756102 shutdown_connections 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.120+0000 7f9ae7ba3700 1 --2- 172.21.15.77:0/3762756102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 0x7f9ae007c720 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:36.859 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.120+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/3762756102 >> 172.21.15.77:0/3762756102 conn(0x7f9ae0103320 msgr2=0x7f9ae0105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.120+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/3762756102 shutdown_connections 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.120+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/3762756102 wait complete. 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.121+0000 7f9ae7ba3700 1 Processor -- start 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.121+0000 7f9ae7ba3700 1 -- start start 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.121+0000 7f9ae7ba3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 0x7f9ae019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.121+0000 7f9ae7ba3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae010e330 con 0x7f9ae007c300 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.121+0000 7f9ae593f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 0x7f9ae019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.121+0000 7f9ae593f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 0x7f9ae019def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50212/0 (socket says 172.21.15.77:50212) 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.121+0000 7f9ae593f700 1 -- 172.21.15.77:0/1033753042 learned_addr learned my addr 172.21.15.77:0/1033753042 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.121+0000 7f9ae593f700 1 -- 172.21.15.77:0/1033753042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae019e430 con 0x7f9ae007c300 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.121+0000 7f9ae593f700 1 --2- 172.21.15.77:0/1033753042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 0x7f9ae019def0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9adc011670 tx=0x7f9adc011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.121+0000 7f9ad6ffd700 1 -- 172.21.15.77:0/1033753042 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9adc0039c0 con 0x7f9ae007c300 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.122+0000 7f9ad6ffd700 1 -- 172.21.15.77:0/1033753042 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9adc011e50 con 0x7f9ae007c300 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.122+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/1033753042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae019e6c0 con 0x7f9ae007c300 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.122+0000 7f9ad6ffd700 1 -- 172.21.15.77:0/1033753042 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9adc01a5c0 con 0x7f9ae007c300 2024-03-16T10:07:36.860 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.122+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/1033753042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae01a13b0 con 0x7f9ae007c300 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.122+0000 7f9ad6ffd700 1 -- 172.21.15.77:0/1033753042 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f9adc01a870 con 0x7f9ae007c300 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.122+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/1033753042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ae0063900 con 0x7f9ae007c300 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.122+0000 7f9ad6ffd700 1 --2- 172.21.15.77:0/1033753042 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f9acc038f20 0x7f9acc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.123+0000 7f9ad6ffd700 1 -- 172.21.15.77:0/1033753042 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9adc04d2d0 con 0x7f9ae007c300 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.123+0000 7f9ae513e700 1 --2- 172.21.15.77:0/1033753042 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f9acc038f20 0x7f9acc03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.123+0000 7f9ae513e700 1 --2- 172.21.15.77:0/1033753042 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f9acc038f20 0x7f9acc03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0006fd0 tx=0x7f9ad0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.127+0000 7f9ad6ffd700 1 -- 172.21.15.77:0/1033753042 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9adc019c70 con 0x7f9ae007c300 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.326+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/1033753042 --> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f9ae0104500 con 0x7f9acc038f20 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.327+0000 7f9ad6ffd700 1 -- 172.21.15.77:0/1033753042 <== mgr.14120 v2:172.21.15.77:6800/1353424963 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f9ae0104500 con 0x7f9acc038f20 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.329+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/1033753042 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f9acc038f20 msgr2=0x7f9acc03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.329+0000 7f9ae7ba3700 1 --2- 172.21.15.77:0/1033753042 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f9acc038f20 0x7f9acc03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0006fd0 tx=0x7f9ad0006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.329+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/1033753042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 msgr2=0x7f9ae019def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.329+0000 7f9ae7ba3700 1 --2- 172.21.15.77:0/1033753042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 0x7f9ae019def0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9adc011670 tx=0x7f9adc011b60 comp rx=0 tx=0).stop 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.329+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/1033753042 shutdown_connections 2024-03-16T10:07:36.861 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.329+0000 7f9ae7ba3700 1 --2- 172.21.15.77:0/1033753042 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f9acc038f20 0x7f9acc03b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:36.862 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.329+0000 7f9ae7ba3700 1 --2- 172.21.15.77:0/1033753042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ae007c300 0x7f9ae019def0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:36.862 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.329+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/1033753042 >> 172.21.15.77:0/1033753042 conn(0x7f9ae0103320 msgr2=0x7f9ae0103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:36.862 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.329+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/1033753042 shutdown_connections 2024-03-16T10:07:36.862 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:36.329+0000 7f9ae7ba3700 1 -- 172.21.15.77:0/1033753042 wait complete. 2024-03-16T10:07:36.862 INFO:teuthology.orchestra.run.smithi077.stdout:Generating ssh key... 2024-03-16T10:07:37.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:36 smithi077 ceph-mon[32400]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:07:37.956 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: Generating public/private rsa key pair. 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: Your identification has been saved in /tmp/tmpjzcnftbv/key. 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: Your public key has been saved in /tmp/tmpjzcnftbv/key.pub. 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: The key fingerprint is: 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: SHA256:nUrw0H4ACZ7L1XFV6JKK3MGud4lsfOSmSElsIrukctg ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: The key's randomart image is: 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: +---[RSA 3072]----+ 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: | ..... ...o. | 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: | . ..+ o . | 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: | o +.+ o | 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: | . + =ooo.. | 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: | . +.++So+. | 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: | o +oo+o. | 2024-03-16T10:07:37.957 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: | oo o+.+ . | 2024-03-16T10:07:37.958 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: |ooE. ...* * | 2024-03-16T10:07:37.958 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: |o.. .o.= | 2024-03-16T10:07:37.958 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: +----[SHA256]-----+ 2024-03-16T10:07:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-mon[32400]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:07:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:37 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:38.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.301+0000 7fa60cb7b700 1 Processor -- start 2024-03-16T10:07:38.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.301+0000 7fa60cb7b700 1 -- start start 2024-03-16T10:07:38.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.302+0000 7fa60cb7b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 0x7fa608108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:38.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.302+0000 7fa60cb7b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa608108640 con 0x7fa608107c50 2024-03-16T10:07:38.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.302+0000 7fa60659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 0x7fa608108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:38.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.302+0000 7fa60659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 0x7fa608108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50224/0 (socket says 172.21.15.77:50224) 2024-03-16T10:07:38.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.302+0000 7fa60659c700 1 -- 172.21.15.77:0/2040456518 learned_addr learned my addr 172.21.15.77:0/2040456518 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:38.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.302+0000 7fa60659c700 1 -- 172.21.15.77:0/2040456518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa608108e50 con 0x7fa608107c50 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.302+0000 7fa60659c700 1 --2- 172.21.15.77:0/2040456518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 0x7fa608108070 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0098a0 tx=0x7fa5fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09380c6058c5424 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.302+0000 7fa60559a700 1 -- 172.21.15.77:0/2040456518 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5fc004160 con 0x7fa608107c50 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.302+0000 7fa60559a700 1 -- 172.21.15.77:0/2040456518 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa5fc004300 con 0x7fa608107c50 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.302+0000 7fa60559a700 1 -- 172.21.15.77:0/2040456518 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5fc011b50 con 0x7fa608107c50 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.303+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2040456518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 msgr2=0x7fa608108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.303+0000 7fa60cb7b700 1 --2- 172.21.15.77:0/2040456518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 0x7fa608108070 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0098a0 tx=0x7fa5fc009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.303+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2040456518 shutdown_connections 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.303+0000 7fa60cb7b700 1 --2- 172.21.15.77:0/2040456518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 0x7fa608108070 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.303+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2040456518 >> 172.21.15.77:0/2040456518 conn(0x7fa608103210 msgr2=0x7fa608105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.303+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2040456518 shutdown_connections 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.303+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2040456518 wait complete. 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.304+0000 7fa60cb7b700 1 Processor -- start 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.304+0000 7fa60cb7b700 1 -- start start 2024-03-16T10:07:38.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.304+0000 7fa60cb7b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 0x7fa60819bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.304+0000 7fa60cb7b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa60810c200 con 0x7fa608107c50 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.304+0000 7fa60659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 0x7fa60819bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.304+0000 7fa60659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 0x7fa60819bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50236/0 (socket says 172.21.15.77:50236) 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.304+0000 7fa60659c700 1 -- 172.21.15.77:0/2999362657 learned_addr learned my addr 172.21.15.77:0/2999362657 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.304+0000 7fa60659c700 1 -- 172.21.15.77:0/2999362657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60819c450 con 0x7fa608107c50 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.304+0000 7fa60659c700 1 --2- 172.21.15.77:0/2999362657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 0x7fa60819bf10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc011670 tx=0x7fa5fc011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.305+0000 7fa5f77fe700 1 -- 172.21.15.77:0/2999362657 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5fc0039c0 con 0x7fa608107c50 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.305+0000 7fa5f77fe700 1 -- 172.21.15.77:0/2999362657 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa5fc011e50 con 0x7fa608107c50 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.305+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2999362657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa60819c6e0 con 0x7fa608107c50 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.305+0000 7fa5f77fe700 1 -- 172.21.15.77:0/2999362657 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5fc01a5c0 con 0x7fa608107c50 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.305+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2999362657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa60819f3d0 con 0x7fa608107c50 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.305+0000 7fa5f77fe700 1 -- 172.21.15.77:0/2999362657 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fa5fc01a870 con 0x7fa608107c50 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.306+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2999362657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa608063900 con 0x7fa608107c50 2024-03-16T10:07:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.306+0000 7fa5f77fe700 1 --2- 172.21.15.77:0/2999362657 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa5f0038f20 0x7fa5f003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.306+0000 7fa5f77fe700 1 -- 172.21.15.77:0/2999362657 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa5fc04d250 con 0x7fa608107c50 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.306+0000 7fa605d9b700 1 --2- 172.21.15.77:0/2999362657 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa5f0038f20 0x7fa5f003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.306+0000 7fa605d9b700 1 --2- 172.21.15.77:0/2999362657 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa5f0038f20 0x7fa5f003b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8006fd0 tx=0x7fa5f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.311+0000 7fa5f77fe700 1 -- 172.21.15.77:0/2999362657 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa5fc019c70 con 0x7fa608107c50 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.505+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2999362657 --> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fa60802dbe0 con 0x7fa5f0038f20 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.648+0000 7fa5f77fe700 1 -- 172.21.15.77:0/2999362657 <== mgr.14120 v2:172.21.15.77:6800/1353424963 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa60802dbe0 con 0x7fa5f0038f20 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.651+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2999362657 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa5f0038f20 msgr2=0x7fa5f003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.651+0000 7fa60cb7b700 1 --2- 172.21.15.77:0/2999362657 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa5f0038f20 0x7fa5f003b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8006fd0 tx=0x7fa5f8006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.651+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2999362657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 msgr2=0x7fa60819bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.651+0000 7fa60cb7b700 1 --2- 172.21.15.77:0/2999362657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 0x7fa60819bf10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc011670 tx=0x7fa5fc011b60 comp rx=0 tx=0).stop 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.651+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2999362657 shutdown_connections 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.651+0000 7fa60cb7b700 1 --2- 172.21.15.77:0/2999362657 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa5f0038f20 0x7fa5f003b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.651+0000 7fa60cb7b700 1 --2- 172.21.15.77:0/2999362657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa608107c50 0x7fa60819bf10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:38.259 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.651+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2999362657 >> 172.21.15.77:0/2999362657 conn(0x7fa608103210 msgr2=0x7fa608103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:38.260 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.651+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2999362657 shutdown_connections 2024-03-16T10:07:38.260 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:37.651+0000 7fa60cb7b700 1 -- 172.21.15.77:0/2999362657 wait complete. 2024-03-16T10:07:39.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:38 smithi077 ceph-mon[32400]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:07:39.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:38 smithi077 ceph-mon[32400]: Generating ssh key... 2024-03-16T10:07:39.337 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVEc01skJFwdDf7zHfFjZBTdTewuMYhdFJMMB2nH5YqZq7xAQzSv8OaVztVaY0jGchi9iLjvB70v2zZGkn/0Fi5JitQKaz82BzhNPDX/djIW2oxNs3Ve32PxEVtsVeJPIcE7nCmjasETWtJis35v1I05e3uthV4XTmIAY64QzLDXzcVN6i+q7qPQe6i2zNZhmB875dXLu9loLTT/lBXhJSSQVWawl0dCl7LTa7Ljr+NZljG6Kujp5CqgIuTB0xIYpmxYLunEuoB6YWvESm5hiD1/Tpi0SPMNuQBhyR+r+C3F6/CrzDi9xyEqg1KcaKOq0BYEv7D1he4gB2x3eHcZIAU55O5cBUmGYHkmNwKOM2pvJfIFIRaOl8KZA1OZUUd1onmRiWxZYzOuzRnpx8Gl51xso76eJRE0E6+R6mwl/TfoUVHUp9Fe0prRobkHIsbUpXp1CfW7Jhi3xnY1PE52YslFWreKaeZpnapIIGk6DKHNz9oVFSutnZGG2PwgLW/l8= ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:07:39.337 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.643+0000 7fa7e9c96700 1 Processor -- start 2024-03-16T10:07:39.337 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.643+0000 7fa7e9c96700 1 -- start start 2024-03-16T10:07:39.337 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.643+0000 7fa7e9c96700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 0x7fa7e41080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:39.337 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.643+0000 7fa7e9c96700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4108670 con 0x7fa7e4107c80 2024-03-16T10:07:39.337 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.644+0000 7fa7e37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 0x7fa7e41080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.644+0000 7fa7e37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 0x7fa7e41080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50250/0 (socket says 172.21.15.77:50250) 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.644+0000 7fa7e37fe700 1 -- 172.21.15.77:0/3755009905 learned_addr learned my addr 172.21.15.77:0/3755009905 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.644+0000 7fa7e37fe700 1 -- 172.21.15.77:0/3755009905 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e4108e80 con 0x7fa7e4107c80 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.644+0000 7fa7e37fe700 1 --2- 172.21.15.77:0/3755009905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 0x7fa7e41080a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40098a0 tx=0x7fa7d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bbb4b53bbb0d4ea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.644+0000 7fa7e27fc700 1 -- 172.21.15.77:0/3755009905 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7d4004160 con 0x7fa7e4107c80 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.644+0000 7fa7e27fc700 1 -- 172.21.15.77:0/3755009905 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa7d4004300 con 0x7fa7e4107c80 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.644+0000 7fa7e27fc700 1 -- 172.21.15.77:0/3755009905 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7d4011b50 con 0x7fa7e4107c80 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.645+0000 7fa7e9c96700 1 -- 172.21.15.77:0/3755009905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 msgr2=0x7fa7e41080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.645+0000 7fa7e9c96700 1 --2- 172.21.15.77:0/3755009905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 0x7fa7e41080a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40098a0 tx=0x7fa7d4009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.645+0000 7fa7e9c96700 1 -- 172.21.15.77:0/3755009905 shutdown_connections 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.645+0000 7fa7e9c96700 1 --2- 172.21.15.77:0/3755009905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 0x7fa7e41080a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.645+0000 7fa7e9c96700 1 -- 172.21.15.77:0/3755009905 >> 172.21.15.77:0/3755009905 conn(0x7fa7e4103240 msgr2=0x7fa7e4105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.645+0000 7fa7e9c96700 1 -- 172.21.15.77:0/3755009905 shutdown_connections 2024-03-16T10:07:39.338 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.645+0000 7fa7e9c96700 1 -- 172.21.15.77:0/3755009905 wait complete. 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.645+0000 7fa7e9c96700 1 Processor -- start 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e9c96700 1 -- start start 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e9c96700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 0x7fa7e419bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e9c96700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e410c230 con 0x7fa7e4107c80 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 0x7fa7e419bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 0x7fa7e419bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50256/0 (socket says 172.21.15.77:50256) 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e37fe700 1 -- 172.21.15.77:0/50210224 learned_addr learned my addr 172.21.15.77:0/50210224 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e37fe700 1 -- 172.21.15.77:0/50210224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e419c1b0 con 0x7fa7e4107c80 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e37fe700 1 --2- 172.21.15.77:0/50210224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 0x7fa7e419bc70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4011670 tx=0x7fa7d4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e0ff9700 1 -- 172.21.15.77:0/50210224 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7d40039c0 con 0x7fa7e4107c80 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e9c96700 1 -- 172.21.15.77:0/50210224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e419c440 con 0x7fa7e4107c80 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e0ff9700 1 -- 172.21.15.77:0/50210224 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa7d4011e50 con 0x7fa7e4107c80 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e0ff9700 1 -- 172.21.15.77:0/50210224 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7d401a5c0 con 0x7fa7e4107c80 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.646+0000 7fa7e9c96700 1 -- 172.21.15.77:0/50210224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e419f130 con 0x7fa7e4107c80 2024-03-16T10:07:39.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.647+0000 7fa7e0ff9700 1 -- 172.21.15.77:0/50210224 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fa7d4024430 con 0x7fa7e4107c80 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.647+0000 7fa7e9c96700 1 -- 172.21.15.77:0/50210224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e4063900 con 0x7fa7e4107c80 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.647+0000 7fa7e0ff9700 1 --2- 172.21.15.77:0/50210224 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa7d0038ed0 0x7fa7d003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.647+0000 7fa7e0ff9700 1 -- 172.21.15.77:0/50210224 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa7d404d310 con 0x7fa7e4107c80 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.648+0000 7fa7e2ffd700 1 --2- 172.21.15.77:0/50210224 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa7d0038ed0 0x7fa7d003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.648+0000 7fa7e2ffd700 1 --2- 172.21.15.77:0/50210224 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa7d0038ed0 0x7fa7d003b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc006fd0 tx=0x7fa7cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.652+0000 7fa7e0ff9700 1 -- 172.21.15.77:0/50210224 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa7d402d910 con 0x7fa7e4107c80 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.849+0000 7fa7e9c96700 1 -- 172.21.15.77:0/50210224 --> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fa7e4104460 con 0x7fa7d0038ed0 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.850+0000 7fa7e0ff9700 1 -- 172.21.15.77:0/50210224 <== mgr.14120 v2:172.21.15.77:6800/1353424963 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fa7e4104460 con 0x7fa7d0038ed0 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.852+0000 7fa7e9c96700 1 -- 172.21.15.77:0/50210224 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa7d0038ed0 msgr2=0x7fa7d003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.852+0000 7fa7e9c96700 1 --2- 172.21.15.77:0/50210224 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa7d0038ed0 0x7fa7d003b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc006fd0 tx=0x7fa7cc006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.852+0000 7fa7e9c96700 1 -- 172.21.15.77:0/50210224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 msgr2=0x7fa7e419bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.852+0000 7fa7e9c96700 1 --2- 172.21.15.77:0/50210224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 0x7fa7e419bc70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4011670 tx=0x7fa7d4011b60 comp rx=0 tx=0).stop 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.852+0000 7fa7e9c96700 1 -- 172.21.15.77:0/50210224 shutdown_connections 2024-03-16T10:07:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.852+0000 7fa7e9c96700 1 --2- 172.21.15.77:0/50210224 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa7d0038ed0 0x7fa7d003b390 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:39.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.852+0000 7fa7e9c96700 1 --2- 172.21.15.77:0/50210224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7e4107c80 0x7fa7e419bc70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:39.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.852+0000 7fa7e9c96700 1 -- 172.21.15.77:0/50210224 >> 172.21.15.77:0/50210224 conn(0x7fa7e4103240 msgr2=0x7fa7e4103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:39.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.853+0000 7fa7e9c96700 1 -- 172.21.15.77:0/50210224 shutdown_connections 2024-03-16T10:07:39.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:38.853+0000 7fa7e9c96700 1 -- 172.21.15.77:0/50210224 wait complete. 2024-03-16T10:07:39.341 INFO:teuthology.orchestra.run.smithi077.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-16T10:07:39.341 INFO:teuthology.orchestra.run.smithi077.stdout:Adding key to root@localhost authorized_keys... 2024-03-16T10:07:39.341 INFO:teuthology.orchestra.run.smithi077.stdout:Adding host smithi077... 2024-03-16T10:07:40.192 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:39 smithi077 ceph-mon[32400]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:07:41.133 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:40 smithi077 ceph-mon[32400]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "addr": "172.21.15.77", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:07:42.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:41 smithi077 ceph-mon[32400]: Deploying cephadm binary to smithi077 2024-03-16T10:07:44.802 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Added host 'smithi077' with addr '172.21.15.77' 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.731+0000 7fa6743fd700 1 Processor -- start 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.731+0000 7fa6743fd700 1 -- start start 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.731+0000 7fa6743fd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 0x7fa66c106330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.731+0000 7fa6743fd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c106900 con 0x7fa66c105f10 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.731+0000 7fa672199700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 0x7fa66c106330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.731+0000 7fa672199700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 0x7fa66c106330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50272/0 (socket says 172.21.15.77:50272) 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.731+0000 7fa672199700 1 -- 172.21.15.77:0/2608316535 learned_addr learned my addr 172.21.15.77:0/2608316535 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.731+0000 7fa672199700 1 -- 172.21.15.77:0/2608316535 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66c108f40 con 0x7fa66c105f10 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.732+0000 7fa672199700 1 --2- 172.21.15.77:0/2608316535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 0x7fa66c106330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa6680098a0 tx=0x7fa668009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b0f14bf0a29cd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.732+0000 7fa671197700 1 -- 172.21.15.77:0/2608316535 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa668004160 con 0x7fa66c105f10 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.732+0000 7fa671197700 1 -- 172.21.15.77:0/2608316535 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa668004300 con 0x7fa66c105f10 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.732+0000 7fa671197700 1 -- 172.21.15.77:0/2608316535 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa668011b50 con 0x7fa66c105f10 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.732+0000 7fa6743fd700 1 -- 172.21.15.77:0/2608316535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 msgr2=0x7fa66c106330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.732+0000 7fa6743fd700 1 --2- 172.21.15.77:0/2608316535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 0x7fa66c106330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa6680098a0 tx=0x7fa668009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.732+0000 7fa6743fd700 1 -- 172.21.15.77:0/2608316535 shutdown_connections 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.732+0000 7fa6743fd700 1 --2- 172.21.15.77:0/2608316535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 0x7fa66c106330 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.732+0000 7fa6743fd700 1 -- 172.21.15.77:0/2608316535 >> 172.21.15.77:0/2608316535 conn(0x7fa66c101880 msgr2=0x7fa66c103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.732+0000 7fa6743fd700 1 -- 172.21.15.77:0/2608316535 shutdown_connections 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.732+0000 7fa6743fd700 1 -- 172.21.15.77:0/2608316535 wait complete. 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.733+0000 7fa6743fd700 1 Processor -- start 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.733+0000 7fa6743fd700 1 -- start start 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.733+0000 7fa6743fd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 0x7fa66c19bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.733+0000 7fa6743fd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c10c170 con 0x7fa66c105f10 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.733+0000 7fa672199700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 0x7fa66c19bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.733+0000 7fa672199700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 0x7fa66c19bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50284/0 (socket says 172.21.15.77:50284) 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.733+0000 7fa672199700 1 -- 172.21.15.77:0/2704228319 learned_addr learned my addr 172.21.15.77:0/2704228319 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.734+0000 7fa672199700 1 -- 172.21.15.77:0/2704228319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66c19c1d0 con 0x7fa66c105f10 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.734+0000 7fa672199700 1 --2- 172.21.15.77:0/2704228319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 0x7fa66c19bc90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa668000c00 tx=0x7fa668011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.734+0000 7fa6637fe700 1 -- 172.21.15.77:0/2704228319 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa668003b20 con 0x7fa66c105f10 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.734+0000 7fa6637fe700 1 -- 172.21.15.77:0/2704228319 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa668003cc0 con 0x7fa66c105f10 2024-03-16T10:07:44.804 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.734+0000 7fa6743fd700 1 -- 172.21.15.77:0/2704228319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66c19c460 con 0x7fa66c105f10 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.734+0000 7fa6743fd700 1 -- 172.21.15.77:0/2704228319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa66c19f150 con 0x7fa66c105f10 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.734+0000 7fa6637fe700 1 -- 172.21.15.77:0/2704228319 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa668024910 con 0x7fa66c105f10 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.735+0000 7fa6637fe700 1 -- 172.21.15.77:0/2704228319 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fa668022020 con 0x7fa66c105f10 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.735+0000 7fa6743fd700 1 -- 172.21.15.77:0/2704228319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa650005360 con 0x7fa66c105f10 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.735+0000 7fa6637fe700 1 --2- 172.21.15.77:0/2704228319 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa658038f70 0x7fa65803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.735+0000 7fa6637fe700 1 -- 172.21.15.77:0/2704228319 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa668011e00 con 0x7fa66c105f10 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.735+0000 7fa671998700 1 --2- 172.21.15.77:0/2704228319 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa658038f70 0x7fa65803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.735+0000 7fa671998700 1 --2- 172.21.15.77:0/2704228319 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa658038f70 0x7fa65803b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa65c006fd0 tx=0x7fa65c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.740+0000 7fa6637fe700 1 -- 172.21.15.77:0/2704228319 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa6680172b0 con 0x7fa66c105f10 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:39.938+0000 7fa6743fd700 1 -- 172.21.15.77:0/2704228319 --> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi077", "addr": "172.21.15.77", "target": ["mon-mgr", ""]}) -- 0x7fa650000bf0 con 0x7fa658038f70 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:44.270+0000 7fa6637fe700 1 -- 172.21.15.77:0/2704228319 <== mgr.14120 v2:172.21.15.77:6800/1353424963 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fa650000bf0 con 0x7fa658038f70 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:44.272+0000 7fa6743fd700 1 -- 172.21.15.77:0/2704228319 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa658038f70 msgr2=0x7fa65803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:44.272+0000 7fa6743fd700 1 --2- 172.21.15.77:0/2704228319 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa658038f70 0x7fa65803b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa65c006fd0 tx=0x7fa65c006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:44.272+0000 7fa6743fd700 1 -- 172.21.15.77:0/2704228319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 msgr2=0x7fa66c19bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:44.272+0000 7fa6743fd700 1 --2- 172.21.15.77:0/2704228319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 0x7fa66c19bc90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa668000c00 tx=0x7fa668011c10 comp rx=0 tx=0).stop 2024-03-16T10:07:44.805 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:44.272+0000 7fa6743fd700 1 -- 172.21.15.77:0/2704228319 shutdown_connections 2024-03-16T10:07:44.806 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:44.272+0000 7fa6743fd700 1 --2- 172.21.15.77:0/2704228319 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fa658038f70 0x7fa65803b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:44.806 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:44.272+0000 7fa6743fd700 1 --2- 172.21.15.77:0/2704228319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa66c105f10 0x7fa66c19bc90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:44.806 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:44.272+0000 7fa6743fd700 1 -- 172.21.15.77:0/2704228319 >> 172.21.15.77:0/2704228319 conn(0x7fa66c101880 msgr2=0x7fa66c102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:44.806 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:44.272+0000 7fa6743fd700 1 -- 172.21.15.77:0/2704228319 shutdown_connections 2024-03-16T10:07:44.806 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:44.272+0000 7fa6743fd700 1 -- 172.21.15.77:0/2704228319 wait complete. 2024-03-16T10:07:44.806 INFO:teuthology.orchestra.run.smithi077.stdout:Deploying unmanaged mon service... 2024-03-16T10:07:45.459 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:45 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:45.460 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:45 smithi077 ceph-mon[32400]: Added host smithi077 2024-03-16T10:07:45.460 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:45 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:07:46.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-16T10:07:46.339 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.283+0000 7fc44e110700 1 Processor -- start 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.283+0000 7fc44e110700 1 -- start start 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.283+0000 7fc44e110700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 0x7fc448106a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.283+0000 7fc44e110700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc44807be30 con 0x7fc448106620 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.283+0000 7fc4477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 0x7fc448106a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.284+0000 7fc4477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 0x7fc448106a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34366/0 (socket says 172.21.15.77:34366) 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.284+0000 7fc4477fe700 1 -- 172.21.15.77:0/538752780 learned_addr learned my addr 172.21.15.77:0/538752780 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.284+0000 7fc4477fe700 1 -- 172.21.15.77:0/538752780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc448106f80 con 0x7fc448106620 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.284+0000 7fc4477fe700 1 --2- 172.21.15.77:0/538752780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 0x7fc448106a40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc4380098a0 tx=0x7fc438009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=833c9d4c7405efc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.284+0000 7fc4467fc700 1 -- 172.21.15.77:0/538752780 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc438004160 con 0x7fc448106620 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.284+0000 7fc4467fc700 1 -- 172.21.15.77:0/538752780 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc438004300 con 0x7fc448106620 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.284+0000 7fc4467fc700 1 -- 172.21.15.77:0/538752780 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc438011b50 con 0x7fc448106620 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.285+0000 7fc44e110700 1 -- 172.21.15.77:0/538752780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 msgr2=0x7fc448106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.285+0000 7fc44e110700 1 --2- 172.21.15.77:0/538752780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 0x7fc448106a40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc4380098a0 tx=0x7fc438009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.285+0000 7fc44e110700 1 -- 172.21.15.77:0/538752780 shutdown_connections 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.285+0000 7fc44e110700 1 --2- 172.21.15.77:0/538752780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 0x7fc448106a40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.285+0000 7fc44e110700 1 -- 172.21.15.77:0/538752780 >> 172.21.15.77:0/538752780 conn(0x7fc448102220 msgr2=0x7fc448104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.285+0000 7fc44e110700 1 -- 172.21.15.77:0/538752780 shutdown_connections 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.285+0000 7fc44e110700 1 -- 172.21.15.77:0/538752780 wait complete. 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.285+0000 7fc44e110700 1 Processor -- start 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.285+0000 7fc44e110700 1 -- start start 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.286+0000 7fc44e110700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 0x7fc44819e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.286+0000 7fc44e110700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc448108010 con 0x7fc448106620 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.286+0000 7fc4477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 0x7fc44819e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.286+0000 7fc4477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 0x7fc44819e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34368/0 (socket says 172.21.15.77:34368) 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.286+0000 7fc4477fe700 1 -- 172.21.15.77:0/393401182 learned_addr learned my addr 172.21.15.77:0/393401182 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.286+0000 7fc4477fe700 1 -- 172.21.15.77:0/393401182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44819e6f0 con 0x7fc448106620 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.286+0000 7fc4477fe700 1 --2- 172.21.15.77:0/393401182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 0x7fc44819e1b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc438011b60 tx=0x7fc438011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.286+0000 7fc444ff9700 1 -- 172.21.15.77:0/393401182 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc438003b20 con 0x7fc448106620 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.286+0000 7fc444ff9700 1 -- 172.21.15.77:0/393401182 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc438003cc0 con 0x7fc448106620 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.286+0000 7fc444ff9700 1 -- 172.21.15.77:0/393401182 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc438024840 con 0x7fc448106620 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.286+0000 7fc44e110700 1 -- 172.21.15.77:0/393401182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc44819e980 con 0x7fc448106620 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.286+0000 7fc44e110700 1 -- 172.21.15.77:0/393401182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4481a1670 con 0x7fc448106620 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.287+0000 7fc444ff9700 1 -- 172.21.15.77:0/393401182 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fc438022020 con 0x7fc448106620 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.287+0000 7fc44e110700 1 -- 172.21.15.77:0/393401182 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc448063900 con 0x7fc448106620 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.287+0000 7fc444ff9700 1 --2- 172.21.15.77:0/393401182 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fc430038f70 0x7fc43003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.287+0000 7fc444ff9700 1 -- 172.21.15.77:0/393401182 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc438011e00 con 0x7fc448106620 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.287+0000 7fc446ffd700 1 --2- 172.21.15.77:0/393401182 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fc430038f70 0x7fc43003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.288+0000 7fc446ffd700 1 --2- 172.21.15.77:0/393401182 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fc430038f70 0x7fc43003b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc43c006fd0 tx=0x7fc43c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.292+0000 7fc444ff9700 1 -- 172.21.15.77:0/393401182 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc43800ba30 con 0x7fc448106620 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.510+0000 7fc44e110700 1 -- 172.21.15.77:0/393401182 --> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc448062710 con 0x7fc430038f70 2024-03-16T10:07:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.518+0000 7fc444ff9700 1 -- 172.21.15.77:0/393401182 <== mgr.14120 v2:172.21.15.77:6800/1353424963 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc448062710 con 0x7fc430038f70 2024-03-16T10:07:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.520+0000 7fc44e110700 1 -- 172.21.15.77:0/393401182 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fc430038f70 msgr2=0x7fc43003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.520+0000 7fc44e110700 1 --2- 172.21.15.77:0/393401182 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fc430038f70 0x7fc43003b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc43c006fd0 tx=0x7fc43c006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.520+0000 7fc44e110700 1 -- 172.21.15.77:0/393401182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 msgr2=0x7fc44819e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.520+0000 7fc44e110700 1 --2- 172.21.15.77:0/393401182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 0x7fc44819e1b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc438011b60 tx=0x7fc438011c40 comp rx=0 tx=0).stop 2024-03-16T10:07:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.520+0000 7fc44e110700 1 -- 172.21.15.77:0/393401182 shutdown_connections 2024-03-16T10:07:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.520+0000 7fc44e110700 1 --2- 172.21.15.77:0/393401182 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fc430038f70 0x7fc43003b430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.520+0000 7fc44e110700 1 --2- 172.21.15.77:0/393401182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc448106620 0x7fc44819e1b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.520+0000 7fc44e110700 1 -- 172.21.15.77:0/393401182 >> 172.21.15.77:0/393401182 conn(0x7fc448102220 msgr2=0x7fc448194ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.520+0000 7fc44e110700 1 -- 172.21.15.77:0/393401182 shutdown_connections 2024-03-16T10:07:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:45.520+0000 7fc44e110700 1 -- 172.21.15.77:0/393401182 wait complete. 2024-03-16T10:07:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:Deploying unmanaged mgr service... 2024-03-16T10:07:46.634 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:46 smithi077 ceph-mon[32400]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:07:46.634 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:46 smithi077 ceph-mon[32400]: Saving service mon spec with placement count:5 2024-03-16T10:07:46.634 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:46 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:47.848 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-16T10:07:47.848 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.731+0000 7f610c16e700 1 Processor -- start 2024-03-16T10:07:47.848 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.732+0000 7f610c16e700 1 -- start start 2024-03-16T10:07:47.848 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.732+0000 7f610c16e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 0x7f610410abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:47.848 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.732+0000 7f610c16e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f610407be70 con 0x7f61041087f0 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.732+0000 7f6109f0a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 0x7f610410abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.732+0000 7f6109f0a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 0x7f610410abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34376/0 (socket says 172.21.15.77:34376) 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.732+0000 7f6109f0a700 1 -- 172.21.15.77:0/1286413452 learned_addr learned my addr 172.21.15.77:0/1286413452 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.732+0000 7f6109f0a700 1 -- 172.21.15.77:0/1286413452 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610410b120 con 0x7f61041087f0 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.732+0000 7f6109f0a700 1 --2- 172.21.15.77:0/1286413452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 0x7f610410abe0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f60fc009e80 tx=0x7f60fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c49d8b157556af server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.732+0000 7f6108f08700 1 -- 172.21.15.77:0/1286413452 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60fc017070 con 0x7f61041087f0 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.733+0000 7f6108f08700 1 -- 172.21.15.77:0/1286413452 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f60fc00b8c0 con 0x7f61041087f0 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.733+0000 7f6108f08700 1 -- 172.21.15.77:0/1286413452 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60fc0045c0 con 0x7f61041087f0 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.733+0000 7f610c16e700 1 -- 172.21.15.77:0/1286413452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 msgr2=0x7f610410abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.733+0000 7f610c16e700 1 --2- 172.21.15.77:0/1286413452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 0x7f610410abe0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f60fc009e80 tx=0x7f60fc00b330 comp rx=0 tx=0).stop 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.733+0000 7f610c16e700 1 -- 172.21.15.77:0/1286413452 shutdown_connections 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.733+0000 7f610c16e700 1 --2- 172.21.15.77:0/1286413452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 0x7f610410abe0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.733+0000 7f610c16e700 1 -- 172.21.15.77:0/1286413452 >> 172.21.15.77:0/1286413452 conn(0x7f6104102220 msgr2=0x7f6104104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.733+0000 7f610c16e700 1 -- 172.21.15.77:0/1286413452 shutdown_connections 2024-03-16T10:07:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.733+0000 7f610c16e700 1 -- 172.21.15.77:0/1286413452 wait complete. 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.734+0000 7f610c16e700 1 Processor -- start 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.734+0000 7f610c16e700 1 -- start start 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.734+0000 7f610c16e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 0x7f6104108070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.734+0000 7f610c16e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f610410c140 con 0x7f61041087f0 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.734+0000 7f6109f0a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 0x7f6104108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.734+0000 7f6109f0a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 0x7f6104108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34384/0 (socket says 172.21.15.77:34384) 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.734+0000 7f6109f0a700 1 -- 172.21.15.77:0/3307994718 learned_addr learned my addr 172.21.15.77:0/3307994718 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.734+0000 7f6109f0a700 1 -- 172.21.15.77:0/3307994718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61041085b0 con 0x7f61041087f0 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.734+0000 7f6109f0a700 1 --2- 172.21.15.77:0/3307994718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 0x7f6104108070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0072a0 tx=0x7f60fc00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.734+0000 7f6102ffd700 1 -- 172.21.15.77:0/3307994718 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60fc020020 con 0x7f61041087f0 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.735+0000 7f6102ffd700 1 -- 172.21.15.77:0/3307994718 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f60fc003b50 con 0x7f61041087f0 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.735+0000 7f610c16e700 1 -- 172.21.15.77:0/3307994718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61041067a0 con 0x7f61041087f0 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.735+0000 7f6102ffd700 1 -- 172.21.15.77:0/3307994718 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60fc017040 con 0x7f61041087f0 2024-03-16T10:07:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.735+0000 7f610c16e700 1 -- 172.21.15.77:0/3307994718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6104106ce0 con 0x7f61041087f0 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.735+0000 7f6102ffd700 1 -- 172.21.15.77:0/3307994718 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f60fc023020 con 0x7f61041087f0 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.735+0000 7f610c16e700 1 -- 172.21.15.77:0/3307994718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6104063900 con 0x7f61041087f0 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.735+0000 7f6102ffd700 1 --2- 172.21.15.77:0/3307994718 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f60ec038f20 0x7f60ec03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.736+0000 7f6102ffd700 1 -- 172.21.15.77:0/3307994718 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f60fc00f050 con 0x7f61041087f0 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.736+0000 7f6109709700 1 --2- 172.21.15.77:0/3307994718 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f60ec038f20 0x7f60ec03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.736+0000 7f6109709700 1 --2- 172.21.15.77:0/3307994718 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f60ec038f20 0x7f60ec03b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f60f8006fd0 tx=0x7f60f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.740+0000 7f6102ffd700 1 -- 172.21.15.77:0/3307994718 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f60fc02a020 con 0x7f61041087f0 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.948+0000 7f610c16e700 1 -- 172.21.15.77:0/3307994718 --> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f6104195750 con 0x7f60ec038f20 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.955+0000 7f6102ffd700 1 -- 172.21.15.77:0/3307994718 <== mgr.14120 v2:172.21.15.77:6800/1353424963 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6104195750 con 0x7f60ec038f20 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.957+0000 7f610c16e700 1 -- 172.21.15.77:0/3307994718 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f60ec038f20 msgr2=0x7f60ec03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.957+0000 7f610c16e700 1 --2- 172.21.15.77:0/3307994718 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f60ec038f20 0x7f60ec03b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f60f8006fd0 tx=0x7f60f8006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.957+0000 7f610c16e700 1 -- 172.21.15.77:0/3307994718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 msgr2=0x7f6104108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.957+0000 7f610c16e700 1 --2- 172.21.15.77:0/3307994718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 0x7f6104108070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0072a0 tx=0x7f60fc00bd50 comp rx=0 tx=0).stop 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.957+0000 7f610c16e700 1 -- 172.21.15.77:0/3307994718 shutdown_connections 2024-03-16T10:07:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.957+0000 7f610c16e700 1 --2- 172.21.15.77:0/3307994718 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f60ec038f20 0x7f60ec03b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:47.852 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.957+0000 7f610c16e700 1 --2- 172.21.15.77:0/3307994718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61041087f0 0x7f6104108070 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:47.852 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.957+0000 7f610c16e700 1 -- 172.21.15.77:0/3307994718 >> 172.21.15.77:0/3307994718 conn(0x7f6104102220 msgr2=0x7f6104102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:47.852 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.957+0000 7f610c16e700 1 -- 172.21.15.77:0/3307994718 shutdown_connections 2024-03-16T10:07:47.852 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:46.957+0000 7f610c16e700 1 -- 172.21.15.77:0/3307994718 wait complete. 2024-03-16T10:07:48.102 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:47 smithi077 ceph-mon[32400]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:07:48.102 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:47 smithi077 ceph-mon[32400]: Saving service mgr spec with placement count:2 2024-03-16T10:07:48.102 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:47 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:48.102 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:47 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:49.071 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.311+0000 7fafa2abe700 1 Processor -- start 2024-03-16T10:07:49.071 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.311+0000 7fafa2abe700 1 -- start start 2024-03-16T10:07:49.071 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.311+0000 7fafa2abe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 0x7faf9c106960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:49.071 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.311+0000 7fafa2abe700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c07be30 con 0x7faf9c106540 2024-03-16T10:07:49.071 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.311+0000 7fafa085a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 0x7faf9c106960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.311+0000 7fafa085a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 0x7faf9c106960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34398/0 (socket says 172.21.15.77:34398) 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.311+0000 7fafa085a700 1 -- 172.21.15.77:0/3540582036 learned_addr learned my addr 172.21.15.77:0/3540582036 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.312+0000 7fafa085a700 1 -- 172.21.15.77:0/3540582036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9c106ea0 con 0x7faf9c106540 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.312+0000 7fafa085a700 1 --2- 172.21.15.77:0/3540582036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 0x7faf9c106960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf900098a0 tx=0x7faf90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7000d6e8ce918f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.312+0000 7faf9b7fe700 1 -- 172.21.15.77:0/3540582036 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf90004160 con 0x7faf9c106540 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.312+0000 7faf9b7fe700 1 -- 172.21.15.77:0/3540582036 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf90004300 con 0x7faf9c106540 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.312+0000 7faf9b7fe700 1 -- 172.21.15.77:0/3540582036 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf90011b20 con 0x7faf9c106540 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.313+0000 7fafa2abe700 1 -- 172.21.15.77:0/3540582036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 msgr2=0x7faf9c106960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.313+0000 7fafa2abe700 1 --2- 172.21.15.77:0/3540582036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 0x7faf9c106960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf900098a0 tx=0x7faf90009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.313+0000 7fafa2abe700 1 -- 172.21.15.77:0/3540582036 shutdown_connections 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.313+0000 7fafa2abe700 1 --2- 172.21.15.77:0/3540582036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 0x7faf9c106960 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.313+0000 7fafa2abe700 1 -- 172.21.15.77:0/3540582036 >> 172.21.15.77:0/3540582036 conn(0x7faf9c1020e0 msgr2=0x7faf9c104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.313+0000 7fafa2abe700 1 -- 172.21.15.77:0/3540582036 shutdown_connections 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.313+0000 7fafa2abe700 1 -- 172.21.15.77:0/3540582036 wait complete. 2024-03-16T10:07:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.314+0000 7fafa2abe700 1 Processor -- start 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.314+0000 7fafa2abe700 1 -- start start 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.314+0000 7fafa2abe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 0x7faf9c19ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.314+0000 7fafa2abe700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c107f70 con 0x7faf9c106540 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.314+0000 7fafa085a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 0x7faf9c19ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.314+0000 7fafa085a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 0x7faf9c19ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34408/0 (socket says 172.21.15.77:34408) 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.314+0000 7fafa085a700 1 -- 172.21.15.77:0/2249071321 learned_addr learned my addr 172.21.15.77:0/2249071321 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.314+0000 7fafa085a700 1 -- 172.21.15.77:0/2249071321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9c19e2f0 con 0x7faf9c106540 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.314+0000 7fafa085a700 1 --2- 172.21.15.77:0/2249071321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 0x7faf9c19ddb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faf900117e0 tx=0x7faf90011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.315+0000 7faf99ffb700 1 -- 172.21.15.77:0/2249071321 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf90003bf0 con 0x7faf9c106540 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.315+0000 7faf99ffb700 1 -- 172.21.15.77:0/2249071321 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf9001a910 con 0x7faf9c106540 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.315+0000 7faf99ffb700 1 -- 172.21.15.77:0/2249071321 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf90024970 con 0x7faf9c106540 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.315+0000 7fafa2abe700 1 -- 172.21.15.77:0/2249071321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf9c19e580 con 0x7faf9c106540 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.315+0000 7fafa2abe700 1 -- 172.21.15.77:0/2249071321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf9c19e9e0 con 0x7faf9c106540 2024-03-16T10:07:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.316+0000 7faf99ffb700 1 -- 172.21.15.77:0/2249071321 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7faf90022020 con 0x7faf9c106540 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.316+0000 7fafa2abe700 1 -- 172.21.15.77:0/2249071321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf9c063900 con 0x7faf9c106540 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.316+0000 7faf99ffb700 1 --2- 172.21.15.77:0/2249071321 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7faf8c038f20 0x7faf8c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.316+0000 7faf99ffb700 1 -- 172.21.15.77:0/2249071321 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7faf9004d180 con 0x7faf9c106540 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.316+0000 7faf9bfff700 1 --2- 172.21.15.77:0/2249071321 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7faf8c038f20 0x7faf8c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.317+0000 7faf9bfff700 1 --2- 172.21.15.77:0/2249071321 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7faf8c038f20 0x7faf8c03b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7faf88006fd0 tx=0x7faf88006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.321+0000 7faf99ffb700 1 -- 172.21.15.77:0/2249071321 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faf90011e20 con 0x7faf9c106540 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.526+0000 7fafa2abe700 1 -- 172.21.15.77:0/2249071321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7faf9c062ee0 con 0x7faf9c106540 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.532+0000 7faf99ffb700 1 -- 172.21.15.77:0/2249071321 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7faf9000b3e0 con 0x7faf9c106540 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.535+0000 7fafa2abe700 1 -- 172.21.15.77:0/2249071321 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7faf8c038f20 msgr2=0x7faf8c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.535+0000 7fafa2abe700 1 --2- 172.21.15.77:0/2249071321 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7faf8c038f20 0x7faf8c03b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7faf88006fd0 tx=0x7faf88006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.535+0000 7fafa2abe700 1 -- 172.21.15.77:0/2249071321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 msgr2=0x7faf9c19ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.535+0000 7fafa2abe700 1 --2- 172.21.15.77:0/2249071321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 0x7faf9c19ddb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faf900117e0 tx=0x7faf90011c10 comp rx=0 tx=0).stop 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.535+0000 7fafa2abe700 1 -- 172.21.15.77:0/2249071321 shutdown_connections 2024-03-16T10:07:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.535+0000 7fafa2abe700 1 --2- 172.21.15.77:0/2249071321 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7faf8c038f20 0x7faf8c03b3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:49.075 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.535+0000 7fafa2abe700 1 --2- 172.21.15.77:0/2249071321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c106540 0x7faf9c19ddb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:49.075 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.535+0000 7fafa2abe700 1 -- 172.21.15.77:0/2249071321 >> 172.21.15.77:0/2249071321 conn(0x7faf9c1020e0 msgr2=0x7faf9c194a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:49.075 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.535+0000 7fafa2abe700 1 -- 172.21.15.77:0/2249071321 shutdown_connections 2024-03-16T10:07:49.075 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:48.535+0000 7fafa2abe700 1 -- 172.21.15.77:0/2249071321 wait complete. 2024-03-16T10:07:49.442 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:49 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:49.442 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:49 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2249071321' entity='client.admin' 2024-03-16T10:07:50.212 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.487+0000 7f9e4de0d700 1 Processor -- start 2024-03-16T10:07:50.212 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.487+0000 7f9e4de0d700 1 -- start start 2024-03-16T10:07:50.212 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.487+0000 7f9e4de0d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 0x7f9e48106fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:50.212 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.487+0000 7f9e4de0d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9e48107590 con 0x7f9e48106ba0 2024-03-16T10:07:50.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.488+0000 7f9e477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 0x7f9e48106fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:50.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.488+0000 7f9e477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 0x7f9e48106fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34424/0 (socket says 172.21.15.77:34424) 2024-03-16T10:07:50.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.488+0000 7f9e477fe700 1 -- 172.21.15.77:0/1999787606 learned_addr learned my addr 172.21.15.77:0/1999787606 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:50.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.488+0000 7f9e477fe700 1 -- 172.21.15.77:0/1999787606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e48107da0 con 0x7f9e48106ba0 2024-03-16T10:07:50.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.488+0000 7f9e477fe700 1 --2- 172.21.15.77:0/1999787606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 0x7f9e48106fc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9e380098a0 tx=0x7f9e38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9611079717d0dc1f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:50.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.488+0000 7f9e467fc700 1 -- 172.21.15.77:0/1999787606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e38004160 con 0x7f9e48106ba0 2024-03-16T10:07:50.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.488+0000 7f9e467fc700 1 -- 172.21.15.77:0/1999787606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e38004300 con 0x7f9e48106ba0 2024-03-16T10:07:50.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.488+0000 7f9e467fc700 1 -- 172.21.15.77:0/1999787606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e38011b20 con 0x7f9e48106ba0 2024-03-16T10:07:50.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.489+0000 7f9e4de0d700 1 -- 172.21.15.77:0/1999787606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 msgr2=0x7f9e48106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:50.213 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.489+0000 7f9e4de0d700 1 --2- 172.21.15.77:0/1999787606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 0x7f9e48106fc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9e380098a0 tx=0x7f9e38009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.489+0000 7f9e4de0d700 1 -- 172.21.15.77:0/1999787606 shutdown_connections 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.489+0000 7f9e4de0d700 1 --2- 172.21.15.77:0/1999787606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 0x7f9e48106fc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.489+0000 7f9e4de0d700 1 -- 172.21.15.77:0/1999787606 >> 172.21.15.77:0/1999787606 conn(0x7f9e48102120 msgr2=0x7f9e48104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.489+0000 7f9e4de0d700 1 -- 172.21.15.77:0/1999787606 shutdown_connections 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.489+0000 7f9e4de0d700 1 -- 172.21.15.77:0/1999787606 wait complete. 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.490+0000 7f9e4de0d700 1 Processor -- start 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.490+0000 7f9e4de0d700 1 -- start start 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.490+0000 7f9e4de0d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 0x7f9e4807ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.490+0000 7f9e4de0d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4810b150 con 0x7f9e48106ba0 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.490+0000 7f9e477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 0x7f9e4807ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.491+0000 7f9e477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 0x7f9e4807ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34430/0 (socket says 172.21.15.77:34430) 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.491+0000 7f9e477fe700 1 -- 172.21.15.77:0/2188681688 learned_addr learned my addr 172.21.15.77:0/2188681688 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.491+0000 7f9e477fe700 1 -- 172.21.15.77:0/2188681688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e4819ec30 con 0x7f9e48106ba0 2024-03-16T10:07:50.214 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.491+0000 7f9e477fe700 1 --2- 172.21.15.77:0/2188681688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 0x7f9e4807ba70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9e380117e0 tx=0x7f9e38011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.491+0000 7f9e44ff9700 1 -- 172.21.15.77:0/2188681688 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e38003bf0 con 0x7f9e48106ba0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.491+0000 7f9e44ff9700 1 -- 172.21.15.77:0/2188681688 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e3801a910 con 0x7f9e48106ba0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.491+0000 7f9e4de0d700 1 -- 172.21.15.77:0/2188681688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e4819ee00 con 0x7f9e48106ba0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.492+0000 7f9e44ff9700 1 -- 172.21.15.77:0/2188681688 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e38024970 con 0x7f9e48106ba0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.492+0000 7f9e4de0d700 1 -- 172.21.15.77:0/2188681688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e4819f1a0 con 0x7f9e48106ba0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.492+0000 7f9e44ff9700 1 -- 172.21.15.77:0/2188681688 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f9e38022020 con 0x7f9e48106ba0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.492+0000 7f9e44ff9700 1 --2- 172.21.15.77:0/2188681688 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f9e30038f70 0x7f9e3003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.492+0000 7f9e44ff9700 1 -- 172.21.15.77:0/2188681688 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9e3804d460 con 0x7f9e48106ba0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.492+0000 7f9e46ffd700 1 --2- 172.21.15.77:0/2188681688 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f9e30038f70 0x7f9e3003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.493+0000 7f9e4de0d700 1 -- 172.21.15.77:0/2188681688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e48063900 con 0x7f9e48106ba0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.493+0000 7f9e46ffd700 1 --2- 172.21.15.77:0/2188681688 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f9e30038f70 0x7f9e3003b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c006fd0 tx=0x7f9e3c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.498+0000 7f9e44ff9700 1 -- 172.21.15.77:0/2188681688 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9e3801aac0 con 0x7f9e48106ba0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.705+0000 7f9e4de0d700 1 -- 172.21.15.77:0/2188681688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f9e4802db70 con 0x7f9e48106ba0 2024-03-16T10:07:50.215 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.708+0000 7f9e44ff9700 1 -- 172.21.15.77:0/2188681688 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f9e380172b0 con 0x7f9e48106ba0 2024-03-16T10:07:50.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.711+0000 7f9e4de0d700 1 -- 172.21.15.77:0/2188681688 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f9e30038f70 msgr2=0x7f9e3003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:50.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.711+0000 7f9e4de0d700 1 --2- 172.21.15.77:0/2188681688 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f9e30038f70 0x7f9e3003b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c006fd0 tx=0x7f9e3c006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:50.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.711+0000 7f9e4de0d700 1 -- 172.21.15.77:0/2188681688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 msgr2=0x7f9e4807ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:50.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.711+0000 7f9e4de0d700 1 --2- 172.21.15.77:0/2188681688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 0x7f9e4807ba70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9e380117e0 tx=0x7f9e38011ba0 comp rx=0 tx=0).stop 2024-03-16T10:07:50.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.711+0000 7f9e4de0d700 1 -- 172.21.15.77:0/2188681688 shutdown_connections 2024-03-16T10:07:50.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.711+0000 7f9e4de0d700 1 --2- 172.21.15.77:0/2188681688 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f9e30038f70 0x7f9e3003b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:50.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.711+0000 7f9e4de0d700 1 --2- 172.21.15.77:0/2188681688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e48106ba0 0x7f9e4807ba70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:50.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.711+0000 7f9e4de0d700 1 -- 172.21.15.77:0/2188681688 >> 172.21.15.77:0/2188681688 conn(0x7f9e48102120 msgr2=0x7f9e48102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:50.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.711+0000 7f9e4de0d700 1 -- 172.21.15.77:0/2188681688 shutdown_connections 2024-03-16T10:07:50.216 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:49.711+0000 7f9e4de0d700 1 -- 172.21.15.77:0/2188681688 wait complete. 2024-03-16T10:07:50.216 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling the dashboard module... 2024-03-16T10:07:50.963 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:50 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2188681688' entity='client.admin' 2024-03-16T10:07:51.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:51 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2881887563' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-16T10:07:51.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:51 smithi077 ceph-mon[32400]: from='mgr.14120 172.21.15.77:0/157813487' entity='mgr.x' 2024-03-16T10:07:51.995 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:51 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: ignoring --setuser ceph since I am not root 2024-03-16T10:07:51.995 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:51 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: ignoring --setgroup ceph since I am not root 2024-03-16T10:07:52.379 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:52 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:52.284+0000 7f64fbe23200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T10:07:52.379 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:52 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:52.378+0000 7f64fbe23200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T10:07:52.423 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.701+0000 7f2349b12700 1 Processor -- start 2024-03-16T10:07:52.423 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.701+0000 7f2349b12700 1 -- start start 2024-03-16T10:07:52.423 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.702+0000 7f2349b12700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 0x7f234410abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:52.423 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.702+0000 7f2349b12700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f234407be70 con 0x7f23441087f0 2024-03-16T10:07:52.423 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.702+0000 7f23437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 0x7f234410abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:52.423 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.702+0000 7f23437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 0x7f234410abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34434/0 (socket says 172.21.15.77:34434) 2024-03-16T10:07:52.423 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.702+0000 7f23437fe700 1 -- 172.21.15.77:0/2370610996 learned_addr learned my addr 172.21.15.77:0/2370610996 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.702+0000 7f23437fe700 1 -- 172.21.15.77:0/2370610996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234410b120 con 0x7f23441087f0 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.702+0000 7f23437fe700 1 --2- 172.21.15.77:0/2370610996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 0x7f234410abe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23380098a0 tx=0x7f2338009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a6cbaec6748832 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.702+0000 7f23427fc700 1 -- 172.21.15.77:0/2370610996 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2338004160 con 0x7f23441087f0 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.702+0000 7f23427fc700 1 -- 172.21.15.77:0/2370610996 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2338004300 con 0x7f23441087f0 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.702+0000 7f23427fc700 1 -- 172.21.15.77:0/2370610996 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2338011b20 con 0x7f23441087f0 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.703+0000 7f2349b12700 1 -- 172.21.15.77:0/2370610996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 msgr2=0x7f234410abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.703+0000 7f2349b12700 1 --2- 172.21.15.77:0/2370610996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 0x7f234410abe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23380098a0 tx=0x7f2338009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.703+0000 7f2349b12700 1 -- 172.21.15.77:0/2370610996 shutdown_connections 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.703+0000 7f2349b12700 1 --2- 172.21.15.77:0/2370610996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 0x7f234410abe0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.703+0000 7f2349b12700 1 -- 172.21.15.77:0/2370610996 >> 172.21.15.77:0/2370610996 conn(0x7f2344102220 msgr2=0x7f2344104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.703+0000 7f2349b12700 1 -- 172.21.15.77:0/2370610996 shutdown_connections 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.703+0000 7f2349b12700 1 -- 172.21.15.77:0/2370610996 wait complete. 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.704+0000 7f2349b12700 1 Processor -- start 2024-03-16T10:07:52.424 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.704+0000 7f2349b12700 1 -- start start 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.704+0000 7f2349b12700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 0x7f2344108070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.704+0000 7f2349b12700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f234410c180 con 0x7f23441087f0 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.704+0000 7f23437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 0x7f2344108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.704+0000 7f23437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 0x7f2344108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34442/0 (socket says 172.21.15.77:34442) 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.704+0000 7f23437fe700 1 -- 172.21.15.77:0/2881887563 learned_addr learned my addr 172.21.15.77:0/2881887563 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.704+0000 7f23437fe700 1 -- 172.21.15.77:0/2881887563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23441085b0 con 0x7f23441087f0 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.705+0000 7f23437fe700 1 --2- 172.21.15.77:0/2881887563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 0x7f2344108070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f23380117e0 tx=0x7f2338011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.705+0000 7f2340ff9700 1 -- 172.21.15.77:0/2881887563 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2338003bf0 con 0x7f23441087f0 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.705+0000 7f2340ff9700 1 -- 172.21.15.77:0/2881887563 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f233801a910 con 0x7f23441087f0 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.705+0000 7f2340ff9700 1 -- 172.21.15.77:0/2881887563 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2338024970 con 0x7f23441087f0 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.705+0000 7f2349b12700 1 -- 172.21.15.77:0/2881887563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23441067a0 con 0x7f23441087f0 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.705+0000 7f2349b12700 1 -- 172.21.15.77:0/2881887563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2344106c60 con 0x7f23441087f0 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.705+0000 7f2340ff9700 1 -- 172.21.15.77:0/2881887563 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f2338022020 con 0x7f23441087f0 2024-03-16T10:07:52.425 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.706+0000 7f2349b12700 1 -- 172.21.15.77:0/2881887563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2344063900 con 0x7f23441087f0 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.706+0000 7f2340ff9700 1 --2- 172.21.15.77:0/2881887563 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f232c038f20 0x7f232c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.706+0000 7f2340ff9700 1 -- 172.21.15.77:0/2881887563 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f233804d290 con 0x7f23441087f0 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.706+0000 7f2342ffd700 1 --2- 172.21.15.77:0/2881887563 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f232c038f20 0x7f232c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.706+0000 7f2342ffd700 1 --2- 172.21.15.77:0/2881887563 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f232c038f20 0x7f232c03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2334006fd0 tx=0x7f2334006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.711+0000 7f2340ff9700 1 -- 172.21.15.77:0/2881887563 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2338011e20 con 0x7f23441087f0 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:50.962+0000 7f2349b12700 1 -- 172.21.15.77:0/2881887563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f2344195600 con 0x7f23441087f0 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:51.715+0000 7f2340ff9700 1 -- 172.21.15.77:0/2881887563 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f2338004300 con 0x7f23441087f0 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:51.717+0000 7f2340ff9700 1 -- 172.21.15.77:0/2881887563 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f233801a430 con 0x7f23441087f0 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:51.719+0000 7f2349b12700 1 -- 172.21.15.77:0/2881887563 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f232c038f20 msgr2=0x7f232c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:51.719+0000 7f2349b12700 1 --2- 172.21.15.77:0/2881887563 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f232c038f20 0x7f232c03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2334006fd0 tx=0x7f2334006e40 comp rx=0 tx=0).stop 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:51.719+0000 7f2349b12700 1 -- 172.21.15.77:0/2881887563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 msgr2=0x7f2344108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:51.719+0000 7f2349b12700 1 --2- 172.21.15.77:0/2881887563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 0x7f2344108070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f23380117e0 tx=0x7f2338011c10 comp rx=0 tx=0).stop 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:51.719+0000 7f2349b12700 1 -- 172.21.15.77:0/2881887563 shutdown_connections 2024-03-16T10:07:52.426 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:51.719+0000 7f2349b12700 1 --2- 172.21.15.77:0/2881887563 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f232c038f20 0x7f232c03b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:52.427 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:51.719+0000 7f2349b12700 1 --2- 172.21.15.77:0/2881887563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23441087f0 0x7f2344108070 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:52.427 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:51.719+0000 7f2349b12700 1 -- 172.21.15.77:0/2881887563 >> 172.21.15.77:0/2881887563 conn(0x7f2344102220 msgr2=0x7f2344102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:52.427 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:51.719+0000 7f2349b12700 1 -- 172.21.15.77:0/2881887563 shutdown_connections 2024-03-16T10:07:52.427 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:51.719+0000 7f2349b12700 1 -- 172.21.15.77:0/2881887563 wait complete. 2024-03-16T10:07:52.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:52 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2881887563' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-16T10:07:52.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:52 smithi077 ceph-mon[32400]: mgrmap e9: x(active, since 18s) 2024-03-16T10:07:53.380 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:53 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:53.038+0000 7f64fbe23200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T10:07:53.380 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:53 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:53.134+0000 7f64fbe23200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T10:07:53.380 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:53 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:53.281+0000 7f64fbe23200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T10:07:53.380 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:53 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:53.379+0000 7f64fbe23200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T10:07:53.644 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:53 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:53.463+0000 7f64fbe23200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T10:07:53.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:07:53.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-16T10:07:53.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T10:07:53.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-16T10:07:53.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T10:07:53.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:07:53.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.850+0000 7f72fd4e9700 1 Processor -- start 2024-03-16T10:07:53.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.850+0000 7f72fd4e9700 1 -- start start 2024-03-16T10:07:53.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.850+0000 7f72fd4e9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 0x7f72f8106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.850+0000 7f72fd4e9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72f8107580 con 0x7f72f8106b90 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.851+0000 7f72f6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 0x7f72f8106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.851+0000 7f72f6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 0x7f72f8106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50910/0 (socket says 172.21.15.77:50910) 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.851+0000 7f72f6ffd700 1 -- 172.21.15.77:0/4145680361 learned_addr learned my addr 172.21.15.77:0/4145680361 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.851+0000 7f72f6ffd700 1 -- 172.21.15.77:0/4145680361 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f8107d90 con 0x7f72f8106b90 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.851+0000 7f72f6ffd700 1 --2- 172.21.15.77:0/4145680361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 0x7f72f8106fb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0098a0 tx=0x7f72ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd64aee8449e582b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.851+0000 7f72f5ffb700 1 -- 172.21.15.77:0/4145680361 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72ec004160 con 0x7f72f8106b90 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.851+0000 7f72f5ffb700 1 -- 172.21.15.77:0/4145680361 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72ec004300 con 0x7f72f8106b90 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.851+0000 7f72f5ffb700 1 -- 172.21.15.77:0/4145680361 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72ec011b20 con 0x7f72f8106b90 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.852+0000 7f72fd4e9700 1 -- 172.21.15.77:0/4145680361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 msgr2=0x7f72f8106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.852+0000 7f72fd4e9700 1 --2- 172.21.15.77:0/4145680361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 0x7f72f8106fb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0098a0 tx=0x7f72ec009bb0 comp rx=0 tx=0).stop 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.852+0000 7f72fd4e9700 1 -- 172.21.15.77:0/4145680361 shutdown_connections 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.852+0000 7f72fd4e9700 1 --2- 172.21.15.77:0/4145680361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 0x7f72f8106fb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.852+0000 7f72fd4e9700 1 -- 172.21.15.77:0/4145680361 >> 172.21.15.77:0/4145680361 conn(0x7f72f8102110 msgr2=0x7f72f8104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.852+0000 7f72fd4e9700 1 -- 172.21.15.77:0/4145680361 shutdown_connections 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.852+0000 7f72fd4e9700 1 -- 172.21.15.77:0/4145680361 wait complete. 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.853+0000 7f72fd4e9700 1 Processor -- start 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.853+0000 7f72fd4e9700 1 -- start start 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.853+0000 7f72fd4e9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 0x7f72f8191cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.853+0000 7f72fd4e9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72f810b140 con 0x7f72f8106b90 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.853+0000 7f72f6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 0x7f72f8191cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.853+0000 7f72f6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 0x7f72f8191cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50918/0 (socket says 172.21.15.77:50918) 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.853+0000 7f72f6ffd700 1 -- 172.21.15.77:0/2131990531 learned_addr learned my addr 172.21.15.77:0/2131990531 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.853+0000 7f72f6ffd700 1 -- 172.21.15.77:0/2131990531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f8192230 con 0x7f72f8106b90 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.853+0000 7f72f6ffd700 1 --2- 172.21.15.77:0/2131990531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 0x7f72f8191cf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0117e0 tx=0x7f72ec011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.854+0000 7f72dffff700 1 -- 172.21.15.77:0/2131990531 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72ec003bf0 con 0x7f72f8106b90 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.854+0000 7f72dffff700 1 -- 172.21.15.77:0/2131990531 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72ec01a910 con 0x7f72f8106b90 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.854+0000 7f72fd4e9700 1 -- 172.21.15.77:0/2131990531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72f8190470 con 0x7f72f8106b90 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.854+0000 7f72dffff700 1 -- 172.21.15.77:0/2131990531 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72ec0249a0 con 0x7f72f8106b90 2024-03-16T10:07:53.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.854+0000 7f72fd4e9700 1 -- 172.21.15.77:0/2131990531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72f81908d0 con 0x7f72f8106b90 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.855+0000 7f72dffff700 1 -- 172.21.15.77:0/2131990531 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f72ec022020 con 0x7f72f8106b90 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.855+0000 7f72fd4e9700 1 -- 172.21.15.77:0/2131990531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72f8063900 con 0x7f72f8106b90 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.855+0000 7f72dffff700 1 --2- 172.21.15.77:0/2131990531 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f72e0038f20 0x7f72e003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.855+0000 7f72dffff700 1 -- 172.21.15.77:0/2131990531 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f72ec04d400 con 0x7f72f8106b90 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.855+0000 7f72f67fc700 1 -- 172.21.15.77:0/2131990531 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f72e0038f20 msgr2=0x7f72e003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1353424963 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.855+0000 7f72f67fc700 1 --2- 172.21.15.77:0/2131990531 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f72e0038f20 0x7f72e003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:52.860+0000 7f72dffff700 1 -- 172.21.15.77:0/2131990531 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f72ec011e20 con 0x7f72f8106b90 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.056+0000 7f72f67fc700 1 -- 172.21.15.77:0/2131990531 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f72e0038f20 msgr2=0x7f72e003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1353424963 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.056+0000 7f72f67fc700 1 --2- 172.21.15.77:0/2131990531 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f72e0038f20 0x7f72e003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.122+0000 7f72fd4e9700 1 -- 172.21.15.77:0/2131990531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f72f8195680 con 0x7f72f8106b90 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.122+0000 7f72dffff700 1 -- 172.21.15.77:0/2131990531 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f72ec00b9d0 con 0x7f72f8106b90 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.124+0000 7f72fd4e9700 1 -- 172.21.15.77:0/2131990531 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f72e0038f20 msgr2=0x7f72e003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.124+0000 7f72fd4e9700 1 --2- 172.21.15.77:0/2131990531 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f72e0038f20 0x7f72e003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.124+0000 7f72fd4e9700 1 -- 172.21.15.77:0/2131990531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 msgr2=0x7f72f8191cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:07:53.655 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.124+0000 7f72fd4e9700 1 --2- 172.21.15.77:0/2131990531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 0x7f72f8191cf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0117e0 tx=0x7f72ec011c10 comp rx=0 tx=0).stop 2024-03-16T10:07:53.656 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.124+0000 7f72fd4e9700 1 -- 172.21.15.77:0/2131990531 shutdown_connections 2024-03-16T10:07:53.656 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.124+0000 7f72fd4e9700 1 --2- 172.21.15.77:0/2131990531 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7f72e0038f20 0x7f72e003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:53.656 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.124+0000 7f72fd4e9700 1 --2- 172.21.15.77:0/2131990531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72f8106b90 0x7f72f8191cf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:07:53.656 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.124+0000 7f72fd4e9700 1 -- 172.21.15.77:0/2131990531 >> 172.21.15.77:0/2131990531 conn(0x7f72f8102110 msgr2=0x7f72f8102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:07:53.656 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.124+0000 7f72fd4e9700 1 -- 172.21.15.77:0/2131990531 shutdown_connections 2024-03-16T10:07:53.656 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:53.125+0000 7f72fd4e9700 1 -- 172.21.15.77:0/2131990531 wait complete. 2024-03-16T10:07:53.656 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for the mgr to restart... 2024-03-16T10:07:53.656 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mgr epoch 9... 2024-03-16T10:07:53.935 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:53 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2131990531' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T10:07:53.935 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:53 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:53.732+0000 7f64fbe23200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T10:07:53.935 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:53 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:53.844+0000 7f64fbe23200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T10:07:53.935 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:53 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:53.934+0000 7f64fbe23200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T10:07:54.213 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:54 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:54.032+0000 7f64fbe23200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T10:07:54.495 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:54 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:54.212+0000 7f64fbe23200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T10:07:55.480 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:55 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:55.170+0000 7f64fbe23200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T10:07:55.480 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:55 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:55.300+0000 7f64fbe23200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T10:07:55.745 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:55 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:55.636+0000 7f64fbe23200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T10:07:55.745 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:55 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:55.726+0000 7f64fbe23200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T10:07:56.245 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:55 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:55.925+0000 7f64fbe23200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:07:56.245 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:56 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:56.111+0000 7f64fbe23200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T10:07:56.653 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:56 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:56.396+0000 7f64fbe23200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:07:56.653 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:56 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:56.652+0000 7f64fbe23200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T10:07:56.995 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:56 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:56.891+0000 7f64fbe23200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T10:07:57.995 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:57 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:57.700+0000 7f64fbe23200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T10:07:58.621 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:58.299+0000 7f64fbe23200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T10:07:58.621 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:58.399+0000 7f64fbe23200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T10:07:58.622 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:07:58.620+0000 7f64fbe23200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T10:07:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-mon[32400]: Active manager daemon x restarted 2024-03-16T10:07:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-mon[32400]: Activating manager daemon x 2024-03-16T10:07:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-mon[32400]: osdmap e3: 0 total, 0 up, 0 in 2024-03-16T10:07:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-mon[32400]: mgrmap e10: x(active, starting, since 0.00315899s) 2024-03-16T10:07:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:07:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T10:07:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T10:07:58.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T10:07:58.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T10:07:58.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-mon[32400]: Manager daemon x is now available 2024-03-16T10:07:58.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:07:58.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:07:59.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T10:07:59.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T10:07:59.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:07:59 smithi077 ceph-mon[32400]: mgrmap e11: x(active, since 1.0057s) 2024-03-16T10:08:00.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-03-16T10:08:00.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-16T10:08:00.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T10:08:00.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-03-16T10:08:00.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.071+0000 7fbd0a118700 1 Processor -- start 2024-03-16T10:08:00.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.071+0000 7fbd0a118700 1 -- start start 2024-03-16T10:08:00.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.071+0000 7fbd0a118700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 0x7fbd041080e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:00.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.071+0000 7fbd0a118700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbd041086b0 con 0x7fbd04107cc0 2024-03-16T10:08:00.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.071+0000 7fbd037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 0x7fbd041080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:00.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.071+0000 7fbd037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 0x7fbd041080e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50920/0 (socket says 172.21.15.77:50920) 2024-03-16T10:08:00.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.071+0000 7fbd037fe700 1 -- 172.21.15.77:0/403485630 learned_addr learned my addr 172.21.15.77:0/403485630 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:00.447 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.072+0000 7fbd037fe700 1 -- 172.21.15.77:0/403485630 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd04108ec0 con 0x7fbd04107cc0 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.072+0000 7fbd037fe700 1 --2- 172.21.15.77:0/403485630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 0x7fbd041080e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80098a0 tx=0x7fbcf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6f7c8e1afa6eae server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.072+0000 7fbd027fc700 1 -- 172.21.15.77:0/403485630 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcf8004160 con 0x7fbd04107cc0 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.072+0000 7fbd027fc700 1 -- 172.21.15.77:0/403485630 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcf8004300 con 0x7fbd04107cc0 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.072+0000 7fbd027fc700 1 -- 172.21.15.77:0/403485630 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcf8011b20 con 0x7fbd04107cc0 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.072+0000 7fbd0a118700 1 -- 172.21.15.77:0/403485630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 msgr2=0x7fbd041080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.073+0000 7fbd0a118700 1 --2- 172.21.15.77:0/403485630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 0x7fbd041080e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80098a0 tx=0x7fbcf8009bb0 comp rx=0 tx=0).stop 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.073+0000 7fbd0a118700 1 -- 172.21.15.77:0/403485630 shutdown_connections 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.073+0000 7fbd0a118700 1 --2- 172.21.15.77:0/403485630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 0x7fbd041080e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.073+0000 7fbd0a118700 1 -- 172.21.15.77:0/403485630 >> 172.21.15.77:0/403485630 conn(0x7fbd04103240 msgr2=0x7fbd041056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.073+0000 7fbd0a118700 1 -- 172.21.15.77:0/403485630 shutdown_connections 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.073+0000 7fbd0a118700 1 -- 172.21.15.77:0/403485630 wait complete. 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.073+0000 7fbd0a118700 1 Processor -- start 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.073+0000 7fbd0a118700 1 -- start start 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.074+0000 7fbd0a118700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 0x7fbd0419bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:00.448 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.074+0000 7fbd0a118700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0410c270 con 0x7fbd04107cc0 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.074+0000 7fbd037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 0x7fbd0419bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.074+0000 7fbd037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 0x7fbd0419bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50932/0 (socket says 172.21.15.77:50932) 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.074+0000 7fbd037fe700 1 -- 172.21.15.77:0/4034566565 learned_addr learned my addr 172.21.15.77:0/4034566565 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.074+0000 7fbd037fe700 1 -- 172.21.15.77:0/4034566565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0419c4e0 con 0x7fbd04107cc0 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.074+0000 7fbd037fe700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 0x7fbd0419bfa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80116a0 tx=0x7fbcf8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.074+0000 7fbd00ff9700 1 -- 172.21.15.77:0/4034566565 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcf8003a40 con 0x7fbd04107cc0 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.074+0000 7fbd00ff9700 1 -- 172.21.15.77:0/4034566565 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcf8003be0 con 0x7fbd04107cc0 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.074+0000 7fbd00ff9700 1 -- 172.21.15.77:0/4034566565 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcf801a720 con 0x7fbd04107cc0 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.074+0000 7fbd0a118700 1 -- 172.21.15.77:0/4034566565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd0419c770 con 0x7fbd04107cc0 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.074+0000 7fbd0a118700 1 -- 172.21.15.77:0/4034566565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd0419cbd0 con 0x7fbd04107cc0 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.075+0000 7fbd00ff9700 1 -- 172.21.15.77:0/4034566565 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7fbcf8024aa0 con 0x7fbd04107cc0 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.076+0000 7fbd00ff9700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 0x7fbcec03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:00.449 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.076+0000 7fbd00ff9700 1 -- 172.21.15.77:0/4034566565 --> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbcec03bb50 con 0x7fbcec038f80 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.076+0000 7fbd00ff9700 1 -- 172.21.15.77:0/4034566565 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbcf804d3f0 con 0x7fbd04107cc0 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.076+0000 7fbd02ffd700 1 -- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 msgr2=0x7fbcec03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1353424963 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.076+0000 7fbd02ffd700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 0x7fbcec03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.277+0000 7fbd02ffd700 1 -- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 msgr2=0x7fbcec03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1353424963 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.277+0000 7fbd02ffd700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 0x7fbcec03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.678+0000 7fbd02ffd700 1 -- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 msgr2=0x7fbcec03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1353424963 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:54.678+0000 7fbd02ffd700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 0x7fbcec03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:55.479+0000 7fbd02ffd700 1 -- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 msgr2=0x7fbcec03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1353424963 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:55.479+0000 7fbd02ffd700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 0x7fbcec03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:57.080+0000 7fbd02ffd700 1 -- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 msgr2=0x7fbcec03b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1353424963 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:57.080+0000 7fbd02ffd700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 0x7fbcec03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:58.636+0000 7fbd00ff9700 1 -- 172.21.15.77:0/4034566565 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7fbcf8024430 con 0x7fbd04107cc0 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:58.636+0000 7fbd00ff9700 1 -- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 msgr2=0x7fbcec03b440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:08:00.450 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:58.636+0000 7fbd00ff9700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 0x7fbcec03b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.637+0000 7fbd00ff9700 1 -- 172.21.15.77:0/4034566565 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7fbcf8017970 con 0x7fbd04107cc0 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.637+0000 7fbd00ff9700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbcec03ca30 0x7fbcec03ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.638+0000 7fbd00ff9700 1 -- 172.21.15.77:0/4034566565 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbcec03bb50 con 0x7fbcec03ca30 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.638+0000 7fbd02ffd700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbcec03ca30 0x7fbcec03ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.638+0000 7fbd02ffd700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbcec03ca30 0x7fbcec03ee20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4003a10 tx=0x7fbcf4009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.639+0000 7fbd00ff9700 1 -- 172.21.15.77:0/4034566565 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fbcec03bb50 con 0x7fbcec03ca30 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd0a118700 1 -- 172.21.15.77:0/4034566565 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fbd04063900 con 0x7fbcec03ca30 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd00ff9700 1 -- 172.21.15.77:0/4034566565 <== mgr.14152 v2:172.21.15.77:6800/1427985582 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fbd04063900 con 0x7fbcec03ca30 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd0a118700 1 -- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbcec03ca30 msgr2=0x7fbcec03ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd0a118700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbcec03ca30 0x7fbcec03ee20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4003a10 tx=0x7fbcf4009270 comp rx=0 tx=0).stop 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd0a118700 1 -- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 msgr2=0x7fbd0419bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd0a118700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 0x7fbd0419bfa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80116a0 tx=0x7fbcf8011ba0 comp rx=0 tx=0).stop 2024-03-16T10:08:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd0a118700 1 -- 172.21.15.77:0/4034566565 shutdown_connections 2024-03-16T10:08:00.452 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd0a118700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbcec03ca30 0x7fbcec03ee20 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:00.452 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd0a118700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd04107cc0 0x7fbd0419bfa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:00.452 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd0a118700 1 --2- 172.21.15.77:0/4034566565 >> [v2:172.21.15.77:6800/1353424963,v1:172.21.15.77:6801/1353424963] conn(0x7fbcec038f80 0x7fbcec03b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:00.452 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd0a118700 1 -- 172.21.15.77:0/4034566565 >> 172.21.15.77:0/4034566565 conn(0x7fbd04103240 msgr2=0x7fbd04103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:00.452 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd0a118700 1 -- 172.21.15.77:0/4034566565 shutdown_connections 2024-03-16T10:08:00.452 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:07:59.646+0000 7fbd0a118700 1 -- 172.21.15.77:0/4034566565 wait complete. 2024-03-16T10:08:00.452 INFO:teuthology.orchestra.run.smithi077.stdout:mgr epoch 9 is available 2024-03-16T10:08:00.452 INFO:teuthology.orchestra.run.smithi077.stdout:Generating a dashboard self-signed certificate... 2024-03-16T10:08:00.939 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:00 smithi077 ceph-mon[32400]: [16/Mar/2024:10:07:59] ENGINE Bus STARTING 2024-03-16T10:08:00.939 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:00 smithi077 ceph-mon[32400]: [16/Mar/2024:10:07:59] ENGINE Serving on https://172.21.15.77:7150 2024-03-16T10:08:00.939 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:00 smithi077 ceph-mon[32400]: [16/Mar/2024:10:07:59] ENGINE Serving on http://172.21.15.77:8765 2024-03-16T10:08:00.939 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:00 smithi077 ceph-mon[32400]: [16/Mar/2024:10:07:59] ENGINE Bus STARTED 2024-03-16T10:08:00.939 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:00 smithi077 ceph-mon[32400]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T10:08:00.939 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:00 smithi077 ceph-mon[32400]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T10:08:00.939 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:01.734 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-16T10:08:01.734 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.937+0000 7f6fd844a700 1 Processor -- start 2024-03-16T10:08:01.734 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.937+0000 7f6fd844a700 1 -- start start 2024-03-16T10:08:01.734 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.938+0000 7f6fd844a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 0x7f6fd01085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.938+0000 7f6fd844a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd0108ba0 con 0x7f6fd01081b0 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.938+0000 7f6fd61e6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 0x7f6fd01085d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.938+0000 7f6fd61e6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 0x7f6fd01085d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51010/0 (socket says 172.21.15.77:51010) 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.938+0000 7f6fd61e6700 1 -- 172.21.15.77:0/187051512 learned_addr learned my addr 172.21.15.77:0/187051512 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.938+0000 7f6fd61e6700 1 -- 172.21.15.77:0/187051512 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd0105cb0 con 0x7f6fd01081b0 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.939+0000 7f6fd61e6700 1 --2- 172.21.15.77:0/187051512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 0x7f6fd01085d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0098a0 tx=0x7f6fcc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3f6984bd560bac server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.939+0000 7f6fd51e4700 1 -- 172.21.15.77:0/187051512 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6fcc004160 con 0x7f6fd01081b0 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.939+0000 7f6fd51e4700 1 -- 172.21.15.77:0/187051512 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fcc004300 con 0x7f6fd01081b0 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.939+0000 7f6fd51e4700 1 -- 172.21.15.77:0/187051512 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6fcc011b20 con 0x7f6fd01081b0 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.940+0000 7f6fd844a700 1 -- 172.21.15.77:0/187051512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 msgr2=0x7f6fd01085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.940+0000 7f6fd844a700 1 --2- 172.21.15.77:0/187051512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 0x7f6fd01085d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0098a0 tx=0x7f6fcc009bb0 comp rx=0 tx=0).stop 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.940+0000 7f6fd844a700 1 -- 172.21.15.77:0/187051512 shutdown_connections 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.940+0000 7f6fd844a700 1 --2- 172.21.15.77:0/187051512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 0x7f6fd01085d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.940+0000 7f6fd844a700 1 -- 172.21.15.77:0/187051512 >> 172.21.15.77:0/187051512 conn(0x7f6fd01018d0 msgr2=0x7f6fd0103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:01.735 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.940+0000 7f6fd844a700 1 -- 172.21.15.77:0/187051512 shutdown_connections 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.940+0000 7f6fd844a700 1 -- 172.21.15.77:0/187051512 wait complete. 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.940+0000 7f6fd844a700 1 Processor -- start 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.940+0000 7f6fd844a700 1 -- start start 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.941+0000 7f6fd844a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 0x7f6fd019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.941+0000 7f6fd844a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd0106d70 con 0x7f6fd01081b0 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.941+0000 7f6fd61e6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 0x7f6fd019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.941+0000 7f6fd61e6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 0x7f6fd019dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51014/0 (socket says 172.21.15.77:51014) 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.941+0000 7f6fd61e6700 1 -- 172.21.15.77:0/983968834 learned_addr learned my addr 172.21.15.77:0/983968834 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.941+0000 7f6fd61e6700 1 -- 172.21.15.77:0/983968834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd019e2e0 con 0x7f6fd01081b0 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.941+0000 7f6fd61e6700 1 --2- 172.21.15.77:0/983968834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 0x7f6fd019dda0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0117e0 tx=0x7f6fcc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.941+0000 7f6fc77fe700 1 -- 172.21.15.77:0/983968834 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6fcc003ba0 con 0x7f6fd01081b0 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.941+0000 7f6fc77fe700 1 -- 172.21.15.77:0/983968834 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fcc01a910 con 0x7f6fd01081b0 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.941+0000 7f6fd844a700 1 -- 172.21.15.77:0/983968834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd019e570 con 0x7f6fd01081b0 2024-03-16T10:08:01.736 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.941+0000 7f6fc77fe700 1 -- 172.21.15.77:0/983968834 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6fcc0248f0 con 0x7f6fd01081b0 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.942+0000 7f6fd844a700 1 -- 172.21.15.77:0/983968834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fd019e9d0 con 0x7f6fd01081b0 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.942+0000 7f6fc77fe700 1 -- 172.21.15.77:0/983968834 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f6fcc022020 con 0x7f6fd01081b0 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.943+0000 7f6fd844a700 1 -- 172.21.15.77:0/983968834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fd0063900 con 0x7f6fd01081b0 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.943+0000 7f6fc77fe700 1 --2- 172.21.15.77:0/983968834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6fbc038e50 0x7f6fbc03b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.943+0000 7f6fc77fe700 1 -- 172.21.15.77:0/983968834 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6fcc04d670 con 0x7f6fd01081b0 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.943+0000 7f6fd59e5700 1 --2- 172.21.15.77:0/983968834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6fbc038e50 0x7f6fbc03b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.943+0000 7f6fd59e5700 1 --2- 172.21.15.77:0/983968834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6fbc038e50 0x7f6fbc03b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0006fd0 tx=0x7f6fc0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:00.948+0000 7f6fc77fe700 1 -- 172.21.15.77:0/983968834 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6fcc00b3e0 con 0x7f6fd01081b0 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:01.161+0000 7f6fd844a700 1 -- 172.21.15.77:0/983968834 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f6fd0062710 con 0x7f6fbc038e50 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:01.252+0000 7f6fc77fe700 1 -- 172.21.15.77:0/983968834 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f6fd0062710 con 0x7f6fbc038e50 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:01.254+0000 7f6fd844a700 1 -- 172.21.15.77:0/983968834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6fbc038e50 msgr2=0x7f6fbc03b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:01.254+0000 7f6fd844a700 1 --2- 172.21.15.77:0/983968834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6fbc038e50 0x7f6fbc03b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0006fd0 tx=0x7f6fc0006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:01.254+0000 7f6fd844a700 1 -- 172.21.15.77:0/983968834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 msgr2=0x7f6fd019dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:01.254+0000 7f6fd844a700 1 --2- 172.21.15.77:0/983968834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 0x7f6fd019dda0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0117e0 tx=0x7f6fcc011ba0 comp rx=0 tx=0).stop 2024-03-16T10:08:01.737 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:01.254+0000 7f6fd844a700 1 -- 172.21.15.77:0/983968834 shutdown_connections 2024-03-16T10:08:01.738 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:01.255+0000 7f6fd844a700 1 --2- 172.21.15.77:0/983968834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6fbc038e50 0x7f6fbc03b310 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:01.738 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:01.255+0000 7f6fd844a700 1 --2- 172.21.15.77:0/983968834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd01081b0 0x7f6fd019dda0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:01.738 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:01.255+0000 7f6fd844a700 1 -- 172.21.15.77:0/983968834 >> 172.21.15.77:0/983968834 conn(0x7f6fd01018d0 msgr2=0x7f6fd0194460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:01.738 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:01.255+0000 7f6fd844a700 1 -- 172.21.15.77:0/983968834 shutdown_connections 2024-03-16T10:08:01.738 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:01.255+0000 7f6fd844a700 1 -- 172.21.15.77:0/983968834 wait complete. 2024-03-16T10:08:01.738 INFO:teuthology.orchestra.run.smithi077.stdout:Creating initial admin user... 2024-03-16T10:08:02.586 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:02 smithi077 ceph-mon[32400]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:08:02.586 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:02 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:02.586 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:02 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:02.586 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:02 smithi077 ceph-mon[32400]: mgrmap e12: x(active, since 2s) 2024-03-16T10:08:02.586 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:02 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:03.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$QyQ.y0gTGEvJUhBzQNxBRuyR/B0ldrLqwC89EIp/z9H.2uLVrPND.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710583682, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-16T10:08:03.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.121+0000 7f55f3a5a700 1 Processor -- start 2024-03-16T10:08:03.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.121+0000 7f55f3a5a700 1 -- start start 2024-03-16T10:08:03.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.121+0000 7f55f3a5a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 0x7f55ec1080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:03.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.121+0000 7f55f3a5a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec108690 con 0x7f55ec107ca0 2024-03-16T10:08:03.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.122+0000 7f55f17f6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 0x7f55ec1080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:03.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.122+0000 7f55f17f6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 0x7f55ec1080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40946/0 (socket says 172.21.15.77:40946) 2024-03-16T10:08:03.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.122+0000 7f55f17f6700 1 -- 172.21.15.77:0/688708026 learned_addr learned my addr 172.21.15.77:0/688708026 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:03.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.122+0000 7f55f17f6700 1 -- 172.21.15.77:0/688708026 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ec108ea0 con 0x7f55ec107ca0 2024-03-16T10:08:03.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.122+0000 7f55f17f6700 1 --2- 172.21.15.77:0/688708026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 0x7f55ec1080c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f55e80098a0 tx=0x7f55e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d21312167c6766 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:03.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.122+0000 7f55e3fff700 1 -- 172.21.15.77:0/688708026 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55e8004160 con 0x7f55ec107ca0 2024-03-16T10:08:03.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.122+0000 7f55e3fff700 1 -- 172.21.15.77:0/688708026 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55e8004300 con 0x7f55ec107ca0 2024-03-16T10:08:03.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.122+0000 7f55e3fff700 1 -- 172.21.15.77:0/688708026 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55e8011b20 con 0x7f55ec107ca0 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.123+0000 7f55f3a5a700 1 -- 172.21.15.77:0/688708026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 msgr2=0x7f55ec1080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.123+0000 7f55f3a5a700 1 --2- 172.21.15.77:0/688708026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 0x7f55ec1080c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f55e80098a0 tx=0x7f55e8009bb0 comp rx=0 tx=0).stop 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.123+0000 7f55f3a5a700 1 -- 172.21.15.77:0/688708026 shutdown_connections 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.123+0000 7f55f3a5a700 1 --2- 172.21.15.77:0/688708026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 0x7f55ec1080c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.123+0000 7f55f3a5a700 1 -- 172.21.15.77:0/688708026 >> 172.21.15.77:0/688708026 conn(0x7f55ec103260 msgr2=0x7f55ec105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.123+0000 7f55f3a5a700 1 -- 172.21.15.77:0/688708026 shutdown_connections 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.123+0000 7f55f3a5a700 1 -- 172.21.15.77:0/688708026 wait complete. 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55f3a5a700 1 Processor -- start 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55f3a5a700 1 -- start start 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55f3a5a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 0x7f55ec07db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55f3a5a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec10c250 con 0x7f55ec107ca0 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55f17f6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 0x7f55ec07db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:03.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55f17f6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 0x7f55ec07db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40958/0 (socket says 172.21.15.77:40958) 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55f17f6700 1 -- 172.21.15.77:0/704336201 learned_addr learned my addr 172.21.15.77:0/704336201 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55f17f6700 1 -- 172.21.15.77:0/704336201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ec07e060 con 0x7f55ec107ca0 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55f17f6700 1 --2- 172.21.15.77:0/704336201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 0x7f55ec07db20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f55e80117e0 tx=0x7f55e8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55e27fc700 1 -- 172.21.15.77:0/704336201 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55e8003bf0 con 0x7f55ec107ca0 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55e27fc700 1 -- 172.21.15.77:0/704336201 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55e801a910 con 0x7f55ec107ca0 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55f3a5a700 1 -- 172.21.15.77:0/704336201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55ec07c2a0 con 0x7f55ec107ca0 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55e27fc700 1 -- 172.21.15.77:0/704336201 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55e80249a0 con 0x7f55ec107ca0 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.124+0000 7f55f3a5a700 1 -- 172.21.15.77:0/704336201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55ec07c700 con 0x7f55ec107ca0 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.125+0000 7f55e27fc700 1 -- 172.21.15.77:0/704336201 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f55e8022020 con 0x7f55ec107ca0 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.125+0000 7f55e27fc700 1 --2- 172.21.15.77:0/704336201 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f55d8038f80 0x7f55d803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.125+0000 7f55e27fc700 1 -- 172.21.15.77:0/704336201 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f55e804d2c0 con 0x7f55ec107ca0 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.125+0000 7f55f3a5a700 1 -- 172.21.15.77:0/704336201 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55ec063900 con 0x7f55ec107ca0 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.126+0000 7f55f0ff5700 1 --2- 172.21.15.77:0/704336201 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f55d8038f80 0x7f55d803b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:03.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.126+0000 7f55f0ff5700 1 --2- 172.21.15.77:0/704336201 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f55d8038f80 0x7f55d803b440 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f55dc006fd0 tx=0x7f55dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.131+0000 7f55e27fc700 1 -- 172.21.15.77:0/704336201 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f55e8011e20 con 0x7f55ec107ca0 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.357+0000 7f55f3a5a700 1 -- 172.21.15.77:0/704336201 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f55ec07c980 con 0x7f55d8038f80 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.585+0000 7f55e27fc700 1 -- 172.21.15.77:0/704336201 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f55ec07c980 con 0x7f55d8038f80 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.587+0000 7f55f3a5a700 1 -- 172.21.15.77:0/704336201 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f55d8038f80 msgr2=0x7f55d803b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.587+0000 7f55f3a5a700 1 --2- 172.21.15.77:0/704336201 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f55d8038f80 0x7f55d803b440 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f55dc006fd0 tx=0x7f55dc006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.587+0000 7f55f3a5a700 1 -- 172.21.15.77:0/704336201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 msgr2=0x7f55ec07db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.587+0000 7f55f3a5a700 1 --2- 172.21.15.77:0/704336201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 0x7f55ec07db20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f55e80117e0 tx=0x7f55e8011c10 comp rx=0 tx=0).stop 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.587+0000 7f55f3a5a700 1 -- 172.21.15.77:0/704336201 shutdown_connections 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.587+0000 7f55f3a5a700 1 --2- 172.21.15.77:0/704336201 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f55d8038f80 0x7f55d803b440 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.587+0000 7f55f3a5a700 1 --2- 172.21.15.77:0/704336201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55ec107ca0 0x7f55ec07db20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.587+0000 7f55f3a5a700 1 -- 172.21.15.77:0/704336201 >> 172.21.15.77:0/704336201 conn(0x7f55ec103260 msgr2=0x7f55ec103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.587+0000 7f55f3a5a700 1 -- 172.21.15.77:0/704336201 shutdown_connections 2024-03-16T10:08:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:02.587+0000 7f55f3a5a700 1 -- 172.21.15.77:0/704336201 wait complete. 2024-03-16T10:08:03.112 INFO:teuthology.orchestra.run.smithi077.stdout:Fetching dashboard port number... 2024-03-16T10:08:03.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:03 smithi077 ceph-mon[32400]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:08:03.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:04.365 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 8443 2024-03-16T10:08:04.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.475+0000 7fcc3ac48700 1 Processor -- start 2024-03-16T10:08:04.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.475+0000 7fcc3ac48700 1 -- start start 2024-03-16T10:08:04.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.476+0000 7fcc3ac48700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 0x7fcc34108620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:04.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.476+0000 7fcc3ac48700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34108bf0 con 0x7fcc34108200 2024-03-16T10:08:04.366 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.476+0000 7fcc389e4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 0x7fcc34108620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.476+0000 7fcc389e4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 0x7fcc34108620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40970/0 (socket says 172.21.15.77:40970) 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.476+0000 7fcc389e4700 1 -- 172.21.15.77:0/279320158 learned_addr learned my addr 172.21.15.77:0/279320158 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.476+0000 7fcc389e4700 1 -- 172.21.15.77:0/279320158 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc34106eb0 con 0x7fcc34108200 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.476+0000 7fcc389e4700 1 --2- 172.21.15.77:0/279320158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 0x7fcc34108620 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcc280098a0 tx=0x7fcc28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4fd97ef5629415 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.476+0000 7fcc337fe700 1 -- 172.21.15.77:0/279320158 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc28004160 con 0x7fcc34108200 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.477+0000 7fcc337fe700 1 -- 172.21.15.77:0/279320158 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc28004300 con 0x7fcc34108200 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.477+0000 7fcc337fe700 1 -- 172.21.15.77:0/279320158 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc28011b20 con 0x7fcc34108200 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.477+0000 7fcc3ac48700 1 -- 172.21.15.77:0/279320158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 msgr2=0x7fcc34108620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.477+0000 7fcc3ac48700 1 --2- 172.21.15.77:0/279320158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 0x7fcc34108620 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcc280098a0 tx=0x7fcc28009bb0 comp rx=0 tx=0).stop 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.478+0000 7fcc3ac48700 1 -- 172.21.15.77:0/279320158 shutdown_connections 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.478+0000 7fcc3ac48700 1 --2- 172.21.15.77:0/279320158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 0x7fcc34108620 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.478+0000 7fcc3ac48700 1 -- 172.21.15.77:0/279320158 >> 172.21.15.77:0/279320158 conn(0x7fcc34101a00 msgr2=0x7fcc34103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.478+0000 7fcc3ac48700 1 -- 172.21.15.77:0/279320158 shutdown_connections 2024-03-16T10:08:04.367 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.478+0000 7fcc3ac48700 1 -- 172.21.15.77:0/279320158 wait complete. 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.478+0000 7fcc3ac48700 1 Processor -- start 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.478+0000 7fcc3ac48700 1 -- start start 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.478+0000 7fcc3ac48700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 0x7fcc3419bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.478+0000 7fcc3ac48700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34105da0 con 0x7fcc34108200 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.478+0000 7fcc389e4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 0x7fcc3419bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.479+0000 7fcc389e4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 0x7fcc3419bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40974/0 (socket says 172.21.15.77:40974) 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.479+0000 7fcc389e4700 1 -- 172.21.15.77:0/864755326 learned_addr learned my addr 172.21.15.77:0/864755326 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.479+0000 7fcc389e4700 1 -- 172.21.15.77:0/864755326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3419c4b0 con 0x7fcc34108200 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.479+0000 7fcc389e4700 1 --2- 172.21.15.77:0/864755326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 0x7fcc3419bf70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcc280117e0 tx=0x7fcc28011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.479+0000 7fcc31ffb700 1 -- 172.21.15.77:0/864755326 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc28003bf0 con 0x7fcc34108200 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.479+0000 7fcc31ffb700 1 -- 172.21.15.77:0/864755326 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc2801a910 con 0x7fcc34108200 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.479+0000 7fcc3ac48700 1 -- 172.21.15.77:0/864755326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3419c740 con 0x7fcc34108200 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.479+0000 7fcc31ffb700 1 -- 172.21.15.77:0/864755326 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc28024970 con 0x7fcc34108200 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.479+0000 7fcc3ac48700 1 -- 172.21.15.77:0/864755326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc3419cba0 con 0x7fcc34108200 2024-03-16T10:08:04.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.480+0000 7fcc31ffb700 1 -- 172.21.15.77:0/864755326 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fcc28022020 con 0x7fcc34108200 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.480+0000 7fcc31ffb700 1 --2- 172.21.15.77:0/864755326 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcc24038f70 0x7fcc2403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.480+0000 7fcc31ffb700 1 -- 172.21.15.77:0/864755326 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcc2804d8d0 con 0x7fcc34108200 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.480+0000 7fcc3ac48700 1 -- 172.21.15.77:0/864755326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc34063900 con 0x7fcc34108200 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.480+0000 7fcc33fff700 1 --2- 172.21.15.77:0/864755326 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcc24038f70 0x7fcc2403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.481+0000 7fcc33fff700 1 --2- 172.21.15.77:0/864755326 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcc24038f70 0x7fcc2403b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcc20006fd0 tx=0x7fcc20006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.485+0000 7fcc31ffb700 1 -- 172.21.15.77:0/864755326 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcc2802cdf0 con 0x7fcc34108200 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.690+0000 7fcc3ac48700 1 -- 172.21.15.77:0/864755326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fcc3402db70 con 0x7fcc34108200 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.690+0000 7fcc31ffb700 1 -- 172.21.15.77:0/864755326 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fcc28032030 con 0x7fcc34108200 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.692+0000 7fcc3ac48700 1 -- 172.21.15.77:0/864755326 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcc24038f70 msgr2=0x7fcc2403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.692+0000 7fcc3ac48700 1 --2- 172.21.15.77:0/864755326 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcc24038f70 0x7fcc2403b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcc20006fd0 tx=0x7fcc20006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.692+0000 7fcc3ac48700 1 -- 172.21.15.77:0/864755326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 msgr2=0x7fcc3419bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.692+0000 7fcc3ac48700 1 --2- 172.21.15.77:0/864755326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 0x7fcc3419bf70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcc280117e0 tx=0x7fcc28011ba0 comp rx=0 tx=0).stop 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.692+0000 7fcc3ac48700 1 -- 172.21.15.77:0/864755326 shutdown_connections 2024-03-16T10:08:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.692+0000 7fcc3ac48700 1 --2- 172.21.15.77:0/864755326 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcc24038f70 0x7fcc2403b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:04.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.692+0000 7fcc3ac48700 1 --2- 172.21.15.77:0/864755326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc34108200 0x7fcc3419bf70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:04.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.692+0000 7fcc3ac48700 1 -- 172.21.15.77:0/864755326 >> 172.21.15.77:0/864755326 conn(0x7fcc34101a00 msgr2=0x7fcc34102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:04.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.692+0000 7fcc3ac48700 1 -- 172.21.15.77:0/864755326 shutdown_connections 2024-03-16T10:08:04.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:03.693+0000 7fcc3ac48700 1 -- 172.21.15.77:0/864755326 wait complete. 2024-03-16T10:08:04.372 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T10:08:04.373 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout disabled 2024-03-16T10:08:04.379 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T10:08:04.379 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-03-16T10:08:04.379 INFO:teuthology.orchestra.run.smithi077.stdout:firewalld.service is not enabled 2024-03-16T10:08:04.379 INFO:teuthology.orchestra.run.smithi077.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-16T10:08:04.381 INFO:teuthology.orchestra.run.smithi077.stdout:Ceph Dashboard is now available at: 2024-03-16T10:08:04.381 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:08:04.381 INFO:teuthology.orchestra.run.smithi077.stdout: URL: https://smithi077.front.sepia.ceph.com:8443/ 2024-03-16T10:08:04.381 INFO:teuthology.orchestra.run.smithi077.stdout: User: admin 2024-03-16T10:08:04.381 INFO:teuthology.orchestra.run.smithi077.stdout: Password: hal2v57sbq 2024-03-16T10:08:04.381 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:08:04.382 INFO:teuthology.orchestra.run.smithi077.stdout:Saving cluster configuration to /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config directory 2024-03-16T10:08:04.382 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling autotune for osd_memory_target 2024-03-16T10:08:04.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:04 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/864755326' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-16T10:08:06.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.634+0000 7fa506c96700 1 Processor -- start 2024-03-16T10:08:06.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.634+0000 7fa506c96700 1 -- start start 2024-03-16T10:08:06.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.634+0000 7fa506c96700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 0x7fa500106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:06.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.634+0000 7fa506c96700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa50007be10 con 0x7fa500106560 2024-03-16T10:08:06.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.635+0000 7fa504a32700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 0x7fa500106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:06.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.635+0000 7fa504a32700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 0x7fa500106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40982/0 (socket says 172.21.15.77:40982) 2024-03-16T10:08:06.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.635+0000 7fa504a32700 1 -- 172.21.15.77:0/216999694 learned_addr learned my addr 172.21.15.77:0/216999694 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:06.368 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.635+0000 7fa504a32700 1 -- 172.21.15.77:0/216999694 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa500106ec0 con 0x7fa500106560 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.635+0000 7fa504a32700 1 --2- 172.21.15.77:0/216999694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 0x7fa500106980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009e80 tx=0x7fa4fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c8f0f8c1666e7e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.635+0000 7fa4f77fe700 1 -- 172.21.15.77:0/216999694 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4fc017070 con 0x7fa500106560 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.635+0000 7fa4f77fe700 1 -- 172.21.15.77:0/216999694 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4fc00b8c0 con 0x7fa500106560 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.635+0000 7fa4f77fe700 1 -- 172.21.15.77:0/216999694 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4fc0045c0 con 0x7fa500106560 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.636+0000 7fa506c96700 1 -- 172.21.15.77:0/216999694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 msgr2=0x7fa500106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.636+0000 7fa506c96700 1 --2- 172.21.15.77:0/216999694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 0x7fa500106980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009e80 tx=0x7fa4fc00b330 comp rx=0 tx=0).stop 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.636+0000 7fa506c96700 1 -- 172.21.15.77:0/216999694 shutdown_connections 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.636+0000 7fa506c96700 1 --2- 172.21.15.77:0/216999694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 0x7fa500106980 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.636+0000 7fa506c96700 1 -- 172.21.15.77:0/216999694 >> 172.21.15.77:0/216999694 conn(0x7fa5001020e0 msgr2=0x7fa500104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.636+0000 7fa506c96700 1 -- 172.21.15.77:0/216999694 shutdown_connections 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.636+0000 7fa506c96700 1 -- 172.21.15.77:0/216999694 wait complete. 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.637+0000 7fa506c96700 1 Processor -- start 2024-03-16T10:08:06.369 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.637+0000 7fa506c96700 1 -- start start 2024-03-16T10:08:06.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.637+0000 7fa506c96700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 0x7fa50019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:06.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.637+0000 7fa506c96700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa500107f90 con 0x7fa500106560 2024-03-16T10:08:06.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.637+0000 7fa504a32700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 0x7fa50019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:06.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.637+0000 7fa504a32700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 0x7fa50019de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40992/0 (socket says 172.21.15.77:40992) 2024-03-16T10:08:06.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.637+0000 7fa504a32700 1 -- 172.21.15.77:0/2507104889 learned_addr learned my addr 172.21.15.77:0/2507104889 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:06.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.637+0000 7fa504a32700 1 -- 172.21.15.77:0/2507104889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50019e340 con 0x7fa500106560 2024-03-16T10:08:06.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.637+0000 7fa504a32700 1 --2- 172.21.15.77:0/2507104889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 0x7fa50019de00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00b7b0 tx=0x7fa4fc00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:06.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.638+0000 7fa4f5ffb700 1 -- 172.21.15.77:0/2507104889 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4fc020020 con 0x7fa500106560 2024-03-16T10:08:06.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.638+0000 7fa506c96700 1 -- 172.21.15.77:0/2507104889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50019e5d0 con 0x7fa500106560 2024-03-16T10:08:06.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.638+0000 7fa4f5ffb700 1 -- 172.21.15.77:0/2507104889 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4fc003d90 con 0x7fa500106560 2024-03-16T10:08:06.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.638+0000 7fa4f5ffb700 1 -- 172.21.15.77:0/2507104889 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4fc017040 con 0x7fa500106560 2024-03-16T10:08:06.370 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.638+0000 7fa506c96700 1 -- 172.21.15.77:0/2507104889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5001a12c0 con 0x7fa500106560 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.639+0000 7fa4f5ffb700 1 -- 172.21.15.77:0/2507104889 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fa4fc023020 con 0x7fa500106560 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.639+0000 7fa4f5ffb700 1 --2- 172.21.15.77:0/2507104889 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4e8039020 0x7fa4e803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.639+0000 7fa506c96700 1 -- 172.21.15.77:0/2507104889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa500063900 con 0x7fa500106560 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.639+0000 7fa4f5ffb700 1 -- 172.21.15.77:0/2507104889 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa4fc00f050 con 0x7fa500106560 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.639+0000 7fa4f7fff700 1 --2- 172.21.15.77:0/2507104889 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4e8039020 0x7fa4e803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.640+0000 7fa4f7fff700 1 --2- 172.21.15.77:0/2507104889 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4e8039020 0x7fa4e803b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0006fd0 tx=0x7fa4f0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.644+0000 7fa4f5ffb700 1 -- 172.21.15.77:0/2507104889 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa4fc02a020 con 0x7fa500106560 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.841+0000 7fa506c96700 1 -- 172.21.15.77:0/2507104889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fa50002de30 con 0x7fa500106560 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.841+0000 7fa4f5ffb700 1 -- 172.21.15.77:0/2507104889 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fa4fc007570 con 0x7fa500106560 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.843+0000 7fa506c96700 1 -- 172.21.15.77:0/2507104889 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4e8039020 msgr2=0x7fa4e803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.843+0000 7fa506c96700 1 --2- 172.21.15.77:0/2507104889 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4e8039020 0x7fa4e803b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0006fd0 tx=0x7fa4f0006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.844+0000 7fa506c96700 1 -- 172.21.15.77:0/2507104889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 msgr2=0x7fa50019de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:06.371 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.844+0000 7fa506c96700 1 --2- 172.21.15.77:0/2507104889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 0x7fa50019de00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00b7b0 tx=0x7fa4fc00bd50 comp rx=0 tx=0).stop 2024-03-16T10:08:06.372 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.844+0000 7fa506c96700 1 -- 172.21.15.77:0/2507104889 shutdown_connections 2024-03-16T10:08:06.372 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.844+0000 7fa506c96700 1 --2- 172.21.15.77:0/2507104889 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4e8039020 0x7fa4e803b4e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:06.372 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.844+0000 7fa506c96700 1 --2- 172.21.15.77:0/2507104889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500106560 0x7fa50019de00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:06.372 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.844+0000 7fa506c96700 1 -- 172.21.15.77:0/2507104889 >> 172.21.15.77:0/2507104889 conn(0x7fa5001020e0 msgr2=0x7fa500194a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:06.372 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.844+0000 7fa506c96700 1 -- 172.21.15.77:0/2507104889 shutdown_connections 2024-03-16T10:08:06.372 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:05.844+0000 7fa506c96700 1 -- 172.21.15.77:0/2507104889 wait complete. 2024-03-16T10:08:06.687 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:06 smithi077 ceph-mon[32400]: mgrmap e13: x(active, since 6s) 2024-03-16T10:08:07.682 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.787+0000 7f1e74df6700 1 Processor -- start 2024-03-16T10:08:07.682 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.787+0000 7f1e74df6700 1 -- start start 2024-03-16T10:08:07.682 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.788+0000 7f1e74df6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 0x7f1e7010ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:07.682 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.788+0000 7f1e74df6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7007be50 con 0x7f1e70108770 2024-03-16T10:08:07.682 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.788+0000 7f1e6e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 0x7f1e7010ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:07.682 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.788+0000 7f1e6e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 0x7f1e7010ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41006/0 (socket says 172.21.15.77:41006) 2024-03-16T10:08:07.682 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.788+0000 7f1e6e59c700 1 -- 172.21.15.77:0/2156784077 learned_addr learned my addr 172.21.15.77:0/2156784077 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:07.682 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.788+0000 7f1e6e59c700 1 -- 172.21.15.77:0/2156784077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7010b0a0 con 0x7f1e70108770 2024-03-16T10:08:07.682 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.788+0000 7f1e6e59c700 1 --2- 172.21.15.77:0/2156784077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 0x7f1e7010ab60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1e640098a0 tx=0x7f1e64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba608349433f9ef server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:07.682 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.789+0000 7f1e6d59a700 1 -- 172.21.15.77:0/2156784077 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e64004160 con 0x7f1e70108770 2024-03-16T10:08:07.682 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.789+0000 7f1e6d59a700 1 -- 172.21.15.77:0/2156784077 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e64004300 con 0x7f1e70108770 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.789+0000 7f1e6d59a700 1 -- 172.21.15.77:0/2156784077 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e64011b20 con 0x7f1e70108770 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.789+0000 7f1e74df6700 1 -- 172.21.15.77:0/2156784077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 msgr2=0x7f1e7010ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.789+0000 7f1e74df6700 1 --2- 172.21.15.77:0/2156784077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 0x7f1e7010ab60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1e640098a0 tx=0x7f1e64009bb0 comp rx=0 tx=0).stop 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.789+0000 7f1e74df6700 1 -- 172.21.15.77:0/2156784077 shutdown_connections 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.789+0000 7f1e74df6700 1 --2- 172.21.15.77:0/2156784077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 0x7f1e7010ab60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.789+0000 7f1e74df6700 1 -- 172.21.15.77:0/2156784077 >> 172.21.15.77:0/2156784077 conn(0x7f1e70102120 msgr2=0x7f1e70104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.789+0000 7f1e74df6700 1 -- 172.21.15.77:0/2156784077 shutdown_connections 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.789+0000 7f1e74df6700 1 -- 172.21.15.77:0/2156784077 wait complete. 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.790+0000 7f1e74df6700 1 Processor -- start 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.790+0000 7f1e74df6700 1 -- start start 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.790+0000 7f1e74df6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 0x7f1e70107fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.790+0000 7f1e74df6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7010c0b0 con 0x7f1e70108770 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.790+0000 7f1e6e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 0x7f1e70107fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.790+0000 7f1e6e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 0x7f1e70107fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41008/0 (socket says 172.21.15.77:41008) 2024-03-16T10:08:07.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.790+0000 7f1e6e59c700 1 -- 172.21.15.77:0/155955688 learned_addr learned my addr 172.21.15.77:0/155955688 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.791+0000 7f1e6e59c700 1 -- 172.21.15.77:0/155955688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e701084e0 con 0x7f1e70108770 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.791+0000 7f1e6e59c700 1 --2- 172.21.15.77:0/155955688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 0x7f1e70107fa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1e640116a0 tx=0x7f1e64011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.791+0000 7f1e5f7fe700 1 -- 172.21.15.77:0/155955688 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e64003a40 con 0x7f1e70108770 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.791+0000 7f1e5f7fe700 1 -- 172.21.15.77:0/155955688 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e64003be0 con 0x7f1e70108770 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.791+0000 7f1e5f7fe700 1 -- 172.21.15.77:0/155955688 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e6401a750 con 0x7f1e70108770 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.791+0000 7f1e74df6700 1 -- 172.21.15.77:0/155955688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e70106720 con 0x7f1e70108770 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.791+0000 7f1e74df6700 1 -- 172.21.15.77:0/155955688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e70106b80 con 0x7f1e70108770 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.792+0000 7f1e5f7fe700 1 -- 172.21.15.77:0/155955688 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f1e6401aa00 con 0x7f1e70108770 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.792+0000 7f1e74df6700 1 -- 172.21.15.77:0/155955688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e70063900 con 0x7f1e70108770 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.792+0000 7f1e5f7fe700 1 --2- 172.21.15.77:0/155955688 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1e58039020 0x7f1e5803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.792+0000 7f1e5f7fe700 1 -- 172.21.15.77:0/155955688 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1e6404d770 con 0x7f1e70108770 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.792+0000 7f1e6dd9b700 1 --2- 172.21.15.77:0/155955688 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1e58039020 0x7f1e5803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.792+0000 7f1e6dd9b700 1 --2- 172.21.15.77:0/155955688 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1e58039020 0x7f1e5803b4e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1e60006fd0 tx=0x7f1e60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:06.797+0000 7f1e5f7fe700 1 -- 172.21.15.77:0/155955688 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1e64011e20 con 0x7f1e70108770 2024-03-16T10:08:07.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:07.108+0000 7f1e74df6700 1 -- 172.21.15.77:0/155955688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f1e70195590 con 0x7f1e70108770 2024-03-16T10:08:07.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:07.110+0000 7f1e5f7fe700 1 -- 172.21.15.77:0/155955688 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f1e64019c70 con 0x7f1e70108770 2024-03-16T10:08:07.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:07.112+0000 7f1e74df6700 1 -- 172.21.15.77:0/155955688 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1e58039020 msgr2=0x7f1e5803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:07.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:07.112+0000 7f1e74df6700 1 --2- 172.21.15.77:0/155955688 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1e58039020 0x7f1e5803b4e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1e60006fd0 tx=0x7f1e60006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:07.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:07.112+0000 7f1e74df6700 1 -- 172.21.15.77:0/155955688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 msgr2=0x7f1e70107fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:07.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:07.112+0000 7f1e74df6700 1 --2- 172.21.15.77:0/155955688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 0x7f1e70107fa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1e640116a0 tx=0x7f1e64011ba0 comp rx=0 tx=0).stop 2024-03-16T10:08:07.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:07.112+0000 7f1e74df6700 1 -- 172.21.15.77:0/155955688 shutdown_connections 2024-03-16T10:08:07.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:07.112+0000 7f1e74df6700 1 --2- 172.21.15.77:0/155955688 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1e58039020 0x7f1e5803b4e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:07.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:07.112+0000 7f1e74df6700 1 --2- 172.21.15.77:0/155955688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e70108770 0x7f1e70107fa0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:07.112+0000 7f1e74df6700 1 -- 172.21.15.77:0/155955688 >> 172.21.15.77:0/155955688 conn(0x7f1e70102120 msgr2=0x7f1e70102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:07.113+0000 7f1e74df6700 1 -- 172.21.15.77:0/155955688 shutdown_connections 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-03-16T10:08:07.113+0000 7f1e74df6700 1 -- 172.21.15.77:0/155955688 wait complete. 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout:Or, if you are only running a single cluster on this host: 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:08:07.686 INFO:teuthology.orchestra.run.smithi077.stdout: ceph telemetry on 2024-03-16T10:08:07.687 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:08:07.687 INFO:teuthology.orchestra.run.smithi077.stdout:For more information see: 2024-03-16T10:08:07.687 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:08:07.687 INFO:teuthology.orchestra.run.smithi077.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-16T10:08:07.687 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:08:07.687 INFO:teuthology.orchestra.run.smithi077.stdout:Bootstrap complete. 2024-03-16T10:08:07.720 INFO:tasks.cephadm:Fetching config... 2024-03-16T10:08:07.720 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:08:07.720 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-16T10:08:07.735 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-16T10:08:07.735 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:08:07.735 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-16T10:08:07.790 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-16T10:08:07.790 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:08:07.791 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-16T10:08:07.858 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-16T10:08:07.858 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:08:07.858 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-16T10:08:07.912 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-16T10:08:07.913 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVEc01skJFwdDf7zHfFjZBTdTewuMYhdFJMMB2nH5YqZq7xAQzSv8OaVztVaY0jGchi9iLjvB70v2zZGkn/0Fi5JitQKaz82BzhNPDX/djIW2oxNs3Ve32PxEVtsVeJPIcE7nCmjasETWtJis35v1I05e3uthV4XTmIAY64QzLDXzcVN6i+q7qPQe6i2zNZhmB875dXLu9loLTT/lBXhJSSQVWawl0dCl7LTa7Ljr+NZljG6Kujp5CqgIuTB0xIYpmxYLunEuoB6YWvESm5hiD1/Tpi0SPMNuQBhyR+r+C3F6/CrzDi9xyEqg1KcaKOq0BYEv7D1he4gB2x3eHcZIAU55O5cBUmGYHkmNwKOM2pvJfIFIRaOl8KZA1OZUUd1onmRiWxZYzOuzRnpx8Gl51xso76eJRE0E6+R6mwl/TfoUVHUp9Fe0prRobkHIsbUpXp1CfW7Jhi3xnY1PE52YslFWreKaeZpnapIIGk6DKHNz9oVFSutnZGG2PwgLW/l8= ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T10:08:07.990 INFO:teuthology.orchestra.run.smithi077.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVEc01skJFwdDf7zHfFjZBTdTewuMYhdFJMMB2nH5YqZq7xAQzSv8OaVztVaY0jGchi9iLjvB70v2zZGkn/0Fi5JitQKaz82BzhNPDX/djIW2oxNs3Ve32PxEVtsVeJPIcE7nCmjasETWtJis35v1I05e3uthV4XTmIAY64QzLDXzcVN6i+q7qPQe6i2zNZhmB875dXLu9loLTT/lBXhJSSQVWawl0dCl7LTa7Ljr+NZljG6Kujp5CqgIuTB0xIYpmxYLunEuoB6YWvESm5hiD1/Tpi0SPMNuQBhyR+r+C3F6/CrzDi9xyEqg1KcaKOq0BYEv7D1he4gB2x3eHcZIAU55O5cBUmGYHkmNwKOM2pvJfIFIRaOl8KZA1OZUUd1onmRiWxZYzOuzRnpx8Gl51xso76eJRE0E6+R6mwl/TfoUVHUp9Fe0prRobkHIsbUpXp1CfW7Jhi3xnY1PE52YslFWreKaeZpnapIIGk6DKHNz9oVFSutnZGG2PwgLW/l8= ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:08:08.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVEc01skJFwdDf7zHfFjZBTdTewuMYhdFJMMB2nH5YqZq7xAQzSv8OaVztVaY0jGchi9iLjvB70v2zZGkn/0Fi5JitQKaz82BzhNPDX/djIW2oxNs3Ve32PxEVtsVeJPIcE7nCmjasETWtJis35v1I05e3uthV4XTmIAY64QzLDXzcVN6i+q7qPQe6i2zNZhmB875dXLu9loLTT/lBXhJSSQVWawl0dCl7LTa7Ljr+NZljG6Kujp5CqgIuTB0xIYpmxYLunEuoB6YWvESm5hiD1/Tpi0SPMNuQBhyR+r+C3F6/CrzDi9xyEqg1KcaKOq0BYEv7D1he4gB2x3eHcZIAU55O5cBUmGYHkmNwKOM2pvJfIFIRaOl8KZA1OZUUd1onmRiWxZYzOuzRnpx8Gl51xso76eJRE0E6+R6mwl/TfoUVHUp9Fe0prRobkHIsbUpXp1CfW7Jhi3xnY1PE52YslFWreKaeZpnapIIGk6DKHNz9oVFSutnZGG2PwgLW/l8= ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T10:08:08.050 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVEc01skJFwdDf7zHfFjZBTdTewuMYhdFJMMB2nH5YqZq7xAQzSv8OaVztVaY0jGchi9iLjvB70v2zZGkn/0Fi5JitQKaz82BzhNPDX/djIW2oxNs3Ve32PxEVtsVeJPIcE7nCmjasETWtJis35v1I05e3uthV4XTmIAY64QzLDXzcVN6i+q7qPQe6i2zNZhmB875dXLu9loLTT/lBXhJSSQVWawl0dCl7LTa7Ljr+NZljG6Kujp5CqgIuTB0xIYpmxYLunEuoB6YWvESm5hiD1/Tpi0SPMNuQBhyR+r+C3F6/CrzDi9xyEqg1KcaKOq0BYEv7D1he4gB2x3eHcZIAU55O5cBUmGYHkmNwKOM2pvJfIFIRaOl8KZA1OZUUd1onmRiWxZYzOuzRnpx8Gl51xso76eJRE0E6+R6mwl/TfoUVHUp9Fe0prRobkHIsbUpXp1CfW7Jhi3xnY1PE52YslFWreKaeZpnapIIGk6DKHNz9oVFSutnZGG2PwgLW/l8= ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:08:08.066 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVEc01skJFwdDf7zHfFjZBTdTewuMYhdFJMMB2nH5YqZq7xAQzSv8OaVztVaY0jGchi9iLjvB70v2zZGkn/0Fi5JitQKaz82BzhNPDX/djIW2oxNs3Ve32PxEVtsVeJPIcE7nCmjasETWtJis35v1I05e3uthV4XTmIAY64QzLDXzcVN6i+q7qPQe6i2zNZhmB875dXLu9loLTT/lBXhJSSQVWawl0dCl7LTa7Ljr+NZljG6Kujp5CqgIuTB0xIYpmxYLunEuoB6YWvESm5hiD1/Tpi0SPMNuQBhyR+r+C3F6/CrzDi9xyEqg1KcaKOq0BYEv7D1he4gB2x3eHcZIAU55O5cBUmGYHkmNwKOM2pvJfIFIRaOl8KZA1OZUUd1onmRiWxZYzOuzRnpx8Gl51xso76eJRE0E6+R6mwl/TfoUVHUp9Fe0prRobkHIsbUpXp1CfW7Jhi3xnY1PE52YslFWreKaeZpnapIIGk6DKHNz9oVFSutnZGG2PwgLW/l8= ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T10:08:08.113 INFO:teuthology.orchestra.run.smithi193.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVEc01skJFwdDf7zHfFjZBTdTewuMYhdFJMMB2nH5YqZq7xAQzSv8OaVztVaY0jGchi9iLjvB70v2zZGkn/0Fi5JitQKaz82BzhNPDX/djIW2oxNs3Ve32PxEVtsVeJPIcE7nCmjasETWtJis35v1I05e3uthV4XTmIAY64QzLDXzcVN6i+q7qPQe6i2zNZhmB875dXLu9loLTT/lBXhJSSQVWawl0dCl7LTa7Ljr+NZljG6Kujp5CqgIuTB0xIYpmxYLunEuoB6YWvESm5hiD1/Tpi0SPMNuQBhyR+r+C3F6/CrzDi9xyEqg1KcaKOq0BYEv7D1he4gB2x3eHcZIAU55O5cBUmGYHkmNwKOM2pvJfIFIRaOl8KZA1OZUUd1onmRiWxZYzOuzRnpx8Gl51xso76eJRE0E6+R6mwl/TfoUVHUp9Fe0prRobkHIsbUpXp1CfW7Jhi3xnY1PE52YslFWreKaeZpnapIIGk6DKHNz9oVFSutnZGG2PwgLW/l8= ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:08:08.128 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-16T10:08:08.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:08 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/155955688' entity='client.admin' 2024-03-16T10:08:08.369 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:08:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.053+0000 7f86b368c700 1 -- 172.21.15.77:0/3522784245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac102790 msgr2=0x7f86ac104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.053+0000 7f86b368c700 1 --2- 172.21.15.77:0/3522784245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac102790 0x7f86ac104b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009910 tx=0x7f86a8009c20 comp rx=0 tx=0).stop 2024-03-16T10:08:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.053+0000 7f86b368c700 1 -- 172.21.15.77:0/3522784245 shutdown_connections 2024-03-16T10:08:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.053+0000 7f86b368c700 1 --2- 172.21.15.77:0/3522784245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac102790 0x7f86ac104b80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.053+0000 7f86b368c700 1 -- 172.21.15.77:0/3522784245 >> 172.21.15.77:0/3522784245 conn(0x7f86ac0fc1a0 msgr2=0x7f86ac0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.053+0000 7f86b368c700 1 -- 172.21.15.77:0/3522784245 shutdown_connections 2024-03-16T10:08:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.053+0000 7f86b368c700 1 -- 172.21.15.77:0/3522784245 wait complete. 2024-03-16T10:08:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.053+0000 7f86b368c700 1 Processor -- start 2024-03-16T10:08:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.054+0000 7f86b368c700 1 -- start start 2024-03-16T10:08:10.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.054+0000 7f86b368c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac102790 0x7f86ac197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.054+0000 7f86b368c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86ac107d30 con 0x7f86ac102790 2024-03-16T10:08:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.054+0000 7f86b1428700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac102790 0x7f86ac197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.054+0000 7f86b1428700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac102790 0x7f86ac197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41030/0 (socket says 172.21.15.77:41030) 2024-03-16T10:08:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.054+0000 7f86b1428700 1 -- 172.21.15.77:0/2312185675 learned_addr learned my addr 172.21.15.77:0/2312185675 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.054+0000 7f86b1428700 1 -- 172.21.15.77:0/2312185675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86ac1978b0 con 0x7f86ac102790 2024-03-16T10:08:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.054+0000 7f86b1428700 1 --2- 172.21.15.77:0/2312185675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac102790 0x7f86ac197370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f86a8004fa0 tx=0x7f86a8005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.054+0000 7f86a27fc700 1 -- 172.21.15.77:0/2312185675 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86a8027070 con 0x7f86ac102790 2024-03-16T10:08:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.054+0000 7f86a27fc700 1 -- 172.21.15.77:0/2312185675 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86a800be40 con 0x7f86ac102790 2024-03-16T10:08:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.055+0000 7f86b368c700 1 -- 172.21.15.77:0/2312185675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86ac197ba0 con 0x7f86ac102790 2024-03-16T10:08:10.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.055+0000 7f86b368c700 1 -- 172.21.15.77:0/2312185675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86ac19a830 con 0x7f86ac102790 2024-03-16T10:08:10.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.055+0000 7f86a27fc700 1 -- 172.21.15.77:0/2312185675 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86a800f460 con 0x7f86ac102790 2024-03-16T10:08:10.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.055+0000 7f86a27fc700 1 -- 172.21.15.77:0/2312185675 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f86a802a070 con 0x7f86ac102790 2024-03-16T10:08:10.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.056+0000 7f86a27fc700 1 --2- 172.21.15.77:0/2312185675 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8698039020 0x7f869803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:10.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.056+0000 7f86a27fc700 1 -- 172.21.15.77:0/2312185675 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f86a801e030 con 0x7f86ac102790 2024-03-16T10:08:10.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.056+0000 7f86b368c700 1 -- 172.21.15.77:0/2312185675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86ac1084b0 con 0x7f86ac102790 2024-03-16T10:08:10.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.056+0000 7f86b0c27700 1 --2- 172.21.15.77:0/2312185675 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8698039020 0x7f869803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:10.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.056+0000 7f86b0c27700 1 --2- 172.21.15.77:0/2312185675 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8698039020 0x7f869803b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f869c006fd0 tx=0x7f869c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:10.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.061+0000 7f86a27fc700 1 -- 172.21.15.77:0/2312185675 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f86a8013070 con 0x7f86ac102790 2024-03-16T10:08:10.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.271+0000 7f86b368c700 1 -- 172.21.15.77:0/2312185675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f86ac02dce0 con 0x7f86ac102790 2024-03-16T10:08:10.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.274+0000 7f86a27fc700 1 -- 172.21.15.77:0/2312185675 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f86a8035030 con 0x7f86ac102790 2024-03-16T10:08:10.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.276+0000 7f86b368c700 1 -- 172.21.15.77:0/2312185675 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8698039020 msgr2=0x7f869803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:10.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.276+0000 7f86b368c700 1 --2- 172.21.15.77:0/2312185675 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8698039020 0x7f869803b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f869c006fd0 tx=0x7f869c006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:10.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.276+0000 7f86b368c700 1 -- 172.21.15.77:0/2312185675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac102790 msgr2=0x7f86ac197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:10.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.276+0000 7f86b368c700 1 --2- 172.21.15.77:0/2312185675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac102790 0x7f86ac197370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f86a8004fa0 tx=0x7f86a8005080 comp rx=0 tx=0).stop 2024-03-16T10:08:10.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.276+0000 7f86b368c700 1 -- 172.21.15.77:0/2312185675 shutdown_connections 2024-03-16T10:08:10.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.276+0000 7f86b368c700 1 --2- 172.21.15.77:0/2312185675 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8698039020 0x7f869803b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:10.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.276+0000 7f86b368c700 1 --2- 172.21.15.77:0/2312185675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac102790 0x7f86ac197370 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:10.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.276+0000 7f86b368c700 1 -- 172.21.15.77:0/2312185675 >> 172.21.15.77:0/2312185675 conn(0x7f86ac0fc1a0 msgr2=0x7f86ac0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:10.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.276+0000 7f86b368c700 1 -- 172.21.15.77:0/2312185675 shutdown_connections 2024-03-16T10:08:10.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:10.276+0000 7f86b368c700 1 -- 172.21.15.77:0/2312185675 wait complete. 2024-03-16T10:08:11.154 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-16T10:08:11.154 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-16T10:08:11.388 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:08:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:11 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2312185675' entity='client.admin' 2024-03-16T10:08:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:08:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:08:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:12.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.825+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2740001884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e4102790 msgr2=0x7fd0e4104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:12.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.825+0000 7fd0e99c9700 1 --2- 172.21.15.77:0/2740001884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e4102790 0x7fd0e4104b80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009910 tx=0x7fd0d4009c20 comp rx=0 tx=0).stop 2024-03-16T10:08:12.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.825+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2740001884 shutdown_connections 2024-03-16T10:08:12.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.825+0000 7fd0e99c9700 1 --2- 172.21.15.77:0/2740001884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e4102790 0x7fd0e4104b80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.825+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2740001884 >> 172.21.15.77:0/2740001884 conn(0x7fd0e40fc1a0 msgr2=0x7fd0e40fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.825+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2740001884 shutdown_connections 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.825+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2740001884 wait complete. 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.825+0000 7fd0e99c9700 1 Processor -- start 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.826+0000 7fd0e99c9700 1 -- start start 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.826+0000 7fd0e99c9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e4102790 0x7fd0e4199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.826+0000 7fd0e99c9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e4107d30 con 0x7fd0e4102790 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.826+0000 7fd0e2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e4102790 0x7fd0e4199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.826+0000 7fd0e2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e4102790 0x7fd0e4199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41946/0 (socket says 172.21.15.77:41946) 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.826+0000 7fd0e2ffd700 1 -- 172.21.15.77:0/2283117681 learned_addr learned my addr 172.21.15.77:0/2283117681 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.826+0000 7fd0e2ffd700 1 -- 172.21.15.77:0/2283117681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e4199ad0 con 0x7fd0e4102790 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.826+0000 7fd0e2ffd700 1 --2- 172.21.15.77:0/2283117681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e4102790 0x7fd0e4199590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4004fa0 tx=0x7fd0d4005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.826+0000 7fd0e89c7700 1 -- 172.21.15.77:0/2283117681 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0d4027070 con 0x7fd0e4102790 2024-03-16T10:08:12.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.827+0000 7fd0e89c7700 1 -- 172.21.15.77:0/2283117681 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0d400be40 con 0x7fd0e4102790 2024-03-16T10:08:12.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.827+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2283117681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e4199dc0 con 0x7fd0e4102790 2024-03-16T10:08:12.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.827+0000 7fd0e89c7700 1 -- 172.21.15.77:0/2283117681 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0d400f460 con 0x7fd0e4102790 2024-03-16T10:08:12.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.827+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2283117681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e419ca50 con 0x7fd0e4102790 2024-03-16T10:08:12.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.827+0000 7fd0e89c7700 1 -- 172.21.15.77:0/2283117681 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fd0d402a070 con 0x7fd0e4102790 2024-03-16T10:08:12.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.828+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2283117681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0e41084b0 con 0x7fd0e4102790 2024-03-16T10:08:12.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.828+0000 7fd0e89c7700 1 --2- 172.21.15.77:0/2283117681 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd0d0038fc0 0x7fd0d003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:12.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.828+0000 7fd0e89c7700 1 -- 172.21.15.77:0/2283117681 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd0d401e030 con 0x7fd0e4102790 2024-03-16T10:08:12.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.828+0000 7fd0e27fc700 1 --2- 172.21.15.77:0/2283117681 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd0d0038fc0 0x7fd0d003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:12.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.828+0000 7fd0e27fc700 1 --2- 172.21.15.77:0/2283117681 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd0d0038fc0 0x7fd0d003b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc006fd0 tx=0x7fd0cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:12.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:12.833+0000 7fd0e89c7700 1 -- 172.21.15.77:0/2283117681 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd0d4013070 con 0x7fd0e4102790 2024-03-16T10:08:13.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:13.031+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2283117681 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fd0e402dcf0 con 0x7fd0d0038fc0 2024-03-16T10:08:13.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:13.035+0000 7fd0e89c7700 1 -- 172.21.15.77:0/2283117681 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd0e402dcf0 con 0x7fd0d0038fc0 2024-03-16T10:08:13.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:13.037+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2283117681 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd0d0038fc0 msgr2=0x7fd0d003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:13.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:13.037+0000 7fd0e99c9700 1 --2- 172.21.15.77:0/2283117681 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd0d0038fc0 0x7fd0d003b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc006fd0 tx=0x7fd0cc006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:13.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:13.037+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2283117681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e4102790 msgr2=0x7fd0e4199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:13.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:13.037+0000 7fd0e99c9700 1 --2- 172.21.15.77:0/2283117681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e4102790 0x7fd0e4199590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4004fa0 tx=0x7fd0d4005080 comp rx=0 tx=0).stop 2024-03-16T10:08:13.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:13.037+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2283117681 shutdown_connections 2024-03-16T10:08:13.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:13.037+0000 7fd0e99c9700 1 --2- 172.21.15.77:0/2283117681 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd0d0038fc0 0x7fd0d003b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:13.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:13.037+0000 7fd0e99c9700 1 --2- 172.21.15.77:0/2283117681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0e4102790 0x7fd0e4199590 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:13.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:13.037+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2283117681 >> 172.21.15.77:0/2283117681 conn(0x7fd0e40fc1a0 msgr2=0x7fd0e40fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:13.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:13.037+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2283117681 shutdown_connections 2024-03-16T10:08:13.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:13.037+0000 7fd0e99c9700 1 -- 172.21.15.77:0/2283117681 wait complete. 2024-03-16T10:08:13.715 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2024-03-16T10:08:13.715 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-16T10:08:13.715 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2024-03-16T10:08:13.734 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-16T10:08:13.734 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:08:13.791 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2024-03-16T10:08:13.791 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch host add smithi192 2024-03-16T10:08:14.027 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:08:14.331 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:14 smithi077 ceph-mon[32400]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:08:14.331 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:14.331 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:08:14.331 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:08:14.331 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:08:14.331 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:14 smithi077 ceph-mon[32400]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-16T10:08:14.331 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:14 smithi077 ceph-mon[32400]: Updating smithi077:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:08:14.331 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:14 smithi077 ceph-mon[32400]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:08:14.331 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:14 smithi077 ceph-mon[32400]: Updating smithi077:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T10:08:14.331 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:14.332 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:14.332 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.548+0000 7ff2a551e700 1 -- 172.21.15.77:0/3642464960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2a0100e30 msgr2=0x7ff2a0101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.548+0000 7ff2a551e700 1 --2- 172.21.15.77:0/3642464960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2a0100e30 0x7ff2a0101210 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff294009910 tx=0x7ff294009c20 comp rx=0 tx=0).stop 2024-03-16T10:08:15.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.548+0000 7ff2a551e700 1 -- 172.21.15.77:0/3642464960 shutdown_connections 2024-03-16T10:08:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.548+0000 7ff2a551e700 1 --2- 172.21.15.77:0/3642464960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2a0100e30 0x7ff2a0101210 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.548+0000 7ff2a551e700 1 -- 172.21.15.77:0/3642464960 >> 172.21.15.77:0/3642464960 conn(0x7ff2a00fc720 msgr2=0x7ff2a00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.548+0000 7ff2a551e700 1 -- 172.21.15.77:0/3642464960 shutdown_connections 2024-03-16T10:08:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.548+0000 7ff2a551e700 1 -- 172.21.15.77:0/3642464960 wait complete. 2024-03-16T10:08:15.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.549+0000 7ff2a551e700 1 Processor -- start 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.549+0000 7ff2a551e700 1 -- start start 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.549+0000 7ff2a551e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2a0100e30 0x7ff2a019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.549+0000 7ff2a551e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a010b200 con 0x7ff2a0100e30 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.549+0000 7ff29effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2a0100e30 0x7ff2a019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.549+0000 7ff29effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2a0100e30 0x7ff2a019d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41964/0 (socket says 172.21.15.77:41964) 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.549+0000 7ff29effd700 1 -- 172.21.15.77:0/2472369757 learned_addr learned my addr 172.21.15.77:0/2472369757 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.549+0000 7ff29effd700 1 -- 172.21.15.77:0/2472369757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a019d980 con 0x7ff2a0100e30 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.550+0000 7ff29effd700 1 --2- 172.21.15.77:0/2472369757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2a0100e30 0x7ff2a019d440 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff294005850 tx=0x7ff294005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.550+0000 7ff287fff700 1 -- 172.21.15.77:0/2472369757 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff294026070 con 0x7ff2a0100e30 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.550+0000 7ff287fff700 1 -- 172.21.15.77:0/2472369757 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff29400bbd0 con 0x7ff2a0100e30 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.550+0000 7ff2a551e700 1 -- 172.21.15.77:0/2472369757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2a019dc70 con 0x7ff2a0100e30 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.550+0000 7ff287fff700 1 -- 172.21.15.77:0/2472369757 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff294003970 con 0x7ff2a0100e30 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.550+0000 7ff2a551e700 1 -- 172.21.15.77:0/2472369757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2a0197600 con 0x7ff2a0100e30 2024-03-16T10:08:15.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.551+0000 7ff287fff700 1 -- 172.21.15.77:0/2472369757 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ff294029070 con 0x7ff2a0100e30 2024-03-16T10:08:15.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.551+0000 7ff287fff700 1 --2- 172.21.15.77:0/2472369757 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff288038fc0 0x7ff28803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:15.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.551+0000 7ff2a551e700 1 -- 172.21.15.77:0/2472369757 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2a0102310 con 0x7ff2a0100e30 2024-03-16T10:08:15.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.551+0000 7ff287fff700 1 -- 172.21.15.77:0/2472369757 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff29401e070 con 0x7ff2a0100e30 2024-03-16T10:08:15.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.551+0000 7ff29e7fc700 1 --2- 172.21.15.77:0/2472369757 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff288038fc0 0x7ff28803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:15.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.551+0000 7ff29e7fc700 1 --2- 172.21.15.77:0/2472369757 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff288038fc0 0x7ff28803b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff290006fd0 tx=0x7ff290006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:15.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.556+0000 7ff287fff700 1 -- 172.21.15.77:0/2472369757 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff294013070 con 0x7ff2a0100e30 2024-03-16T10:08:15.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:15.753+0000 7ff2a551e700 1 -- 172.21.15.77:0/2472369757 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}) -- 0x7ff2a0198b30 con 0x7ff288038fc0 2024-03-16T10:08:16.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:15 smithi077 ceph-mon[32400]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:08:17.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:16 smithi077 ceph-mon[32400]: Deploying cephadm binary to smithi192 2024-03-16T10:08:19.948 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:19 smithi077 ceph-mon[32400]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:19.980 INFO:teuthology.orchestra.run.smithi077.stdout:Added host 'smithi192' with addr '172.21.15.192' 2024-03-16T10:08:19.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:19.979+0000 7ff287fff700 1 -- 172.21.15.77:0/2472369757 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7ff2a0198b30 con 0x7ff288038fc0 2024-03-16T10:08:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:19.981+0000 7ff2a551e700 1 -- 172.21.15.77:0/2472369757 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff288038fc0 msgr2=0x7ff28803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:19.981+0000 7ff2a551e700 1 --2- 172.21.15.77:0/2472369757 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff288038fc0 0x7ff28803b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff290006fd0 tx=0x7ff290006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:19.981+0000 7ff2a551e700 1 -- 172.21.15.77:0/2472369757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2a0100e30 msgr2=0x7ff2a019d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:19.981+0000 7ff2a551e700 1 --2- 172.21.15.77:0/2472369757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2a0100e30 0x7ff2a019d440 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff294005850 tx=0x7ff294005030 comp rx=0 tx=0).stop 2024-03-16T10:08:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:19.981+0000 7ff2a551e700 1 -- 172.21.15.77:0/2472369757 shutdown_connections 2024-03-16T10:08:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:19.981+0000 7ff2a551e700 1 --2- 172.21.15.77:0/2472369757 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff288038fc0 0x7ff28803b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:19.981+0000 7ff2a551e700 1 --2- 172.21.15.77:0/2472369757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2a0100e30 0x7ff2a019d440 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:19.981+0000 7ff2a551e700 1 -- 172.21.15.77:0/2472369757 >> 172.21.15.77:0/2472369757 conn(0x7ff2a00fc720 msgr2=0x7ff2a006a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:19.982+0000 7ff2a551e700 1 -- 172.21.15.77:0/2472369757 shutdown_connections 2024-03-16T10:08:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:19.982+0000 7ff2a551e700 1 -- 172.21.15.77:0/2472369757 wait complete. 2024-03-16T10:08:20.569 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T10:08:20.803 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:08:21.203 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:20 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:21.203 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:20 smithi077 ceph-mon[32400]: Added host smithi192 2024-03-16T10:08:21.203 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:20 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:08:21.203 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:20 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:21.203 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:20 smithi077 ceph-mon[32400]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:21 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:22.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.292+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/4285999241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e04101f10 msgr2=0x7f8e041022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:22.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.292+0000 7f8e0bf0c700 1 --2- 172.21.15.77:0/4285999241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e04101f10 0x7f8e041022f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009910 tx=0x7f8df4009c20 comp rx=0 tx=0).stop 2024-03-16T10:08:22.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.292+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/4285999241 shutdown_connections 2024-03-16T10:08:22.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.292+0000 7f8e0bf0c700 1 --2- 172.21.15.77:0/4285999241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e04101f10 0x7f8e041022f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:22.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.292+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/4285999241 >> 172.21.15.77:0/4285999241 conn(0x7f8e040fd7e0 msgr2=0x7f8e040ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:22.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.292+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/4285999241 shutdown_connections 2024-03-16T10:08:22.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.292+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/4285999241 wait complete. 2024-03-16T10:08:22.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.293+0000 7f8e0bf0c700 1 Processor -- start 2024-03-16T10:08:22.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.293+0000 7f8e0bf0c700 1 -- start start 2024-03-16T10:08:22.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.293+0000 7f8e0bf0c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e04101f10 0x7f8e0419d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:22.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.293+0000 7f8e0bf0c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0410c2e0 con 0x7f8e04101f10 2024-03-16T10:08:22.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.293+0000 7f8e09ca8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e04101f10 0x7f8e0419d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:22.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.293+0000 7f8e09ca8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e04101f10 0x7f8e0419d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55572/0 (socket says 172.21.15.77:55572) 2024-03-16T10:08:22.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.293+0000 7f8e09ca8700 1 -- 172.21.15.77:0/288268792 learned_addr learned my addr 172.21.15.77:0/288268792 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:22.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.293+0000 7f8e09ca8700 1 -- 172.21.15.77:0/288268792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e0419db20 con 0x7f8e04101f10 2024-03-16T10:08:22.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.294+0000 7f8e09ca8700 1 --2- 172.21.15.77:0/288268792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e04101f10 0x7f8e0419d5e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8df4006010 tx=0x7f8df4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:22.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.294+0000 7f8dfaffd700 1 -- 172.21.15.77:0/288268792 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8df4026070 con 0x7f8e04101f10 2024-03-16T10:08:22.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.294+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/288268792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e0419de10 con 0x7f8e04101f10 2024-03-16T10:08:22.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.294+0000 7f8dfaffd700 1 -- 172.21.15.77:0/288268792 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8df400bbd0 con 0x7f8e04101f10 2024-03-16T10:08:22.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.294+0000 7f8dfaffd700 1 -- 172.21.15.77:0/288268792 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8df4003910 con 0x7f8e04101f10 2024-03-16T10:08:22.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.294+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/288268792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e04197880 con 0x7f8e04101f10 2024-03-16T10:08:22.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.295+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/288268792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e041033f0 con 0x7f8e04101f10 2024-03-16T10:08:22.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.295+0000 7f8dfaffd700 1 -- 172.21.15.77:0/288268792 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f8df4029070 con 0x7f8e04101f10 2024-03-16T10:08:22.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.295+0000 7f8dfaffd700 1 --2- 172.21.15.77:0/288268792 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8df0038fc0 0x7f8df003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:22.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.295+0000 7f8dfaffd700 1 -- 172.21.15.77:0/288268792 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8df401e070 con 0x7f8e04101f10 2024-03-16T10:08:22.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.295+0000 7f8e094a7700 1 --2- 172.21.15.77:0/288268792 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8df0038fc0 0x7f8df003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:22.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.296+0000 7f8e094a7700 1 --2- 172.21.15.77:0/288268792 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8df0038fc0 0x7f8df003b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8e00006fd0 tx=0x7f8e00006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:22.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.300+0000 7f8dfaffd700 1 -- 172.21.15.77:0/288268792 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8df4013070 con 0x7f8e04101f10 2024-03-16T10:08:22.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.503+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/288268792 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8e04198be0 con 0x7f8df0038fc0 2024-03-16T10:08:22.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.504+0000 7f8dfaffd700 1 -- 172.21.15.77:0/288268792 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f8e04198be0 con 0x7f8df0038fc0 2024-03-16T10:08:22.505 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:08:22.505 INFO:teuthology.orchestra.run.smithi077.stdout:[{"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2024-03-16T10:08:22.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.506+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/288268792 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8df0038fc0 msgr2=0x7f8df003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:22.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.506+0000 7f8e0bf0c700 1 --2- 172.21.15.77:0/288268792 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8df0038fc0 0x7f8df003b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8e00006fd0 tx=0x7f8e00006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:22.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.506+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/288268792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e04101f10 msgr2=0x7f8e0419d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:22.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.506+0000 7f8e0bf0c700 1 --2- 172.21.15.77:0/288268792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e04101f10 0x7f8e0419d5e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8df4006010 tx=0x7f8df4005030 comp rx=0 tx=0).stop 2024-03-16T10:08:22.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.507+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/288268792 shutdown_connections 2024-03-16T10:08:22.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.507+0000 7f8e0bf0c700 1 --2- 172.21.15.77:0/288268792 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8df0038fc0 0x7f8df003b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:22.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.507+0000 7f8e0bf0c700 1 --2- 172.21.15.77:0/288268792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e04101f10 0x7f8e0419d5e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:22.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.507+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/288268792 >> 172.21.15.77:0/288268792 conn(0x7f8e040fd7e0 msgr2=0x7f8e04104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:22.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.507+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/288268792 shutdown_connections 2024-03-16T10:08:22.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:22.507+0000 7f8e0bf0c700 1 -- 172.21.15.77:0/288268792 wait complete. 2024-03-16T10:08:23.138 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:23 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:23.139 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:23 smithi077 ceph-mon[32400]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:08:23.139 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:23 smithi077 ceph-mon[32400]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:23.168 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi193 2024-03-16T10:08:23.168 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-16T10:08:23.168 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.conf 2024-03-16T10:08:23.186 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-16T10:08:23.186 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:08:23.244 INFO:tasks.cephadm:Adding host smithi193 to orchestrator... 2024-03-16T10:08:23.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch host add smithi193 2024-03-16T10:08:23.489 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:08:24.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.865+0000 7fcfab786700 1 -- 172.21.15.77:0/4263719491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa4101f10 msgr2=0x7fcfa41022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:24.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.865+0000 7fcfab786700 1 --2- 172.21.15.77:0/4263719491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa4101f10 0x7fcfa41022f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009910 tx=0x7fcf94009c20 comp rx=0 tx=0).stop 2024-03-16T10:08:24.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.865+0000 7fcfab786700 1 -- 172.21.15.77:0/4263719491 shutdown_connections 2024-03-16T10:08:24.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.865+0000 7fcfab786700 1 --2- 172.21.15.77:0/4263719491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa4101f10 0x7fcfa41022f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:24.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.865+0000 7fcfab786700 1 -- 172.21.15.77:0/4263719491 >> 172.21.15.77:0/4263719491 conn(0x7fcfa40fd7e0 msgr2=0x7fcfa40ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:24.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.865+0000 7fcfab786700 1 -- 172.21.15.77:0/4263719491 shutdown_connections 2024-03-16T10:08:24.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.865+0000 7fcfab786700 1 -- 172.21.15.77:0/4263719491 wait complete. 2024-03-16T10:08:24.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.866+0000 7fcfab786700 1 Processor -- start 2024-03-16T10:08:24.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.866+0000 7fcfab786700 1 -- start start 2024-03-16T10:08:24.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.866+0000 7fcfab786700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa4101f10 0x7fcfa419d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:24.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.866+0000 7fcfab786700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa410c2e0 con 0x7fcfa4101f10 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.866+0000 7fcfa9522700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa4101f10 0x7fcfa419d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.866+0000 7fcfa9522700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa4101f10 0x7fcfa419d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55602/0 (socket says 172.21.15.77:55602) 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.866+0000 7fcfa9522700 1 -- 172.21.15.77:0/1506364215 learned_addr learned my addr 172.21.15.77:0/1506364215 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.866+0000 7fcfa9522700 1 -- 172.21.15.77:0/1506364215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa419db20 con 0x7fcfa4101f10 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.867+0000 7fcfa9522700 1 --2- 172.21.15.77:0/1506364215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa4101f10 0x7fcfa419d5e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcf94005850 tx=0x7fcf94005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.867+0000 7fcf9a7fc700 1 -- 172.21.15.77:0/1506364215 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf94026070 con 0x7fcfa4101f10 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.867+0000 7fcf9a7fc700 1 -- 172.21.15.77:0/1506364215 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf9400bbd0 con 0x7fcfa4101f10 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.867+0000 7fcfab786700 1 -- 172.21.15.77:0/1506364215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa419de10 con 0x7fcfa4101f10 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.867+0000 7fcf9a7fc700 1 -- 172.21.15.77:0/1506364215 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf94003970 con 0x7fcfa4101f10 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.867+0000 7fcfab786700 1 -- 172.21.15.77:0/1506364215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa4197880 con 0x7fcfa4101f10 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.868+0000 7fcf9a7fc700 1 -- 172.21.15.77:0/1506364215 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fcf94029070 con 0x7fcfa4101f10 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.868+0000 7fcfab786700 1 -- 172.21.15.77:0/1506364215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa41033f0 con 0x7fcfa4101f10 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.868+0000 7fcf9a7fc700 1 --2- 172.21.15.77:0/1506364215 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf90038fc0 0x7fcf9003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.868+0000 7fcf9a7fc700 1 -- 172.21.15.77:0/1506364215 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcf9401e070 con 0x7fcfa4101f10 2024-03-16T10:08:24.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.868+0000 7fcfa8d21700 1 --2- 172.21.15.77:0/1506364215 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf90038fc0 0x7fcf9003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:24.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.868+0000 7fcfa8d21700 1 --2- 172.21.15.77:0/1506364215 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf90038fc0 0x7fcf9003b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0006fd0 tx=0x7fcfa0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:24.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:24.873+0000 7fcf9a7fc700 1 -- 172.21.15.77:0/1506364215 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcf94013070 con 0x7fcfa4101f10 2024-03-16T10:08:25.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:25.071+0000 7fcfab786700 1 -- 172.21.15.77:0/1506364215 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}) -- 0x7fcfa4198be0 con 0x7fcf90038fc0 2024-03-16T10:08:25.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:25 smithi077 ceph-mon[32400]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:25.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:25.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:25.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:25.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:25.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:08:25.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:08:25.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:08:26.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:26 smithi077 ceph-mon[32400]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:08:26.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:26 smithi077 ceph-mon[32400]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-16T10:08:26.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:26 smithi077 ceph-mon[32400]: Updating smithi192:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:08:26.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:26 smithi077 ceph-mon[32400]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:08:26.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:26 smithi077 ceph-mon[32400]: Deploying cephadm binary to smithi193 2024-03-16T10:08:26.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:26 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:26.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:26 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:26.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:26 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:27.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:27 smithi077 ceph-mon[32400]: Updating smithi192:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T10:08:27.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:27 smithi077 ceph-mon[32400]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:29.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:29.265+0000 7fcf9a7fc700 1 -- 172.21.15.77:0/1506364215 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fcfa4198be0 con 0x7fcf90038fc0 2024-03-16T10:08:29.267 INFO:teuthology.orchestra.run.smithi077.stdout:Added host 'smithi193' with addr '172.21.15.193' 2024-03-16T10:08:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:29.268+0000 7fcfab786700 1 -- 172.21.15.77:0/1506364215 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf90038fc0 msgr2=0x7fcf9003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:29.268+0000 7fcfab786700 1 --2- 172.21.15.77:0/1506364215 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf90038fc0 0x7fcf9003b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0006fd0 tx=0x7fcfa0006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:29.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:29.268+0000 7fcfab786700 1 -- 172.21.15.77:0/1506364215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa4101f10 msgr2=0x7fcfa419d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:29.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:29.268+0000 7fcfab786700 1 --2- 172.21.15.77:0/1506364215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa4101f10 0x7fcfa419d5e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcf94005850 tx=0x7fcf94005030 comp rx=0 tx=0).stop 2024-03-16T10:08:29.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:29.268+0000 7fcfab786700 1 -- 172.21.15.77:0/1506364215 shutdown_connections 2024-03-16T10:08:29.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:29.268+0000 7fcfab786700 1 --2- 172.21.15.77:0/1506364215 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf90038fc0 0x7fcf9003b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:29.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:29.268+0000 7fcfab786700 1 --2- 172.21.15.77:0/1506364215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa4101f10 0x7fcfa419d5e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:29.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:29.268+0000 7fcfab786700 1 -- 172.21.15.77:0/1506364215 >> 172.21.15.77:0/1506364215 conn(0x7fcfa40fd7e0 msgr2=0x7fcfa4104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:29.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:29.268+0000 7fcfab786700 1 -- 172.21.15.77:0/1506364215 shutdown_connections 2024-03-16T10:08:29.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:29.268+0000 7fcfab786700 1 -- 172.21.15.77:0/1506364215 wait complete. 2024-03-16T10:08:29.852 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:29 smithi077 ceph-mon[32400]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:29.852 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:29 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:29.852 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:29 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:08:29.883 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T10:08:30.127 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:08:30.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:30 smithi077 ceph-mon[32400]: Added host smithi193 2024-03-16T10:08:30.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:30 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:30.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:30 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.437+0000 7f22a2a23700 1 -- 172.21.15.77:0/4018579036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f229c101f00 msgr2=0x7f229c1022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.437+0000 7f22a2a23700 1 --2- 172.21.15.77:0/4018579036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f229c101f00 0x7f229c1022e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f228c009910 tx=0x7f228c009c20 comp rx=0 tx=0).stop 2024-03-16T10:08:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.437+0000 7f22a2a23700 1 -- 172.21.15.77:0/4018579036 shutdown_connections 2024-03-16T10:08:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.437+0000 7f22a2a23700 1 --2- 172.21.15.77:0/4018579036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f229c101f00 0x7f229c1022e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.437+0000 7f22a2a23700 1 -- 172.21.15.77:0/4018579036 >> 172.21.15.77:0/4018579036 conn(0x7f229c0fd810 msgr2=0x7f229c0ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.437+0000 7f22a2a23700 1 -- 172.21.15.77:0/4018579036 shutdown_connections 2024-03-16T10:08:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.437+0000 7f22a2a23700 1 -- 172.21.15.77:0/4018579036 wait complete. 2024-03-16T10:08:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.438+0000 7f22a2a23700 1 Processor -- start 2024-03-16T10:08:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.438+0000 7f22a2a23700 1 -- start start 2024-03-16T10:08:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.438+0000 7f22a2a23700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f229c101f00 0x7f229c19d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.438+0000 7f22a2a23700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f229c10c2d0 con 0x7f229c101f00 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.439+0000 7f229bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f229c101f00 0x7f229c19d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.439+0000 7f229bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f229c101f00 0x7f229c19d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55622/0 (socket says 172.21.15.77:55622) 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.439+0000 7f229bfff700 1 -- 172.21.15.77:0/1734957039 learned_addr learned my addr 172.21.15.77:0/1734957039 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.439+0000 7f229bfff700 1 -- 172.21.15.77:0/1734957039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229c19db10 con 0x7f229c101f00 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.439+0000 7f229bfff700 1 --2- 172.21.15.77:0/1734957039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f229c101f00 0x7f229c19d5d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f228c005850 tx=0x7f228c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.439+0000 7f22997fa700 1 -- 172.21.15.77:0/1734957039 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f228c026070 con 0x7f229c101f00 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.439+0000 7f22a2a23700 1 -- 172.21.15.77:0/1734957039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f229c19de00 con 0x7f229c101f00 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.439+0000 7f22997fa700 1 -- 172.21.15.77:0/1734957039 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f228c00bbd0 con 0x7f229c101f00 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.439+0000 7f22997fa700 1 -- 172.21.15.77:0/1734957039 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f228c003970 con 0x7f229c101f00 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.439+0000 7f22a2a23700 1 -- 172.21.15.77:0/1734957039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f229c197870 con 0x7f229c101f00 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.440+0000 7f22997fa700 1 -- 172.21.15.77:0/1734957039 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f228c029070 con 0x7f229c101f00 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.440+0000 7f22a2a23700 1 -- 172.21.15.77:0/1734957039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f229c1033e0 con 0x7f229c101f00 2024-03-16T10:08:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.440+0000 7f22997fa700 1 --2- 172.21.15.77:0/1734957039 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2284038fc0 0x7f228403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:31.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.440+0000 7f22997fa700 1 -- 172.21.15.77:0/1734957039 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f228c01e070 con 0x7f229c101f00 2024-03-16T10:08:31.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.440+0000 7f229b7fe700 1 --2- 172.21.15.77:0/1734957039 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2284038fc0 0x7f228403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:31.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.441+0000 7f229b7fe700 1 --2- 172.21.15.77:0/1734957039 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2284038fc0 0x7f228403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2290006fd0 tx=0x7f2290006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:31.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.445+0000 7f22997fa700 1 -- 172.21.15.77:0/1734957039 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f228c013070 con 0x7f229c101f00 2024-03-16T10:08:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.646+0000 7f22a2a23700 1 -- 172.21.15.77:0/1734957039 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f229c198bd0 con 0x7f2284038fc0 2024-03-16T10:08:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.648+0000 7f22997fa700 1 -- 172.21.15.77:0/1734957039 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f229c198bd0 con 0x7f2284038fc0 2024-03-16T10:08:31.650 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:08:31.651 INFO:teuthology.orchestra.run.smithi077.stdout:[{"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}, {"addr": "172.21.15.193", "hostname": "smithi193", "labels": [], "status": ""}] 2024-03-16T10:08:31.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.650+0000 7f22a2a23700 1 -- 172.21.15.77:0/1734957039 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2284038fc0 msgr2=0x7f228403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:31.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.650+0000 7f22a2a23700 1 --2- 172.21.15.77:0/1734957039 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2284038fc0 0x7f228403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2290006fd0 tx=0x7f2290006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:31.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.650+0000 7f22a2a23700 1 -- 172.21.15.77:0/1734957039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f229c101f00 msgr2=0x7f229c19d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:31.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.650+0000 7f22a2a23700 1 --2- 172.21.15.77:0/1734957039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f229c101f00 0x7f229c19d5d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f228c005850 tx=0x7f228c005030 comp rx=0 tx=0).stop 2024-03-16T10:08:31.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.650+0000 7f22a2a23700 1 -- 172.21.15.77:0/1734957039 shutdown_connections 2024-03-16T10:08:31.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.650+0000 7f22a2a23700 1 --2- 172.21.15.77:0/1734957039 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2284038fc0 0x7f228403b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:31.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.650+0000 7f22a2a23700 1 --2- 172.21.15.77:0/1734957039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f229c101f00 0x7f229c19d5d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:31.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.650+0000 7f22a2a23700 1 -- 172.21.15.77:0/1734957039 >> 172.21.15.77:0/1734957039 conn(0x7f229c0fd810 msgr2=0x7f229c104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:31.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.650+0000 7f22a2a23700 1 -- 172.21.15.77:0/1734957039 shutdown_connections 2024-03-16T10:08:31.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:31.650+0000 7f22a2a23700 1 -- 172.21.15.77:0/1734957039 wait complete. 2024-03-16T10:08:31.981 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:31 smithi077 ceph-mon[32400]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:32.259 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-16T10:08:32.259 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-16T10:08:32.493 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:08:32.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:32 smithi077 ceph-mon[32400]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:08:32.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:32 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:33.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.841+0000 7f389e899700 1 -- 172.21.15.77:0/585456194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898104130 msgr2=0x7f3898104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:33.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.841+0000 7f389e899700 1 --2- 172.21.15.77:0/585456194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898104130 0x7f3898104510 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3888009910 tx=0x7f3888009c20 comp rx=0 tx=0).stop 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.841+0000 7f389e899700 1 -- 172.21.15.77:0/585456194 shutdown_connections 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.841+0000 7f389e899700 1 --2- 172.21.15.77:0/585456194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898104130 0x7f3898104510 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.841+0000 7f389e899700 1 -- 172.21.15.77:0/585456194 >> 172.21.15.77:0/585456194 conn(0x7f38980ff9e0 msgr2=0x7f3898101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.841+0000 7f389e899700 1 -- 172.21.15.77:0/585456194 shutdown_connections 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.841+0000 7f389e899700 1 -- 172.21.15.77:0/585456194 wait complete. 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.842+0000 7f389e899700 1 Processor -- start 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.842+0000 7f389e899700 1 -- start start 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.842+0000 7f389e899700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898104130 0x7f3898199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.842+0000 7f389e899700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f389810e500 con 0x7f3898104130 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.843+0000 7f3897fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898104130 0x7f3898199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.843+0000 7f3897fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898104130 0x7f3898199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37478/0 (socket says 172.21.15.77:37478) 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.843+0000 7f3897fff700 1 -- 172.21.15.77:0/2228419147 learned_addr learned my addr 172.21.15.77:0/2228419147 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.843+0000 7f3897fff700 1 -- 172.21.15.77:0/2228419147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389819a010 con 0x7f3898104130 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.843+0000 7f3897fff700 1 --2- 172.21.15.77:0/2228419147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898104130 0x7f3898199ad0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3888005850 tx=0x7f3888005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:33.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.843+0000 7f38957fa700 1 -- 172.21.15.77:0/2228419147 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3888026070 con 0x7f3898104130 2024-03-16T10:08:33.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.843+0000 7f38957fa700 1 -- 172.21.15.77:0/2228419147 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f388800bbd0 con 0x7f3898104130 2024-03-16T10:08:33.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.843+0000 7f389e899700 1 -- 172.21.15.77:0/2228419147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f389819a300 con 0x7f3898104130 2024-03-16T10:08:33.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.843+0000 7f38957fa700 1 -- 172.21.15.77:0/2228419147 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3888003970 con 0x7f3898104130 2024-03-16T10:08:33.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.843+0000 7f389e899700 1 -- 172.21.15.77:0/2228419147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f389819e330 con 0x7f3898104130 2024-03-16T10:08:33.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.844+0000 7f38957fa700 1 -- 172.21.15.77:0/2228419147 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f3888029070 con 0x7f3898104130 2024-03-16T10:08:33.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.844+0000 7f389e899700 1 -- 172.21.15.77:0/2228419147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3898105610 con 0x7f3898104130 2024-03-16T10:08:33.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.844+0000 7f38957fa700 1 --2- 172.21.15.77:0/2228419147 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3880038fc0 0x7f388003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:33.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.844+0000 7f38957fa700 1 -- 172.21.15.77:0/2228419147 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f388801e070 con 0x7f3898104130 2024-03-16T10:08:33.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.844+0000 7f38977fe700 1 --2- 172.21.15.77:0/2228419147 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3880038fc0 0x7f388003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:33.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.845+0000 7f38977fe700 1 --2- 172.21.15.77:0/2228419147 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3880038fc0 0x7f388003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f388c006fd0 tx=0x7f388c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:33.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:33.849+0000 7f38957fa700 1 -- 172.21.15.77:0/2228419147 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3888013070 con 0x7f3898104130 2024-03-16T10:08:33.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:33 smithi077 ceph-mon[32400]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:34.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:34.050+0000 7f389e899700 1 -- 172.21.15.77:0/2228419147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f3898063900 con 0x7f3898104130 2024-03-16T10:08:34.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:34.758+0000 7f38957fa700 1 -- 172.21.15.77:0/2228419147 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f3888030020 con 0x7f3898104130 2024-03-16T10:08:34.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:34.760+0000 7f389e899700 1 -- 172.21.15.77:0/2228419147 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3880038fc0 msgr2=0x7f388003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:34.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:34.760+0000 7f389e899700 1 --2- 172.21.15.77:0/2228419147 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3880038fc0 0x7f388003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f388c006fd0 tx=0x7f388c006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:34.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:34.760+0000 7f389e899700 1 -- 172.21.15.77:0/2228419147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898104130 msgr2=0x7f3898199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:34.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:34.760+0000 7f389e899700 1 --2- 172.21.15.77:0/2228419147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898104130 0x7f3898199ad0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3888005850 tx=0x7f3888005030 comp rx=0 tx=0).stop 2024-03-16T10:08:34.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:34.760+0000 7f389e899700 1 -- 172.21.15.77:0/2228419147 shutdown_connections 2024-03-16T10:08:34.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:34.761+0000 7f389e899700 1 --2- 172.21.15.77:0/2228419147 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3880038fc0 0x7f388003b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:34.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:34.761+0000 7f389e899700 1 --2- 172.21.15.77:0/2228419147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3898104130 0x7f3898199ad0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:34.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:34.761+0000 7f389e899700 1 -- 172.21.15.77:0/2228419147 >> 172.21.15.77:0/2228419147 conn(0x7f38980ff9e0 msgr2=0x7f3898105b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:34.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:34.761+0000 7f389e899700 1 -- 172.21.15.77:0/2228419147 shutdown_connections 2024-03-16T10:08:34.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:08:34.761+0000 7f389e899700 1 -- 172.21.15.77:0/2228419147 wait complete. 2024-03-16T10:08:34.763 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2024-03-16T10:08:35.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:34 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2228419147' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T10:08:35.370 INFO:tasks.cephadm:Adding mon.a on smithi077 2024-03-16T10:08:35.370 INFO:tasks.cephadm:Adding mon.b on smithi192 2024-03-16T10:08:35.370 INFO:tasks.cephadm:Adding mon.c on smithi193 2024-03-16T10:08:35.370 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi077:172.21.15.77=a;smithi192:172.21.15.192=b;smithi193:172.21.15.193=c' 2024-03-16T10:08:35.603 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-16T10:08:35.651 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-16T10:08:36.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:35 smithi077 ceph-mon[32400]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:36.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:35 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2228419147' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T10:08:36.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:35 smithi077 ceph-mon[32400]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:08:36.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:36.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:36.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:36.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:36.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:08:36.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:08:36.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:08:37.157 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.153+0000 7f6218cd9700 1 -- 172.21.15.193:0/288741539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214100e00 msgr2=0x7f62141011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:37.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.153+0000 7f6218cd9700 1 --2- 172.21.15.193:0/288741539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214100e00 0x7f62141011e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6208009910 tx=0x7f6208009c20 comp rx=0 tx=0).stop 2024-03-16T10:08:37.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.153+0000 7f6218cd9700 1 -- 172.21.15.193:0/288741539 shutdown_connections 2024-03-16T10:08:37.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.153+0000 7f6218cd9700 1 --2- 172.21.15.193:0/288741539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214100e00 0x7f62141011e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:37.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.153+0000 7f6218cd9700 1 -- 172.21.15.193:0/288741539 >> 172.21.15.193:0/288741539 conn(0x7f62140fc6b0 msgr2=0x7f62140fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:37.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.153+0000 7f6218cd9700 1 -- 172.21.15.193:0/288741539 shutdown_connections 2024-03-16T10:08:37.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.153+0000 7f6218cd9700 1 -- 172.21.15.193:0/288741539 wait complete. 2024-03-16T10:08:37.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.154+0000 7f6218cd9700 1 Processor -- start 2024-03-16T10:08:37.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.154+0000 7f6218cd9700 1 -- start start 2024-03-16T10:08:37.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.154+0000 7f6218cd9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214100e00 0x7f6214196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:37.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.154+0000 7f6218cd9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f621410b1d0 con 0x7f6214100e00 2024-03-16T10:08:37.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.155+0000 7f621259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214100e00 0x7f6214196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:37.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.155+0000 7f621259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214100e00 0x7f6214196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.193:44538/0 (socket says 172.21.15.193:44538) 2024-03-16T10:08:37.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.155+0000 7f621259c700 1 -- 172.21.15.193:0/3718358799 learned_addr learned my addr 172.21.15.193:0/3718358799 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-16T10:08:37.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.155+0000 7f621259c700 1 -- 172.21.15.193:0/3718358799 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621406af10 con 0x7f6214100e00 2024-03-16T10:08:37.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.155+0000 7f621259c700 1 --2- 172.21.15.193:0/3718358799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214100e00 0x7f6214196470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6208005850 tx=0x7f6208005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:37.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.156+0000 7f62037fe700 1 -- 172.21.15.193:0/3718358799 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6208026070 con 0x7f6214100e00 2024-03-16T10:08:37.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.156+0000 7f62037fe700 1 -- 172.21.15.193:0/3718358799 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f620800bbd0 con 0x7f6214100e00 2024-03-16T10:08:37.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.156+0000 7f6218cd9700 1 -- 172.21.15.193:0/3718358799 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62141969b0 con 0x7f6214100e00 2024-03-16T10:08:37.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.156+0000 7f62037fe700 1 -- 172.21.15.193:0/3718358799 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6208003970 con 0x7f6214100e00 2024-03-16T10:08:37.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.156+0000 7f6218cd9700 1 -- 172.21.15.193:0/3718358799 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6214196e70 con 0x7f6214100e00 2024-03-16T10:08:37.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.157+0000 7f62037fe700 1 -- 172.21.15.193:0/3718358799 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f6208029070 con 0x7f6214100e00 2024-03-16T10:08:37.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.157+0000 7f62037fe700 1 --2- 172.21.15.193:0/3718358799 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f61fc038bd0 0x7f61fc03b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:37.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.157+0000 7f62037fe700 1 -- 172.21.15.193:0/3718358799 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f620801e070 con 0x7f6214100e00 2024-03-16T10:08:37.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.157+0000 7f6211d9b700 1 --2- 172.21.15.193:0/3718358799 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f61fc038bd0 0x7f61fc03b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:37.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.157+0000 7f6218cd9700 1 -- 172.21.15.193:0/3718358799 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62141022e0 con 0x7f6214100e00 2024-03-16T10:08:37.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.158+0000 7f6211d9b700 1 --2- 172.21.15.193:0/3718358799 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f61fc038bd0 0x7f61fc03b090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6204006fd0 tx=0x7f6204006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:37.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.163+0000 7f62037fe700 1 -- 172.21.15.193:0/3718358799 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6208013070 con 0x7f6214100e00 2024-03-16T10:08:37.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:36 smithi077 ceph-mon[32400]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-16T10:08:37.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:36 smithi077 ceph-mon[32400]: Updating smithi193:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:08:37.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:36 smithi077 ceph-mon[32400]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:08:37.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:36 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:37.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:36 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:37.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:36 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:37.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.369+0000 7f6218cd9700 1 -- 172.21.15.193:0/3718358799 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi077:172.21.15.77=a;smithi192:172.21.15.192=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}) -- 0x7f621406a700 con 0x7f61fc038bd0 2024-03-16T10:08:37.377 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.377+0000 7f62037fe700 1 -- 172.21.15.193:0/3718358799 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f621406a700 con 0x7f61fc038bd0 2024-03-16T10:08:37.377 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mon update... 2024-03-16T10:08:37.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.378+0000 7f6218cd9700 1 -- 172.21.15.193:0/3718358799 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f61fc038bd0 msgr2=0x7f61fc03b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:37.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.378+0000 7f6218cd9700 1 --2- 172.21.15.193:0/3718358799 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f61fc038bd0 0x7f61fc03b090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6204006fd0 tx=0x7f6204006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:37.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.378+0000 7f6218cd9700 1 -- 172.21.15.193:0/3718358799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214100e00 msgr2=0x7f6214196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:37.379 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.378+0000 7f6218cd9700 1 --2- 172.21.15.193:0/3718358799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214100e00 0x7f6214196470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6208005850 tx=0x7f6208005030 comp rx=0 tx=0).stop 2024-03-16T10:08:37.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.378+0000 7f6218cd9700 1 -- 172.21.15.193:0/3718358799 shutdown_connections 2024-03-16T10:08:37.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.378+0000 7f6218cd9700 1 --2- 172.21.15.193:0/3718358799 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f61fc038bd0 0x7f61fc03b090 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:37.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.378+0000 7f6218cd9700 1 --2- 172.21.15.193:0/3718358799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214100e00 0x7f6214196470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:37.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.378+0000 7f6218cd9700 1 -- 172.21.15.193:0/3718358799 >> 172.21.15.193:0/3718358799 conn(0x7f62140fc6b0 msgr2=0x7f6214103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:37.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.378+0000 7f6218cd9700 1 -- 172.21.15.193:0/3718358799 shutdown_connections 2024-03-16T10:08:37.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:37.378+0000 7f6218cd9700 1 -- 172.21.15.193:0/3718358799 wait complete. 2024-03-16T10:08:38.012 DEBUG:teuthology.orchestra.run.smithi192:mon.b> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.b.service 2024-03-16T10:08:38.015 DEBUG:teuthology.orchestra.run.smithi193:mon.c> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.c.service 2024-03-16T10:08:38.017 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T10:08:38.017 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T10:08:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:37 smithi077 ceph-mon[32400]: Updating smithi193:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T10:08:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:37 smithi077 ceph-mon[32400]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:08:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:08:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:08:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:08:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:08:39.111 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:08:39.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:38 smithi077 ceph-mon[32400]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi077:172.21.15.77=a;smithi192:172.21.15.192=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:08:39.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:38 smithi077 ceph-mon[32400]: Saving service mon spec with placement smithi077:172.21.15.77=a;smithi192:172.21.15.192=b;smithi193:172.21.15.193=c;count:3 2024-03-16T10:08:39.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:38 smithi077 ceph-mon[32400]: Deploying daemon mon.c on smithi193 2024-03-16T10:08:40.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:39 smithi077 ceph-mon[32400]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:40.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.982+0000 7f9fce0fa700 1 -- 172.21.15.193:0/4153832814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fc8102e70 msgr2=0x7f9fc8103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:40.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.982+0000 7f9fce0fa700 1 --2- 172.21.15.193:0/4153832814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fc8102e70 0x7f9fc8103250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8009910 tx=0x7f9fb8009c20 comp rx=0 tx=0).stop 2024-03-16T10:08:40.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.982+0000 7f9fce0fa700 1 -- 172.21.15.193:0/4153832814 shutdown_connections 2024-03-16T10:08:40.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.982+0000 7f9fce0fa700 1 --2- 172.21.15.193:0/4153832814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fc8102e70 0x7f9fc8103250 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:40.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.982+0000 7f9fce0fa700 1 -- 172.21.15.193:0/4153832814 >> 172.21.15.193:0/4153832814 conn(0x7f9fc80fc6a0 msgr2=0x7f9fc80feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:40.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.982+0000 7f9fce0fa700 1 -- 172.21.15.193:0/4153832814 shutdown_connections 2024-03-16T10:08:40.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.982+0000 7f9fce0fa700 1 -- 172.21.15.193:0/4153832814 wait complete. 2024-03-16T10:08:40.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.983+0000 7f9fce0fa700 1 Processor -- start 2024-03-16T10:08:40.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.983+0000 7f9fce0fa700 1 -- start start 2024-03-16T10:08:40.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.983+0000 7f9fce0fa700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fc8102e70 0x7f9fc8100ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:40.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.983+0000 7f9fce0fa700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9fc810c360 con 0x7f9fc8102e70 2024-03-16T10:08:40.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.984+0000 7f9fc77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fc8102e70 0x7f9fc8100ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:40.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.984+0000 7f9fc77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fc8102e70 0x7f9fc8100ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.193:36552/0 (socket says 172.21.15.193:36552) 2024-03-16T10:08:40.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.984+0000 7f9fc77fe700 1 -- 172.21.15.193:0/3601515565 learned_addr learned my addr 172.21.15.193:0/3601515565 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-16T10:08:40.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.984+0000 7f9fc77fe700 1 -- 172.21.15.193:0/3601515565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fc81029c0 con 0x7f9fc8102e70 2024-03-16T10:08:40.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.984+0000 7f9fc77fe700 1 --2- 172.21.15.193:0/3601515565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fc8102e70 0x7f9fc8100ad0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8006010 tx=0x7f9fb8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:40.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.984+0000 7f9fc4ff9700 1 -- 172.21.15.193:0/3601515565 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fb8026070 con 0x7f9fc8102e70 2024-03-16T10:08:40.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.984+0000 7f9fc4ff9700 1 -- 172.21.15.193:0/3601515565 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fb800bc30 con 0x7f9fc8102e70 2024-03-16T10:08:40.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.984+0000 7f9fce0fa700 1 -- 172.21.15.193:0/3601515565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fc8101010 con 0x7f9fc8102e70 2024-03-16T10:08:40.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.984+0000 7f9fc4ff9700 1 -- 172.21.15.193:0/3601515565 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9fb80039a0 con 0x7f9fc8102e70 2024-03-16T10:08:40.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.984+0000 7f9fce0fa700 1 -- 172.21.15.193:0/3601515565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fc81014d0 con 0x7f9fc8102e70 2024-03-16T10:08:40.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.985+0000 7f9fce0fa700 1 -- 172.21.15.193:0/3601515565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fc806a9f0 con 0x7f9fc8102e70 2024-03-16T10:08:40.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.985+0000 7f9fc4ff9700 1 -- 172.21.15.193:0/3601515565 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f9fb8029070 con 0x7f9fc8102e70 2024-03-16T10:08:40.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.985+0000 7f9fc4ff9700 1 --2- 172.21.15.193:0/3601515565 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9fb0038fc0 0x7f9fb003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:40.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.985+0000 7f9fc4ff9700 1 -- 172.21.15.193:0/3601515565 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9fb801e070 con 0x7f9fc8102e70 2024-03-16T10:08:40.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.986+0000 7f9fc6ffd700 1 --2- 172.21.15.193:0/3601515565 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9fb0038fc0 0x7f9fb003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:40.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.986+0000 7f9fc6ffd700 1 --2- 172.21.15.193:0/3601515565 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9fb0038fc0 0x7f9fb003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc006fd0 tx=0x7f9fbc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:40.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:40.991+0000 7f9fc4ff9700 1 -- 172.21.15.193:0/3601515565 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9fb8013070 con 0x7f9fc8102e70 2024-03-16T10:08:41.273 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:41.272+0000 7f9fce0fa700 1 -- 172.21.15.193:0/3601515565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9fc81020a0 con 0x7f9fc8102e70 2024-03-16T10:08:41.273 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:41.273+0000 7f9fc4ff9700 1 -- 172.21.15.193:0/3601515565 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f9fb8030020 con 0x7f9fc8102e70 2024-03-16T10:08:41.273 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:08:41.273 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","modified":"2024-03-16T10:06:54.961708Z","created":"2024-03-16T10:06:54.961708Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-16T10:08:41.276 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:41.275+0000 7f9fce0fa700 1 -- 172.21.15.193:0/3601515565 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9fb0038fc0 msgr2=0x7f9fb003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:41.276 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:41.275+0000 7f9fce0fa700 1 --2- 172.21.15.193:0/3601515565 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9fb0038fc0 0x7f9fb003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc006fd0 tx=0x7f9fbc006e40 comp rx=0 tx=0).stop 2024-03-16T10:08:41.276 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:41.275+0000 7f9fce0fa700 1 -- 172.21.15.193:0/3601515565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fc8102e70 msgr2=0x7f9fc8100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:41.276 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:41.275+0000 7f9fce0fa700 1 --2- 172.21.15.193:0/3601515565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fc8102e70 0x7f9fc8100ad0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8006010 tx=0x7f9fb8005030 comp rx=0 tx=0).stop 2024-03-16T10:08:41.276 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:41.275+0000 7f9fce0fa700 1 -- 172.21.15.193:0/3601515565 shutdown_connections 2024-03-16T10:08:41.276 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:41.275+0000 7f9fce0fa700 1 --2- 172.21.15.193:0/3601515565 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9fb0038fc0 0x7f9fb003b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:41.276 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:41.275+0000 7f9fce0fa700 1 --2- 172.21.15.193:0/3601515565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fc8102e70 0x7f9fc8100ad0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:41.276 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:41.275+0000 7f9fce0fa700 1 -- 172.21.15.193:0/3601515565 >> 172.21.15.193:0/3601515565 conn(0x7f9fc80fc6a0 msgr2=0x7f9fc8103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:41.276 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:41.275+0000 7f9fce0fa700 1 -- 172.21.15.193:0/3601515565 shutdown_connections 2024-03-16T10:08:41.276 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:41.275+0000 7f9fce0fa700 1 -- 172.21.15.193:0/3601515565 wait complete. 2024-03-16T10:08:41.276 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-16T10:08:42.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:41 smithi077 ceph-mon[32400]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:42.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:41 smithi077 ceph-mon[32400]: from='client.? 172.21.15.193:0/3601515565' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T10:08:42.976 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T10:08:42.976 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T10:08:43.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:42 smithi193 systemd[1]: Started Ceph mon.c for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:08:43.208 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:08:48.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.087+0000 7fb4ec416700 1 -- 172.21.15.193:0/1265225129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4e4100ef0 msgr2=0x7fb4d0005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:48.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.087+0000 7fb4ec416700 1 --2- 172.21.15.193:0/1265225129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4e4100ef0 0x7fb4d0005610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009c10 tx=0x7fb4dc009f20 comp rx=0 tx=0).stop 2024-03-16T10:08:48.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.088+0000 7fb4ec416700 1 -- 172.21.15.193:0/1265225129 shutdown_connections 2024-03-16T10:08:48.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.088+0000 7fb4ec416700 1 --2- 172.21.15.193:0/1265225129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4e4100ef0 0x7fb4d0005610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:48.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.088+0000 7fb4ec416700 1 -- 172.21.15.193:0/1265225129 >> 172.21.15.193:0/1265225129 conn(0x7fb4e40fc780 msgr2=0x7fb4e40feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:48.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.088+0000 7fb4ec416700 1 -- 172.21.15.193:0/1265225129 shutdown_connections 2024-03-16T10:08:48.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.088+0000 7fb4ec416700 1 -- 172.21.15.193:0/1265225129 wait complete. 2024-03-16T10:08:48.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4ec416700 1 Processor -- start 2024-03-16T10:08:48.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4ec416700 1 -- start start 2024-03-16T10:08:48.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4ec416700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4e4100ef0 0x7fb4e419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:48.091 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4ec416700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4e419fd60 0x7fb4e41999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4ec416700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e4108d20 con 0x7fb4e4100ef0 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4ec416700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e4108ba0 con 0x7fb4e419fd60 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4e99b1700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4e419fd60 0x7fb4e41999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4e99b1700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4e419fd60 0x7fb4e41999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:41956/0 (socket says 172.21.15.193:41956) 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4e99b1700 1 -- 172.21.15.193:0/1041920429 learned_addr learned my addr 172.21.15.193:0/1041920429 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4ea1b2700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4e4100ef0 0x7fb4e419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4e99b1700 1 -- 172.21.15.193:0/1041920429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4e419fd60 msgr2=0x7fb4e41999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4e99b1700 1 -- 172.21.15.193:0/1041920429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4e419fd60 msgr2=0x7fb4e41999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4e99b1700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4e419fd60 0x7fb4e41999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.089+0000 7fb4e99b1700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4e419fd60 0x7fb4e41999e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.090+0000 7fb4ea1b2700 1 -- 172.21.15.193:0/1041920429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4e419fd60 msgr2=0x7fb4e41999e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.090+0000 7fb4ea1b2700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4e419fd60 0x7fb4e41999e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.090+0000 7fb4ea1b2700 1 -- 172.21.15.193:0/1041920429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4e41a03a0 con 0x7fb4e4100ef0 2024-03-16T10:08:48.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.090+0000 7fb4ea1b2700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4e4100ef0 0x7fb4e419f820 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc007950 tx=0x7fb4dc005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:48.093 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.090+0000 7fb4db7fe700 1 -- 172.21.15.193:0/1041920429 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb4dc0195d0 con 0x7fb4e4100ef0 2024-03-16T10:08:48.093 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.090+0000 7fb4ec416700 1 -- 172.21.15.193:0/1041920429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4e4199f20 con 0x7fb4e4100ef0 2024-03-16T10:08:48.093 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.090+0000 7fb4db7fe700 1 -- 172.21.15.193:0/1041920429 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4dc019c50 con 0x7fb4e4100ef0 2024-03-16T10:08:48.093 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.090+0000 7fb4db7fe700 1 -- 172.21.15.193:0/1041920429 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb4dc021bd0 con 0x7fb4e4100ef0 2024-03-16T10:08:48.093 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.090+0000 7fb4ec416700 1 -- 172.21.15.193:0/1041920429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4e419a4b0 con 0x7fb4e4100ef0 2024-03-16T10:08:48.094 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.091+0000 7fb4db7fe700 1 -- 172.21.15.193:0/1041920429 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fb4dc015070 con 0x7fb4e4100ef0 2024-03-16T10:08:48.094 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.092+0000 7fb4ec416700 1 -- 172.21.15.193:0/1041920429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4e41023d0 con 0x7fb4e4100ef0 2024-03-16T10:08:48.094 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.092+0000 7fb4db7fe700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4c8038d60 0x7fb4c803b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:48.094 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.092+0000 7fb4db7fe700 1 -- 172.21.15.193:0/1041920429 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb4dc027030 con 0x7fb4e4100ef0 2024-03-16T10:08:48.094 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.092+0000 7fb4e99b1700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4c8038d60 0x7fb4c803b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:48.094 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.093+0000 7fb4e99b1700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4c8038d60 0x7fb4c803b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4009180 tx=0x7fb4d4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:48.097 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.097+0000 7fb4db7fe700 1 -- 172.21.15.193:0/1041920429 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb4dc01d030 con 0x7fb4e4100ef0 2024-03-16T10:08:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:08:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: mon.a calling monitor election 2024-03-16T10:08:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: mon.c calling monitor election 2024-03-16T10:08:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: monmap epoch 2 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: last_changed 2024-03-16T10:08:42.808394+0000 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: created 2024-03-16T10:06:54.961708+0000 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: min_mon_release 19 (squid) 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: election_strategy: 1 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: fsmap 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:08:48.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: mgrmap e13: x(active, since 49s) 2024-03-16T10:08:48.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: overall HEALTH_OK 2024-03-16T10:08:48.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:48.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:48.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:48.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:08:48.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:08:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:08:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: mon.a calling monitor election 2024-03-16T10:08:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: mon.c calling monitor election 2024-03-16T10:08:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: monmap epoch 2 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: last_changed 2024-03-16T10:08:42.808394+0000 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: created 2024-03-16T10:06:54.961708+0000 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: min_mon_release 19 (squid) 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: election_strategy: 1 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: fsmap 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: mgrmap e13: x(active, since 49s) 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: overall HEALTH_OK 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:48.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:48.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:08:48.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:08:48.374 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.374+0000 7fb4ec416700 1 -- 172.21.15.193:0/1041920429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fb4e419af10 con 0x7fb4e4100ef0 2024-03-16T10:08:48.375 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.375+0000 7fb4db7fe700 1 -- 172.21.15.193:0/1041920429 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fb4dc016c00 con 0x7fb4e4100ef0 2024-03-16T10:08:48.375 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:08:48.375 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","modified":"2024-03-16T10:08:42.808394Z","created":"2024-03-16T10:06:54.961708Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-16T10:08:48.377 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.377+0000 7fb4ec416700 1 -- 172.21.15.193:0/1041920429 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4c8038d60 msgr2=0x7fb4c803b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:48.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.377+0000 7fb4ec416700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4c8038d60 0x7fb4c803b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4009180 tx=0x7fb4d4008040 comp rx=0 tx=0).stop 2024-03-16T10:08:48.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.377+0000 7fb4ec416700 1 -- 172.21.15.193:0/1041920429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4e4100ef0 msgr2=0x7fb4e419f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:48.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.377+0000 7fb4ec416700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4e4100ef0 0x7fb4e419f820 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc007950 tx=0x7fb4dc005b00 comp rx=0 tx=0).stop 2024-03-16T10:08:48.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.377+0000 7fb4ec416700 1 -- 172.21.15.193:0/1041920429 shutdown_connections 2024-03-16T10:08:48.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.377+0000 7fb4ec416700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4c8038d60 0x7fb4c803b220 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:48.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.377+0000 7fb4ec416700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4e4100ef0 0x7fb4e419f820 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:48.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.377+0000 7fb4ec416700 1 --2- 172.21.15.193:0/1041920429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4e419fd60 0x7fb4e41999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:48.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.377+0000 7fb4ec416700 1 -- 172.21.15.193:0/1041920429 >> 172.21.15.193:0/1041920429 conn(0x7fb4e40fc780 msgr2=0x7fb4e406a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:48.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.377+0000 7fb4ec416700 1 -- 172.21.15.193:0/1041920429 shutdown_connections 2024-03-16T10:08:48.378 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:48.377+0000 7fb4ec416700 1 -- 172.21.15.193:0/1041920429 wait complete. 2024-03-16T10:08:48.379 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-03-16T10:08:49.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:48 smithi077 ceph-mon[32400]: Deploying daemon mon.b on smithi192 2024-03-16T10:08:49.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:48 smithi077 ceph-mon[32400]: from='client.? 172.21.15.193:0/1041920429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T10:08:49.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:48 smithi077 ceph-mon[32400]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:49.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:49.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:48 smithi193 ceph-mon[36311]: Deploying daemon mon.b on smithi192 2024-03-16T10:08:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:48 smithi193 ceph-mon[36311]: from='client.? 172.21.15.193:0/1041920429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T10:08:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:48 smithi193 ceph-mon[36311]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:49.875 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T10:08:49.875 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T10:08:50.119 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:08:50.245 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:08:49 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:08:49.808+0000 7f64b1fc5700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-16T10:08:51.582 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:51 smithi192 ceph-mon[36354]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-16T10:08:51.582 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:51 smithi192 bash[36333]: 632d7ca9fce96d32df18434062e77000e0659db114ac4218fa6d718db3df47a9 2024-03-16T10:08:51.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:51 smithi192 systemd[1]: Started Ceph mon.b for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:08:56.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: mon.c calling monitor election 2024-03-16T10:08:56.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: mon.a calling monitor election 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: monmap epoch 3 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:08:56.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: last_changed 2024-03-16T10:08:51.565283+0000 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: created 2024-03-16T10:06:54.961708+0000 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: min_mon_release 19 (squid) 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: election_strategy: 1 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: 2: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: fsmap 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: mgrmap e13: x(active, since 57s) 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: mon.b (rank 2) addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] is down (out of quorum) 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:56.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:56.998 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:57.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: mon.c calling monitor election 2024-03-16T10:08:57.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:08:57.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: mon.a calling monitor election 2024-03-16T10:08:57.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:57.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:57.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:57.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:57.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:57.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:57.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:57.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: monmap epoch 3 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: last_changed 2024-03-16T10:08:51.565283+0000 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: created 2024-03-16T10:06:54.961708+0000 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: min_mon_release 19 (squid) 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: election_strategy: 1 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: 2: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: fsmap 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: mgrmap e13: x(active, since 57s) 2024-03-16T10:08:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T10:08:57.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T10:08:57.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T10:08:57.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: mon.b (rank 2) addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] is down (out of quorum) 2024-03-16T10:08:57.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:57.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:57.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:57.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:57.581 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.580+0000 7f086747f700 1 -- 172.21.15.193:0/70450428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c0084b0 msgr2=0x7f084c00a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:57.581 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.580+0000 7f086747f700 1 --2- 172.21.15.193:0/70450428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c0084b0 0x7f084c00a920 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0860101d60 tx=0x7f085000fe90 comp rx=0 tx=0).stop 2024-03-16T10:08:57.581 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.580+0000 7f086747f700 1 -- 172.21.15.193:0/70450428 shutdown_connections 2024-03-16T10:08:57.581 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.580+0000 7f086747f700 1 --2- 172.21.15.193:0/70450428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f084c0084b0 0x7f084c00a920 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:57.581 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.580+0000 7f086747f700 1 -- 172.21.15.193:0/70450428 >> 172.21.15.193:0/70450428 conn(0x7f08600fc790 msgr2=0x7f08600febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:57.581 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.580+0000 7f086747f700 1 -- 172.21.15.193:0/70450428 shutdown_connections 2024-03-16T10:08:57.582 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.580+0000 7f086747f700 1 -- 172.21.15.193:0/70450428 wait complete. 2024-03-16T10:08:57.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.581+0000 7f086747f700 1 Processor -- start 2024-03-16T10:08:57.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.581+0000 7f086747f700 1 -- start start 2024-03-16T10:08:57.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.581+0000 7f086747f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f084c0084b0 0x7f086019b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.581+0000 7f086747f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f086019bb50 0x7f08601956e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.581+0000 7f086747f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0860195c20 0x7f08601960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.581+0000 7f086747f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0860107d00 con 0x7f0860195c20 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.581+0000 7f086747f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f0860107b80 con 0x7f086019bb50 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.581+0000 7f086747f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0860107e80 con 0x7f084c0084b0 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.581+0000 7f086521b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f084c0084b0 0x7f086019b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f086521b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f084c0084b0 0x7f086019b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:60694/0 (socket says 172.21.15.193:60694) 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f086521b700 1 -- 172.21.15.193:0/1271552281 learned_addr learned my addr 172.21.15.193:0/1271552281 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f086521b700 1 -- 172.21.15.193:0/1271552281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f084c0084b0 msgr2=0x7f086019b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f086521b700 1 -- 172.21.15.193:0/1271552281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f084c0084b0 msgr2=0x7f086019b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f086521b700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f084c0084b0 0x7f086019b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f0865a1c700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0860195c20 0x7f08601960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f086521b700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f084c0084b0 0x7f086019b610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:08:57.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f0864a1a700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f086019bb50 0x7f08601956e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f0864a1a700 1 -- 172.21.15.193:0/1271552281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f086019bb50 msgr2=0x7f08601956e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f0864a1a700 1 -- 172.21.15.193:0/1271552281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f086019bb50 msgr2=0x7f08601956e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f0864a1a700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f086019bb50 0x7f08601956e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f0864a1a700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f086019bb50 0x7f08601956e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f0865a1c700 1 -- 172.21.15.193:0/1271552281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f084c0084b0 msgr2=0x7f086019b610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f0865a1c700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f084c0084b0 0x7f086019b610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f0865a1c700 1 -- 172.21.15.193:0/1271552281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f086019bb50 msgr2=0x7f08601956e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f0865a1c700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f086019bb50 0x7f08601956e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f0865a1c700 1 -- 172.21.15.193:0/1271552281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08601a23c0 con 0x7f0860195c20 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f0865a1c700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0860195c20 0x7f08601960d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f085800bd50 tx=0x7f085800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f08567fc700 1 -- 172.21.15.193:0/1271552281 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0858019070 con 0x7f0860195c20 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f08567fc700 1 -- 172.21.15.193:0/1271552281 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f085800cd10 con 0x7f0860195c20 2024-03-16T10:08:57.586 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f08567fc700 1 -- 172.21.15.193:0/1271552281 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08580127e0 con 0x7f0860195c20 2024-03-16T10:08:57.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.582+0000 7f086747f700 1 -- 172.21.15.193:0/1271552281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08601a2650 con 0x7f0860195c20 2024-03-16T10:08:57.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.583+0000 7f086747f700 1 -- 172.21.15.193:0/1271552281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08601a2b40 con 0x7f0860195c20 2024-03-16T10:08:57.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.583+0000 7f086747f700 1 -- 172.21.15.193:0/1271552281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08601023e0 con 0x7f0860195c20 2024-03-16T10:08:57.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.583+0000 7f08567fc700 1 -- 172.21.15.193:0/1271552281 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f085801c070 con 0x7f0860195c20 2024-03-16T10:08:57.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.584+0000 7f08567fc700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0844039450 0x7f084403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:08:57.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.584+0000 7f08567fc700 1 -- 172.21.15.193:0/1271552281 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f085804b9d0 con 0x7f0860195c20 2024-03-16T10:08:57.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.584+0000 7f086521b700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0844039450 0x7f084403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:08:57.587 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.584+0000 7f086521b700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0844039450 0x7f084403b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0850005b40 tx=0x7f0850005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:08:57.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.589+0000 7f08567fc700 1 -- 172.21.15.193:0/1271552281 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f085801bdd0 con 0x7f0860195c20 2024-03-16T10:08:57.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.869+0000 7f086747f700 1 -- 172.21.15.193:0/1271552281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f086002db70 con 0x7f0860195c20 2024-03-16T10:08:57.870 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:57 smithi193 ceph-mon[36311]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:57.870 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:08:57.870 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:57.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.870+0000 7f08567fc700 1 -- 172.21.15.193:0/1271552281 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f0858014440 con 0x7f0860195c20 2024-03-16T10:08:57.871 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-16T10:08:57.871 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":3,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","modified":"2024-03-16T10:08:51.565283Z","created":"2024-03-16T10:06:54.961708Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-16T10:08:57.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.872+0000 7f086747f700 1 -- 172.21.15.193:0/1271552281 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0844039450 msgr2=0x7f084403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:57.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.872+0000 7f086747f700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0844039450 0x7f084403b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0850005b40 tx=0x7f0850005a90 comp rx=0 tx=0).stop 2024-03-16T10:08:57.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.872+0000 7f086747f700 1 -- 172.21.15.193:0/1271552281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0860195c20 msgr2=0x7f08601960d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:08:57.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.872+0000 7f086747f700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0860195c20 0x7f08601960d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f085800bd50 tx=0x7f085800be30 comp rx=0 tx=0).stop 2024-03-16T10:08:57.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.872+0000 7f086747f700 1 -- 172.21.15.193:0/1271552281 shutdown_connections 2024-03-16T10:08:57.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.872+0000 7f086747f700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0844039450 0x7f084403b910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:57.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.872+0000 7f086747f700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0860195c20 0x7f08601960d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:57.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.872+0000 7f086747f700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f084c0084b0 0x7f086019b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:57.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.872+0000 7f086747f700 1 --2- 172.21.15.193:0/1271552281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f086019bb50 0x7f08601956e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:08:57.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.872+0000 7f086747f700 1 -- 172.21.15.193:0/1271552281 >> 172.21.15.193:0/1271552281 conn(0x7f08600fc790 msgr2=0x7f08600fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:08:57.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.872+0000 7f086747f700 1 -- 172.21.15.193:0/1271552281 shutdown_connections 2024-03-16T10:08:57.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:08:57.872+0000 7f086747f700 1 -- 172.21.15.193:0/1271552281 wait complete. 2024-03-16T10:08:57.873 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 3 2024-03-16T10:08:57.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:57 smithi077 ceph-mon[32400]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:57.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:08:57.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:58.672 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-16T10:08:58.672 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-16T10:08:58.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: mon.c calling monitor election 2024-03-16T10:08:58.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:08:58.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: mon.a calling monitor election 2024-03-16T10:08:58.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:58.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:08:58.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:58.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:58.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: monmap epoch 3 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: last_changed 2024-03-16T10:08:51.565283+0000 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: created 2024-03-16T10:06:54.961708+0000 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: min_mon_release 19 (squid) 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: election_strategy: 1 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-03-16T10:08:58.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: 2: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: fsmap 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: mgrmap e13: x(active, since 57s) 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: mon.b (rank 2) addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] is down (out of quorum) 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:08:58.920 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:08:59.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: mon.b calling monitor election 2024-03-16T10:08:59.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-16T10:08:59.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-16T10:08:59.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-16T10:08:59.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: Updating smithi077:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:08:59.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: Updating smithi193:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:08:59.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: Updating smithi192:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:08:59.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: mon.b calling monitor election 2024-03-16T10:08:59.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: mon.c calling monitor election 2024-03-16T10:08:59.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: mon.a calling monitor election 2024-03-16T10:08:59.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: monmap epoch 3 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: last_changed 2024-03-16T10:08:51.565283+0000 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: created 2024-03-16T10:06:54.961708+0000 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: min_mon_release 19 (squid) 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: election_strategy: 1 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: 2: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: fsmap 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: mgrmap e13: x(active, since 59s) 2024-03-16T10:08:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: Cluster is now healthy 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: overall HEALTH_OK 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T10:08:59.748 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:08:59.749 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:08:59.749 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:08:59.749 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:08:59 smithi077 ceph-mon[32400]: Reconfiguring daemon mon.a on smithi077 2024-03-16T10:08:59.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: mon.b calling monitor election 2024-03-16T10:08:59.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-16T10:08:59.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-16T10:08:59.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: Updating smithi077:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: Updating smithi193:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: Updating smithi192:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: mon.b calling monitor election 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: mon.c calling monitor election 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: mon.a calling monitor election 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: monmap epoch 3 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: last_changed 2024-03-16T10:08:51.565283+0000 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: created 2024-03-16T10:06:54.961708+0000 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: min_mon_release 19 (squid) 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: election_strategy: 1 2024-03-16T10:08:59.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: 2: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: fsmap 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: mgrmap e13: x(active, since 59s) 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: Cluster is now healthy 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: overall HEALTH_OK 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:08:59.920 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.920 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.920 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.920 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.920 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:08:59.920 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T10:08:59.920 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:08:59.920 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:08:59.920 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:08:59.920 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:08:59 smithi192 ceph-mon[36354]: Reconfiguring daemon mon.a on smithi077 2024-03-16T10:09:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: mon.b calling monitor election 2024-03-16T10:09:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: Updating smithi077:/etc/ceph/ceph.conf 2024-03-16T10:09:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-16T10:09:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-16T10:09:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: Updating smithi077:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:09:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: Updating smithi193:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:09:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: Updating smithi192:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T10:09:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: mon.b calling monitor election 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: mon.c calling monitor election 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: mon.a calling monitor election 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: monmap epoch 3 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: last_changed 2024-03-16T10:08:51.565283+0000 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: created 2024-03-16T10:06:54.961708+0000 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: min_mon_release 19 (squid) 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: election_strategy: 1 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: 2: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: fsmap 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: mgrmap e13: x(active, since 59s) 2024-03-16T10:09:00.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: Cluster is now healthy 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: overall HEALTH_OK 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T10:09:00.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:09:00.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:09:00.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:00.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:08:59 smithi193 ceph-mon[36311]: Reconfiguring daemon mon.a on smithi077 2024-03-16T10:09:00.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.442+0000 7f37e566f700 1 -- 172.21.15.77:0/1844796158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37e0102e50 msgr2=0x7f37e0103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:00.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.442+0000 7f37e566f700 1 --2- 172.21.15.77:0/1844796158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37e0102e50 0x7f37e0103230 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f37d0009910 tx=0x7f37d0009c20 comp rx=0 tx=0).stop 2024-03-16T10:09:00.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.442+0000 7f37e566f700 1 -- 172.21.15.77:0/1844796158 shutdown_connections 2024-03-16T10:09:00.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.442+0000 7f37e566f700 1 --2- 172.21.15.77:0/1844796158 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37e0102e50 0x7f37e0103230 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:00.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.442+0000 7f37e566f700 1 -- 172.21.15.77:0/1844796158 >> 172.21.15.77:0/1844796158 conn(0x7f37e00fc660 msgr2=0x7f37e00fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:09:00.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.442+0000 7f37e566f700 1 -- 172.21.15.77:0/1844796158 shutdown_connections 2024-03-16T10:09:00.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.442+0000 7f37e566f700 1 -- 172.21.15.77:0/1844796158 wait complete. 2024-03-16T10:09:00.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.442+0000 7f37e566f700 1 Processor -- start 2024-03-16T10:09:00.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37e566f700 1 -- start start 2024-03-16T10:09:00.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37e566f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37e0102e50 0x7f37e019d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:00.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37e566f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37e019dc70 0x7f37e0197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:00.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37e566f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37e0197ed0 0x7f37e0198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:00.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37e566f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f37e010c2f0 con 0x7f37e019dc70 2024-03-16T10:09:00.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37e566f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f37e010c170 con 0x7f37e0102e50 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37e566f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f37e010c470 con 0x7f37e0197ed0 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37d67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37e019dc70 0x7f37e0197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37d67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37e019dc70 0x7f37e0197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36988/0 (socket says 172.21.15.77:36988) 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37d67fc700 1 -- 172.21.15.77:0/2229283195 learned_addr learned my addr 172.21.15.77:0/2229283195 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37deffd700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37e0102e50 0x7f37e019d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37d67fc700 1 -- 172.21.15.77:0/2229283195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37e0197ed0 msgr2=0x7f37e0198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37d67fc700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37e0197ed0 0x7f37e0198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37d67fc700 1 -- 172.21.15.77:0/2229283195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37e0102e50 msgr2=0x7f37e019d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37d67fc700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37e0102e50 0x7f37e019d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37d67fc700 1 -- 172.21.15.77:0/2229283195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e01a4600 con 0x7f37e019dc70 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.443+0000 7f37d67fc700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37e019dc70 0x7f37e0197990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f37c800b780 tx=0x7f37c800bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.444+0000 7f37dcff9700 1 -- 172.21.15.77:0/2229283195 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8004230 con 0x7f37e019dc70 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.444+0000 7f37dcff9700 1 -- 172.21.15.77:0/2229283195 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37c80043d0 con 0x7f37e019dc70 2024-03-16T10:09:00.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.444+0000 7f37e566f700 1 -- 172.21.15.77:0/2229283195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e01a4830 con 0x7f37e019dc70 2024-03-16T10:09:00.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.444+0000 7f37dcff9700 1 -- 172.21.15.77:0/2229283195 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8011c60 con 0x7f37e019dc70 2024-03-16T10:09:00.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.444+0000 7f37e566f700 1 -- 172.21.15.77:0/2229283195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e01a4dc0 con 0x7f37e019dc70 2024-03-16T10:09:00.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.444+0000 7f37dcff9700 1 -- 172.21.15.77:0/2229283195 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f37c801a400 con 0x7f37e019dc70 2024-03-16T10:09:00.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.445+0000 7f37dcff9700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f37cc039450 0x7f37cc03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:00.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.445+0000 7f37dcff9700 1 -- 172.21.15.77:0/2229283195 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f37c804cc00 con 0x7f37e019dc70 2024-03-16T10:09:00.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.445+0000 7f37deffd700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f37cc039450 0x7f37cc03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:00.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.445+0000 7f37e566f700 1 -- 172.21.15.77:0/2229283195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37e0063900 con 0x7f37e019dc70 2024-03-16T10:09:00.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.445+0000 7f37deffd700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f37cc039450 0x7f37cc03b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f37d0009750 tx=0x7f37d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:09:00.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.450+0000 7f37dcff9700 1 -- 172.21.15.77:0/2229283195 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f37c8020500 con 0x7f37e019dc70 2024-03-16T10:09:00.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.653+0000 7f37e566f700 1 -- 172.21.15.77:0/2229283195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f37e01a5080 con 0x7f37e019dc70 2024-03-16T10:09:00.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.653+0000 7f37dcff9700 1 -- 172.21.15.77:0/2229283195 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f37c80202a0 con 0x7f37e019dc70 2024-03-16T10:09:00.654 INFO:teuthology.orchestra.run.smithi077.stdout:# minimal ceph.conf for c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:09:00.654 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-03-16T10:09:00.654 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:09:00.654 INFO:teuthology.orchestra.run.smithi077.stdout: mon_host = [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2024-03-16T10:09:00.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.655+0000 7f37e566f700 1 -- 172.21.15.77:0/2229283195 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f37cc039450 msgr2=0x7f37cc03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:00.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.655+0000 7f37e566f700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f37cc039450 0x7f37cc03b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f37d0009750 tx=0x7f37d0005fd0 comp rx=0 tx=0).stop 2024-03-16T10:09:00.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.655+0000 7f37e566f700 1 -- 172.21.15.77:0/2229283195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37e019dc70 msgr2=0x7f37e0197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:00.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.655+0000 7f37e566f700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37e019dc70 0x7f37e0197990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f37c800b780 tx=0x7f37c800bb40 comp rx=0 tx=0).stop 2024-03-16T10:09:00.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.655+0000 7f37e566f700 1 -- 172.21.15.77:0/2229283195 shutdown_connections 2024-03-16T10:09:00.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.655+0000 7f37e566f700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f37cc039450 0x7f37cc03b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:00.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.655+0000 7f37e566f700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37e0197ed0 0x7f37e0198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:00.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.655+0000 7f37e566f700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37e0102e50 0x7f37e019d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:00.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.655+0000 7f37e566f700 1 --2- 172.21.15.77:0/2229283195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37e019dc70 0x7f37e0197990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:00.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.655+0000 7f37e566f700 1 -- 172.21.15.77:0/2229283195 >> 172.21.15.77:0/2229283195 conn(0x7f37e00fc660 msgr2=0x7f37e0105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:09:00.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.655+0000 7f37e566f700 1 -- 172.21.15.77:0/2229283195 shutdown_connections 2024-03-16T10:09:00.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:00.655+0000 7f37e566f700 1 -- 172.21.15.77:0/2229283195 wait complete. 2024-03-16T10:09:00.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:09:00.926 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:09:01.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:09:01.810 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:01 smithi077 ceph-mon[32400]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:01.810 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:01 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2229283195' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:01.810 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:01 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:01.810 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:01 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:01.831 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:01 smithi192 ceph-mon[36354]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:01.831 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:01 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2229283195' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:01.831 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:01 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:01.832 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:01 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:01.930 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-16T10:09:01.930 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:09:01.930 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T10:09:01.960 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:09:01.960 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:09:02.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:01 smithi193 ceph-mon[36311]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:02.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:01 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2229283195' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:02.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:01 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:02.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:01 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:02.028 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-16T10:09:02.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T10:09:02.059 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-16T10:09:02.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:09:02.126 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-16T10:09:02.126 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T10:09:02.160 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-16T10:09:02.160 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T10:09:02.229 INFO:tasks.cephadm:Adding mgr.x on smithi077 2024-03-16T10:09:02.229 INFO:tasks.cephadm:Adding mgr.y on smithi192 2024-03-16T10:09:02.230 INFO:tasks.cephadm:Adding mgr.z on smithi193 2024-03-16T10:09:02.230 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi077=x;smithi192=y;smithi193=z' 2024-03-16T10:09:02.503 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:09:02.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:02 smithi192 ceph-mon[36354]: Reconfiguring mon.b (monmap changed)... 2024-03-16T10:09:02.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:02 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:09:02.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:02 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:09:02.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:02 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:02.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:02 smithi192 ceph-mon[36354]: Reconfiguring daemon mon.b on smithi192 2024-03-16T10:09:02.811 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:02 smithi193 ceph-mon[36311]: Reconfiguring mon.b (monmap changed)... 2024-03-16T10:09:02.811 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:02 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:09:02.811 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:02 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:09:02.811 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:02 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:02.811 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:02 smithi193 ceph-mon[36311]: Reconfiguring daemon mon.b on smithi192 2024-03-16T10:09:02.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:02 smithi077 ceph-mon[32400]: Reconfiguring mon.b (monmap changed)... 2024-03-16T10:09:02.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:02 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:09:02.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:02 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:09:02.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:02 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:02.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:02 smithi077 ceph-mon[32400]: Reconfiguring daemon mon.b on smithi192 2024-03-16T10:09:03.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:03 smithi193 ceph-mon[36311]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:03.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:03.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:03.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:09:03.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:09:03.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:03.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:03 smithi192 ceph-mon[36354]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:03.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:03.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:03.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:09:03.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:09:03.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:03.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:03 smithi077 ceph-mon[32400]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:03.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:03.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:03.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T10:09:03.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T10:09:03.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:04.036 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.036+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/1002325113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd81023c0 msgr2=0x7fdcd81027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:04.036 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.036+0000 7fdcdd2ce700 1 --2- 172.21.15.193:0/1002325113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd81023c0 0x7fdcd81027e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8009910 tx=0x7fdcc8009c20 comp rx=0 tx=0).stop 2024-03-16T10:09:04.036 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.036+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/1002325113 shutdown_connections 2024-03-16T10:09:04.036 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.036+0000 7fdcdd2ce700 1 --2- 172.21.15.193:0/1002325113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd81023c0 0x7fdcd81027e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:04.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.036+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/1002325113 >> 172.21.15.193:0/1002325113 conn(0x7fdcd80fba80 msgr2=0x7fdcd80fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:09:04.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.036+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/1002325113 shutdown_connections 2024-03-16T10:09:04.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.036+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/1002325113 wait complete. 2024-03-16T10:09:04.040 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.036+0000 7fdcdd2ce700 1 Processor -- start 2024-03-16T10:09:04.040 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcdd2ce700 1 -- start start 2024-03-16T10:09:04.040 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcdd2ce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81023c0 0x7fdcd81009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:04.040 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcdd2ce700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd819a370 0x7fdcd819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:04.040 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcdd2ce700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd819ccf0 0x7fdcd819f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:04.040 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcdd2ce700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd8106e40 con 0x7fdcd819a370 2024-03-16T10:09:04.040 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcdd2ce700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd8106cc0 con 0x7fdcd81023c0 2024-03-16T10:09:04.040 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcdd2ce700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd8106fc0 con 0x7fdcd819ccf0 2024-03-16T10:09:04.040 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcd77fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd819ccf0 0x7fdcd819f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:04.040 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcd77fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd819ccf0 0x7fdcd819f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:60702/0 (socket says 172.21.15.193:60702) 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcd77fe700 1 -- 172.21.15.193:0/3620060703 learned_addr learned my addr 172.21.15.193:0/3620060703 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcd67fc700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd819a370 0x7fdcd819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcd6ffd700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81023c0 0x7fdcd81009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcd77fe700 1 -- 172.21.15.193:0/3620060703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81023c0 msgr2=0x7fdcd81009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcd77fe700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81023c0 0x7fdcd81009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcd77fe700 1 -- 172.21.15.193:0/3620060703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd819a370 msgr2=0x7fdcd819c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcd77fe700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd819a370 0x7fdcd819c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcd77fe700 1 -- 172.21.15.193:0/3620060703 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd819fa50 con 0x7fdcd819ccf0 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcd6ffd700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81023c0 0x7fdcd81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.037+0000 7fdcd77fe700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd819ccf0 0x7fdcd819f180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000ba50 tx=0x7fdcd000bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.038+0000 7fdcbffff700 1 -- 172.21.15.193:0/3620060703 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0019070 con 0x7fdcd819ccf0 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.038+0000 7fdcbffff700 1 -- 172.21.15.193:0/3620060703 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcd0014440 con 0x7fdcd819ccf0 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.038+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/3620060703 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd819fd40 con 0x7fdcd819ccf0 2024-03-16T10:09:04.041 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.038+0000 7fdcbffff700 1 -- 172.21.15.193:0/3620060703 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd00075b0 con 0x7fdcd819ccf0 2024-03-16T10:09:04.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.038+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/3620060703 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd81a02b0 con 0x7fdcd819ccf0 2024-03-16T10:09:04.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.039+0000 7fdcbffff700 1 -- 172.21.15.193:0/3620060703 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fdcd001c070 con 0x7fdcd819ccf0 2024-03-16T10:09:04.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.039+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/3620060703 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcd8063900 con 0x7fdcd819ccf0 2024-03-16T10:09:04.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.039+0000 7fdcbffff700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdcc0039450 0x7fdcc003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:04.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.039+0000 7fdcbffff700 1 -- 172.21.15.193:0/3620060703 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdcd004b630 con 0x7fdcd819ccf0 2024-03-16T10:09:04.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.039+0000 7fdcd6ffd700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdcc0039450 0x7fdcc003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:04.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.040+0000 7fdcd6ffd700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdcc0039450 0x7fdcc003b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8005330 tx=0x7fdcc8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:09:04.044 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.044+0000 7fdcbffff700 1 -- 172.21.15.193:0/3620060703 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdcd0012500 con 0x7fdcd819ccf0 2024-03-16T10:09:04.252 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.252+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/3620060703 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi077=x;smithi192=y;smithi193=z", "target": ["mon-mgr", ""]}) -- 0x7fdcd80627a0 con 0x7fdcc0039450 2024-03-16T10:09:04.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.258+0000 7fdcbffff700 1 -- 172.21.15.193:0/3620060703 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fdcd80627a0 con 0x7fdcc0039450 2024-03-16T10:09:04.259 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mgr update... 2024-03-16T10:09:04.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.261+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/3620060703 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdcc0039450 msgr2=0x7fdcc003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:04.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.261+0000 7fdcdd2ce700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdcc0039450 0x7fdcc003b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8005330 tx=0x7fdcc8005fd0 comp rx=0 tx=0).stop 2024-03-16T10:09:04.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.261+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/3620060703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd819ccf0 msgr2=0x7fdcd819f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:04.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.261+0000 7fdcdd2ce700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd819ccf0 0x7fdcd819f180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000ba50 tx=0x7fdcd000bd60 comp rx=0 tx=0).stop 2024-03-16T10:09:04.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.261+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/3620060703 shutdown_connections 2024-03-16T10:09:04.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.261+0000 7fdcdd2ce700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdcc0039450 0x7fdcc003b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:04.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.261+0000 7fdcdd2ce700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd819ccf0 0x7fdcd819f180 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:04.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.261+0000 7fdcdd2ce700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81023c0 0x7fdcd81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:04.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.261+0000 7fdcdd2ce700 1 --2- 172.21.15.193:0/3620060703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd819a370 0x7fdcd819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:04.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.261+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/3620060703 >> 172.21.15.193:0/3620060703 conn(0x7fdcd80fba80 msgr2=0x7fdcd80fc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:09:04.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.261+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/3620060703 shutdown_connections 2024-03-16T10:09:04.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:09:04.261+0000 7fdcdd2ce700 1 -- 172.21.15.193:0/3620060703 wait complete. 2024-03-16T10:09:04.843 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:04 smithi193 ceph-mon[36311]: Reconfiguring mon.c (monmap changed)... 2024-03-16T10:09:04.843 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:04 smithi193 ceph-mon[36311]: Reconfiguring daemon mon.c on smithi193 2024-03-16T10:09:04.843 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:04 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:04.868 DEBUG:teuthology.orchestra.run.smithi192:mgr.y> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.y.service 2024-03-16T10:09:04.870 DEBUG:teuthology.orchestra.run.smithi193:mgr.z> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.z.service 2024-03-16T10:09:04.872 INFO:tasks.cephadm:Deploying OSDs... 2024-03-16T10:09:04.873 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:09:04.873 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T10:09:04.893 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T10:09:04.893 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-03-16T10:09:04.896 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:04 smithi192 ceph-mon[36354]: Reconfiguring mon.c (monmap changed)... 2024-03-16T10:09:04.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:04 smithi192 ceph-mon[36354]: Reconfiguring daemon mon.c on smithi193 2024-03-16T10:09:04.897 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:04 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:04.950 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T10:09:04.950 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:09:04.950 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-16T10:09:04.950 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:09:04.950 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:09:04.950 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-16 10:08:08.609109014 +0000 2024-03-16T10:09:04.951 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-16 10:03:16.384222091 +0000 2024-03-16T10:09:04.951 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-16 10:03:16.384222091 +0000 2024-03-16T10:09:04.951 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-16 10:03:16.384222091 +0000 2024-03-16T10:09:04.951 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T10:09:04.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:04 smithi077 ceph-mon[32400]: Reconfiguring mon.c (monmap changed)... 2024-03-16T10:09:04.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:04 smithi077 ceph-mon[32400]: Reconfiguring daemon mon.c on smithi193 2024-03-16T10:09:04.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:04 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:05.021 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-16T10:09:05.021 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-16T10:09:05.021 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000295526 s, 1.7 MB/s 2024-03-16T10:09:05.023 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T10:09:05.081 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-03-16T10:09:05.138 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T10:09:05.138 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:09:05.138 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-16T10:09:05.138 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:09:05.138 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:09:05.138 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-16 10:08:08.642108302 +0000 2024-03-16T10:09:05.138 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-16 10:03:16.733214543 +0000 2024-03-16T10:09:05.138 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-16 10:03:16.733214543 +0000 2024-03-16T10:09:05.139 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-16 10:03:16.733214543 +0000 2024-03-16T10:09:05.139 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T10:09:05.206 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-16T10:09:05.206 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-16T10:09:05.206 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000202376 s, 2.5 MB/s 2024-03-16T10:09:05.207 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T10:09:05.264 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-03-16T10:09:05.320 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T10:09:05.320 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:09:05.320 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-16T10:09:05.321 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:09:05.321 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:09:05.321 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-16 10:08:08.675107589 +0000 2024-03-16T10:09:05.321 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-16 10:03:17.062207428 +0000 2024-03-16T10:09:05.321 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-16 10:03:17.062207428 +0000 2024-03-16T10:09:05.321 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-16 10:03:17.062207428 +0000 2024-03-16T10:09:05.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T10:09:05.388 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-16T10:09:05.388 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-16T10:09:05.389 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000197961 s, 2.6 MB/s 2024-03-16T10:09:05.390 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T10:09:05.448 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-03-16T10:09:05.503 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T10:09:05.503 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:09:05.503 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-16T10:09:05.503 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:09:05.503 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:09:05.503 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-16 10:08:08.705106941 +0000 2024-03-16T10:09:05.503 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-16 10:03:17.400200119 +0000 2024-03-16T10:09:05.504 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-16 10:03:17.400200119 +0000 2024-03-16T10:09:05.504 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-16 10:03:17.400200119 +0000 2024-03-16T10:09:05.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T10:09:05.571 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-03-16T10:09:05.571 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-03-16T10:09:05.571 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000175561 s, 2.9 MB/s 2024-03-16T10:09:05.572 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T10:09:05.628 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-16T10:09:05.628 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T10:09:05.646 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T10:09:05.646 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-03-16T10:09:05.702 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T10:09:05.702 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:09:05.703 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-16T10:09:05.703 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:09:05.703 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:09:05.703 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-16 10:08:51.041671457 +0000 2024-03-16T10:09:05.703 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-16 10:03:40.164486544 +0000 2024-03-16T10:09:05.703 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-16 10:03:40.164486544 +0000 2024-03-16T10:09:05.703 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-16 10:03:40.164486544 +0000 2024-03-16T10:09:05.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T10:09:05.744 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:05 smithi077 ceph-mon[32400]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi077=x;smithi192=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:09:05.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:05 smithi077 ceph-mon[32400]: Saving service mgr spec with placement smithi077=x;smithi192=y;smithi193=z;count:3 2024-03-16T10:09:05.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:05 smithi077 ceph-mon[32400]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:05.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:05 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:05.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:05 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:05.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:05 smithi193 ceph-mon[36311]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi077=x;smithi192=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:09:05.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:05 smithi193 ceph-mon[36311]: Saving service mgr spec with placement smithi077=x;smithi192=y;smithi193=z;count:3 2024-03-16T10:09:05.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:05 smithi193 ceph-mon[36311]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:05.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:05 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:05.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:05 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:05.771 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-16T10:09:05.771 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-16T10:09:05.771 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000314485 s, 1.6 MB/s 2024-03-16T10:09:05.772 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T10:09:05.830 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-03-16T10:09:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T10:09:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:09:05.887 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-16T10:09:05.887 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:09:05.887 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:09:05.887 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-16 10:08:51.042671438 +0000 2024-03-16T10:09:05.887 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-16 10:03:40.519479411 +0000 2024-03-16T10:09:05.887 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-16 10:03:40.519479411 +0000 2024-03-16T10:09:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-16 10:03:40.519479411 +0000 2024-03-16T10:09:05.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T10:09:05.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:05 smithi192 ceph-mon[36354]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi077=x;smithi192=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:09:05.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:05 smithi192 ceph-mon[36354]: Saving service mgr spec with placement smithi077=x;smithi192=y;smithi193=z;count:3 2024-03-16T10:09:05.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:05 smithi192 ceph-mon[36354]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:05.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:05 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:05.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:05 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:05.943 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-16T10:09:05.943 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-16T10:09:05.943 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.00022448 s, 2.3 MB/s 2024-03-16T10:09:05.944 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T10:09:06.001 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-03-16T10:09:06.056 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T10:09:06.057 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:09:06.057 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-16T10:09:06.057 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:09:06.057 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:09:06.057 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-16 10:08:51.042671438 +0000 2024-03-16T10:09:06.057 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-16 10:03:40.858472599 +0000 2024-03-16T10:09:06.057 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-16 10:03:40.858472599 +0000 2024-03-16T10:09:06.057 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-16 10:03:40.858472599 +0000 2024-03-16T10:09:06.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T10:09:06.123 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-16T10:09:06.123 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-16T10:09:06.123 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000308144 s, 1.7 MB/s 2024-03-16T10:09:06.124 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T10:09:06.180 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-03-16T10:09:06.234 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T10:09:06.234 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:09:06.235 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-16T10:09:06.235 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:09:06.235 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:09:06.235 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-16 10:08:51.042671438 +0000 2024-03-16T10:09:06.235 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-16 10:03:41.199465747 +0000 2024-03-16T10:09:06.235 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-16 10:03:41.199465747 +0000 2024-03-16T10:09:06.235 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-16 10:03:41.199465747 +0000 2024-03-16T10:09:06.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T10:09:06.299 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-16T10:09:06.299 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-16T10:09:06.299 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.00019346 s, 2.6 MB/s 2024-03-16T10:09:06.301 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T10:09:06.355 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-16T10:09:06.355 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T10:09:06.370 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T10:09:06.370 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-03-16T10:09:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T10:09:06.425 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:09:06.425 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-16T10:09:06.425 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:09:06.425 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:09:06.425 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-16 10:08:42.288430549 +0000 2024-03-16T10:09:06.426 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-16 10:03:45.269979913 +0000 2024-03-16T10:09:06.426 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-16 10:03:45.269979913 +0000 2024-03-16T10:09:06.426 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-16 10:03:45.269979913 +0000 2024-03-16T10:09:06.426 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T10:09:06.491 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-16T10:09:06.491 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-16T10:09:06.491 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000256973 s, 2.0 MB/s 2024-03-16T10:09:06.492 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T10:09:06.548 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-03-16T10:09:06.603 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T10:09:06.603 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:09:06.603 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-16T10:09:06.603 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:09:06.603 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:09:06.603 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-16 10:08:42.289430533 +0000 2024-03-16T10:09:06.603 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-16 10:03:45.621974301 +0000 2024-03-16T10:09:06.604 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-16 10:03:45.621974301 +0000 2024-03-16T10:09:06.604 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-16 10:03:45.621974301 +0000 2024-03-16T10:09:06.604 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T10:09:06.671 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-16T10:09:06.672 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-16T10:09:06.672 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000239241 s, 2.1 MB/s 2024-03-16T10:09:06.673 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T10:09:06.728 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-03-16T10:09:06.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:06 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:06.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:06 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:06.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:06 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:06.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:06 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:06.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:06 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:09:06.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:06 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T10:09:06.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:06 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:09:06.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:06 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:06.767 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T10:09:06.767 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:09:06.767 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-16T10:09:06.767 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:09:06.767 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:09:06.767 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-16 10:08:42.289430533 +0000 2024-03-16T10:09:06.767 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-16 10:03:45.957968945 +0000 2024-03-16T10:09:06.767 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-16 10:03:45.957968945 +0000 2024-03-16T10:09:06.767 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-16 10:03:45.957968945 +0000 2024-03-16T10:09:06.767 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T10:09:06.832 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-16T10:09:06.832 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-16T10:09:06.832 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000197153 s, 2.6 MB/s 2024-03-16T10:09:06.833 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T10:09:06.888 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-03-16T10:09:06.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:06 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:06.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:06 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:06.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:06 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:06.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:06 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:06.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:06 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:09:06.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:06 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T10:09:06.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:06 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:09:06.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:06 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:06.943 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T10:09:06.943 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T10:09:06.943 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-16T10:09:06.943 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:09:06.943 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T10:09:06.943 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-16 10:08:42.289430533 +0000 2024-03-16T10:09:06.944 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-16 10:03:46.291963620 +0000 2024-03-16T10:09:06.944 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-16 10:03:46.291963620 +0000 2024-03-16T10:09:06.944 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-16 10:03:46.291963620 +0000 2024-03-16T10:09:06.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T10:09:06.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:06 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:06.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:06 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:06.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:06 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:06.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:06 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:06.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:06 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:09:06.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:06 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T10:09:06.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:06 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:09:06.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:06 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:07.008 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-16T10:09:07.008 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-16T10:09:07.008 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000210002 s, 2.4 MB/s 2024-03-16T10:09:07.010 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T10:09:07.065 INFO:tasks.cephadm:Deploying osd.0 on smithi077 with /dev/vg_nvme/lv_4... 2024-03-16T10:09:07.065 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T10:09:07.298 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:09:07.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:07 smithi077 ceph-mon[32400]: Deploying daemon mgr.z on smithi193 2024-03-16T10:09:07.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:07 smithi077 ceph-mon[32400]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:07.843 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:07 smithi193 ceph-mon[36311]: Deploying daemon mgr.z on smithi193 2024-03-16T10:09:07.843 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:07 smithi193 ceph-mon[36311]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:07.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:07 smithi192 ceph-mon[36354]: Deploying daemon mgr.z on smithi193 2024-03-16T10:09:07.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:07 smithi192 ceph-mon[36354]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:08.226 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:08 smithi193 systemd[1]: Starting Ceph mgr.z for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:09:08.492 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:08 smithi193 podman[38153]: 2024-03-16 10:09:08.129031792 +0000 UTC m=+0.018090592 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:09:08.492 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:08 smithi193 podman[38153]: 2024-03-16 10:09:08.246598877 +0000 UTC m=+0.135657669 container create 2fdf6e064b0be34be34ed9180e57fd87c4755b710d8ddb0ae1c559905b5f1395 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-16T10:09:08.492 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:08 smithi193 podman[38153]: 2024-03-16 10:09:08.465604102 +0000 UTC m=+0.354662913 container init 2fdf6e064b0be34be34ed9180e57fd87c4755b710d8ddb0ae1c559905b5f1395 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-16T10:09:08.492 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:08 smithi193 podman[38153]: 2024-03-16 10:09:08.468132366 +0000 UTC m=+0.357191151 container start 2fdf6e064b0be34be34ed9180e57fd87c4755b710d8ddb0ae1c559905b5f1395 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307) 2024-03-16T10:09:08.753 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:08 smithi193 ceph-mgr[38172]: -- 172.21.15.193:0/66826267 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b1353565a0 con 0x55b13532ac00 2024-03-16T10:09:08.753 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:08 smithi193 bash[38153]: 2fdf6e064b0be34be34ed9180e57fd87c4755b710d8ddb0ae1c559905b5f1395 2024-03-16T10:09:08.753 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:08 smithi193 systemd[1]: Started Ceph mgr.z for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:09:09.155 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T10:09:09.155 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T10:09:09.155 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-03-16T10:09:09.155 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-03-16T10:09:09.155 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126943 s, 826 MB/s 2024-03-16T10:09:09.156 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-03-16T10:09:09.252 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:09.001+0000 7f1960e94200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T10:09:09.253 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:09.091+0000 7f1960e94200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T10:09:09.730 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:09.730 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:09.730 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:09.730 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:09:09.730 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T10:09:09.731 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:09:09.731 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:09.731 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:09 smithi077 ceph-mon[32400]: Deploying daemon mgr.y on smithi192 2024-03-16T10:09:09.731 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:09 smithi077 ceph-mon[32400]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:09.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_4 2024-03-16T10:09:09.872 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:09.727+0000 7f1960e94200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T10:09:09.872 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:09.820+0000 7f1960e94200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T10:09:09.872 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:09.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:09.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:09.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:09:09.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T10:09:09.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:09:09.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:09.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-mon[36311]: Deploying daemon mgr.y on smithi192 2024-03-16T10:09:09.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-mon[36311]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:09.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:09.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:09.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:09.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:09:09.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T10:09:09.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:09:09.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:09.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:09 smithi192 ceph-mon[36354]: Deploying daemon mgr.y on smithi192 2024-03-16T10:09:09.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:09 smithi192 ceph-mon[36354]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:09.994 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:09:10.153 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:09 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:09.968+0000 7f1960e94200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T10:09:10.153 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:10 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:10.068+0000 7f1960e94200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T10:09:10.153 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:10 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:10.152+0000 7f1960e94200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T10:09:10.709 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:10 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:10.418+0000 7f1960e94200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T10:09:10.710 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:10 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:10.527+0000 7f1960e94200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T10:09:10.710 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:10 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:10.612+0000 7f1960e94200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T10:09:10.710 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:10 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:10.708+0000 7f1960e94200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T10:09:10.876 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:10 smithi192 systemd[1]: Starting Ceph mgr.y for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:09:11.003 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:10 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:10.885+0000 7f1960e94200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T10:09:11.153 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:10 smithi192 podman[37734]: 2024-03-16 10:09:10.875739989 +0000 UTC m=+0.106317708 container create 48b8155497ba5445f552cda164d4ff31640bbe63c3bb99d11bf575238c77fbcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:09:11.153 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:10 smithi192 podman[37734]: 2024-03-16 10:09:10.787483285 +0000 UTC m=+0.018061026 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:09:11.154 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:11 smithi192 podman[37734]: 2024-03-16 10:09:11.078379704 +0000 UTC m=+0.308957433 container init 48b8155497ba5445f552cda164d4ff31640bbe63c3bb99d11bf575238c77fbcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T10:09:11.154 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:11 smithi192 podman[37734]: 2024-03-16 10:09:11.081021057 +0000 UTC m=+0.311598790 container start 48b8155497ba5445f552cda164d4ff31640bbe63c3bb99d11bf575238c77fbcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:09:11.154 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:11 smithi192 ceph-mgr[37753]: -- 172.21.15.192:0/4127802455 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562cf7c8a5a0 con 0x562cf7c5ec00 2024-03-16T10:09:11.416 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:11 smithi192 bash[37734]: 48b8155497ba5445f552cda164d4ff31640bbe63c3bb99d11bf575238c77fbcf 2024-03-16T10:09:11.417 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:11 smithi192 systemd[1]: Started Ceph mgr.y for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:09:11.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.491+0000 7f043a6dc700 1 -- 172.21.15.77:0/3542153806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04341034f0 msgr2=0x7f0434103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:11.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.491+0000 7f043a6dc700 1 --2- 172.21.15.77:0/3542153806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04341034f0 0x7f0434103950 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f04280099c0 tx=0x7f0428009cd0 comp rx=0 tx=0).stop 2024-03-16T10:09:11.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.491+0000 7f043a6dc700 1 -- 172.21.15.77:0/3542153806 shutdown_connections 2024-03-16T10:09:11.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.491+0000 7f043a6dc700 1 --2- 172.21.15.77:0/3542153806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0434103e90 0x7f043410a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:11.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.491+0000 7f043a6dc700 1 --2- 172.21.15.77:0/3542153806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04341022f0 0x7f0434102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:11.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.491+0000 7f043a6dc700 1 --2- 172.21.15.77:0/3542153806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04341034f0 0x7f0434103950 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:11.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.491+0000 7f043a6dc700 1 -- 172.21.15.77:0/3542153806 >> 172.21.15.77:0/3542153806 conn(0x7f04340fd8b0 msgr2=0x7f04340ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:09:11.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.492+0000 7f043a6dc700 1 -- 172.21.15.77:0/3542153806 shutdown_connections 2024-03-16T10:09:11.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.492+0000 7f043a6dc700 1 -- 172.21.15.77:0/3542153806 wait complete. 2024-03-16T10:09:11.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.492+0000 7f043a6dc700 1 Processor -- start 2024-03-16T10:09:11.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.492+0000 7f043a6dc700 1 -- start start 2024-03-16T10:09:11.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.492+0000 7f043a6dc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04341022f0 0x7f0434076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:11.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.492+0000 7f043a6dc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04341034f0 0x7f04340746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:11.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.492+0000 7f043a6dc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0434103e90 0x7f0434074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.492+0000 7f043a6dc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f043410df50 con 0x7f0434103e90 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.492+0000 7f043a6dc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f043410ddd0 con 0x7f04341022f0 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.492+0000 7f043a6dc700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f043410e0d0 con 0x7f04341034f0 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f0438c79700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0434103e90 0x7f0434074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f0438c79700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0434103e90 0x7f0434074c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44898/0 (socket says 172.21.15.77:44898) 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f0438c79700 1 -- 172.21.15.77:0/2820421901 learned_addr learned my addr 172.21.15.77:0/2820421901 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f04337fe700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04341034f0 0x7f04340746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f0433fff700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04341022f0 0x7f0434076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f0438c79700 1 -- 172.21.15.77:0/2820421901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04341034f0 msgr2=0x7f04340746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f0438c79700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04341034f0 0x7f04340746c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f0438c79700 1 -- 172.21.15.77:0/2820421901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04341022f0 msgr2=0x7f0434076070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f0438c79700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04341022f0 0x7f0434076070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:11.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f0438c79700 1 -- 172.21.15.77:0/2820421901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04340754b0 con 0x7f0434103e90 2024-03-16T10:09:11.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f0438c79700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0434103e90 0x7f0434074c00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f042c00b970 tx=0x7f042c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:09:11.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f04317fa700 1 -- 172.21.15.77:0/2820421901 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c00d860 con 0x7f0434103e90 2024-03-16T10:09:11.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f04317fa700 1 -- 172.21.15.77:0/2820421901 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f042c00e430 con 0x7f0434103e90 2024-03-16T10:09:11.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f043a6dc700 1 -- 172.21.15.77:0/2820421901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04341a26c0 con 0x7f0434103e90 2024-03-16T10:09:11.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f04317fa700 1 -- 172.21.15.77:0/2820421901 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c017660 con 0x7f0434103e90 2024-03-16T10:09:11.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.493+0000 7f043a6dc700 1 -- 172.21.15.77:0/2820421901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04341a2c00 con 0x7f0434103e90 2024-03-16T10:09:11.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.494+0000 7f04317fa700 1 -- 172.21.15.77:0/2820421901 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f042c0203f0 con 0x7f0434103e90 2024-03-16T10:09:11.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.494+0000 7f043a6dc700 1 -- 172.21.15.77:0/2820421901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0434068f90 con 0x7f0434103e90 2024-03-16T10:09:11.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.494+0000 7f04317fa700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f041c039200 0x7f041c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:11.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.494+0000 7f04317fa700 1 -- 172.21.15.77:0/2820421901 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f042c04bcb0 con 0x7f0434103e90 2024-03-16T10:09:11.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.495+0000 7f0433fff700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f041c039200 0x7f041c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:11.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.495+0000 7f0433fff700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f041c039200 0x7f041c03b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0434103350 tx=0x7f042400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:09:11.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.499+0000 7f04317fa700 1 -- 172.21.15.77:0/2820421901 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f042c013030 con 0x7f0434103e90 2024-03-16T10:09:11.704 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:11 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:11.661+0000 7fd848e43200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T10:09:11.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:11.704+0000 7f043a6dc700 1 -- 172.21.15.77:0/2820421901 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0434062710 con 0x7f041c039200 2024-03-16T10:09:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:11 smithi077 ceph-mon[32400]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:12.003 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:11 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:11.865+0000 7f1960e94200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T10:09:12.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:11 smithi193 ceph-mon[36311]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:12.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:12.167 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:11 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:11.761+0000 7fd848e43200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T10:09:12.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:11 smithi192 ceph-mon[36354]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:12.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:12.264 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:12.008+0000 7f1960e94200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T10:09:12.643 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:12.350+0000 7f1960e94200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T10:09:12.643 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:12.440+0000 7f1960e94200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T10:09:12.643 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:12.641+0000 7f1960e94200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:09:12.760 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:12.508+0000 7fd848e43200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T10:09:12.760 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:12.613+0000 7fd848e43200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T10:09:12.760 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-mon[36354]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:09:12.760 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:09:12.760 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:09:12.760 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:12.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:12.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:12.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:12 smithi077 ceph-mon[32400]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:09:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:09:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:09:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:13.002 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:12.826+0000 7f1960e94200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T10:09:13.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-mon[36311]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:09:13.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:09:13.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:09:13.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:13.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:13.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:13.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:13.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:13.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:13.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:13.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:13.167 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:12.759+0000 7fd848e43200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T10:09:13.167 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:12.854+0000 7fd848e43200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T10:09:13.167 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:12 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:12.936+0000 7fd848e43200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T10:09:13.366 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:13 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:13.105+0000 7f1960e94200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:09:13.366 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:13 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:13.364+0000 7f1960e94200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T10:09:13.488 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:13.199+0000 7fd848e43200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T10:09:13.488 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:13.308+0000 7fd848e43200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T10:09:13.489 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:13.393+0000 7fd848e43200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T10:09:13.489 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:13.487+0000 7fd848e43200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T10:09:13.706 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:13 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:13.599+0000 7f1960e94200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T10:09:13.917 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:13.664+0000 7fd848e43200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T10:09:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:13 smithi192 ceph-mon[36354]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:13 smithi192 ceph-mon[36354]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T10:09:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:13 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:09:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:13 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:09:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:13 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:13 smithi192 ceph-mon[36354]: Reconfiguring daemon mgr.x on smithi077 2024-03-16T10:09:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:13 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:13.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:13 smithi077 ceph-mon[32400]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:13.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:13 smithi077 ceph-mon[32400]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T10:09:13.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:13 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:09:13.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:13 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:09:13.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:13 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:13.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:13 smithi077 ceph-mon[32400]: Reconfiguring daemon mgr.x on smithi077 2024-03-16T10:09:13.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:13 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:14.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:13 smithi193 ceph-mon[36311]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:14.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:13 smithi193 ceph-mon[36311]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T10:09:14.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:13 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T10:09:14.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:13 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T10:09:14.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:13 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:14.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:13 smithi193 ceph-mon[36311]: Reconfiguring daemon mgr.x on smithi077 2024-03-16T10:09:14.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:13 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:14.707 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:14 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:14.395+0000 7f1960e94200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T10:09:14.917 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:14 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:14.636+0000 7fd848e43200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T10:09:14.917 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:14 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:14.764+0000 7fd848e43200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T10:09:14.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:14 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/292637568' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b40ed61e-36f0-43e0-8f47-e80bc6001587"}]: dispatch 2024-03-16T10:09:14.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:14 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/292637568' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b40ed61e-36f0-43e0-8f47-e80bc6001587"}]': finished 2024-03-16T10:09:14.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:14 smithi192 ceph-mon[36354]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T10:09:14.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:14 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:14.969 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:14 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/292637568' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b40ed61e-36f0-43e0-8f47-e80bc6001587"}]: dispatch 2024-03-16T10:09:14.969 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:14 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/292637568' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b40ed61e-36f0-43e0-8f47-e80bc6001587"}]': finished 2024-03-16T10:09:14.969 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:14 smithi193 ceph-mon[36311]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T10:09:14.969 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:14 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:14.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:14 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/292637568' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b40ed61e-36f0-43e0-8f47-e80bc6001587"}]: dispatch 2024-03-16T10:09:14.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:14 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/292637568' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b40ed61e-36f0-43e0-8f47-e80bc6001587"}]': finished 2024-03-16T10:09:14.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:14 smithi077 ceph-mon[32400]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T10:09:14.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:15.253 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:14 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:14.967+0000 7f1960e94200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T10:09:15.253 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:15 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:15.063+0000 7f1960e94200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T10:09:15.380 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:15.098+0000 7fd848e43200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T10:09:15.381 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:15.187+0000 7fd848e43200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T10:09:15.381 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:15.379+0000 7fd848e43200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:09:15.666 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:15.565+0000 7fd848e43200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T10:09:15.708 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:09:15 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z[38168]: 2024-03-16T10:09:15.278+0000 7f1960e94200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T10:09:15.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:15 smithi077 ceph-mon[32400]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:15.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:15 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/335866377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:09:15.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:15 smithi077 ceph-mon[32400]: Standby manager daemon z started 2024-03-16T10:09:15.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:15 smithi077 ceph-mon[32400]: from='mgr.? 172.21.15.193:0/2640545274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T10:09:15.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:15 smithi077 ceph-mon[32400]: from='mgr.? 172.21.15.193:0/2640545274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T10:09:15.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:15 smithi077 ceph-mon[32400]: from='mgr.? 172.21.15.193:0/2640545274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T10:09:15.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:15 smithi077 ceph-mon[32400]: from='mgr.? 172.21.15.193:0/2640545274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T10:09:16.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:15 smithi193 ceph-mon[36311]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:16.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:15 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/335866377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:09:16.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:15 smithi193 ceph-mon[36311]: Standby manager daemon z started 2024-03-16T10:09:16.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:15 smithi193 ceph-mon[36311]: from='mgr.? 172.21.15.193:0/2640545274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T10:09:16.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:15 smithi193 ceph-mon[36311]: from='mgr.? 172.21.15.193:0/2640545274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T10:09:16.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:15 smithi193 ceph-mon[36311]: from='mgr.? 172.21.15.193:0/2640545274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T10:09:16.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:15 smithi193 ceph-mon[36311]: from='mgr.? 172.21.15.193:0/2640545274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T10:09:16.109 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:15.847+0000 7fd848e43200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T10:09:16.109 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:16 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:16.108+0000 7fd848e43200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T10:09:16.109 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:15 smithi192 ceph-mon[36354]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:16.109 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:15 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/335866377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:09:16.109 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:15 smithi192 ceph-mon[36354]: Standby manager daemon z started 2024-03-16T10:09:16.109 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:15 smithi192 ceph-mon[36354]: from='mgr.? 172.21.15.193:0/2640545274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T10:09:16.109 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:15 smithi192 ceph-mon[36354]: from='mgr.? 172.21.15.193:0/2640545274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T10:09:16.110 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:15 smithi192 ceph-mon[36354]: from='mgr.? 172.21.15.193:0/2640545274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T10:09:16.110 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:15 smithi192 ceph-mon[36354]: from='mgr.? 172.21.15.193:0/2640545274' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T10:09:16.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:16.249+0000 7f04317fa700 1 -- 172.21.15.77:0/2820421901 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 14) ==== 91698+0+0 (secure 0 0 0) 0x7f042c015c20 con 0x7f0434103e90 2024-03-16T10:09:16.416 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:16 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:16.341+0000 7fd848e43200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T10:09:17.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:17 smithi192 ceph-mon[36354]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-16T10:09:17.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:17 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T10:09:17.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:17 smithi192 ceph-mon[36354]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:17.417 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:17 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:17.132+0000 7fd848e43200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T10:09:17.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:17 smithi077 ceph-mon[32400]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-16T10:09:17.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:17 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T10:09:17.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:17 smithi077 ceph-mon[32400]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:17.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:17 smithi193 ceph-mon[36311]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-16T10:09:17.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:17 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T10:09:17.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:17 smithi193 ceph-mon[36311]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:18.010 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:17 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:17.702+0000 7fd848e43200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T10:09:18.011 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:17 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:17.796+0000 7fd848e43200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T10:09:18.011 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:09:18 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y[37749]: 2024-03-16T10:09:18.009+0000 7fd848e43200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T10:09:18.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:18.265+0000 7f04317fa700 1 -- 172.21.15.77:0/2820421901 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f042c020c40 con 0x7f0434103e90 2024-03-16T10:09:18.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:18 smithi192 ceph-mon[36354]: Standby manager daemon y started 2024-03-16T10:09:18.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:18 smithi192 ceph-mon[36354]: from='mgr.? 172.21.15.192:0/3604523160' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T10:09:18.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:18 smithi192 ceph-mon[36354]: from='mgr.? 172.21.15.192:0/3604523160' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T10:09:18.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:18 smithi192 ceph-mon[36354]: from='mgr.? 172.21.15.192:0/3604523160' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T10:09:18.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:18 smithi192 ceph-mon[36354]: from='mgr.? 172.21.15.192:0/3604523160' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T10:09:18.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:18 smithi077 ceph-mon[32400]: Standby manager daemon y started 2024-03-16T10:09:18.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:18 smithi077 ceph-mon[32400]: from='mgr.? 172.21.15.192:0/3604523160' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T10:09:18.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:18 smithi077 ceph-mon[32400]: from='mgr.? 172.21.15.192:0/3604523160' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T10:09:18.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:18 smithi077 ceph-mon[32400]: from='mgr.? 172.21.15.192:0/3604523160' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T10:09:18.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:18 smithi077 ceph-mon[32400]: from='mgr.? 172.21.15.192:0/3604523160' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T10:09:18.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:18 smithi193 ceph-mon[36311]: Standby manager daemon y started 2024-03-16T10:09:18.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:18 smithi193 ceph-mon[36311]: from='mgr.? 172.21.15.192:0/3604523160' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T10:09:18.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:18 smithi193 ceph-mon[36311]: from='mgr.? 172.21.15.192:0/3604523160' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T10:09:18.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:18 smithi193 ceph-mon[36311]: from='mgr.? 172.21.15.192:0/3604523160' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T10:09:18.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:18 smithi193 ceph-mon[36311]: from='mgr.? 172.21.15.192:0/3604523160' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T10:09:19.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:19 smithi077 ceph-mon[32400]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-16T10:09:19.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T10:09:19.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:19 smithi077 ceph-mon[32400]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:19.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:19 smithi192 ceph-mon[36354]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-16T10:09:19.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T10:09:19.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:19 smithi192 ceph-mon[36354]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:19.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:19 smithi193 ceph-mon[36311]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-16T10:09:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T10:09:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:19 smithi193 ceph-mon[36311]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:20.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:20 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:20.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:20 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:20.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:20 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:20.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:20 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:20.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:20 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:20.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:20 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:20.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:20 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:20.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:20 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:20.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:20 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:20.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:20 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:20.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:20 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:20.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:20 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:20.998 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:20 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:20.998 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:20 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:20.998 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:20 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:20.998 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:20 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:20.998 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:20 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:20.999 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:20 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:21.893 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:21 smithi192 ceph-mon[36354]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:21.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:21 smithi077 ceph-mon[32400]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:22.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:21 smithi193 ceph-mon[36311]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:22.896 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T10:09:22.896 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:22.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T10:09:22.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:23.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T10:09:23.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:23.828 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:23 smithi077 ceph-mon[32400]: Deploying daemon osd.0 on smithi077 2024-03-16T10:09:23.828 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:23 smithi077 ceph-mon[32400]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:23.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:23 smithi192 ceph-mon[36354]: Deploying daemon osd.0 on smithi077 2024-03-16T10:09:23.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:23 smithi192 ceph-mon[36354]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:24.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:23 smithi193 ceph-mon[36311]: Deploying daemon osd.0 on smithi077 2024-03-16T10:09:24.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:23 smithi193 ceph-mon[36311]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:25.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:25 smithi077 ceph-mon[32400]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:26.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:25 smithi193 ceph-mon[36311]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:26.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:25 smithi192 ceph-mon[36354]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:27.998 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:27 smithi077 ceph-mon[32400]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:27.998 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:27.998 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:27.998 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:28.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:27 smithi193 ceph-mon[36311]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:28.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:28.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:28.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:28.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:27 smithi192 ceph-mon[36354]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:28.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:28.168 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:28.168 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:29.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:29 smithi077 ceph-mon[32400]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:30.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:29 smithi193 ceph-mon[36311]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:30.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:29 smithi192 ceph-mon[36354]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:31.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.121+0000 7f04317fa700 1 -- 172.21.15.77:0/2820421901 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0434062710 con 0x7f041c039200 2024-03-16T10:09:31.123 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 0 on host 'smithi077' 2024-03-16T10:09:31.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.123+0000 7f043a6dc700 1 -- 172.21.15.77:0/2820421901 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f041c039200 msgr2=0x7f041c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:31.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.123+0000 7f043a6dc700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f041c039200 0x7f041c03b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0434103350 tx=0x7f042400b410 comp rx=0 tx=0).stop 2024-03-16T10:09:31.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.123+0000 7f043a6dc700 1 -- 172.21.15.77:0/2820421901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0434103e90 msgr2=0x7f0434074c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:31.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.123+0000 7f043a6dc700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0434103e90 0x7f0434074c00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f042c00b970 tx=0x7f042c00bc80 comp rx=0 tx=0).stop 2024-03-16T10:09:31.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.124+0000 7f043a6dc700 1 -- 172.21.15.77:0/2820421901 shutdown_connections 2024-03-16T10:09:31.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.124+0000 7f043a6dc700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f041c039200 0x7f041c03b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:31.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.124+0000 7f043a6dc700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0434103e90 0x7f0434074c00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:31.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.124+0000 7f043a6dc700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04341022f0 0x7f0434076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:31.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.124+0000 7f043a6dc700 1 --2- 172.21.15.77:0/2820421901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04341034f0 0x7f04340746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:31.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.124+0000 7f043a6dc700 1 -- 172.21.15.77:0/2820421901 >> 172.21.15.77:0/2820421901 conn(0x7f04340fd8b0 msgr2=0x7f04341067f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:09:31.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.124+0000 7f043a6dc700 1 -- 172.21.15.77:0/2820421901 shutdown_connections 2024-03-16T10:09:31.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:09:31.124+0000 7f043a6dc700 1 -- 172.21.15.77:0/2820421901 wait complete. 2024-03-16T10:09:31.311 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.311 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.311 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:31.311 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:31.311 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.311 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:31 smithi077 ceph-mon[32400]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:31.311 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.311 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.311 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:31.311 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:31 smithi077 ceph-mon[32400]: from='osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T10:09:31.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:31.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:31.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:31 smithi193 ceph-mon[36311]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:31.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:31.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:31 smithi193 ceph-mon[36311]: from='osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T10:09:31.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:31.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:31.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:31 smithi192 ceph-mon[36354]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:31.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:31.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:31.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:31 smithi192 ceph-mon[36354]: from='osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T10:09:31.779 DEBUG:teuthology.orchestra.run.smithi077:osd.0> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.0.service 2024-03-16T10:09:31.781 INFO:tasks.cephadm:Deploying osd.1 on smithi192 with /dev/vg_nvme/lv_4... 2024-03-16T10:09:31.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T10:09:32.022 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:09:33.232 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:33 smithi077 ceph-mon[32400]: from='osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T10:09:33.232 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:33 smithi077 ceph-mon[32400]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T10:09:33.232 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:33 smithi077 ceph-mon[32400]: from='osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T10:09:33.233 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:33.233 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:33 smithi077 ceph-mon[32400]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:33.233 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:09:33 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0[44735]: 2024-03-16T10:09:33.106+0000 7f56857c3700 -1 osd.0 0 waiting for initial osdmap 2024-03-16T10:09:33.233 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:09:33 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0[44735]: 2024-03-16T10:09:33.111+0000 7f56815b6700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:09:33.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:33 smithi192 ceph-mon[36354]: from='osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T10:09:33.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:33 smithi192 ceph-mon[36354]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T10:09:33.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:33 smithi192 ceph-mon[36354]: from='osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T10:09:33.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:33.334 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:33 smithi192 ceph-mon[36354]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:33.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:33 smithi193 ceph-mon[36311]: from='osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T10:09:33.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:33 smithi193 ceph-mon[36311]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T10:09:33.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:33 smithi193 ceph-mon[36311]: from='osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T10:09:33.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:33.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:33 smithi193 ceph-mon[36311]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:33.870 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T10:09:33.870 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T10:09:33.870 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-16T10:09:33.870 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-16T10:09:33.870 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011433 s, 917 MB/s 2024-03-16T10:09:33.870 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-16T10:09:34.365 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:34 smithi077 ceph-mon[32400]: from='osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T10:09:34.365 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:34 smithi077 ceph-mon[32400]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T10:09:34.365 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:34 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:34.366 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:34 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:34.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:34 smithi192 ceph-mon[36354]: from='osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T10:09:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:34 smithi192 ceph-mon[36354]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T10:09:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:34 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:34 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:34.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:34 smithi193 ceph-mon[36311]: from='osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T10:09:34.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:34 smithi193 ceph-mon[36311]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T10:09:34.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:34 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:34.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:34 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:34.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2024-03-16T10:09:34.772 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:09:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: purged_snaps scrub starts 2024-03-16T10:09:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: purged_snaps scrub ok 2024-03-16T10:09:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613] boot 2024-03-16T10:09:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T10:09:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: Detected new or changed devices on smithi077 2024-03-16T10:09:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:09:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi077 to 17088M 2024-03-16T10:09:35.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:35.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:35.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:35.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:35.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: purged_snaps scrub starts 2024-03-16T10:09:35.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: purged_snaps scrub ok 2024-03-16T10:09:35.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:35.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613] boot 2024-03-16T10:09:35.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T10:09:35.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:35.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:35.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: Detected new or changed devices on smithi077 2024-03-16T10:09:35.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:35.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:35.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:09:35.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi077 to 17088M 2024-03-16T10:09:35.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:35.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:35.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:35.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:35.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: purged_snaps scrub starts 2024-03-16T10:09:35.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: purged_snaps scrub ok 2024-03-16T10:09:35.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:35.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: osd.0 [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613] boot 2024-03-16T10:09:35.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T10:09:35.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T10:09:35.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T10:09:35.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: Detected new or changed devices on smithi077 2024-03-16T10:09:35.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:35.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:35.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:09:35.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi077 to 17088M 2024-03-16T10:09:35.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:35.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:35.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:35.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:36.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.240+0000 7f738f5f5700 1 -- 172.21.15.192:0/2831364150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73881023f0 msgr2=0x7f7388102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:36.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.240+0000 7f738f5f5700 1 --2- 172.21.15.192:0/2831364150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73881023f0 0x7f7388102850 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f73780099c0 tx=0x7f7378009cd0 comp rx=0 tx=0).stop 2024-03-16T10:09:36.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.240+0000 7f738f5f5700 1 -- 172.21.15.192:0/2831364150 shutdown_connections 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.240+0000 7f738f5f5700 1 --2- 172.21.15.192:0/2831364150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7388102d90 0x7f7388109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.240+0000 7f738f5f5700 1 --2- 172.21.15.192:0/2831364150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73881011f0 0x7f7388101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.240+0000 7f738f5f5700 1 --2- 172.21.15.192:0/2831364150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73881023f0 0x7f7388102850 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.240+0000 7f738f5f5700 1 -- 172.21.15.192:0/2831364150 >> 172.21.15.192:0/2831364150 conn(0x7f73880fc790 msgr2=0x7f73880febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.240+0000 7f738f5f5700 1 -- 172.21.15.192:0/2831364150 shutdown_connections 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.240+0000 7f738f5f5700 1 -- 172.21.15.192:0/2831364150 wait complete. 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738f5f5700 1 Processor -- start 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738f5f5700 1 -- start start 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738f5f5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73881011f0 0x7f7388198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738f5f5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73881023f0 0x7f7388198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738f5f5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7388102d90 0x7f738819fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738f5f5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f738810ce50 con 0x7f73881023f0 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738f5f5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f738810ccd0 con 0x7f73881011f0 2024-03-16T10:09:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738f5f5700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f738810cfd0 con 0x7f7388102d90 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738d391700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73881011f0 0x7f7388198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738d391700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73881011f0 0x7f7388198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:46894/0 (socket says 172.21.15.192:46894) 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738d391700 1 -- 172.21.15.192:0/598255561 learned_addr learned my addr 172.21.15.192:0/598255561 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738cb90700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73881023f0 0x7f7388198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738db92700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7388102d90 0x7f738819fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738d391700 1 -- 172.21.15.192:0/598255561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7388102d90 msgr2=0x7f738819fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738d391700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7388102d90 0x7f738819fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738d391700 1 -- 172.21.15.192:0/598255561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73881023f0 msgr2=0x7f7388198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738d391700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73881023f0 0x7f7388198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.241+0000 7f738d391700 1 -- 172.21.15.192:0/598255561 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73881a01b0 con 0x7f73881011f0 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.242+0000 7f738d391700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73881011f0 0x7f7388198550 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f738400b500 tx=0x7f738400b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.242+0000 7f737e7fc700 1 -- 172.21.15.192:0/598255561 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7384015020 con 0x7f73881011f0 2024-03-16T10:09:36.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.242+0000 7f738f5f5700 1 -- 172.21.15.192:0/598255561 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73881a04a0 con 0x7f73881011f0 2024-03-16T10:09:36.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.242+0000 7f737e7fc700 1 -- 172.21.15.192:0/598255561 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7384011d10 con 0x7f73881011f0 2024-03-16T10:09:36.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.242+0000 7f738f5f5700 1 -- 172.21.15.192:0/598255561 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73881a0a60 con 0x7f73881011f0 2024-03-16T10:09:36.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.242+0000 7f737e7fc700 1 -- 172.21.15.192:0/598255561 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738400f2e0 con 0x7f73881011f0 2024-03-16T10:09:36.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.243+0000 7f738f5f5700 1 -- 172.21.15.192:0/598255561 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7388068f90 con 0x7f73881011f0 2024-03-16T10:09:36.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.244+0000 7f737e7fc700 1 -- 172.21.15.192:0/598255561 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f7384018020 con 0x7f73881011f0 2024-03-16T10:09:36.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.244+0000 7f737e7fc700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f73740a2800 0x7f73740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:09:36.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.244+0000 7f737e7fc700 1 -- 172.21.15.192:0/598255561 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f73840cc6f0 con 0x7f73881011f0 2024-03-16T10:09:36.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.244+0000 7f738cb90700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f73740a2800 0x7f73740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:09:36.247 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.245+0000 7f738cb90700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f73740a2800 0x7f73740a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7378000c00 tx=0x7f7378005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:09:36.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.249+0000 7f737e7fc700 1 -- 172.21.15.192:0/598255561 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f738401f020 con 0x7f73881011f0 2024-03-16T10:09:36.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:36 smithi192 ceph-mon[36354]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T10:09:36.452 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:36.452+0000 7f738f5f5700 1 -- 172.21.15.192:0/598255561 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f73881a0d80 con 0x7f73740a2800 2024-03-16T10:09:36.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:36 smithi077 ceph-mon[32400]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T10:09:36.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:36 smithi193 ceph-mon[36311]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T10:09:37.307 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:37 smithi192 ceph-mon[36354]: from='client.24134 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:09:37.308 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:09:37.308 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:09:37.308 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:37.308 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:37 smithi192 ceph-mon[36354]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:37 smithi077 ceph-mon[32400]: from='client.24134 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:09:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:09:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:09:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:37 smithi077 ceph-mon[32400]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:37 smithi193 ceph-mon[36311]: from='client.24134 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:09:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:09:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:09:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:37 smithi193 ceph-mon[36311]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:39.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:39 smithi077 ceph-mon[32400]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:39.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:39 smithi077 ceph-mon[32400]: from='client.? 172.21.15.192:0/1810233531' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7d45429-9695-4b62-a44f-edc170f07838"}]: dispatch 2024-03-16T10:09:39.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:39 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7d45429-9695-4b62-a44f-edc170f07838"}]: dispatch 2024-03-16T10:09:39.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:39 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7d45429-9695-4b62-a44f-edc170f07838"}]': finished 2024-03-16T10:09:39.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:39 smithi077 ceph-mon[32400]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T10:09:39.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:39 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:39.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:39 smithi077 ceph-mon[32400]: from='client.? 172.21.15.192:0/719690263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:09:40.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:39 smithi193 ceph-mon[36311]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:40.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:39 smithi193 ceph-mon[36311]: from='client.? 172.21.15.192:0/1810233531' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7d45429-9695-4b62-a44f-edc170f07838"}]: dispatch 2024-03-16T10:09:40.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:39 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7d45429-9695-4b62-a44f-edc170f07838"}]: dispatch 2024-03-16T10:09:40.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:39 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7d45429-9695-4b62-a44f-edc170f07838"}]': finished 2024-03-16T10:09:40.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:39 smithi193 ceph-mon[36311]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T10:09:40.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:39 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:40.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:39 smithi193 ceph-mon[36311]: from='client.? 172.21.15.192:0/719690263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:09:40.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:39 smithi192 ceph-mon[36354]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:40.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:39 smithi192 ceph-mon[36354]: from='client.? 172.21.15.192:0/1810233531' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7d45429-9695-4b62-a44f-edc170f07838"}]: dispatch 2024-03-16T10:09:40.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:39 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7d45429-9695-4b62-a44f-edc170f07838"}]: dispatch 2024-03-16T10:09:40.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:39 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7d45429-9695-4b62-a44f-edc170f07838"}]': finished 2024-03-16T10:09:40.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:39 smithi192 ceph-mon[36354]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T10:09:40.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:39 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:40.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:39 smithi192 ceph-mon[36354]: from='client.? 172.21.15.192:0/719690263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:09:41.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:41 smithi077 ceph-mon[32400]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:42.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:41 smithi193 ceph-mon[36311]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:42.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:41 smithi192 ceph-mon[36354]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:43.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:43 smithi077 ceph-mon[32400]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:43.996 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:43 smithi192 ceph-mon[36354]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:44.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:43 smithi193 ceph-mon[36311]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:45.983 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:45 smithi192 ceph-mon[36354]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:45.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:45 smithi077 ceph-mon[32400]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:46.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:45 smithi193 ceph-mon[36311]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:46.902 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:46 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T10:09:46.903 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:46 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:46 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T10:09:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:46 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:47.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:46 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T10:09:47.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:46 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:47.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:47 smithi077 ceph-mon[32400]: Deploying daemon osd.1 on smithi192 2024-03-16T10:09:47.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:47 smithi077 ceph-mon[32400]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:48.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:47 smithi193 ceph-mon[36311]: Deploying daemon osd.1 on smithi192 2024-03-16T10:09:48.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:47 smithi193 ceph-mon[36311]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:48.017 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:47 smithi192 ceph-mon[36354]: Deploying daemon osd.1 on smithi192 2024-03-16T10:09:48.017 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:47 smithi192 ceph-mon[36354]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:49.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:49 smithi192 ceph-mon[36354]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:49.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:49 smithi077 ceph-mon[32400]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:50.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:49 smithi193 ceph-mon[36311]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:51.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:51 smithi077 ceph-mon[32400]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:51.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:51 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:51.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:51 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:51.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:51 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:52.001 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:51 smithi192 ceph-mon[36354]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:52.002 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:51 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:52.002 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:51 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:52.002 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:51 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:52.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:51 smithi193 ceph-mon[36311]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:52.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:51 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:52.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:51 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:52.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:51 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:53.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:53 smithi077 ceph-mon[32400]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:53.998 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:53 smithi192 ceph-mon[36354]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:54.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:53 smithi193 ceph-mon[36311]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:55.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.217+0000 7f737e7fc700 1 -- 172.21.15.192:0/598255561 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f73881a0d80 con 0x7f73740a2800 2024-03-16T10:09:55.219 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 1 on host 'smithi192' 2024-03-16T10:09:55.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.219+0000 7f738f5f5700 1 -- 172.21.15.192:0/598255561 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f73740a2800 msgr2=0x7f73740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:55.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.219+0000 7f738f5f5700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f73740a2800 0x7f73740a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7378000c00 tx=0x7f7378005fb0 comp rx=0 tx=0).stop 2024-03-16T10:09:55.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.219+0000 7f738f5f5700 1 -- 172.21.15.192:0/598255561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73881011f0 msgr2=0x7f7388198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:09:55.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.219+0000 7f738f5f5700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73881011f0 0x7f7388198550 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f738400b500 tx=0x7f738400b810 comp rx=0 tx=0).stop 2024-03-16T10:09:55.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.219+0000 7f738f5f5700 1 -- 172.21.15.192:0/598255561 shutdown_connections 2024-03-16T10:09:55.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.219+0000 7f738f5f5700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f73740a2800 0x7f73740a4cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:55.221 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.219+0000 7f738f5f5700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7388102d90 0x7f738819fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:55.221 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.219+0000 7f738f5f5700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73881011f0 0x7f7388198550 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:55.221 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.219+0000 7f738f5f5700 1 --2- 172.21.15.192:0/598255561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73881023f0 0x7f7388198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:09:55.221 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.219+0000 7f738f5f5700 1 -- 172.21.15.192:0/598255561 >> 172.21.15.192:0/598255561 conn(0x7f73880fc790 msgr2=0x7f73880fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:09:55.221 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.219+0000 7f738f5f5700 1 -- 172.21.15.192:0/598255561 shutdown_connections 2024-03-16T10:09:55.221 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:09:55.219+0000 7f738f5f5700 1 -- 172.21.15.192:0/598255561 wait complete. 2024-03-16T10:09:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:55 smithi193 ceph-mon[36311]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:55 smithi193 ceph-mon[36311]: from='osd.1 [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T10:09:55.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:55 smithi193 ceph-mon[36311]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T10:09:55.613 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.613 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.613 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:55.614 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:55.614 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.614 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:55 smithi192 ceph-mon[36354]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:55.614 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.614 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.614 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:55.614 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:55 smithi192 ceph-mon[36354]: from='osd.1 [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T10:09:55.614 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:55 smithi192 ceph-mon[36354]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T10:09:55.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:55.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:55.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:55 smithi077 ceph-mon[32400]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:55.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:55.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:09:55.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:55 smithi077 ceph-mon[32400]: from='osd.1 [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T10:09:55.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:55 smithi077 ceph-mon[32400]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T10:09:55.933 DEBUG:teuthology.orchestra.run.smithi192:osd.1> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.1.service 2024-03-16T10:09:55.935 INFO:tasks.cephadm:Deploying osd.2 on smithi193 with /dev/vg_nvme/lv_4... 2024-03-16T10:09:55.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T10:09:56.180 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:09:57.327 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:57 smithi193 ceph-mon[36311]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T10:09:57.327 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:57 smithi193 ceph-mon[36311]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T10:09:57.327 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:57 smithi193 ceph-mon[36311]: from='osd.1 [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:09:57.327 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:57.327 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:57 smithi193 ceph-mon[36311]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:09:57.327 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:57 smithi193 ceph-mon[36311]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:57.359 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:57 smithi192 ceph-mon[36354]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T10:09:57.359 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:57 smithi192 ceph-mon[36354]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T10:09:57.360 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:57 smithi192 ceph-mon[36354]: from='osd.1 [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:09:57.360 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:57.360 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:57 smithi192 ceph-mon[36354]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:09:57.360 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:57 smithi192 ceph-mon[36354]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:57 smithi077 ceph-mon[32400]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T10:09:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:57 smithi077 ceph-mon[32400]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T10:09:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:57 smithi077 ceph-mon[32400]: from='osd.1 [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:09:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:57 smithi077 ceph-mon[32400]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:09:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:57 smithi077 ceph-mon[32400]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:57.898 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T10:09:57.898 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T10:09:57.898 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-16T10:09:57.898 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-16T10:09:57.898 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0120942 s, 867 MB/s 2024-03-16T10:09:57.898 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-16T10:09:57.917 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:09:57 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1[40959]: 2024-03-16T10:09:57.661+0000 7fec8b94a700 -1 osd.1 0 waiting for initial osdmap 2024-03-16T10:09:57.917 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:09:57 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1[40959]: 2024-03-16T10:09:57.665+0000 7fec82f34700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:09:58.410 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_4 2024-03-16T10:09:58.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:58 smithi077 ceph-mon[32400]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-16T10:09:58.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:58 smithi077 ceph-mon[32400]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T10:09:58.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:58.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:58.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:58 smithi077 ceph-mon[32400]: from='osd.1 ' entity='osd.1' 2024-03-16T10:09:58.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:58.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:58 smithi193 ceph-mon[36311]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-16T10:09:58.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:58 smithi193 ceph-mon[36311]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T10:09:58.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:58.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:58.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:58 smithi193 ceph-mon[36311]: from='osd.1 ' entity='osd.1' 2024-03-16T10:09:58.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:58.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:58 smithi192 ceph-mon[36354]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-16T10:09:58.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:58 smithi192 ceph-mon[36354]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T10:09:58.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:58.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:58.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:58 smithi192 ceph-mon[36354]: from='osd.1 ' entity='osd.1' 2024-03-16T10:09:58.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:58.666 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:09:59.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: purged_snaps scrub starts 2024-03-16T10:09:59.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: purged_snaps scrub ok 2024-03-16T10:09:59.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:59.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: osd.1 [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545] boot 2024-03-16T10:09:59.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T10:09:59.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:59.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:59.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:59.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:09:59.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:59.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:59.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:59.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:09:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:59.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: purged_snaps scrub starts 2024-03-16T10:09:59.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: purged_snaps scrub ok 2024-03-16T10:09:59.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:59.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: osd.1 [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545] boot 2024-03-16T10:09:59.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T10:09:59.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:59.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:59.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:59.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:09:59.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:59.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:59.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:59.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:09:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: purged_snaps scrub starts 2024-03-16T10:09:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: purged_snaps scrub ok 2024-03-16T10:09:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T10:09:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: osd.1 [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545] boot 2024-03-16T10:09:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T10:09:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T10:09:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:09:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:09:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:09:59.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:09:59.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:09:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:00.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.139+0000 7f007013b700 1 -- 172.21.15.193:0/979209642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00681023d0 msgr2=0x7f0068102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:00.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.139+0000 7f007013b700 1 --2- 172.21.15.193:0/979209642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00681023d0 0x7f0068102830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0058009990 tx=0x7f0058009ca0 comp rx=0 tx=0).stop 2024-03-16T10:10:00.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.139+0000 7f007013b700 1 -- 172.21.15.193:0/979209642 shutdown_connections 2024-03-16T10:10:00.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.139+0000 7f007013b700 1 --2- 172.21.15.193:0/979209642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00681011d0 0x7f00681015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:00.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.139+0000 7f007013b700 1 --2- 172.21.15.193:0/979209642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0068102d70 0x7f0068109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:00.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.139+0000 7f007013b700 1 --2- 172.21.15.193:0/979209642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00681023d0 0x7f0068102830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:00.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.139+0000 7f007013b700 1 -- 172.21.15.193:0/979209642 >> 172.21.15.193:0/979209642 conn(0x7f00680fc770 msgr2=0x7f00680febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:10:00.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.139+0000 7f007013b700 1 -- 172.21.15.193:0/979209642 shutdown_connections 2024-03-16T10:10:00.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.139+0000 7f007013b700 1 -- 172.21.15.193:0/979209642 wait complete. 2024-03-16T10:10:00.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f007013b700 1 Processor -- start 2024-03-16T10:10:00.142 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f007013b700 1 -- start start 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f007013b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00681011d0 0x7f00681984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f007013b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00681023d0 0x7f0068198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f007013b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0068102d70 0x7f006819fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f007013b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f006810cd90 con 0x7f00681023d0 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f007013b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f006810cc10 con 0x7f0068102d70 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f007013b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f006810cf10 con 0x7f00681011d0 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f006ded7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00681011d0 0x7f00681984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f006ded7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00681011d0 0x7f00681984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:42726/0 (socket says 172.21.15.193:42726) 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f006ded7700 1 -- 172.21.15.193:0/766561275 learned_addr learned my addr 172.21.15.193:0/766561275 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f006d6d6700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00681023d0 0x7f0068198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f006e6d8700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0068102d70 0x7f006819fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f006ded7700 1 -- 172.21.15.193:0/766561275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0068102d70 msgr2=0x7f006819fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f006ded7700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0068102d70 0x7f006819fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:00.143 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f006ded7700 1 -- 172.21.15.193:0/766561275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00681023d0 msgr2=0x7f0068198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:00.144 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f006ded7700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00681023d0 0x7f0068198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:00.144 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.140+0000 7f006ded7700 1 -- 172.21.15.193:0/766561275 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00681a0120 con 0x7f00681011d0 2024-03-16T10:10:00.144 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.141+0000 7f006ded7700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00681011d0 0x7f00681984c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f006400c9a0 tx=0x7f006400ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:10:00.144 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.141+0000 7f005effd700 1 -- 172.21.15.193:0/766561275 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0064004220 con 0x7f00681011d0 2024-03-16T10:10:00.144 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.141+0000 7f005effd700 1 -- 172.21.15.193:0/766561275 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00640043c0 con 0x7f00681011d0 2024-03-16T10:10:00.144 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.141+0000 7f007013b700 1 -- 172.21.15.193:0/766561275 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00681a0410 con 0x7f00681011d0 2024-03-16T10:10:00.144 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.141+0000 7f005effd700 1 -- 172.21.15.193:0/766561275 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0064003c90 con 0x7f00681011d0 2024-03-16T10:10:00.144 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.141+0000 7f007013b700 1 -- 172.21.15.193:0/766561275 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00681a0950 con 0x7f00681011d0 2024-03-16T10:10:00.145 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.142+0000 7f007013b700 1 -- 172.21.15.193:0/766561275 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0068068f90 con 0x7f00681011d0 2024-03-16T10:10:00.145 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.143+0000 7f005effd700 1 -- 172.21.15.193:0/766561275 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f00640076d0 con 0x7f00681011d0 2024-03-16T10:10:00.145 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.143+0000 7f005effd700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00540a2800 0x7f00540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:10:00.145 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.144+0000 7f005effd700 1 -- 172.21.15.193:0/766561275 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f00640ccd70 con 0x7f00681011d0 2024-03-16T10:10:00.145 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.144+0000 7f006d6d6700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00540a2800 0x7f00540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:10:00.145 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.144+0000 7f006d6d6700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00540a2800 0x7f00540a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f005800b5c0 tx=0x7f0058003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:10:00.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.148+0000 7f005effd700 1 -- 172.21.15.193:0/766561275 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0064014030 con 0x7f00681011d0 2024-03-16T10:10:00.355 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:00.355+0000 7f007013b700 1 -- 172.21.15.193:0/766561275 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f00680fe940 con 0x7f00540a2800 2024-03-16T10:10:00.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:00 smithi077 ceph-mon[32400]: Detected new or changed devices on smithi192 2024-03-16T10:10:00.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:00 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi192 to 17080M 2024-03-16T10:10:00.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:00 smithi077 ceph-mon[32400]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T10:10:00.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:00 smithi077 ceph-mon[32400]: overall HEALTH_OK 2024-03-16T10:10:00.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:00 smithi193 ceph-mon[36311]: Detected new or changed devices on smithi192 2024-03-16T10:10:00.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:00 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi192 to 17080M 2024-03-16T10:10:00.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:00 smithi193 ceph-mon[36311]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T10:10:00.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:00 smithi193 ceph-mon[36311]: overall HEALTH_OK 2024-03-16T10:10:00.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:00 smithi192 ceph-mon[36354]: Detected new or changed devices on smithi192 2024-03-16T10:10:00.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:00 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi192 to 17080M 2024-03-16T10:10:00.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:00 smithi192 ceph-mon[36354]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T10:10:00.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:00 smithi192 ceph-mon[36354]: overall HEALTH_OK 2024-03-16T10:10:01.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:01 smithi077 ceph-mon[32400]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:10:01.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:01 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:10:01.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:01 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:10:01.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:01 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:01.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:01 smithi077 ceph-mon[32400]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:01.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:01 smithi193 ceph-mon[36311]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:10:01.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:01 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:10:01.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:01 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:10:01.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:01 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:01.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:01 smithi193 ceph-mon[36311]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:01.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:01 smithi192 ceph-mon[36354]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:10:01.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:01 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:10:01.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:01 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:10:01.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:01 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:01.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:01 smithi192 ceph-mon[36354]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:03.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:03 smithi077 ceph-mon[32400]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:03.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:03 smithi077 ceph-mon[32400]: from='client.? 172.21.15.193:0/1760958946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc"}]: dispatch 2024-03-16T10:10:03.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:03 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc"}]: dispatch 2024-03-16T10:10:03.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:03 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc"}]': finished 2024-03-16T10:10:03.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:03 smithi077 ceph-mon[32400]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T10:10:03.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:03.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:03 smithi077 ceph-mon[32400]: from='client.? 172.21.15.193:0/3708144223' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:10:04.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:03 smithi193 ceph-mon[36311]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:04.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:03 smithi193 ceph-mon[36311]: from='client.? 172.21.15.193:0/1760958946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc"}]: dispatch 2024-03-16T10:10:04.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:03 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc"}]: dispatch 2024-03-16T10:10:04.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:03 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc"}]': finished 2024-03-16T10:10:04.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:03 smithi193 ceph-mon[36311]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T10:10:04.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:04.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:03 smithi193 ceph-mon[36311]: from='client.? 172.21.15.193:0/3708144223' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:10:04.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:03 smithi192 ceph-mon[36354]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:04.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:03 smithi192 ceph-mon[36354]: from='client.? 172.21.15.193:0/1760958946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc"}]: dispatch 2024-03-16T10:10:04.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:03 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc"}]: dispatch 2024-03-16T10:10:04.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:03 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc"}]': finished 2024-03-16T10:10:04.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:03 smithi192 ceph-mon[36354]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T10:10:04.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:04.168 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:03 smithi192 ceph-mon[36354]: from='client.? 172.21.15.193:0/3708144223' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:10:05.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:05 smithi077 ceph-mon[32400]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:06.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:05 smithi193 ceph-mon[36311]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:05 smithi192 ceph-mon[36354]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:07.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:07 smithi077 ceph-mon[32400]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:08.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:07 smithi193 ceph-mon[36311]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:08.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:07 smithi192 ceph-mon[36354]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:09.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:09 smithi077 ceph-mon[32400]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:10.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:09 smithi193 ceph-mon[36311]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:10.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:09 smithi192 ceph-mon[36354]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:10.814 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:10 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T10:10:10.814 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:10 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:10.985 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:10 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T10:10:10.985 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:10 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:10.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:10 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T10:10:10.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:10 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:11.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:11 smithi193 ceph-mon[36311]: Deploying daemon osd.2 on smithi193 2024-03-16T10:10:11.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:11 smithi193 ceph-mon[36311]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:11 smithi077 ceph-mon[32400]: Deploying daemon osd.2 on smithi193 2024-03-16T10:10:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:11 smithi077 ceph-mon[32400]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:12.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:11 smithi192 ceph-mon[36354]: Deploying daemon osd.2 on smithi193 2024-03-16T10:10:12.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:11 smithi192 ceph-mon[36354]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:13.988 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:13 smithi193 ceph-mon[36311]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:13.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:13 smithi077 ceph-mon[32400]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:14.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:13 smithi192 ceph-mon[36354]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:15.856 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:15 smithi193 ceph-mon[36311]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:15.856 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:15 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:15.856 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:15 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:15.856 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:15 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:10:15.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:15 smithi077 ceph-mon[32400]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:15.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:15 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:15.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:15 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:15.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:15 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:10:16.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:15 smithi192 ceph-mon[36354]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:16.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:15 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:16.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:15 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:16.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:15 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:10:17.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:17 smithi077 ceph-mon[32400]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:17.997 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:17 smithi193 ceph-mon[36311]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:18.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:17 smithi192 ceph-mon[36354]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:18.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.570+0000 7f005effd700 1 -- 172.21.15.193:0/766561275 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f00680fe940 con 0x7f00540a2800 2024-03-16T10:10:18.572 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 2 on host 'smithi193' 2024-03-16T10:10:18.574 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.572+0000 7f007013b700 1 -- 172.21.15.193:0/766561275 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00540a2800 msgr2=0x7f00540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:18.574 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.572+0000 7f007013b700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00540a2800 0x7f00540a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f005800b5c0 tx=0x7f0058003960 comp rx=0 tx=0).stop 2024-03-16T10:10:18.574 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.572+0000 7f007013b700 1 -- 172.21.15.193:0/766561275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00681011d0 msgr2=0x7f00681984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:18.574 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.572+0000 7f007013b700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00681011d0 0x7f00681984c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f006400c9a0 tx=0x7f006400ccb0 comp rx=0 tx=0).stop 2024-03-16T10:10:18.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.573+0000 7f007013b700 1 -- 172.21.15.193:0/766561275 shutdown_connections 2024-03-16T10:10:18.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.573+0000 7f007013b700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00540a2800 0x7f00540a4cc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:18.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.573+0000 7f007013b700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0068102d70 0x7f006819fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:18.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.573+0000 7f007013b700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00681011d0 0x7f00681984c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:18.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.573+0000 7f007013b700 1 --2- 172.21.15.193:0/766561275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00681023d0 0x7f0068198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:18.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.573+0000 7f007013b700 1 -- 172.21.15.193:0/766561275 >> 172.21.15.193:0/766561275 conn(0x7f00680fc770 msgr2=0x7f00680fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:10:18.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.573+0000 7f007013b700 1 -- 172.21.15.193:0/766561275 shutdown_connections 2024-03-16T10:10:18.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:10:18.573+0000 7f007013b700 1 -- 172.21.15.193:0/766561275 wait complete. 2024-03-16T10:10:18.851 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:18 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:18.851 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:18 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:18.851 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:18 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:18.851 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:18 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:10:18.851 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:18 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:18.851 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:18 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:18.851 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:18 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:10:18.851 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:18 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:19.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:18 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:19.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:18 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:19.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:18 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:19.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:18 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:10:19.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:18 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:19.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:18 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:19.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:18 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:10:19.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:18 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:19.202 DEBUG:teuthology.orchestra.run.smithi193:osd.2> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.2.service 2024-03-16T10:10:19.204 INFO:tasks.cephadm:Deploying osd.3 on smithi077 with /dev/vg_nvme/lv_3... 2024-03-16T10:10:19.204 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T10:10:19.234 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:18 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:19.234 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:18 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:19.234 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:18 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:19.234 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:18 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:10:19.234 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:18 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:19.234 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:18 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:19.234 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:18 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:10:19.234 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:18 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:19.451 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:10:20.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:19 smithi193 ceph-mon[36311]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:20.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:19 smithi193 ceph-mon[36311]: from='osd.2 [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T10:10:20.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:19 smithi193 ceph-mon[36311]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T10:10:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:19 smithi192 ceph-mon[36354]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:20.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:19 smithi192 ceph-mon[36354]: from='osd.2 [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T10:10:20.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:19 smithi192 ceph-mon[36354]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T10:10:20.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:19 smithi077 ceph-mon[32400]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:20.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:19 smithi077 ceph-mon[32400]: from='osd.2 [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T10:10:20.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:19 smithi077 ceph-mon[32400]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T10:10:20.946 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:20 smithi077 ceph-mon[32400]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T10:10:20.946 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:20 smithi077 ceph-mon[32400]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T10:10:20.946 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:20 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:20.946 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:20 smithi077 ceph-mon[32400]: from='osd.2 [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:10:20.946 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:20 smithi077 ceph-mon[32400]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:10:21.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:20 smithi193 ceph-mon[36311]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T10:10:21.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:20 smithi193 ceph-mon[36311]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T10:10:21.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:20 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:21.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:20 smithi193 ceph-mon[36311]: from='osd.2 [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:10:21.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:20 smithi193 ceph-mon[36311]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:10:21.003 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:10:20 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2[41492]: 2024-03-16T10:10:20.780+0000 7f8a12bb3700 -1 osd.2 0 waiting for initial osdmap 2024-03-16T10:10:21.003 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:10:20 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2[41492]: 2024-03-16T10:10:20.784+0000 7f8a0e9a6700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:10:21.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:20 smithi192 ceph-mon[36354]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T10:10:21.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:20 smithi192 ceph-mon[36354]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T10:10:21.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:20 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:21.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:20 smithi192 ceph-mon[36354]: from='osd.2 [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:10:21.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:20 smithi192 ceph-mon[36354]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:10:21.515 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T10:10:21.515 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T10:10:21.515 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-03-16T10:10:21.516 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-03-16T10:10:21.516 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0151257 s, 693 MB/s 2024-03-16T10:10:21.516 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-03-16T10:10:21.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:21 smithi193 ceph-mon[36311]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:21.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:21 smithi193 ceph-mon[36311]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-16T10:10:21.874 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:21 smithi193 ceph-mon[36311]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T10:10:21.874 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:21 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:21.874 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:21 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:21.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:21 smithi192 ceph-mon[36354]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:21.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:21 smithi192 ceph-mon[36354]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-16T10:10:21.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:21 smithi192 ceph-mon[36354]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T10:10:21.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:21 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:21.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:21 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:22.177 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:21 smithi077 ceph-mon[32400]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:22.177 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:21 smithi077 ceph-mon[32400]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-16T10:10:22.177 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:21 smithi077 ceph-mon[32400]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T10:10:22.177 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:21 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:22.177 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:21 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:22.203 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_3 2024-03-16T10:10:22.489 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:10:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:22 smithi077 ceph-mon[32400]: purged_snaps scrub starts 2024-03-16T10:10:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:22 smithi077 ceph-mon[32400]: purged_snaps scrub ok 2024-03-16T10:10:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:22 smithi077 ceph-mon[32400]: osd.2 [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947] boot 2024-03-16T10:10:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:22 smithi077 ceph-mon[32400]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T10:10:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:23.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:10:23.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:23.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:23.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:10:23.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:22 smithi193 ceph-mon[36311]: purged_snaps scrub starts 2024-03-16T10:10:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:22 smithi193 ceph-mon[36311]: purged_snaps scrub ok 2024-03-16T10:10:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:22 smithi193 ceph-mon[36311]: osd.2 [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947] boot 2024-03-16T10:10:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:22 smithi193 ceph-mon[36311]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T10:10:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:10:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:10:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:23.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:22 smithi192 ceph-mon[36354]: purged_snaps scrub starts 2024-03-16T10:10:23.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:22 smithi192 ceph-mon[36354]: purged_snaps scrub ok 2024-03-16T10:10:23.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:22 smithi192 ceph-mon[36354]: osd.2 [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947] boot 2024-03-16T10:10:23.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:22 smithi192 ceph-mon[36354]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T10:10:23.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T10:10:23.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:23.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:23.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:10:23.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:23.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:23.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:10:23.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:24.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.208+0000 7fd67782a700 1 -- 172.21.15.77:0/1956852807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670101200 msgr2=0x7fd670101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.208+0000 7fd67782a700 1 --2- 172.21.15.77:0/1956852807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670101200 0x7fd670101620 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0099c0 tx=0x7fd66c009cd0 comp rx=0 tx=0).stop 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.208+0000 7fd67782a700 1 -- 172.21.15.77:0/1956852807 shutdown_connections 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.208+0000 7fd67782a700 1 --2- 172.21.15.77:0/1956852807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670101200 0x7fd670101620 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.208+0000 7fd67782a700 1 --2- 172.21.15.77:0/1956852807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd670102da0 0x7fd670109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.208+0000 7fd67782a700 1 --2- 172.21.15.77:0/1956852807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd670102400 0x7fd670102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.208+0000 7fd67782a700 1 -- 172.21.15.77:0/1956852807 >> 172.21.15.77:0/1956852807 conn(0x7fd6700fc7a0 msgr2=0x7fd6700febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.208+0000 7fd67782a700 1 -- 172.21.15.77:0/1956852807 shutdown_connections 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.208+0000 7fd67782a700 1 -- 172.21.15.77:0/1956852807 wait complete. 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.208+0000 7fd67782a700 1 Processor -- start 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.208+0000 7fd67782a700 1 -- start start 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd67782a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd670101200 0x7fd6701985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd67782a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd670102400 0x7fd670198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:10:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd67782a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670102da0 0x7fd67019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd67782a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd67010cdf0 con 0x7fd670102da0 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd67782a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd67010cc70 con 0x7fd670102400 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd67782a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd67010cf70 con 0x7fd670101200 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd675dc7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670102da0 0x7fd67019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd6755c6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd670101200 0x7fd6701985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd674dc5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd670102400 0x7fd670198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd6755c6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd670101200 0x7fd6701985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:45040/0 (socket says 172.21.15.77:45040) 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd675dc7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670102da0 0x7fd67019fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54952/0 (socket says 172.21.15.77:54952) 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd674dc5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd670102400 0x7fd670198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:49246/0 (socket says 172.21.15.77:49246) 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd6755c6700 1 -- 172.21.15.77:0/2745059834 learned_addr learned my addr 172.21.15.77:0/2745059834 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd6755c6700 1 -- 172.21.15.77:0/2745059834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd670102400 msgr2=0x7fd670198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd6755c6700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd670102400 0x7fd670198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd6755c6700 1 -- 172.21.15.77:0/2745059834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670102da0 msgr2=0x7fd67019fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd6755c6700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670102da0 0x7fd67019fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd6755c6700 1 -- 172.21.15.77:0/2745059834 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6701a0200 con 0x7fd670101200 2024-03-16T10:10:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd675dc7700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670102da0 0x7fd67019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:10:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.209+0000 7fd6755c6700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd670101200 0x7fd6701985a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00fab0 tx=0x7fd66c00fae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:10:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.210+0000 7fd6667fc700 1 -- 172.21.15.77:0/2745059834 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c027070 con 0x7fd670101200 2024-03-16T10:10:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.210+0000 7fd67782a700 1 -- 172.21.15.77:0/2745059834 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6701a0490 con 0x7fd670101200 2024-03-16T10:10:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.210+0000 7fd6667fc700 1 -- 172.21.15.77:0/2745059834 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd66c00fd00 con 0x7fd670101200 2024-03-16T10:10:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.210+0000 7fd6667fc700 1 -- 172.21.15.77:0/2745059834 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c0173f0 con 0x7fd670101200 2024-03-16T10:10:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.210+0000 7fd67782a700 1 -- 172.21.15.77:0/2745059834 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6701a0970 con 0x7fd670101200 2024-03-16T10:10:24.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.211+0000 7fd67782a700 1 -- 172.21.15.77:0/2745059834 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd670068f90 con 0x7fd670101200 2024-03-16T10:10:24.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.212+0000 7fd6667fc700 1 -- 172.21.15.77:0/2745059834 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7fd66c02a070 con 0x7fd670101200 2024-03-16T10:10:24.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.213+0000 7fd6667fc700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd65c0a2800 0x7fd65c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:10:24.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.213+0000 7fd674dc5700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd65c0a2800 0x7fd65c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:10:24.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.213+0000 7fd6667fc700 1 -- 172.21.15.77:0/2745059834 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7fd66c0ce2a0 con 0x7fd670101200 2024-03-16T10:10:24.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.213+0000 7fd674dc5700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd65c0a2800 0x7fd65c0a4cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd660009c60 tx=0x7fd660009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:10:24.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.216+0000 7fd6667fc700 1 -- 172.21.15.77:0/2745059834 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd66c013070 con 0x7fd670101200 2024-03-16T10:10:24.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:23 smithi077 ceph-mon[32400]: Detected new or changed devices on smithi193 2024-03-16T10:10:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:23 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi193 to 17080M 2024-03-16T10:10:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:23 smithi077 ceph-mon[32400]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:23 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:10:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:23 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T10:10:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:23 smithi077 ceph-mon[32400]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T10:10:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:23 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:10:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:23 smithi193 ceph-mon[36311]: Detected new or changed devices on smithi193 2024-03-16T10:10:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:23 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi193 to 17080M 2024-03-16T10:10:24.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:23 smithi193 ceph-mon[36311]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:24.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:23 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:10:24.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:23 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T10:10:24.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:23 smithi193 ceph-mon[36311]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T10:10:24.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:23 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:10:24.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:23 smithi192 ceph-mon[36354]: Detected new or changed devices on smithi193 2024-03-16T10:10:24.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:23 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi193 to 17080M 2024-03-16T10:10:24.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:23 smithi192 ceph-mon[36354]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T10:10:24.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:23 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:10:24.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:23 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T10:10:24.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:23 smithi192 ceph-mon[36354]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T10:10:24.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:23 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:10:24.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:24.429+0000 7fd67782a700 1 -- 172.21.15.77:0/2745059834 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd67006a690 con 0x7fd65c0a2800 2024-03-16T10:10:24.594 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:10:24 smithi077 sudo[49429]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T10:10:24.594 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:10:24 smithi077 sudo[49429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:10:24.594 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:10:24 smithi077 sudo[49429]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:10:24.594 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:10:24 smithi077 sudo[49429]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:10:24.980 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:10:24 smithi077 sudo[49499]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T10:10:24.981 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:10:24 smithi077 sudo[49499]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:10:24.981 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:10:24 smithi077 sudo[49499]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:10:24.981 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:10:24 smithi077 sudo[49499]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:10:25.120 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:10:24 smithi192 sudo[44954]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T10:10:25.120 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:10:24 smithi192 sudo[44954]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:10:25.120 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:10:24 smithi192 sudo[44954]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:10:25.120 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:10:24 smithi192 sudo[44954]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:10:25.120 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T10:10:25.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:24 smithi192 ceph-mon[36354]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T10:10:25.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:24 smithi192 ceph-mon[36354]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:10:25.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:10:25.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:10:25.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:25.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:25.121 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:24 smithi192 ceph-mon[36354]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:25.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T10:10:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:24 smithi077 ceph-mon[32400]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T10:10:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:24 smithi077 ceph-mon[32400]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:10:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:10:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:10:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:24 smithi077 ceph-mon[32400]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:25.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T10:10:25.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:24 smithi193 ceph-mon[36311]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T10:10:25.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:24 smithi193 ceph-mon[36311]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:10:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:10:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:10:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:24 smithi193 ceph-mon[36311]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:25.416 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:10:25 smithi192 sudo[44957]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T10:10:25.416 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:10:25 smithi192 sudo[44957]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:10:25.416 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:10:25 smithi192 sudo[44957]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:10:25.416 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:10:25 smithi192 sudo[44957]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:10:25.645 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:10:25 smithi193 sudo[45430]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T10:10:25.645 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:10:25 smithi193 sudo[45430]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:10:25.645 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:10:25 smithi193 sudo[45430]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:10:25.645 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:10:25 smithi193 sudo[45430]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:10:26.002 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:10:25 smithi193 sudo[45433]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T10:10:26.003 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:10:25 smithi193 sudo[45433]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:10:26.003 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:10:25 smithi193 sudo[45433]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:10:26.003 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:10:25 smithi193 sudo[45433]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:10:26.177 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:25 smithi077 sudo[49580]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T10:10:26.177 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:25 smithi077 sudo[49580]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:10:26.177 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:25 smithi077 sudo[49580]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:10:26.178 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:25 smithi077 sudo[49580]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:10:26.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:26 smithi193 ceph-mon[36311]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T10:10:26.437 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:26 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:26.437 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:26 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:26.437 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:26 smithi193 ceph-mon[36311]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:10:26.437 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:26 smithi193 sudo[45436]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T10:10:26.437 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:26 smithi193 sudo[45436]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:10:26.437 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:26 smithi193 sudo[45436]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:10:26.437 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:26 smithi193 sudo[45436]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:10:26.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:26 smithi077 sudo[49604]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T10:10:26.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:26 smithi077 sudo[49604]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:10:26.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:26 smithi077 sudo[49604]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:10:26.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:26 smithi077 sudo[49604]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:10:26.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:26 smithi077 ceph-mon[32400]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T10:10:26.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:26 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:26.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:26 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:26.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:26 smithi077 ceph-mon[32400]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:10:26.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:26 smithi192 ceph-mon[36354]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T10:10:26.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:26 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:26.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:26 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:26.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:26 smithi192 ceph-mon[36354]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:10:26.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:26.683+0000 7fd6667fc700 1 -- 172.21.15.77:0/2745059834 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd66c0987f0 con 0x7fd670101200 2024-03-16T10:10:26.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:26 smithi193 sudo[45439]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T10:10:26.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:26 smithi193 sudo[45439]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:10:26.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:26 smithi193 sudo[45439]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:10:26.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:26 smithi193 sudo[45439]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:10:27.176 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:26 smithi192 sudo[44960]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T10:10:27.176 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:26 smithi192 sudo[44960]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:10:27.176 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:26 smithi192 sudo[44960]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:10:27.176 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:26 smithi192 sudo[44960]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:10:27.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:10:27.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:10:27.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:10:27.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:10:27.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:10:27.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T10:10:27.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:10:27.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:10:27.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:10:27.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:27.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:10:27.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:10:27.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:10:27.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:10:27.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:10:27.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3969421506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d140b161-7795-4be8-bbb4-8a010ea7707e"}]: dispatch 2024-03-16T10:10:27.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:10:27.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:27 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3969421506' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d140b161-7795-4be8-bbb4-8a010ea7707e"}]': finished 2024-03-16T10:10:27.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:10:27.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:10:27.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:10:27.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:10:27.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:10:27.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T10:10:27.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:10:27.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:10:27.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:10:27.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:27.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:10:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:10:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:10:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:10:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:10:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3969421506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d140b161-7795-4be8-bbb4-8a010ea7707e"}]: dispatch 2024-03-16T10:10:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:10:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:27 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3969421506' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d140b161-7795-4be8-bbb4-8a010ea7707e"}]': finished 2024-03-16T10:10:27.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 sudo[44963]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T10:10:27.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 sudo[44963]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T10:10:27.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 sudo[44963]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T10:10:27.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 sudo[44963]: pam_unix(sudo:session): session closed for user root 2024-03-16T10:10:27.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:10:27.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:10:27.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:10:27.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:10:27.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:10:27.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T10:10:27.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:10:27.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:10:27.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:10:27.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:27.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:10:27.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T10:10:27.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T10:10:27.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T10:10:27.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T10:10:27.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3969421506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d140b161-7795-4be8-bbb4-8a010ea7707e"}]: dispatch 2024-03-16T10:10:27.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T10:10:27.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:27 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3969421506' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d140b161-7795-4be8-bbb4-8a010ea7707e"}]': finished 2024-03-16T10:10:28.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:28 smithi077 ceph-mon[32400]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T10:10:28.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:28.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:28 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/26925525' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:10:28.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:28 smithi193 ceph-mon[36311]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T10:10:28.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:28.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:28 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/26925525' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:10:28.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:28 smithi192 ceph-mon[36354]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T10:10:28.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:28.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:28 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/26925525' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:10:29.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:29 smithi077 ceph-mon[32400]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:29.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:29 smithi193 ceph-mon[36311]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:29.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:29 smithi192 ceph-mon[36354]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:31.973 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:31 smithi077 ceph-mon[32400]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-16T10:10:32.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:31 smithi193 ceph-mon[36311]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-16T10:10:32.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:31 smithi192 ceph-mon[36354]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-16T10:10:33.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:33 smithi077 ceph-mon[32400]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-16T10:10:34.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:33 smithi193 ceph-mon[36311]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-16T10:10:34.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:33 smithi192 ceph-mon[36354]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-16T10:10:34.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:34 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T10:10:34.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:34 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:35.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:34 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T10:10:35.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:34 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:35.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:34 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T10:10:35.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:34 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:35.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:35 smithi077 ceph-mon[32400]: Deploying daemon osd.3 on smithi077 2024-03-16T10:10:35.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:35 smithi077 ceph-mon[32400]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 189 KiB/s wr, 12 op/s 2024-03-16T10:10:36.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:35 smithi193 ceph-mon[36311]: Deploying daemon osd.3 on smithi077 2024-03-16T10:10:36.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:35 smithi193 ceph-mon[36311]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 189 KiB/s wr, 12 op/s 2024-03-16T10:10:36.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:35 smithi192 ceph-mon[36354]: Deploying daemon osd.3 on smithi077 2024-03-16T10:10:36.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:35 smithi192 ceph-mon[36354]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 189 KiB/s wr, 12 op/s 2024-03-16T10:10:37.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:37 smithi077 ceph-mon[32400]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-16T10:10:38.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:37 smithi193 ceph-mon[36311]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-16T10:10:38.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:37 smithi192 ceph-mon[36354]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-16T10:10:39.834 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:39 smithi077 ceph-mon[32400]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 159 KiB/s wr, 10 op/s 2024-03-16T10:10:40.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:39 smithi193 ceph-mon[36311]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 159 KiB/s wr, 10 op/s 2024-03-16T10:10:40.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:39 smithi192 ceph-mon[36354]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 159 KiB/s wr, 10 op/s 2024-03-16T10:10:41.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:41 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:41.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:41 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:41.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:41 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:10:41.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:41 smithi077 ceph-mon[32400]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-16T10:10:41.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:41 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:41.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:41 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:41.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:41 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:10:41.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:41 smithi192 ceph-mon[36354]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-16T10:10:41.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:41 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:41.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:41 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:41.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:41 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:10:41.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:41 smithi193 ceph-mon[36311]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-16T10:10:43.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.508+0000 7fd6667fc700 1 -- 172.21.15.77:0/2745059834 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd67006a690 con 0x7fd65c0a2800 2024-03-16T10:10:43.510 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 3 on host 'smithi077' 2024-03-16T10:10:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.510+0000 7fd67782a700 1 -- 172.21.15.77:0/2745059834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd65c0a2800 msgr2=0x7fd65c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.510+0000 7fd67782a700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd65c0a2800 0x7fd65c0a4cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd660009c60 tx=0x7fd660009260 comp rx=0 tx=0).stop 2024-03-16T10:10:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.510+0000 7fd67782a700 1 -- 172.21.15.77:0/2745059834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd670101200 msgr2=0x7fd6701985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.510+0000 7fd67782a700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd670101200 0x7fd6701985a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00fab0 tx=0x7fd66c00fae0 comp rx=0 tx=0).stop 2024-03-16T10:10:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.510+0000 7fd67782a700 1 -- 172.21.15.77:0/2745059834 shutdown_connections 2024-03-16T10:10:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.510+0000 7fd67782a700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd65c0a2800 0x7fd65c0a4cc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.510+0000 7fd67782a700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd670102da0 0x7fd67019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.511+0000 7fd67782a700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd670101200 0x7fd6701985a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.511+0000 7fd67782a700 1 --2- 172.21.15.77:0/2745059834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd670102400 0x7fd670198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.511+0000 7fd67782a700 1 -- 172.21.15.77:0/2745059834 >> 172.21.15.77:0/2745059834 conn(0x7fd6700fc7a0 msgr2=0x7fd670069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:10:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.511+0000 7fd67782a700 1 -- 172.21.15.77:0/2745059834 shutdown_connections 2024-03-16T10:10:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:10:43.511+0000 7fd67782a700 1 -- 172.21.15.77:0/2745059834 wait complete. 2024-03-16T10:10:43.979 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:43 smithi077 ceph-mon[32400]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:43.980 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:43 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:43.980 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:43 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:44.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:43 smithi193 ceph-mon[36311]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:44.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:43 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:44.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:43 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:44.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:43 smithi192 ceph-mon[36354]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:44.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:43 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:44.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:43 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:44.264 DEBUG:teuthology.orchestra.run.smithi077:osd.3> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.3.service 2024-03-16T10:10:44.266 INFO:tasks.cephadm:Deploying osd.4 on smithi192 with /dev/vg_nvme/lv_3... 2024-03-16T10:10:44.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T10:10:44.510 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:10:44.914 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:44 smithi192 ceph-mon[36354]: from='osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T10:10:44.914 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:44 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:44.915 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:44 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:44.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:44 smithi077 ceph-mon[32400]: from='osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T10:10:44.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:44 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:44.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:44 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:45.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:44 smithi193 ceph-mon[36311]: from='osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T10:10:45.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:44 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:45.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:44 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:45 smithi192 ceph-mon[36354]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:45 smithi192 ceph-mon[36354]: from='osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T10:10:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:45 smithi192 ceph-mon[36354]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T10:10:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:45 smithi192 ceph-mon[36354]: from='osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T10:10:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:45 smithi077 ceph-mon[32400]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:45 smithi077 ceph-mon[32400]: from='osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T10:10:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:45 smithi077 ceph-mon[32400]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T10:10:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:45 smithi077 ceph-mon[32400]: from='osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T10:10:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:45.995 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:10:45 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3[51690]: 2024-03-16T10:10:45.735+0000 7ff255245700 -1 osd.3 0 waiting for initial osdmap 2024-03-16T10:10:45.995 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:10:45 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3[51690]: 2024-03-16T10:10:45.742+0000 7ff251038700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:10:46.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:45 smithi193 ceph-mon[36311]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:45 smithi193 ceph-mon[36311]: from='osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T10:10:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:45 smithi193 ceph-mon[36311]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T10:10:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:45 smithi193 ceph-mon[36311]: from='osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T10:10:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:46.567 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T10:10:46.567 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T10:10:46.568 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-16T10:10:46.568 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-16T10:10:46.568 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112568 s, 932 MB/s 2024-03-16T10:10:46.568 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-16T10:10:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:46 smithi077 ceph-mon[32400]: from='osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T10:10:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:46 smithi077 ceph-mon[32400]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T10:10:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:46 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:46 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:46 smithi077 ceph-mon[32400]: osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423] boot 2024-03-16T10:10:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:46 smithi077 ceph-mon[32400]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T10:10:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:46 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:47.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:46 smithi193 ceph-mon[36311]: from='osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T10:10:47.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:46 smithi193 ceph-mon[36311]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T10:10:47.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:46 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:47.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:46 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:47.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:46 smithi193 ceph-mon[36311]: osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423] boot 2024-03-16T10:10:47.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:46 smithi193 ceph-mon[36311]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T10:10:47.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:46 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:47.037 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:46 smithi192 ceph-mon[36354]: from='osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T10:10:47.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:46 smithi192 ceph-mon[36354]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T10:10:47.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:46 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:47.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:46 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:47.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:46 smithi192 ceph-mon[36354]: osd.3 [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423] boot 2024-03-16T10:10:47.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:46 smithi192 ceph-mon[36354]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T10:10:47.038 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:46 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T10:10:47.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2024-03-16T10:10:47.430 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:10:47.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:47 smithi192 ceph-mon[36354]: purged_snaps scrub starts 2024-03-16T10:10:47.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:47 smithi192 ceph-mon[36354]: purged_snaps scrub ok 2024-03-16T10:10:47.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:47 smithi192 ceph-mon[36354]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:47.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:47 smithi192 ceph-mon[36354]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T10:10:48.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:47 smithi193 ceph-mon[36311]: purged_snaps scrub starts 2024-03-16T10:10:48.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:47 smithi193 ceph-mon[36311]: purged_snaps scrub ok 2024-03-16T10:10:48.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:47 smithi193 ceph-mon[36311]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:48.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:47 smithi193 ceph-mon[36311]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T10:10:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:47 smithi077 ceph-mon[32400]: purged_snaps scrub starts 2024-03-16T10:10:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:47 smithi077 ceph-mon[32400]: purged_snaps scrub ok 2024-03-16T10:10:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:47 smithi077 ceph-mon[32400]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T10:10:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:47 smithi077 ceph-mon[32400]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T10:10:48.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.957+0000 7f306aa77700 1 -- 172.21.15.192:0/1753577919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3064101200 msgr2=0x7f3064101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:48.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.957+0000 7f306aa77700 1 --2- 172.21.15.192:0/1753577919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3064101200 0x7f3064101620 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f30580099c0 tx=0x7f3058009cd0 comp rx=0 tx=0).stop 2024-03-16T10:10:48.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.957+0000 7f306aa77700 1 -- 172.21.15.192:0/1753577919 shutdown_connections 2024-03-16T10:10:48.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.957+0000 7f306aa77700 1 --2- 172.21.15.192:0/1753577919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3064102da0 0x7f3064109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:48.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.957+0000 7f306aa77700 1 --2- 172.21.15.192:0/1753577919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3064101200 0x7f3064101620 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:48.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.957+0000 7f306aa77700 1 --2- 172.21.15.192:0/1753577919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3064102400 0x7f3064102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:48.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.957+0000 7f306aa77700 1 -- 172.21.15.192:0/1753577919 >> 172.21.15.192:0/1753577919 conn(0x7f30640fc7a0 msgr2=0x7f30640febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:10:48.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.957+0000 7f306aa77700 1 -- 172.21.15.192:0/1753577919 shutdown_connections 2024-03-16T10:10:48.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.957+0000 7f306aa77700 1 -- 172.21.15.192:0/1753577919 wait complete. 2024-03-16T10:10:48.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f306aa77700 1 Processor -- start 2024-03-16T10:10:48.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f306aa77700 1 -- start start 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f306aa77700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3064101200 0x7f306406aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f306aa77700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3064102400 0x7f306419d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f306aa77700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3064102da0 0x7f306419f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f306aa77700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f306410ce40 con 0x7f3064102400 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f306aa77700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f306410ccc0 con 0x7f3064101200 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f306aa77700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f306410cfc0 con 0x7f3064102da0 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f3068813700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3064101200 0x7f306406aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f3068813700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3064101200 0x7f306406aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:46268/0 (socket says 172.21.15.192:46268) 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f3068813700 1 -- 172.21.15.192:0/2947535181 learned_addr learned my addr 172.21.15.192:0/2947535181 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f3063fff700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3064102400 0x7f306419d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f3069014700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3064102da0 0x7f306419f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f3068813700 1 -- 172.21.15.192:0/2947535181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3064102da0 msgr2=0x7f306419f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f3068813700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3064102da0 0x7f306419f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f3068813700 1 -- 172.21.15.192:0/2947535181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3064102400 msgr2=0x7f306419d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:10:48.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f3068813700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3064102400 0x7f306419d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:10:48.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f3068813700 1 -- 172.21.15.192:0/2947535181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30641a0270 con 0x7f3064101200 2024-03-16T10:10:48.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.958+0000 7f3068813700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3064101200 0x7f306406aae0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3058003890 tx=0x7f3058009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:10:48.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.959+0000 7f3061ffb700 1 -- 172.21.15.192:0/2947535181 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3058027070 con 0x7f3064101200 2024-03-16T10:10:48.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.959+0000 7f3061ffb700 1 -- 172.21.15.192:0/2947535181 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30580058e0 con 0x7f3064101200 2024-03-16T10:10:48.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.959+0000 7f306aa77700 1 -- 172.21.15.192:0/2947535181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3064197130 con 0x7f3064101200 2024-03-16T10:10:48.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.959+0000 7f3061ffb700 1 -- 172.21.15.192:0/2947535181 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3058022ce0 con 0x7f3064101200 2024-03-16T10:10:48.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.959+0000 7f306aa77700 1 -- 172.21.15.192:0/2947535181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30641976e0 con 0x7f3064101200 2024-03-16T10:10:48.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.960+0000 7f306aa77700 1 -- 172.21.15.192:0/2947535181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3064068f90 con 0x7f3064101200 2024-03-16T10:10:48.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.961+0000 7f3061ffb700 1 -- 172.21.15.192:0/2947535181 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f305802a070 con 0x7f3064101200 2024-03-16T10:10:48.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.961+0000 7f3061ffb700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f30540a2910 0x7f30540a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:10:48.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.962+0000 7f3061ffb700 1 -- 172.21.15.192:0/2947535181 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f30580ce6d0 con 0x7f3064101200 2024-03-16T10:10:48.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.962+0000 7f3063fff700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f30540a2910 0x7f30540a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:10:48.963 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.962+0000 7f3063fff700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f30540a2910 0x7f30540a4dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f306406b060 tx=0x7f3050009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:10:48.967 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:48.966+0000 7f3061ffb700 1 -- 172.21.15.192:0/2947535181 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f305805e740 con 0x7f3064101200 2024-03-16T10:10:49.146 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: Detected new or changed devices on smithi077 2024-03-16T10:10:49.146 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.146 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.146 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:10:49.146 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:10:49.146 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi077 to 8544M 2024-03-16T10:10:49.147 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.147 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:49.147 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:10:49.147 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.147 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:10:49.147 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:49.147 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:10:49.147 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.147 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:10:49.147 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:48 smithi077 ceph-mon[32400]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T10:10:49.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: Detected new or changed devices on smithi077 2024-03-16T10:10:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:10:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:10:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi077 to 8544M 2024-03-16T10:10:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:10:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:10:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:10:49.168 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.168 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:10:49.168 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:48 smithi192 ceph-mon[36354]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T10:10:49.172 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:10:49.171+0000 7f306aa77700 1 -- 172.21.15.192:0/2947535181 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f3064062710 con 0x7f30540a2910 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: Detected new or changed devices on smithi077 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi077 to 8544M 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:10:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:10:49.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:10:49.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:48 smithi193 ceph-mon[36311]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T10:10:50.355 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:50 smithi077 ceph-mon[32400]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:10:50.355 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:50 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:10:50.356 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:50 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:10:50.356 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:50 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:50.356 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:50 smithi077 ceph-mon[32400]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T10:10:50.369 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:50 smithi192 ceph-mon[36354]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:10:50.369 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:50 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:10:50.369 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:50 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:10:50.369 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:50 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:50.370 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:50 smithi192 ceph-mon[36354]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T10:10:50.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:50 smithi193 ceph-mon[36311]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:10:50.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:50 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:10:50.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:50 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:10:50.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:50 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:10:50.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:50 smithi193 ceph-mon[36311]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T10:10:51.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:51 smithi077 ceph-mon[32400]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T10:10:51.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:51 smithi077 ceph-mon[32400]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T10:10:52.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:51 smithi193 ceph-mon[36311]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T10:10:52.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:51 smithi193 ceph-mon[36311]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T10:10:52.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:51 smithi192 ceph-mon[36354]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T10:10:52.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:51 smithi192 ceph-mon[36354]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T10:10:52.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:52 smithi077 ceph-mon[32400]: from='client.? 172.21.15.192:0/1520143182' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5101c752-e69f-4cff-af41-8818a614d0ba"}]: dispatch 2024-03-16T10:10:52.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:52 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5101c752-e69f-4cff-af41-8818a614d0ba"}]: dispatch 2024-03-16T10:10:52.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:52 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5101c752-e69f-4cff-af41-8818a614d0ba"}]': finished 2024-03-16T10:10:52.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:52 smithi077 ceph-mon[32400]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T10:10:52.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:52 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:10:52.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:52 smithi077 ceph-mon[32400]: from='client.? 172.21.15.192:0/4241960404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:10:53.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:52 smithi193 ceph-mon[36311]: from='client.? 172.21.15.192:0/1520143182' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5101c752-e69f-4cff-af41-8818a614d0ba"}]: dispatch 2024-03-16T10:10:53.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:52 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5101c752-e69f-4cff-af41-8818a614d0ba"}]: dispatch 2024-03-16T10:10:53.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:52 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5101c752-e69f-4cff-af41-8818a614d0ba"}]': finished 2024-03-16T10:10:53.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:52 smithi193 ceph-mon[36311]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T10:10:53.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:52 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:10:53.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:52 smithi193 ceph-mon[36311]: from='client.? 172.21.15.192:0/4241960404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:10:53.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:52 smithi192 ceph-mon[36354]: from='client.? 172.21.15.192:0/1520143182' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5101c752-e69f-4cff-af41-8818a614d0ba"}]: dispatch 2024-03-16T10:10:53.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:52 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5101c752-e69f-4cff-af41-8818a614d0ba"}]: dispatch 2024-03-16T10:10:53.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:52 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5101c752-e69f-4cff-af41-8818a614d0ba"}]': finished 2024-03-16T10:10:53.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:52 smithi192 ceph-mon[36354]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T10:10:53.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:52 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:10:53.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:52 smithi192 ceph-mon[36354]: from='client.? 172.21.15.192:0/4241960404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:10:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:53 smithi192 ceph-mon[36354]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T10:10:54.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:53 smithi077 ceph-mon[32400]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T10:10:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:53 smithi193 ceph-mon[36311]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T10:10:56.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:55 smithi192 ceph-mon[36354]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-16T10:10:56.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:55 smithi077 ceph-mon[32400]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-16T10:10:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:55 smithi193 ceph-mon[36311]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-16T10:10:58.009 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:57 smithi192 ceph-mon[36354]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:10:58.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:57 smithi077 ceph-mon[32400]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:10:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:57 smithi193 ceph-mon[36311]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:00.046 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:59 smithi192 ceph-mon[36354]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:00.046 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T10:11:00.046 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:10:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:00.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:59 smithi077 ceph-mon[32400]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:00.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T10:11:00.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:10:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:59 smithi193 ceph-mon[36311]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T10:11:00.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:10:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:01.027 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:00 smithi192 ceph-mon[36354]: Deploying daemon osd.4 on smithi192 2024-03-16T10:11:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:00 smithi077 ceph-mon[32400]: Deploying daemon osd.4 on smithi192 2024-03-16T10:11:01.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:00 smithi193 ceph-mon[36311]: Deploying daemon osd.4 on smithi192 2024-03-16T10:11:01.998 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:01 smithi192 ceph-mon[36354]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:02.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:01 smithi077 ceph-mon[32400]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:01 smithi193 ceph-mon[36311]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:04.106 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:03 smithi192 ceph-mon[36354]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:04.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:03 smithi077 ceph-mon[32400]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:03 smithi193 ceph-mon[36311]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:05.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:05 smithi192 ceph-mon[36354]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:05.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:05 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:05.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:05 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:05.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:05 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:05.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:05 smithi077 ceph-mon[32400]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:05.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:05 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:05.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:05 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:05.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:05 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:06.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:05 smithi193 ceph-mon[36311]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:06.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:05 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:06.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:05 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:06.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:05 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:07.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:07 smithi077 ceph-mon[32400]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:07.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:07 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:07.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:07 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:07.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:07 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:07.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:07 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:07.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:07 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:08.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:07 smithi193 ceph-mon[36311]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:08.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:07 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:08.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:07 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:08.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:07 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:08.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:07 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:08.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:07 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:08.130 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:07 smithi192 ceph-mon[36354]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:08.130 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:07 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:08.130 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:07 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:08.131 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:07 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:08.131 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:07 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:08.131 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:07 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:08.535 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.534+0000 7f3061ffb700 1 -- 172.21.15.192:0/2947535181 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3064062710 con 0x7f30540a2910 2024-03-16T10:11:08.536 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 4 on host 'smithi192' 2024-03-16T10:11:08.538 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.537+0000 7f306aa77700 1 -- 172.21.15.192:0/2947535181 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f30540a2910 msgr2=0x7f30540a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:11:08.538 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.537+0000 7f306aa77700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f30540a2910 0x7f30540a4dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f306406b060 tx=0x7f3050009190 comp rx=0 tx=0).stop 2024-03-16T10:11:08.538 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.537+0000 7f306aa77700 1 -- 172.21.15.192:0/2947535181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3064101200 msgr2=0x7f306406aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:11:08.538 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.537+0000 7f306aa77700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3064101200 0x7f306406aae0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3058003890 tx=0x7f3058009580 comp rx=0 tx=0).stop 2024-03-16T10:11:08.538 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.537+0000 7f306aa77700 1 -- 172.21.15.192:0/2947535181 shutdown_connections 2024-03-16T10:11:08.538 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.537+0000 7f306aa77700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f30540a2910 0x7f30540a4dd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:08.538 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.537+0000 7f306aa77700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3064102da0 0x7f306419f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:08.538 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.537+0000 7f306aa77700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3064101200 0x7f306406aae0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:08.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.537+0000 7f306aa77700 1 --2- 172.21.15.192:0/2947535181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3064102400 0x7f306419d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:08.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.537+0000 7f306aa77700 1 -- 172.21.15.192:0/2947535181 >> 172.21.15.192:0/2947535181 conn(0x7f30640fc7a0 msgr2=0x7f30641057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:11:08.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.537+0000 7f306aa77700 1 -- 172.21.15.192:0/2947535181 shutdown_connections 2024-03-16T10:11:08.539 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:11:08.537+0000 7f306aa77700 1 -- 172.21.15.192:0/2947535181 wait complete. 2024-03-16T10:11:09.149 DEBUG:teuthology.orchestra.run.smithi192:osd.4> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.4.service 2024-03-16T10:11:09.151 INFO:tasks.cephadm:Deploying osd.5 on smithi193 with /dev/vg_nvme/lv_3... 2024-03-16T10:11:09.151 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T10:11:09.392 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:11:09.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:09.719 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:09.719 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:09.719 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:09 smithi192 ceph-mon[36354]: from='osd.4 [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T10:11:09.719 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:09 smithi192 ceph-mon[36354]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T10:11:09.719 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:09 smithi192 ceph-mon[36354]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:09.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:09.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:09.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:09.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:09 smithi077 ceph-mon[32400]: from='osd.4 [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T10:11:09.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:09 smithi077 ceph-mon[32400]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T10:11:09.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:09 smithi077 ceph-mon[32400]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:09.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:09.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:09.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:09.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:09 smithi193 ceph-mon[36311]: from='osd.4 [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T10:11:09.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:09 smithi193 ceph-mon[36311]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T10:11:09.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:09 smithi193 ceph-mon[36311]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:10.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:10 smithi192 ceph-mon[36354]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T10:11:10.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:10 smithi192 ceph-mon[36354]: from='osd.4 [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:11:10.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:10 smithi192 ceph-mon[36354]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T10:11:10.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:10 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:10.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:10 smithi192 ceph-mon[36354]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:11:10.665 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:11:10 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4[47783]: 2024-03-16T10:11:10.503+0000 7f8d1dc54700 -1 osd.4 0 waiting for initial osdmap 2024-03-16T10:11:10.665 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:11:10 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4[47783]: 2024-03-16T10:11:10.508+0000 7f8d19a47700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:11:10.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:10 smithi193 ceph-mon[36311]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T10:11:10.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:10 smithi193 ceph-mon[36311]: from='osd.4 [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:11:10.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:10 smithi193 ceph-mon[36311]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T10:11:10.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:10 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:10.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:10 smithi193 ceph-mon[36311]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:11:10.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:10 smithi077 ceph-mon[32400]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T10:11:10.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:10 smithi077 ceph-mon[32400]: from='osd.4 [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:11:10.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:10 smithi077 ceph-mon[32400]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T10:11:10.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:10 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:10.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:10 smithi077 ceph-mon[32400]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:11:11.381 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T10:11:11.381 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T10:11:11.381 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-16T10:11:11.381 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-16T10:11:11.381 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121861 s, 860 MB/s 2024-03-16T10:11:11.382 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-16T10:11:11.703 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:11 smithi193 ceph-mon[36311]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-16T10:11:11.703 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:11 smithi193 ceph-mon[36311]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T10:11:11.703 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:11.703 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:11.703 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:11 smithi193 ceph-mon[36311]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:11.704 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:11.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:11 smithi192 ceph-mon[36354]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-16T10:11:11.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:11 smithi192 ceph-mon[36354]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T10:11:11.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:11.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:11.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:11 smithi192 ceph-mon[36354]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:11.863 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:11.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:11 smithi077 ceph-mon[32400]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-16T10:11:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:11 smithi077 ceph-mon[32400]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T10:11:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:11 smithi077 ceph-mon[32400]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-16T10:11:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:12.046 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_3 2024-03-16T10:11:12.279 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:11:12.688 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: purged_snaps scrub starts 2024-03-16T10:11:12.688 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: purged_snaps scrub ok 2024-03-16T10:11:12.688 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: osd.4 [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951] boot 2024-03-16T10:11:12.688 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T10:11:12.688 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:12.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:12.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:12.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:11:12.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:11:12.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:12.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:12.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:12.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:12 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:12.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: purged_snaps scrub starts 2024-03-16T10:11:12.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: purged_snaps scrub ok 2024-03-16T10:11:12.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: osd.4 [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951] boot 2024-03-16T10:11:12.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T10:11:12.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:12.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:12.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:12.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:11:12.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:11:12.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:12.918 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:12.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:12.919 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:12 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: purged_snaps scrub starts 2024-03-16T10:11:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: purged_snaps scrub ok 2024-03-16T10:11:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: osd.4 [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951] boot 2024-03-16T10:11:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T10:11:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T10:11:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:11:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:11:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:12.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:12.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:12 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:13.643 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:13 smithi193 ceph-mon[36311]: Detected new or changed devices on smithi192 2024-03-16T10:11:13.644 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:13 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi192 to 8540M 2024-03-16T10:11:13.644 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:13 smithi193 ceph-mon[36311]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T10:11:13.644 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:13 smithi193 ceph-mon[36311]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:13.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.862+0000 7f2248e59700 1 -- 172.21.15.193:0/3137170704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22441011f0 msgr2=0x7f2244101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:11:13.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.862+0000 7f2248e59700 1 --2- 172.21.15.193:0/3137170704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22441011f0 0x7f2244101610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2234009990 tx=0x7f2234009ca0 comp rx=0 tx=0).stop 2024-03-16T10:11:13.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.862+0000 7f2248e59700 1 -- 172.21.15.193:0/3137170704 shutdown_connections 2024-03-16T10:11:13.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.862+0000 7f2248e59700 1 --2- 172.21.15.193:0/3137170704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2244102d90 0x7f2244109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:13.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.862+0000 7f2248e59700 1 --2- 172.21.15.193:0/3137170704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22441011f0 0x7f2244101610 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:13.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.862+0000 7f2248e59700 1 --2- 172.21.15.193:0/3137170704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22441023f0 0x7f2244102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:13.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.862+0000 7f2248e59700 1 -- 172.21.15.193:0/3137170704 >> 172.21.15.193:0/3137170704 conn(0x7f22440fc790 msgr2=0x7f22440febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:11:13.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.862+0000 7f2248e59700 1 -- 172.21.15.193:0/3137170704 shutdown_connections 2024-03-16T10:11:13.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.862+0000 7f2248e59700 1 -- 172.21.15.193:0/3137170704 wait complete. 2024-03-16T10:11:13.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f2248e59700 1 Processor -- start 2024-03-16T10:11:13.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f2248e59700 1 -- start start 2024-03-16T10:11:13.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f2248e59700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22441011f0 0x7f224406b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:11:13.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f2248e59700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22441023f0 0x7f224419d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:11:13.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f2248e59700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2244102d90 0x7f224419f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:11:13.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f2248e59700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f224410cdf0 con 0x7f22441023f0 2024-03-16T10:11:13.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f2248e59700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f224410cc70 con 0x7f2244102d90 2024-03-16T10:11:13.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f2248e59700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f224410cf70 con 0x7f22441011f0 2024-03-16T10:11:13.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f224259c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22441011f0 0x7f224406b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:11:13.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f224259c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22441011f0 0x7f224406b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:43174/0 (socket says 172.21.15.193:43174) 2024-03-16T10:11:13.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f224259c700 1 -- 172.21.15.193:0/1247324629 learned_addr learned my addr 172.21.15.193:0/1247324629 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-16T10:11:13.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f2242d9d700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2244102d90 0x7f224419f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.863+0000 7f2241d9b700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22441023f0 0x7f224419d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.864+0000 7f224259c700 1 -- 172.21.15.193:0/1247324629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2244102d90 msgr2=0x7f224419f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.864+0000 7f224259c700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2244102d90 0x7f224419f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.864+0000 7f224259c700 1 -- 172.21.15.193:0/1247324629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22441023f0 msgr2=0x7f224419d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.864+0000 7f224259c700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22441023f0 0x7f224419d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.864+0000 7f224259c700 1 -- 172.21.15.193:0/1247324629 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22441a01a0 con 0x7f22441011f0 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.864+0000 7f2241d9b700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22441023f0 0x7f224419d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.864+0000 7f224259c700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22441011f0 0x7f224406b370 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f22340039c0 tx=0x7f22340047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.864+0000 7f22337fe700 1 -- 172.21.15.193:0/1247324629 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234003c70 con 0x7f22441011f0 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.864+0000 7f2248e59700 1 -- 172.21.15.193:0/1247324629 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22441a0430 con 0x7f22441011f0 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.864+0000 7f22337fe700 1 -- 172.21.15.193:0/1247324629 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2234004830 con 0x7f22441011f0 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.864+0000 7f22337fe700 1 -- 172.21.15.193:0/1247324629 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234005440 con 0x7f22441011f0 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.864+0000 7f2248e59700 1 -- 172.21.15.193:0/1247324629 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22441a0880 con 0x7f22441011f0 2024-03-16T10:11:13.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.865+0000 7f2248e59700 1 -- 172.21.15.193:0/1247324629 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2244068f90 con 0x7f22441011f0 2024-03-16T10:11:13.868 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.866+0000 7f22337fe700 1 -- 172.21.15.193:0/1247324629 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2234003e10 con 0x7f22441011f0 2024-03-16T10:11:13.868 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.866+0000 7f22337fe700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f222c0a2890 0x7f222c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:11:13.868 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.867+0000 7f2241d9b700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f222c0a2890 0x7f222c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:11:13.868 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.867+0000 7f22337fe700 1 -- 172.21.15.193:0/1247324629 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f22340ceab0 con 0x7f22441011f0 2024-03-16T10:11:13.869 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.867+0000 7f2241d9b700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f222c0a2890 0x7f222c0a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f224406b8f0 tx=0x7f223800b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:11:13.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:13.871+0000 7f22337fe700 1 -- 172.21.15.193:0/1247324629 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2234013070 con 0x7f22441011f0 2024-03-16T10:11:13.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:13 smithi192 ceph-mon[36354]: Detected new or changed devices on smithi192 2024-03-16T10:11:13.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:13 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi192 to 8540M 2024-03-16T10:11:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:13 smithi192 ceph-mon[36354]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T10:11:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:13 smithi192 ceph-mon[36354]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:13.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:13 smithi077 ceph-mon[32400]: Detected new or changed devices on smithi192 2024-03-16T10:11:13.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:13 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi192 to 8540M 2024-03-16T10:11:13.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:13 smithi077 ceph-mon[32400]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T10:11:13.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:13 smithi077 ceph-mon[32400]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:14.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:14.074+0000 7f2248e59700 1 -- 172.21.15.193:0/1247324629 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2244062710 con 0x7f222c0a2890 2024-03-16T10:11:14.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:14 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:11:14.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:14 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:11:14.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:14 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:14.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:14 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:11:14.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:14 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:11:14.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:14 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:14.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:11:14.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:11:14.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:15.875 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:15 smithi193 ceph-mon[36311]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:11:15.875 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:15 smithi193 ceph-mon[36311]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:15.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:15 smithi192 ceph-mon[36354]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:11:15.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:15 smithi192 ceph-mon[36354]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:15.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:15 smithi077 ceph-mon[32400]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:11:15.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:15 smithi077 ceph-mon[32400]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:16.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:16 smithi077 ceph-mon[32400]: from='client.? 172.21.15.193:0/1556461087' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e66d1765-4379-49e9-9fa8-531107546970"}]: dispatch 2024-03-16T10:11:16.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:16 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e66d1765-4379-49e9-9fa8-531107546970"}]: dispatch 2024-03-16T10:11:16.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:16 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e66d1765-4379-49e9-9fa8-531107546970"}]': finished 2024-03-16T10:11:16.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:16 smithi077 ceph-mon[32400]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T10:11:16.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:16 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:17.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:16 smithi193 ceph-mon[36311]: from='client.? 172.21.15.193:0/1556461087' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e66d1765-4379-49e9-9fa8-531107546970"}]: dispatch 2024-03-16T10:11:17.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:16 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e66d1765-4379-49e9-9fa8-531107546970"}]: dispatch 2024-03-16T10:11:17.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:16 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e66d1765-4379-49e9-9fa8-531107546970"}]': finished 2024-03-16T10:11:17.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:16 smithi193 ceph-mon[36311]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T10:11:17.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:16 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:17.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:16 smithi192 ceph-mon[36354]: from='client.? 172.21.15.193:0/1556461087' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e66d1765-4379-49e9-9fa8-531107546970"}]: dispatch 2024-03-16T10:11:17.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:16 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e66d1765-4379-49e9-9fa8-531107546970"}]: dispatch 2024-03-16T10:11:17.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:16 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e66d1765-4379-49e9-9fa8-531107546970"}]': finished 2024-03-16T10:11:17.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:16 smithi192 ceph-mon[36354]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T10:11:17.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:16 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:17.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:17 smithi077 ceph-mon[32400]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:17.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:17 smithi077 ceph-mon[32400]: from='client.? 172.21.15.193:0/2587853094' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:11:18.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:17 smithi193 ceph-mon[36311]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:18.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:17 smithi193 ceph-mon[36311]: from='client.? 172.21.15.193:0/2587853094' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:11:18.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:17 smithi192 ceph-mon[36354]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:18.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:17 smithi192 ceph-mon[36354]: from='client.? 172.21.15.193:0/2587853094' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:11:19.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:19 smithi077 ceph-mon[32400]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:20.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:19 smithi193 ceph-mon[36311]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:19 smithi192 ceph-mon[36354]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:21.986 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:21 smithi193 ceph-mon[36311]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:21.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:21 smithi077 ceph-mon[32400]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:22.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:21 smithi192 ceph-mon[36354]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:23.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:23 smithi077 ceph-mon[32400]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:24.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:23 smithi193 ceph-mon[36311]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:23 smithi192 ceph-mon[36354]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:24.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T10:11:24.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:24.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T10:11:24.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:25.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T10:11:25.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:25.863 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:25 smithi193 ceph-mon[36311]: Deploying daemon osd.5 on smithi193 2024-03-16T10:11:25.863 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:25 smithi193 ceph-mon[36311]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:25 smithi192 ceph-mon[36354]: Deploying daemon osd.5 on smithi193 2024-03-16T10:11:26.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:25 smithi192 ceph-mon[36354]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:25 smithi077 ceph-mon[32400]: Deploying daemon osd.5 on smithi193 2024-03-16T10:11:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:25 smithi077 ceph-mon[32400]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:28.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:27 smithi193 ceph-mon[36311]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:28.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:27 smithi192 ceph-mon[36354]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:27 smithi077 ceph-mon[32400]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:30.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:29 smithi193 ceph-mon[36311]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:30.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:29 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:30.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:29 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:30.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:29 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:30.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:29 smithi192 ceph-mon[36354]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:30.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:29 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:30.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:29 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:30.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:29 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:30.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:29 smithi077 ceph-mon[32400]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:30.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:29 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:30.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:29 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:30.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:29 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:32.031 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:31 smithi193 ceph-mon[36311]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:32.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:31 smithi192 ceph-mon[36354]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:32.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:31 smithi077 ceph-mon[32400]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:32.887 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.885+0000 7f22337fe700 1 -- 172.21.15.193:0/1247324629 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2244062710 con 0x7f222c0a2890 2024-03-16T10:11:32.887 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 5 on host 'smithi193' 2024-03-16T10:11:32.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.887+0000 7f2248e59700 1 -- 172.21.15.193:0/1247324629 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f222c0a2890 msgr2=0x7f222c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:11:32.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.887+0000 7f2248e59700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f222c0a2890 0x7f222c0a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f224406b8f0 tx=0x7f223800b2c0 comp rx=0 tx=0).stop 2024-03-16T10:11:32.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.887+0000 7f2248e59700 1 -- 172.21.15.193:0/1247324629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22441011f0 msgr2=0x7f224406b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:11:32.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.887+0000 7f2248e59700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22441011f0 0x7f224406b370 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f22340039c0 tx=0x7f22340047b0 comp rx=0 tx=0).stop 2024-03-16T10:11:32.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.887+0000 7f2248e59700 1 -- 172.21.15.193:0/1247324629 shutdown_connections 2024-03-16T10:11:32.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.887+0000 7f2248e59700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f222c0a2890 0x7f222c0a4d50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:32.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.887+0000 7f2248e59700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2244102d90 0x7f224419f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:32.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.887+0000 7f2248e59700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22441011f0 0x7f224406b370 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:32.889 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.887+0000 7f2248e59700 1 --2- 172.21.15.193:0/1247324629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22441023f0 0x7f224419d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:32.890 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.887+0000 7f2248e59700 1 -- 172.21.15.193:0/1247324629 >> 172.21.15.193:0/1247324629 conn(0x7f22440fc790 msgr2=0x7f2244105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:11:32.890 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.888+0000 7f2248e59700 1 -- 172.21.15.193:0/1247324629 shutdown_connections 2024-03-16T10:11:32.890 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:11:32.888+0000 7f2248e59700 1 -- 172.21.15.193:0/1247324629 wait complete. 2024-03-16T10:11:33.597 DEBUG:teuthology.orchestra.run.smithi193:osd.5> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.5.service 2024-03-16T10:11:33.600 INFO:tasks.cephadm:Deploying osd.6 on smithi077 with /dev/vg_nvme/lv_2... 2024-03-16T10:11:33.600 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T10:11:33.853 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:11:33.976 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:33 smithi077 ceph-mon[32400]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:33.976 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:33.976 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:33.976 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:33 smithi077 ceph-mon[32400]: from='osd.5 [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T10:11:33.977 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:33 smithi077 ceph-mon[32400]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T10:11:33.977 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:33.977 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:34.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:33 smithi193 ceph-mon[36311]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:34.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:34.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:34.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:33 smithi193 ceph-mon[36311]: from='osd.5 [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T10:11:34.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:33 smithi193 ceph-mon[36311]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T10:11:34.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:34.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:34.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:33 smithi192 ceph-mon[36354]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:34.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:34.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:34.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:33 smithi192 ceph-mon[36354]: from='osd.5 [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T10:11:34.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:33 smithi192 ceph-mon[36354]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T10:11:34.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:34.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:35.134 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:34 smithi077 ceph-mon[32400]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T10:11:35.135 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:34 smithi077 ceph-mon[32400]: from='osd.5 [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:11:35.135 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:34 smithi077 ceph-mon[32400]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T10:11:35.135 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:34 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:35.135 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:34 smithi077 ceph-mon[32400]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:11:35.135 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:34 smithi077 ceph-mon[32400]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:35.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:34 smithi192 ceph-mon[36354]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T10:11:35.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:34 smithi192 ceph-mon[36354]: from='osd.5 [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:11:35.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:34 smithi192 ceph-mon[36354]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T10:11:35.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:34 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:35.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:34 smithi192 ceph-mon[36354]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:11:35.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:34 smithi192 ceph-mon[36354]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:35.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:34 smithi193 ceph-mon[36311]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T10:11:35.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:34 smithi193 ceph-mon[36311]: from='osd.5 [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:11:35.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:34 smithi193 ceph-mon[36311]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T10:11:35.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:34 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:35.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:34 smithi193 ceph-mon[36311]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:11:35.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:34 smithi193 ceph-mon[36311]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-16T10:11:35.607 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:11:35 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:11:35.337+0000 7fd9e0a80700 -1 osd.5 0 waiting for initial osdmap 2024-03-16T10:11:35.608 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:11:35 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:11:35.346+0000 7fd9d806a700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:11:35.839 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T10:11:35.839 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T10:11:35.839 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-03-16T10:11:35.839 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-03-16T10:11:35.839 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124498 s, 842 MB/s 2024-03-16T10:11:35.840 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-03-16T10:11:36.132 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:35 smithi077 ceph-mon[32400]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-16T10:11:36.132 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:35 smithi077 ceph-mon[32400]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T10:11:36.133 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:36.133 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:36.133 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:35 smithi077 ceph-mon[32400]: from='osd.5 ' entity='osd.5' 2024-03-16T10:11:36.133 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:35 smithi077 ceph-mon[32400]: osd.5 [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054] boot 2024-03-16T10:11:36.133 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:35 smithi077 ceph-mon[32400]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T10:11:36.133 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:36.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:35 smithi192 ceph-mon[36354]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-16T10:11:36.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:35 smithi192 ceph-mon[36354]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T10:11:36.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:36.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:36.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:35 smithi192 ceph-mon[36354]: from='osd.5 ' entity='osd.5' 2024-03-16T10:11:36.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:35 smithi192 ceph-mon[36354]: osd.5 [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054] boot 2024-03-16T10:11:36.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:35 smithi192 ceph-mon[36354]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T10:11:36.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:35 smithi193 ceph-mon[36311]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-16T10:11:36.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:35 smithi193 ceph-mon[36311]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T10:11:36.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:36.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:36.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:35 smithi193 ceph-mon[36311]: from='osd.5 ' entity='osd.5' 2024-03-16T10:11:36.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:35 smithi193 ceph-mon[36311]: osd.5 [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054] boot 2024-03-16T10:11:36.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:35 smithi193 ceph-mon[36311]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T10:11:36.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T10:11:36.411 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_2 2024-03-16T10:11:36.659 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:11:36.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:36 smithi077 ceph-mon[32400]: purged_snaps scrub starts 2024-03-16T10:11:36.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:36 smithi077 ceph-mon[32400]: purged_snaps scrub ok 2024-03-16T10:11:36.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:36 smithi077 ceph-mon[32400]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:36.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:36 smithi077 ceph-mon[32400]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T10:11:37.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:36 smithi193 ceph-mon[36311]: purged_snaps scrub starts 2024-03-16T10:11:37.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:36 smithi193 ceph-mon[36311]: purged_snaps scrub ok 2024-03-16T10:11:37.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:36 smithi193 ceph-mon[36311]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:37.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:36 smithi193 ceph-mon[36311]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T10:11:37.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:36 smithi192 ceph-mon[36354]: purged_snaps scrub starts 2024-03-16T10:11:37.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:36 smithi192 ceph-mon[36354]: purged_snaps scrub ok 2024-03-16T10:11:37.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:36 smithi192 ceph-mon[36354]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:37.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:36 smithi192 ceph-mon[36354]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T10:11:38.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.345+0000 7f8dfcac0700 1 -- 172.21.15.77:0/1319335254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df81056b0 msgr2=0x7f8df8107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:11:38.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.345+0000 7f8dfcac0700 1 --2- 172.21.15.77:0/1319335254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df81056b0 0x7f8df8107aa0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8dec009990 tx=0x7f8dec009ca0 comp rx=0 tx=0).stop 2024-03-16T10:11:38.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.345+0000 7f8dfcac0700 1 -- 172.21.15.77:0/1319335254 shutdown_connections 2024-03-16T10:11:38.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.345+0000 7f8dfcac0700 1 --2- 172.21.15.77:0/1319335254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df8107fe0 0x7f8df810a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:38.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.345+0000 7f8dfcac0700 1 --2- 172.21.15.77:0/1319335254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8df8102d80 0x7f8df8105170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:38.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.345+0000 7f8dfcac0700 1 --2- 172.21.15.77:0/1319335254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df81056b0 0x7f8df8107aa0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:38.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.345+0000 7f8dfcac0700 1 -- 172.21.15.77:0/1319335254 >> 172.21.15.77:0/1319335254 conn(0x7f8df80fc790 msgr2=0x7f8df80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:11:38.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.345+0000 7f8dfcac0700 1 -- 172.21.15.77:0/1319335254 shutdown_connections 2024-03-16T10:11:38.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.345+0000 7f8dfcac0700 1 -- 172.21.15.77:0/1319335254 wait complete. 2024-03-16T10:11:38.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8dfcac0700 1 Processor -- start 2024-03-16T10:11:38.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8dfcac0700 1 -- start start 2024-03-16T10:11:38.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8dfcac0700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df8102d80 0x7f8df8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:11:38.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8dfcac0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df81056b0 0x7f8df8198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:11:38.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8dfcac0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8df8107fe0 0x7f8df819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:11:38.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8dfcac0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810df40 con 0x7f8df81056b0 2024-03-16T10:11:38.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8dfcac0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810ddc0 con 0x7f8df8107fe0 2024-03-16T10:11:38.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8dfcac0700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810e0c0 con 0x7f8df8102d80 2024-03-16T10:11:38.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8df5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df81056b0 0x7f8df8198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:11:38.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8df5d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df81056b0 0x7f8df8198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60450/0 (socket says 172.21.15.77:60450) 2024-03-16T10:11:38.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8df5d9b700 1 -- 172.21.15.77:0/2465964221 learned_addr learned my addr 172.21.15.77:0/2465964221 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:11:38.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8df659c700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df8102d80 0x7f8df8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:11:38.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.346+0000 7f8df6d9d700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8df8107fe0 0x7f8df819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:11:38.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.347+0000 7f8df5d9b700 1 -- 172.21.15.77:0/2465964221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df8102d80 msgr2=0x7f8df8198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:11:38.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.347+0000 7f8df5d9b700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df8102d80 0x7f8df8198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:38.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.347+0000 7f8df5d9b700 1 -- 172.21.15.77:0/2465964221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8df8107fe0 msgr2=0x7f8df819fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:11:38.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.347+0000 7f8df5d9b700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8df8107fe0 0x7f8df819fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:38.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.347+0000 7f8df5d9b700 1 -- 172.21.15.77:0/2465964221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df81a0180 con 0x7f8df81056b0 2024-03-16T10:11:38.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.347+0000 7f8df5d9b700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df81056b0 0x7f8df8198a60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8dec004790 tx=0x7f8dec009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:11:38.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.347+0000 7f8de77fe700 1 -- 172.21.15.77:0/2465964221 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec027070 con 0x7f8df81056b0 2024-03-16T10:11:38.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.347+0000 7f8dfcac0700 1 -- 172.21.15.77:0/2465964221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df81a0410 con 0x7f8df81056b0 2024-03-16T10:11:38.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.347+0000 7f8de77fe700 1 -- 172.21.15.77:0/2465964221 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dec004210 con 0x7f8df81056b0 2024-03-16T10:11:38.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.347+0000 7f8de77fe700 1 -- 172.21.15.77:0/2465964221 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec00f460 con 0x7f8df81056b0 2024-03-16T10:11:38.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.347+0000 7f8dfcac0700 1 -- 172.21.15.77:0/2465964221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df81a08e0 con 0x7f8df81056b0 2024-03-16T10:11:38.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.348+0000 7f8dfcac0700 1 -- 172.21.15.77:0/2465964221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8df8068f90 con 0x7f8df81056b0 2024-03-16T10:11:38.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.349+0000 7f8de77fe700 1 -- 172.21.15.77:0/2465964221 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8dec02a070 con 0x7f8df81056b0 2024-03-16T10:11:38.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.350+0000 7f8de77fe700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8de00a2890 0x7f8de00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:11:38.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.350+0000 7f8df659c700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8de00a2890 0x7f8de00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:11:38.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.350+0000 7f8de77fe700 1 -- 172.21.15.77:0/2465964221 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f8dec0ceed0 con 0x7f8df81056b0 2024-03-16T10:11:38.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.350+0000 7f8df659c700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8de00a2890 0x7f8de00a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8de8005950 tx=0x7f8de80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:11:38.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.353+0000 7f8de77fe700 1 -- 172.21.15.77:0/2465964221 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8dec013070 con 0x7f8df81056b0 2024-03-16T10:11:38.355 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: Detected new or changed devices on smithi193 2024-03-16T10:11:38.355 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.355 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.355 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:11:38.355 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:11:38.355 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi193 to 8540M 2024-03-16T10:11:38.355 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.355 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:38.355 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:38.356 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.356 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:38.356 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:38.356 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:38.356 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.356 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:38 smithi077 ceph-mon[32400]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T10:11:38.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: Detected new or changed devices on smithi193 2024-03-16T10:11:38.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:11:38.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:11:38.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi193 to 8540M 2024-03-16T10:11:38.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:38.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:38.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:38.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:38.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:38.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:38 smithi192 ceph-mon[36354]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T10:11:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: Detected new or changed devices on smithi193 2024-03-16T10:11:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:11:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:11:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi193 to 8540M 2024-03-16T10:11:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:38.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:38.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:38.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:38.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:38 smithi193 ceph-mon[36311]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T10:11:38.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:38.560+0000 7f8dfcac0700 1 -- 172.21.15.77:0/2465964221 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8df8062710 con 0x7f8de00a2890 2024-03-16T10:11:39.389 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:39 smithi077 ceph-mon[32400]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:11:39.389 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:39 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:11:39.389 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:39 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:11:39.389 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:39 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:39.389 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:39 smithi077 ceph-mon[32400]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:39.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:39 smithi192 ceph-mon[36354]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:11:39.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:39 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:11:39.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:39 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:11:39.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:39 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:39.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:39 smithi192 ceph-mon[36354]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:39.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:39 smithi193 ceph-mon[36311]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:11:39.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:39 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:11:39.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:39 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:11:39.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:39 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:39.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:39 smithi193 ceph-mon[36311]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:40.408 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:40 smithi077 ceph-mon[32400]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T10:11:40.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:40 smithi192 ceph-mon[36354]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T10:11:40.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:40 smithi193 ceph-mon[36311]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T10:11:41.404 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:41 smithi193 ceph-mon[36311]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T10:11:41.404 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:41 smithi193 ceph-mon[36311]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T10:11:41.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:41 smithi192 ceph-mon[36354]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T10:11:41.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:41 smithi192 ceph-mon[36354]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T10:11:41.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:41 smithi077 ceph-mon[32400]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T10:11:41.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:41 smithi077 ceph-mon[32400]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T10:11:42.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:42 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2530217820' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3ff0ae8-233d-4840-8cd1-a0050cd4aa43"}]: dispatch 2024-03-16T10:11:42.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:42 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2530217820' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3ff0ae8-233d-4840-8cd1-a0050cd4aa43"}]': finished 2024-03-16T10:11:42.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:42 smithi192 ceph-mon[36354]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T10:11:42.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:42 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:11:42.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:42 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2049339993' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:11:42.417 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:42 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2530217820' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3ff0ae8-233d-4840-8cd1-a0050cd4aa43"}]: dispatch 2024-03-16T10:11:42.418 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:42 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2530217820' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3ff0ae8-233d-4840-8cd1-a0050cd4aa43"}]': finished 2024-03-16T10:11:42.418 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:42 smithi193 ceph-mon[36311]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T10:11:42.418 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:42 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:11:42.418 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:42 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2049339993' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:11:42.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:42 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2530217820' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3ff0ae8-233d-4840-8cd1-a0050cd4aa43"}]: dispatch 2024-03-16T10:11:42.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:42 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2530217820' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3ff0ae8-233d-4840-8cd1-a0050cd4aa43"}]': finished 2024-03-16T10:11:42.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:42 smithi077 ceph-mon[32400]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T10:11:42.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:42 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:11:42.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:42 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2049339993' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:11:43.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:43 smithi192 ceph-mon[36354]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-16T10:11:43.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:43 smithi077 ceph-mon[32400]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-16T10:11:43.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:43 smithi193 ceph-mon[36311]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-16T10:11:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:45 smithi077 ceph-mon[32400]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T10:11:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:45 smithi193 ceph-mon[36311]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T10:11:46.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:45 smithi192 ceph-mon[36354]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T10:11:47.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:47 smithi077 ceph-mon[32400]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-16T10:11:48.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:47 smithi193 ceph-mon[36311]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-16T10:11:48.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:47 smithi192 ceph-mon[36354]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-16T10:11:48.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T10:11:48.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:49.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T10:11:49.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:49.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T10:11:49.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:49.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:49 smithi077 ceph-mon[32400]: Deploying daemon osd.6 on smithi077 2024-03-16T10:11:49.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:49 smithi077 ceph-mon[32400]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:50.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:49 smithi193 ceph-mon[36311]: Deploying daemon osd.6 on smithi077 2024-03-16T10:11:50.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:49 smithi193 ceph-mon[36311]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:50.016 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:49 smithi192 ceph-mon[36354]: Deploying daemon osd.6 on smithi077 2024-03-16T10:11:50.016 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:49 smithi192 ceph-mon[36354]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:51.933 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:51 smithi077 ceph-mon[32400]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:52.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:51 smithi193 ceph-mon[36311]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:52.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:51 smithi192 ceph-mon[36354]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:53.859 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:53 smithi077 ceph-mon[32400]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:54.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:53 smithi193 ceph-mon[36311]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:53 smithi192 ceph-mon[36354]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:55.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:55.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:55.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:55.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:55 smithi077 ceph-mon[32400]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:55.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:55.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:55.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:55.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:55 smithi192 ceph-mon[36354]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:55.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:55.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:55 smithi193 ceph-mon[36311]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:57.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:57 smithi077 ceph-mon[32400]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:57.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:57.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:57.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:57.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:57.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:58.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:57 smithi193 ceph-mon[36311]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:58.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:58.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:58.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:58.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:58.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:57 smithi192 ceph-mon[36354]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:58.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:11:58.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:11:58.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:58.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.227+0000 7f8de77fe700 1 -- 172.21.15.77:0/2465964221 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8df8062710 con 0x7f8de00a2890 2024-03-16T10:11:58.228 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 6 on host 'smithi077' 2024-03-16T10:11:58.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.230+0000 7f8dfcac0700 1 -- 172.21.15.77:0/2465964221 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8de00a2890 msgr2=0x7f8de00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:11:58.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.230+0000 7f8dfcac0700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8de00a2890 0x7f8de00a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8de8005950 tx=0x7f8de80058e0 comp rx=0 tx=0).stop 2024-03-16T10:11:58.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.230+0000 7f8dfcac0700 1 -- 172.21.15.77:0/2465964221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df81056b0 msgr2=0x7f8df8198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:11:58.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.230+0000 7f8dfcac0700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df81056b0 0x7f8df8198a60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8dec004790 tx=0x7f8dec009580 comp rx=0 tx=0).stop 2024-03-16T10:11:58.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.230+0000 7f8dfcac0700 1 -- 172.21.15.77:0/2465964221 shutdown_connections 2024-03-16T10:11:58.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.230+0000 7f8dfcac0700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8de00a2890 0x7f8de00a4d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:58.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.230+0000 7f8dfcac0700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8df8107fe0 0x7f8df819fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:58.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.230+0000 7f8dfcac0700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df8102d80 0x7f8df8198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:58.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.230+0000 7f8dfcac0700 1 --2- 172.21.15.77:0/2465964221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df81056b0 0x7f8df8198a60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:11:58.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.230+0000 7f8dfcac0700 1 -- 172.21.15.77:0/2465964221 >> 172.21.15.77:0/2465964221 conn(0x7f8df80fc790 msgr2=0x7f8df81036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:11:58.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.230+0000 7f8dfcac0700 1 -- 172.21.15.77:0/2465964221 shutdown_connections 2024-03-16T10:11:58.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:11:58.230+0000 7f8dfcac0700 1 -- 172.21.15.77:0/2465964221 wait complete. 2024-03-16T10:11:58.921 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:58 smithi077 ceph-mon[32400]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T10:11:58.922 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:58.922 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:58.922 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:58.947 DEBUG:teuthology.orchestra.run.smithi077:osd.6> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.6.service 2024-03-16T10:11:58.949 INFO:tasks.cephadm:Deploying osd.7 on smithi192 with /dev/vg_nvme/lv_2... 2024-03-16T10:11:58.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T10:11:59.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:58 smithi193 ceph-mon[36311]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T10:11:59.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:59.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:59.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:59.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:58 smithi192 ceph-mon[36354]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T10:11:59.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:59.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:11:59.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:11:59.196 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:11:59.887 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:59 smithi077 ceph-mon[32400]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:59.887 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:59 smithi077 ceph-mon[32400]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T10:11:59.887 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:59 smithi077 ceph-mon[32400]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T10:11:59.887 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:59 smithi077 ceph-mon[32400]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T10:11:59.887 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:11:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:11:59.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:59 smithi192 ceph-mon[36354]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:11:59.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:59 smithi192 ceph-mon[36354]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T10:11:59.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:59 smithi192 ceph-mon[36354]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T10:11:59.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:59 smithi192 ceph-mon[36354]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T10:11:59.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:11:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:00.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:59 smithi193 ceph-mon[36311]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:12:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:59 smithi193 ceph-mon[36311]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T10:12:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:59 smithi193 ceph-mon[36311]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T10:12:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:59 smithi193 ceph-mon[36311]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T10:12:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:11:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:00.495 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:12:00 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:12:00.168+0000 7fc901e0b700 -1 osd.6 0 waiting for initial osdmap 2024-03-16T10:12:00.495 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:12:00 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:12:00.176+0000 7fc8f93f5700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:12:00.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:00 smithi077 ceph-mon[32400]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T10:12:00.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:00 smithi077 ceph-mon[32400]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T10:12:00.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:00.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:00.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:00 smithi077 ceph-mon[32400]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' 2024-03-16T10:12:01.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:00 smithi193 ceph-mon[36311]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T10:12:01.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:00 smithi193 ceph-mon[36311]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T10:12:01.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:01.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:01.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:00 smithi193 ceph-mon[36311]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' 2024-03-16T10:12:01.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:00 smithi192 ceph-mon[36354]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T10:12:01.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:00 smithi192 ceph-mon[36354]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T10:12:01.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:01.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:01.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:00 smithi192 ceph-mon[36354]: from='osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736]' entity='osd.6' 2024-03-16T10:12:01.306 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T10:12:01.306 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T10:12:01.306 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-16T10:12:01.306 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-16T10:12:01.306 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112419 s, 933 MB/s 2024-03-16T10:12:01.307 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-16T10:12:01.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2024-03-16T10:12:02.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:01 smithi193 ceph-mon[36311]: purged_snaps scrub starts 2024-03-16T10:12:02.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:01 smithi193 ceph-mon[36311]: purged_snaps scrub ok 2024-03-16T10:12:02.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:01 smithi193 ceph-mon[36311]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:12:02.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:01 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:02.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:01 smithi193 ceph-mon[36311]: osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736] boot 2024-03-16T10:12:02.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:01 smithi193 ceph-mon[36311]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T10:12:02.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:01 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:02.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:01 smithi192 ceph-mon[36354]: purged_snaps scrub starts 2024-03-16T10:12:02.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:01 smithi192 ceph-mon[36354]: purged_snaps scrub ok 2024-03-16T10:12:02.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:01 smithi192 ceph-mon[36354]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:12:02.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:01 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:02.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:01 smithi192 ceph-mon[36354]: osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736] boot 2024-03-16T10:12:02.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:01 smithi192 ceph-mon[36354]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T10:12:02.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:01 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:02.205 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:12:02.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:01 smithi077 ceph-mon[32400]: purged_snaps scrub starts 2024-03-16T10:12:02.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:01 smithi077 ceph-mon[32400]: purged_snaps scrub ok 2024-03-16T10:12:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:01 smithi077 ceph-mon[32400]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-16T10:12:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:01 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:01 smithi077 ceph-mon[32400]: osd.6 [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736] boot 2024-03-16T10:12:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:01 smithi077 ceph-mon[32400]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T10:12:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:01 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T10:12:03.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: Detected new or changed devices on smithi077 2024-03-16T10:12:03.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:03.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:03.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:03.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:03.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:03.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi077 to 5696M 2024-03-16T10:12:03.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:03.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:03.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:03.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:03.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:03.631 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:03 smithi192 ceph-mon[36354]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T10:12:03.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: Detected new or changed devices on smithi077 2024-03-16T10:12:03.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:03.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:03.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:03.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:03.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:03.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi077 to 5696M 2024-03-16T10:12:03.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:03.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:03.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:03.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:03.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:03.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:03 smithi077 ceph-mon[32400]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T10:12:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: Detected new or changed devices on smithi077 2024-03-16T10:12:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi077 to 5696M 2024-03-16T10:12:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:03.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:03 smithi193 ceph-mon[36311]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T10:12:03.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.863+0000 7eff8dc8b700 1 -- 172.21.15.192:0/1499995807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff88107ff0 msgr2=0x7eff8810a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:03.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.863+0000 7eff8dc8b700 1 --2- 172.21.15.192:0/1499995807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff88107ff0 0x7eff8810a460 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7eff8000b0a0 tx=0x7eff8000b3b0 comp rx=0 tx=0).stop 2024-03-16T10:12:03.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.863+0000 7eff8dc8b700 1 -- 172.21.15.192:0/1499995807 shutdown_connections 2024-03-16T10:12:03.864 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.863+0000 7eff8dc8b700 1 --2- 172.21.15.192:0/1499995807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff88107ff0 0x7eff8810a460 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:03.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.863+0000 7eff8dc8b700 1 --2- 172.21.15.192:0/1499995807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff88102d90 0x7eff88105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:03.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.863+0000 7eff8dc8b700 1 --2- 172.21.15.192:0/1499995807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff881056c0 0x7eff88107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:03.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.863+0000 7eff8dc8b700 1 -- 172.21.15.192:0/1499995807 >> 172.21.15.192:0/1499995807 conn(0x7eff880fc7a0 msgr2=0x7eff880febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:12:03.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.863+0000 7eff8dc8b700 1 -- 172.21.15.192:0/1499995807 shutdown_connections 2024-03-16T10:12:03.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.863+0000 7eff8dc8b700 1 -- 172.21.15.192:0/1499995807 wait complete. 2024-03-16T10:12:03.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff8dc8b700 1 Processor -- start 2024-03-16T10:12:03.865 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff8dc8b700 1 -- start start 2024-03-16T10:12:03.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff8dc8b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff88102d90 0x7eff881026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:12:03.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff8dc8b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff881056c0 0x7eff88100d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:12:03.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff8dc8b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff88107ff0 0x7eff88101280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:12:03.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff8dc8b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810df10 con 0x7eff88102d90 2024-03-16T10:12:03.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff8dc8b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810dd90 con 0x7eff88107ff0 2024-03-16T10:12:03.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff8dc8b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810e090 con 0x7eff881056c0 2024-03-16T10:12:03.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff87fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff88107ff0 0x7eff88101280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:12:03.867 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff87fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff88107ff0 0x7eff88101280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:47006/0 (socket says 172.21.15.192:47006) 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff87fff700 1 -- 172.21.15.192:0/4074968320 learned_addr learned my addr 172.21.15.192:0/4074968320 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff877fe700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff88102d90 0x7eff881026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff86ffd700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff881056c0 0x7eff88100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff87fff700 1 -- 172.21.15.192:0/4074968320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff881056c0 msgr2=0x7eff88100d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff87fff700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff881056c0 0x7eff88100d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff87fff700 1 -- 172.21.15.192:0/4074968320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff88102d90 msgr2=0x7eff881026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff87fff700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff88102d90 0x7eff881026f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.864+0000 7eff87fff700 1 -- 172.21.15.192:0/4074968320 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff88101b30 con 0x7eff88107ff0 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.865+0000 7eff87fff700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff88107ff0 0x7eff88101280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7eff80009bc0 tx=0x7eff80004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.865+0000 7eff84ff9700 1 -- 172.21.15.192:0/4074968320 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff80021070 con 0x7eff88107ff0 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.865+0000 7eff8dc8b700 1 -- 172.21.15.192:0/4074968320 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff881a25c0 con 0x7eff88107ff0 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.865+0000 7eff84ff9700 1 -- 172.21.15.192:0/4074968320 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff80007b80 con 0x7eff88107ff0 2024-03-16T10:12:03.868 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.865+0000 7eff84ff9700 1 -- 172.21.15.192:0/4074968320 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff800047d0 con 0x7eff88107ff0 2024-03-16T10:12:03.869 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.865+0000 7eff8dc8b700 1 -- 172.21.15.192:0/4074968320 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff881a2a20 con 0x7eff88107ff0 2024-03-16T10:12:03.869 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.866+0000 7eff8dc8b700 1 -- 172.21.15.192:0/4074968320 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff88068f90 con 0x7eff88107ff0 2024-03-16T10:12:03.869 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.867+0000 7eff84ff9700 1 -- 172.21.15.192:0/4074968320 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7eff80003a00 con 0x7eff88107ff0 2024-03-16T10:12:03.870 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.867+0000 7eff84ff9700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff700a2890 0x7eff700a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:12:03.870 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.867+0000 7eff84ff9700 1 -- 172.21.15.192:0/4074968320 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7eff800ced80 con 0x7eff88107ff0 2024-03-16T10:12:03.870 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.868+0000 7eff877fe700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff700a2890 0x7eff700a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:12:03.870 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.868+0000 7eff877fe700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff700a2890 0x7eff700a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7eff78009c60 tx=0x7eff78009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:12:03.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:03.871+0000 7eff84ff9700 1 -- 172.21.15.192:0/4074968320 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7eff8009cbc0 con 0x7eff88107ff0 2024-03-16T10:12:04.075 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:04.074+0000 7eff8dc8b700 1 -- 172.21.15.192:0/4074968320 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7eff88062710 con 0x7eff700a2890 2024-03-16T10:12:04.675 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:04 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:12:04.675 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:04 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:12:04.675 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:04 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:04.744 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:04 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:12:04.744 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:04 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:12:04.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:04 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:04.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:04 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:12:04.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:04 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:12:04.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:04 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:05.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:05 smithi192 ceph-mon[36354]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:12:05.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:05 smithi192 ceph-mon[36354]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:05.744 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:05 smithi077 ceph-mon[32400]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:12:05.744 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:05 smithi077 ceph-mon[32400]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:05.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:05 smithi193 ceph-mon[36311]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:12:05.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:05 smithi193 ceph-mon[36311]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:07.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:07 smithi077 ceph-mon[32400]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:07.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:07 smithi077 ceph-mon[32400]: from='client.? 172.21.15.192:0/2774969981' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c422662-cf7a-478c-ae70-309d21cbb34d"}]: dispatch 2024-03-16T10:12:07.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:07 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c422662-cf7a-478c-ae70-309d21cbb34d"}]: dispatch 2024-03-16T10:12:07.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:07 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c422662-cf7a-478c-ae70-309d21cbb34d"}]': finished 2024-03-16T10:12:07.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:07 smithi077 ceph-mon[32400]: osdmap e48: 8 total, 7 up, 8 in 2024-03-16T10:12:07.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:07 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:07.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:07 smithi077 ceph-mon[32400]: from='client.? 172.21.15.192:0/346023767' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:12:08.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:07 smithi193 ceph-mon[36311]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:08.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:07 smithi193 ceph-mon[36311]: from='client.? 172.21.15.192:0/2774969981' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c422662-cf7a-478c-ae70-309d21cbb34d"}]: dispatch 2024-03-16T10:12:08.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:07 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c422662-cf7a-478c-ae70-309d21cbb34d"}]: dispatch 2024-03-16T10:12:08.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:07 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c422662-cf7a-478c-ae70-309d21cbb34d"}]': finished 2024-03-16T10:12:08.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:07 smithi193 ceph-mon[36311]: osdmap e48: 8 total, 7 up, 8 in 2024-03-16T10:12:08.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:07 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:08.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:07 smithi193 ceph-mon[36311]: from='client.? 172.21.15.192:0/346023767' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:12:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:07 smithi192 ceph-mon[36354]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:07 smithi192 ceph-mon[36354]: from='client.? 172.21.15.192:0/2774969981' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c422662-cf7a-478c-ae70-309d21cbb34d"}]: dispatch 2024-03-16T10:12:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:07 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c422662-cf7a-478c-ae70-309d21cbb34d"}]: dispatch 2024-03-16T10:12:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:07 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c422662-cf7a-478c-ae70-309d21cbb34d"}]': finished 2024-03-16T10:12:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:07 smithi192 ceph-mon[36354]: osdmap e48: 8 total, 7 up, 8 in 2024-03-16T10:12:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:07 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:07 smithi192 ceph-mon[36354]: from='client.? 172.21.15.192:0/346023767' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:12:09.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:09 smithi077 ceph-mon[32400]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:10.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:09 smithi193 ceph-mon[36311]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:10.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:09 smithi192 ceph-mon[36354]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:11 smithi077 ceph-mon[32400]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:12.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:11 smithi193 ceph-mon[36311]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:12.046 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:11 smithi192 ceph-mon[36354]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:13 smithi192 ceph-mon[36354]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:13.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:13 smithi077 ceph-mon[32400]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:14.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:13 smithi193 ceph-mon[36311]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:15.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:14 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T10:12:15.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:14 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:15.017 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:14 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T10:12:15.017 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:14 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:15.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T10:12:15.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:14 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:15.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:15 smithi192 ceph-mon[36354]: Deploying daemon osd.7 on smithi192 2024-03-16T10:12:15.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:15 smithi192 ceph-mon[36354]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:16.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:15 smithi193 ceph-mon[36311]: Deploying daemon osd.7 on smithi192 2024-03-16T10:12:16.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:15 smithi193 ceph-mon[36311]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:16.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:15 smithi077 ceph-mon[32400]: Deploying daemon osd.7 on smithi192 2024-03-16T10:12:16.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:15 smithi077 ceph-mon[32400]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:18.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:17 smithi193 ceph-mon[36311]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:18.069 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:17 smithi192 ceph-mon[36354]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:18.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:17 smithi077 ceph-mon[32400]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:20.047 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:19 smithi077 ceph-mon[32400]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:19 smithi192 ceph-mon[36354]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:19 smithi193 ceph-mon[36311]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:21.317 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:21 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:21.317 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:21 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:12:21.318 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:21 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:21.318 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:21 smithi077 ceph-mon[32400]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:21.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:21 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:21.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:21 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:12:21.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:21 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:21.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:21 smithi192 ceph-mon[36354]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:21.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:21 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:21.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:21 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:12:21.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:21 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:21.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:21 smithi193 ceph-mon[36311]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:23.541 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.540+0000 7eff84ff9700 1 -- 172.21.15.192:0/4074968320 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7eff88062710 con 0x7eff700a2890 2024-03-16T10:12:23.541 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 7 on host 'smithi192' 2024-03-16T10:12:23.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.543+0000 7eff8dc8b700 1 -- 172.21.15.192:0/4074968320 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff700a2890 msgr2=0x7eff700a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:23.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.543+0000 7eff8dc8b700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff700a2890 0x7eff700a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7eff78009c60 tx=0x7eff78009260 comp rx=0 tx=0).stop 2024-03-16T10:12:23.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.543+0000 7eff8dc8b700 1 -- 172.21.15.192:0/4074968320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff88107ff0 msgr2=0x7eff88101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:23.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.543+0000 7eff8dc8b700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff88107ff0 0x7eff88101280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7eff80009bc0 tx=0x7eff80004ce0 comp rx=0 tx=0).stop 2024-03-16T10:12:23.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.543+0000 7eff8dc8b700 1 -- 172.21.15.192:0/4074968320 shutdown_connections 2024-03-16T10:12:23.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.543+0000 7eff8dc8b700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff700a2890 0x7eff700a4d50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:23.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.543+0000 7eff8dc8b700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff88102d90 0x7eff881026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:23.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.543+0000 7eff8dc8b700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff88107ff0 0x7eff88101280 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:23.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.543+0000 7eff8dc8b700 1 --2- 172.21.15.192:0/4074968320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff881056c0 0x7eff88100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:23.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.543+0000 7eff8dc8b700 1 -- 172.21.15.192:0/4074968320 >> 172.21.15.192:0/4074968320 conn(0x7eff880fc7a0 msgr2=0x7eff88103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:12:23.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.543+0000 7eff8dc8b700 1 -- 172.21.15.192:0/4074968320 shutdown_connections 2024-03-16T10:12:23.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:12:23.543+0000 7eff8dc8b700 1 -- 172.21.15.192:0/4074968320 wait complete. 2024-03-16T10:12:23.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:23 smithi077 ceph-mon[32400]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:23.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:23 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:23.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:23 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:24.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:23 smithi193 ceph-mon[36311]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:24.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:23 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:24.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:23 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:23 smithi192 ceph-mon[36354]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:23 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:23 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:24.206 DEBUG:teuthology.orchestra.run.smithi192:osd.7> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.7.service 2024-03-16T10:12:24.208 INFO:tasks.cephadm:Deploying osd.8 on smithi193 with /dev/vg_nvme/lv_2... 2024-03-16T10:12:24.208 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T10:12:24.456 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:12:24.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:24 smithi193 ceph-mon[36311]: from='osd.7 [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T10:12:24.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:24 smithi193 ceph-mon[36311]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T10:12:24.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:24.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:24.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:24 smithi077 ceph-mon[32400]: from='osd.7 [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T10:12:24.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:24 smithi077 ceph-mon[32400]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T10:12:24.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:24.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:24.998 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:24 smithi192 ceph-mon[36354]: from='osd.7 [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T10:12:24.998 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:24 smithi192 ceph-mon[36354]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T10:12:24.998 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:24.998 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:25.842 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:25 smithi193 ceph-mon[36311]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:25.842 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:25 smithi193 ceph-mon[36311]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T10:12:25.842 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:25 smithi193 ceph-mon[36311]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T10:12:25.842 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:25 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:25.843 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:25 smithi193 ceph-mon[36311]: from='osd.7 [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:12:25.843 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:25 smithi193 ceph-mon[36311]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:12:26.085 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:25 smithi192 ceph-mon[36354]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:26.085 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:25 smithi192 ceph-mon[36354]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T10:12:26.085 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:25 smithi192 ceph-mon[36354]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T10:12:26.086 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:25 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:26.086 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:25 smithi192 ceph-mon[36354]: from='osd.7 [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:12:26.086 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:25 smithi192 ceph-mon[36354]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:12:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:25 smithi077 ceph-mon[32400]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:25 smithi077 ceph-mon[32400]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T10:12:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:25 smithi077 ceph-mon[32400]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T10:12:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:25 smithi077 ceph-mon[32400]: from='osd.7 [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:12:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:25 smithi077 ceph-mon[32400]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:12:26.348 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:12:26 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:12:26.177+0000 7f42226cc700 -1 osd.7 0 waiting for initial osdmap 2024-03-16T10:12:26.348 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:12:26 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:12:26.186+0000 7f421a4b7700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:12:26.453 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T10:12:26.453 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T10:12:26.453 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-16T10:12:26.453 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-16T10:12:26.453 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125301 s, 837 MB/s 2024-03-16T10:12:26.454 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-16T10:12:26.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:26 smithi193 ceph-mon[36311]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-16T10:12:26.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:26 smithi193 ceph-mon[36311]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T10:12:26.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:26 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:26.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:26 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:26.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:26 smithi193 ceph-mon[36311]: from='osd.7 ' entity='osd.7' 2024-03-16T10:12:27.041 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:26 smithi192 ceph-mon[36354]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-16T10:12:27.041 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:26 smithi192 ceph-mon[36354]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T10:12:27.041 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:26 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:27.041 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:26 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:27.041 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:26 smithi192 ceph-mon[36354]: from='osd.7 ' entity='osd.7' 2024-03-16T10:12:27.155 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_2 2024-03-16T10:12:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:26 smithi077 ceph-mon[32400]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-16T10:12:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:26 smithi077 ceph-mon[32400]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T10:12:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:26 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:26 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:26 smithi077 ceph-mon[32400]: from='osd.7 ' entity='osd.7' 2024-03-16T10:12:27.397 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:12:28.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:27 smithi193 ceph-mon[36311]: purged_snaps scrub starts 2024-03-16T10:12:28.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:27 smithi193 ceph-mon[36311]: purged_snaps scrub ok 2024-03-16T10:12:28.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:27 smithi193 ceph-mon[36311]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:28.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:27 smithi193 ceph-mon[36311]: osd.7 [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243] boot 2024-03-16T10:12:28.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:27 smithi193 ceph-mon[36311]: osdmap e51: 8 total, 8 up, 8 in 2024-03-16T10:12:28.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:28.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:27 smithi192 ceph-mon[36354]: purged_snaps scrub starts 2024-03-16T10:12:28.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:27 smithi192 ceph-mon[36354]: purged_snaps scrub ok 2024-03-16T10:12:28.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:27 smithi192 ceph-mon[36354]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:28.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:27 smithi192 ceph-mon[36354]: osd.7 [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243] boot 2024-03-16T10:12:28.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:27 smithi192 ceph-mon[36354]: osdmap e51: 8 total, 8 up, 8 in 2024-03-16T10:12:28.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:28.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:27 smithi077 ceph-mon[32400]: purged_snaps scrub starts 2024-03-16T10:12:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:27 smithi077 ceph-mon[32400]: purged_snaps scrub ok 2024-03-16T10:12:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:27 smithi077 ceph-mon[32400]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-16T10:12:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:27 smithi077 ceph-mon[32400]: osd.7 [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243] boot 2024-03-16T10:12:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:27 smithi077 ceph-mon[32400]: osdmap e51: 8 total, 8 up, 8 in 2024-03-16T10:12:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T10:12:28.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T10:12:28.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:28.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:28.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:28.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:28.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:28.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:28.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:28.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:28.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:28.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:12:28.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:28.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:28.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:28.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:28 smithi193 ceph-mon[36311]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T10:12:29.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.108+0000 7fa056f0c700 1 -- 172.21.15.193:0/2612440925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050102400 msgr2=0x7fa050102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:29.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.108+0000 7fa056f0c700 1 --2- 172.21.15.193:0/2612440925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050102400 0x7fa050102860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa03c009990 tx=0x7fa03c009ca0 comp rx=0 tx=0).stop 2024-03-16T10:12:29.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.108+0000 7fa056f0c700 1 -- 172.21.15.193:0/2612440925 shutdown_connections 2024-03-16T10:12:29.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.108+0000 7fa056f0c700 1 --2- 172.21.15.193:0/2612440925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa050101200 0x7fa050101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:29.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.108+0000 7fa056f0c700 1 --2- 172.21.15.193:0/2612440925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa050102da0 0x7fa050109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:29.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.108+0000 7fa056f0c700 1 --2- 172.21.15.193:0/2612440925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050102400 0x7fa050102860 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:29.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.108+0000 7fa056f0c700 1 -- 172.21.15.193:0/2612440925 >> 172.21.15.193:0/2612440925 conn(0x7fa0500fc7c0 msgr2=0x7fa0500febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:12:29.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.108+0000 7fa056f0c700 1 -- 172.21.15.193:0/2612440925 shutdown_connections 2024-03-16T10:12:29.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.108+0000 7fa056f0c700 1 -- 172.21.15.193:0/2612440925 wait complete. 2024-03-16T10:12:29.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa056f0c700 1 Processor -- start 2024-03-16T10:12:29.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa056f0c700 1 -- start start 2024-03-16T10:12:29.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa056f0c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050101200 0x7fa050197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:12:29.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa056f0c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa050102400 0x7fa050197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:12:29.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa056f0c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa050102da0 0x7fa05019e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:12:29.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa056f0c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa05010ce00 con 0x7fa050102400 2024-03-16T10:12:29.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa056f0c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa05010cc80 con 0x7fa050102da0 2024-03-16T10:12:29.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa056f0c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa05010cf80 con 0x7fa050101200 2024-03-16T10:12:29.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa054ca8700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050101200 0x7fa050197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:12:29.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa054ca8700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050101200 0x7fa050197410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:39734/0 (socket says 172.21.15.193:39734) 2024-03-16T10:12:29.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa054ca8700 1 -- 172.21.15.193:0/1633012025 learned_addr learned my addr 172.21.15.193:0/1633012025 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-16T10:12:29.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa04ffff700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa050102400 0x7fa050197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:12:29.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.109+0000 7fa0554a9700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa050102da0 0x7fa05019e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:12:29.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.110+0000 7fa054ca8700 1 -- 172.21.15.193:0/1633012025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa050102da0 msgr2=0x7fa05019e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:29.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.110+0000 7fa054ca8700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa050102da0 0x7fa05019e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:29.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.110+0000 7fa054ca8700 1 -- 172.21.15.193:0/1633012025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa050102400 msgr2=0x7fa050197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:29.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.110+0000 7fa054ca8700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa050102400 0x7fa050197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:29.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.110+0000 7fa054ca8700 1 -- 172.21.15.193:0/1633012025 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05019f070 con 0x7fa050101200 2024-03-16T10:12:29.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.110+0000 7fa0554a9700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa050102da0 0x7fa05019e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:12:29.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.110+0000 7fa054ca8700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050101200 0x7fa050197410 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa04400d9b0 tx=0x7fa04400dd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:12:29.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.110+0000 7fa04dffb700 1 -- 172.21.15.193:0/1633012025 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa044016070 con 0x7fa050101200 2024-03-16T10:12:29.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.110+0000 7fa056f0c700 1 -- 172.21.15.193:0/1633012025 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa05019f360 con 0x7fa050101200 2024-03-16T10:12:29.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.110+0000 7fa04dffb700 1 -- 172.21.15.193:0/1633012025 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa04400bb60 con 0x7fa050101200 2024-03-16T10:12:29.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.110+0000 7fa04dffb700 1 -- 172.21.15.193:0/1633012025 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04400f3c0 con 0x7fa050101200 2024-03-16T10:12:29.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.110+0000 7fa056f0c700 1 -- 172.21.15.193:0/1633012025 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa05019f8f0 con 0x7fa050101200 2024-03-16T10:12:29.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.111+0000 7fa056f0c700 1 -- 172.21.15.193:0/1633012025 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa050068f90 con 0x7fa050101200 2024-03-16T10:12:29.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.112+0000 7fa04dffb700 1 -- 172.21.15.193:0/1633012025 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa044019070 con 0x7fa050101200 2024-03-16T10:12:29.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.113+0000 7fa04dffb700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0400a2890 0x7fa0400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:12:29.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.113+0000 7fa04dffb700 1 -- 172.21.15.193:0/1633012025 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7fa0440cdb40 con 0x7fa050101200 2024-03-16T10:12:29.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.113+0000 7fa04ffff700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0400a2890 0x7fa0400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:12:29.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.113+0000 7fa04ffff700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0400a2890 0x7fa0400a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00b5c0 tx=0x7fa03c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:12:29.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.117+0000 7fa04dffb700 1 -- 172.21.15.193:0/1633012025 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa044020030 con 0x7fa050101200 2024-03-16T10:12:29.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:28 smithi192 ceph-mon[36354]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T10:12:29.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T10:12:29.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:29.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:29.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:29.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:29.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:29.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:29.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:29.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:29.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:29.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:12:29.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:29.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:29.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:29.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:28 smithi077 ceph-mon[32400]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T10:12:29.319 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:29.318+0000 7fa056f0c700 1 -- 172.21.15.193:0/1633012025 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa0500627d0 con 0x7fa0400a2890 2024-03-16T10:12:30.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:29 smithi193 ceph-mon[36311]: Detected new or changed devices on smithi192 2024-03-16T10:12:30.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:29 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi192 to 5693M 2024-03-16T10:12:30.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:29 smithi193 ceph-mon[36311]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:30.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:29 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:12:30.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:29 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:12:30.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:29 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:30.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:29 smithi192 ceph-mon[36354]: Detected new or changed devices on smithi192 2024-03-16T10:12:30.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:29 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi192 to 5693M 2024-03-16T10:12:30.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:29 smithi192 ceph-mon[36354]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:30.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:29 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:12:30.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:29 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:12:30.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:29 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:30.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:29 smithi077 ceph-mon[32400]: Detected new or changed devices on smithi192 2024-03-16T10:12:30.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:29 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi192 to 5693M 2024-03-16T10:12:30.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:29 smithi077 ceph-mon[32400]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:30.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:29 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:12:30.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:29 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:12:30.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:29 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:31.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:30 smithi193 ceph-mon[36311]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:12:31.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:30 smithi193 ceph-mon[36311]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T10:12:31.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:30 smithi192 ceph-mon[36354]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:12:31.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:30 smithi192 ceph-mon[36354]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T10:12:31.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:30 smithi077 ceph-mon[32400]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:12:31.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:30 smithi077 ceph-mon[32400]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T10:12:32.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:31 smithi192 ceph-mon[36354]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:32.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:31 smithi192 ceph-mon[36354]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T10:12:32.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:31 smithi077 ceph-mon[32400]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:32.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:31 smithi077 ceph-mon[32400]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T10:12:32.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:31 smithi193 ceph-mon[36311]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:32.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:31 smithi193 ceph-mon[36311]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T10:12:33.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:32 smithi192 ceph-mon[36354]: from='client.? 172.21.15.193:0/3502921350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09fc6b3a-3c26-42b2-a975-51c1c4617605"}]: dispatch 2024-03-16T10:12:33.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:32 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09fc6b3a-3c26-42b2-a975-51c1c4617605"}]: dispatch 2024-03-16T10:12:33.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:32 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09fc6b3a-3c26-42b2-a975-51c1c4617605"}]': finished 2024-03-16T10:12:33.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:32 smithi192 ceph-mon[36354]: osdmap e56: 9 total, 8 up, 9 in 2024-03-16T10:12:33.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:32 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:32 smithi077 ceph-mon[32400]: from='client.? 172.21.15.193:0/3502921350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09fc6b3a-3c26-42b2-a975-51c1c4617605"}]: dispatch 2024-03-16T10:12:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:32 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09fc6b3a-3c26-42b2-a975-51c1c4617605"}]: dispatch 2024-03-16T10:12:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:32 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09fc6b3a-3c26-42b2-a975-51c1c4617605"}]': finished 2024-03-16T10:12:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:32 smithi077 ceph-mon[32400]: osdmap e56: 9 total, 8 up, 9 in 2024-03-16T10:12:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:32 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:33.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:32 smithi193 ceph-mon[36311]: from='client.? 172.21.15.193:0/3502921350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09fc6b3a-3c26-42b2-a975-51c1c4617605"}]: dispatch 2024-03-16T10:12:33.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:32 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09fc6b3a-3c26-42b2-a975-51c1c4617605"}]: dispatch 2024-03-16T10:12:33.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:32 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09fc6b3a-3c26-42b2-a975-51c1c4617605"}]': finished 2024-03-16T10:12:33.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:32 smithi193 ceph-mon[36311]: osdmap e56: 9 total, 8 up, 9 in 2024-03-16T10:12:33.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:32 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:34.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:33 smithi192 ceph-mon[36354]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:34.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:33 smithi192 ceph-mon[36354]: from='client.? 172.21.15.193:0/695149654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:12:34.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:33 smithi077 ceph-mon[32400]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:34.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:33 smithi077 ceph-mon[32400]: from='client.? 172.21.15.193:0/695149654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:12:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:33 smithi193 ceph-mon[36311]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:33 smithi193 ceph-mon[36311]: from='client.? 172.21.15.193:0/695149654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:12:36.164 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:35 smithi193 ceph-mon[36311]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:36.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:35 smithi192 ceph-mon[36354]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:36.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:35 smithi077 ceph-mon[32400]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:38.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:37 smithi193 ceph-mon[36311]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:38.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:37 smithi192 ceph-mon[36354]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:37 smithi077 ceph-mon[32400]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:40.096 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:39 smithi193 ceph-mon[36311]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:40.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:39 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T10:12:40.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:39 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:40.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:39 smithi192 ceph-mon[36354]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:40.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:39 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T10:12:40.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:39 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:40.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:39 smithi077 ceph-mon[32400]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:40.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:39 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T10:12:40.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:39 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:40.850 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:40 smithi193 ceph-mon[36311]: Deploying daemon osd.8 on smithi193 2024-03-16T10:12:41.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:40 smithi192 ceph-mon[36354]: Deploying daemon osd.8 on smithi193 2024-03-16T10:12:41.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:40 smithi077 ceph-mon[32400]: Deploying daemon osd.8 on smithi193 2024-03-16T10:12:42.100 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:41 smithi193 ceph-mon[36311]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:42.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:41 smithi192 ceph-mon[36354]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:42.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:41 smithi077 ceph-mon[32400]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:43.915 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:43 smithi193 ceph-mon[36311]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:44.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:43 smithi192 ceph-mon[36354]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:44.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:43 smithi077 ceph-mon[32400]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:45 smithi193 ceph-mon[36311]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:12:46.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:45 smithi192 ceph-mon[36354]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:46.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:46.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:46.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:12:46.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:45 smithi077 ceph-mon[32400]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:46.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:46.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:46.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:12:48.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:47 smithi193 ceph-mon[36311]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:48.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:47 smithi192 ceph-mon[36354]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:48.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:47 smithi077 ceph-mon[32400]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:48.790 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.788+0000 7fa04dffb700 1 -- 172.21.15.193:0/1633012025 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa0500627d0 con 0x7fa0400a2890 2024-03-16T10:12:48.790 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 8 on host 'smithi193' 2024-03-16T10:12:48.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.791+0000 7fa056f0c700 1 -- 172.21.15.193:0/1633012025 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0400a2890 msgr2=0x7fa0400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:48.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.791+0000 7fa056f0c700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0400a2890 0x7fa0400a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00b5c0 tx=0x7fa03c005fb0 comp rx=0 tx=0).stop 2024-03-16T10:12:48.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.791+0000 7fa056f0c700 1 -- 172.21.15.193:0/1633012025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050101200 msgr2=0x7fa050197410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:48.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.791+0000 7fa056f0c700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050101200 0x7fa050197410 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa04400d9b0 tx=0x7fa04400dd70 comp rx=0 tx=0).stop 2024-03-16T10:12:48.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.791+0000 7fa056f0c700 1 -- 172.21.15.193:0/1633012025 shutdown_connections 2024-03-16T10:12:48.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.791+0000 7fa056f0c700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0400a2890 0x7fa0400a4d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:48.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.791+0000 7fa056f0c700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa050102da0 0x7fa05019e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:48.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.791+0000 7fa056f0c700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050101200 0x7fa050197410 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:48.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.791+0000 7fa056f0c700 1 --2- 172.21.15.193:0/1633012025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa050102400 0x7fa050197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:48.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.791+0000 7fa056f0c700 1 -- 172.21.15.193:0/1633012025 >> 172.21.15.193:0/1633012025 conn(0x7fa0500fc7c0 msgr2=0x7fa050105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:12:48.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.791+0000 7fa056f0c700 1 -- 172.21.15.193:0/1633012025 shutdown_connections 2024-03-16T10:12:48.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:12:48.791+0000 7fa056f0c700 1 -- 172.21.15.193:0/1633012025 wait complete. 2024-03-16T10:12:49.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:49 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:49 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:49 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:49.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:49 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:49.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:49 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:49 smithi193 ceph-mon[36311]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:49.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:49 smithi193 ceph-mon[36311]: from='osd.8 [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T10:12:49.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:49 smithi193 ceph-mon[36311]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T10:12:49.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:49 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:49 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:49 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:12:49.391 DEBUG:teuthology.orchestra.run.smithi193:osd.8> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.8.service 2024-03-16T10:12:49.394 INFO:tasks.cephadm:Deploying osd.9 on smithi077 with /dev/vg_nvme/lv_1... 2024-03-16T10:12:49.394 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T10:12:49.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:49 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:49 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:49 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:49.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:49 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:49.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:49 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:49 smithi192 ceph-mon[36354]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:49.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:49 smithi192 ceph-mon[36354]: from='osd.8 [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T10:12:49.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:49 smithi192 ceph-mon[36354]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T10:12:49.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:49 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:49 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:49 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:12:49.424 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:49 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.424 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:49 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.424 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:49 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:49.424 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:49 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:49.424 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:49 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.424 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:49 smithi077 ceph-mon[32400]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:49.424 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:49 smithi077 ceph-mon[32400]: from='osd.8 [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T10:12:49.425 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:49 smithi077 ceph-mon[32400]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T10:12:49.425 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:49 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.425 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:49 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:49.425 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:49 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:12:49.639 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:12:50.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:50 smithi193 ceph-mon[36311]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T10:12:50.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:50 smithi193 ceph-mon[36311]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T10:12:50.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:50 smithi193 ceph-mon[36311]: from='osd.8 [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:12:50.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:50 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:50.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:50 smithi193 ceph-mon[36311]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:12:50.286 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:50 smithi077 ceph-mon[32400]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T10:12:50.286 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:50 smithi077 ceph-mon[32400]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T10:12:50.286 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:50 smithi077 ceph-mon[32400]: from='osd.8 [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:12:50.286 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:50 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:50.286 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:50 smithi077 ceph-mon[32400]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:12:50.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:50 smithi192 ceph-mon[36354]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T10:12:50.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:50 smithi192 ceph-mon[36354]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T10:12:50.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:50 smithi192 ceph-mon[36354]: from='osd.8 [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:12:50.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:50 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:50.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:50 smithi192 ceph-mon[36354]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:12:50.598 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:12:50 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:12:50.502+0000 7fde2f59a700 -1 osd.8 0 waiting for initial osdmap 2024-03-16T10:12:50.598 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:12:50 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:12:50.512+0000 7fde27385700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:12:51.203 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:51 smithi077 ceph-mon[32400]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-16T10:12:51.203 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:51 smithi077 ceph-mon[32400]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T10:12:51.203 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:51 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:51.203 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:51 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:51.203 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:51 smithi077 ceph-mon[32400]: from='osd.8 ' entity='osd.8' 2024-03-16T10:12:51.203 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:51 smithi077 ceph-mon[32400]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:51.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:51 smithi193 ceph-mon[36311]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-16T10:12:51.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:51 smithi193 ceph-mon[36311]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T10:12:51.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:51 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:51.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:51 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:51.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:51 smithi193 ceph-mon[36311]: from='osd.8 ' entity='osd.8' 2024-03-16T10:12:51.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:51 smithi193 ceph-mon[36311]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:51.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:51 smithi192 ceph-mon[36354]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-16T10:12:51.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:51 smithi192 ceph-mon[36354]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T10:12:51.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:51 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:51.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:51 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:51.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:51 smithi192 ceph-mon[36354]: from='osd.8 ' entity='osd.8' 2024-03-16T10:12:51.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:51 smithi192 ceph-mon[36354]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-16T10:12:51.814 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T10:12:51.814 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T10:12:51.814 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-03-16T10:12:51.815 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-03-16T10:12:51.815 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124521 s, 842 MB/s 2024-03-16T10:12:51.815 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-03-16T10:12:52.281 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:52 smithi193 ceph-mon[36311]: purged_snaps scrub starts 2024-03-16T10:12:52.281 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:52 smithi193 ceph-mon[36311]: purged_snaps scrub ok 2024-03-16T10:12:52.281 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:52 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:52.281 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:52 smithi193 ceph-mon[36311]: osd.8 [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269] boot 2024-03-16T10:12:52.281 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:52 smithi193 ceph-mon[36311]: osdmap e59: 9 total, 9 up, 9 in 2024-03-16T10:12:52.282 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:52 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:52.347 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:52 smithi077 ceph-mon[32400]: purged_snaps scrub starts 2024-03-16T10:12:52.347 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:52 smithi077 ceph-mon[32400]: purged_snaps scrub ok 2024-03-16T10:12:52.347 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:52 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:52.347 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:52 smithi077 ceph-mon[32400]: osd.8 [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269] boot 2024-03-16T10:12:52.348 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:52 smithi077 ceph-mon[32400]: osdmap e59: 9 total, 9 up, 9 in 2024-03-16T10:12:52.348 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:52 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:52.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:52 smithi192 ceph-mon[36354]: purged_snaps scrub starts 2024-03-16T10:12:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:52 smithi192 ceph-mon[36354]: purged_snaps scrub ok 2024-03-16T10:12:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:52 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:52 smithi192 ceph-mon[36354]: osd.8 [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269] boot 2024-03-16T10:12:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:52 smithi192 ceph-mon[36354]: osdmap e59: 9 total, 9 up, 9 in 2024-03-16T10:12:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:52 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T10:12:52.612 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_1 2024-03-16T10:12:52.893 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:12:53.357 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T10:12:53.357 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: Detected new or changed devices on smithi193 2024-03-16T10:12:53.357 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:53.357 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:53.357 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:53.358 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:53.358 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:53.358 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi193 to 5693M 2024-03-16T10:12:53.358 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:53.358 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:53.358 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:53.358 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:53.358 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:53 smithi077 ceph-mon[32400]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:12:53.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T10:12:53.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: Detected new or changed devices on smithi193 2024-03-16T10:12:53.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:53.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:53.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:53.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:53.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:53.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi193 to 5693M 2024-03-16T10:12:53.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:53.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:53.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:53.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:53.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:53 smithi192 ceph-mon[36354]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:12:53.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T10:12:53.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: Detected new or changed devices on smithi193 2024-03-16T10:12:53.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:53.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:53.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:53.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:53.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:12:53.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi193 to 5693M 2024-03-16T10:12:53.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:53.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:53.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:12:53.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:12:53.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:53 smithi193 ceph-mon[36311]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:12:54.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:54 smithi192 ceph-mon[36354]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T10:12:54.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:54 smithi192 ceph-mon[36354]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T10:12:54.457 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:54 smithi077 ceph-mon[32400]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T10:12:54.458 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:54 smithi077 ceph-mon[32400]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T10:12:54.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:54 smithi193 ceph-mon[36311]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T10:12:54.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:54 smithi193 ceph-mon[36311]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T10:12:54.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.696+0000 7fa6349c1700 1 -- 172.21.15.77:0/2536172376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa630103f40 msgr2=0x7fa63010a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:54.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.696+0000 7fa6349c1700 1 --2- 172.21.15.77:0/2536172376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa630103f40 0x7fa63010a7f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa62800b0a0 tx=0x7fa62800b3b0 comp rx=0 tx=0).stop 2024-03-16T10:12:54.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.696+0000 7fa6349c1700 1 -- 172.21.15.77:0/2536172376 shutdown_connections 2024-03-16T10:12:54.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.696+0000 7fa6349c1700 1 --2- 172.21.15.77:0/2536172376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa630103f40 0x7fa63010a7f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:54.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.696+0000 7fa6349c1700 1 --2- 172.21.15.77:0/2536172376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6301005c0 0x7fa6301009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:54.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.696+0000 7fa6349c1700 1 --2- 172.21.15.77:0/2536172376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6301035a0 0x7fa630103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:54.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.696+0000 7fa6349c1700 1 -- 172.21.15.77:0/2536172376 >> 172.21.15.77:0/2536172376 conn(0x7fa6300fbf40 msgr2=0x7fa6300fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:12:54.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.696+0000 7fa6349c1700 1 -- 172.21.15.77:0/2536172376 shutdown_connections 2024-03-16T10:12:54.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.696+0000 7fa6349c1700 1 -- 172.21.15.77:0/2536172376 wait complete. 2024-03-16T10:12:54.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa6349c1700 1 Processor -- start 2024-03-16T10:12:54.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa6349c1700 1 -- start start 2024-03-16T10:12:54.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa6349c1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6301005c0 0x7fa630102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:12:54.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa6349c1700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6301035a0 0x7fa630101550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:12:54.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa6349c1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa630103f40 0x7fa630101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:12:54.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa6349c1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa63010deb0 con 0x7fa6301005c0 2024-03-16T10:12:54.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa6349c1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa63010dd30 con 0x7fa630103f40 2024-03-16T10:12:54.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa6349c1700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa63010e030 con 0x7fa6301035a0 2024-03-16T10:12:54.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa62e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6301005c0 0x7fa630102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:12:54.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa62e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6301005c0 0x7fa630102f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48122/0 (socket says 172.21.15.77:48122) 2024-03-16T10:12:54.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa62e59c700 1 -- 172.21.15.77:0/2667945834 learned_addr learned my addr 172.21.15.77:0/2667945834 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:12:54.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa62dd9b700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6301035a0 0x7fa630101550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:12:54.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.697+0000 7fa62ed9d700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa630103f40 0x7fa630101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:12:54.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.698+0000 7fa62e59c700 1 -- 172.21.15.77:0/2667945834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6301035a0 msgr2=0x7fa630101550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:54.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.698+0000 7fa62e59c700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6301035a0 0x7fa630101550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:54.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.698+0000 7fa62e59c700 1 -- 172.21.15.77:0/2667945834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa630103f40 msgr2=0x7fa630101a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:12:54.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.698+0000 7fa62e59c700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa630103f40 0x7fa630101a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:12:54.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.698+0000 7fa62e59c700 1 -- 172.21.15.77:0/2667945834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa630102340 con 0x7fa6301005c0 2024-03-16T10:12:54.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.698+0000 7fa62e59c700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6301005c0 0x7fa630102f00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa62000d980 tx=0x7fa62000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:12:54.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.698+0000 7fa6277fe700 1 -- 172.21.15.77:0/2667945834 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa620017030 con 0x7fa6301005c0 2024-03-16T10:12:54.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.698+0000 7fa6349c1700 1 -- 172.21.15.77:0/2667945834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6301a2600 con 0x7fa6301005c0 2024-03-16T10:12:54.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.698+0000 7fa6277fe700 1 -- 172.21.15.77:0/2667945834 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa620004500 con 0x7fa6301005c0 2024-03-16T10:12:54.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.698+0000 7fa6277fe700 1 -- 172.21.15.77:0/2667945834 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6200039e0 con 0x7fa6301005c0 2024-03-16T10:12:54.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.698+0000 7fa6349c1700 1 -- 172.21.15.77:0/2667945834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6301a2b10 con 0x7fa6301005c0 2024-03-16T10:12:54.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.699+0000 7fa6349c1700 1 -- 172.21.15.77:0/2667945834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa630068f90 con 0x7fa6301005c0 2024-03-16T10:12:54.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.700+0000 7fa6277fe700 1 -- 172.21.15.77:0/2667945834 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa62001a070 con 0x7fa6301005c0 2024-03-16T10:12:54.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.701+0000 7fa6277fe700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa61c0a2890 0x7fa61c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:12:54.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.701+0000 7fa62dd9b700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa61c0a2890 0x7fa61c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:12:54.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.701+0000 7fa6277fe700 1 -- 172.21.15.77:0/2667945834 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fa6200cdcb0 con 0x7fa6301005c0 2024-03-16T10:12:54.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.701+0000 7fa62dd9b700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa61c0a2890 0x7fa61c0a4d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa61800abd0 tx=0x7fa61800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:12:54.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.704+0000 7fa6277fe700 1 -- 172.21.15.77:0/2667945834 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa62009b700 con 0x7fa6301005c0 2024-03-16T10:12:54.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:12:54.913+0000 7fa6349c1700 1 -- 172.21.15.77:0/2667945834 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fa630062710 con 0x7fa61c0a2890 2024-03-16T10:12:55.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:55 smithi077 ceph-mon[32400]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T10:12:55.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:55 smithi077 ceph-mon[32400]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:12:55.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:12:55.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:12:55.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:55 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:55 smithi193 ceph-mon[36311]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T10:12:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:55 smithi193 ceph-mon[36311]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:12:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:12:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:12:55.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:55 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:55.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:55 smithi192 ceph-mon[36354]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T10:12:55.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:55 smithi192 ceph-mon[36354]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:12:55.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:12:55.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:12:55.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:55 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:12:56.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:56 smithi077 ceph-mon[32400]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:12:56.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:56 smithi193 ceph-mon[36311]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:12:56.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:56 smithi192 ceph-mon[36354]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:12:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:57 smithi077 ceph-mon[32400]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:12:57.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:57 smithi193 ceph-mon[36311]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:12:57.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:57 smithi192 ceph-mon[36354]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:12:58.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:58 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1664332245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe5dbc58-ce7a-48ac-816f-df30c3560a15"}]: dispatch 2024-03-16T10:12:58.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:58 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1664332245' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe5dbc58-ce7a-48ac-816f-df30c3560a15"}]': finished 2024-03-16T10:12:58.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:58 smithi077 ceph-mon[32400]: osdmap e64: 10 total, 9 up, 10 in 2024-03-16T10:12:58.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:12:58.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:58 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1664332245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe5dbc58-ce7a-48ac-816f-df30c3560a15"}]: dispatch 2024-03-16T10:12:58.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:58 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1664332245' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe5dbc58-ce7a-48ac-816f-df30c3560a15"}]': finished 2024-03-16T10:12:58.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:58 smithi193 ceph-mon[36311]: osdmap e64: 10 total, 9 up, 10 in 2024-03-16T10:12:58.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:12:58.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:58 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1664332245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe5dbc58-ce7a-48ac-816f-df30c3560a15"}]: dispatch 2024-03-16T10:12:58.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:58 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1664332245' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe5dbc58-ce7a-48ac-816f-df30c3560a15"}]': finished 2024-03-16T10:12:58.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:58 smithi192 ceph-mon[36354]: osdmap e64: 10 total, 9 up, 10 in 2024-03-16T10:12:58.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:12:59.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:59 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3572862772' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:12:59.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:12:59 smithi077 ceph-mon[32400]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-16T10:12:59.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:59 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3572862772' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:12:59.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:12:59 smithi193 ceph-mon[36311]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-16T10:12:59.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:59 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3572862772' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:12:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:12:59 smithi192 ceph-mon[36354]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-16T10:13:01.991 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:01 smithi077 ceph-mon[32400]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-16T10:13:02.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:01 smithi193 ceph-mon[36311]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-16T10:13:02.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:01 smithi192 ceph-mon[36354]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-16T10:13:04.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:03 smithi193 ceph-mon[36311]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-16T10:13:04.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:03 smithi077 ceph-mon[32400]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-16T10:13:04.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:03 smithi192 ceph-mon[36354]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-16T10:13:06.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:05 smithi193 ceph-mon[36311]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T10:13:06.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:05 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T10:13:06.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:05 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:06.034 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:05 smithi077 ceph-mon[32400]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T10:13:06.034 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:05 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T10:13:06.034 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:05 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:05 smithi192 ceph-mon[36354]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T10:13:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:05 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T10:13:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:05 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:07.116 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:06 smithi077 ceph-mon[32400]: Deploying daemon osd.9 on smithi077 2024-03-16T10:13:07.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:06 smithi192 ceph-mon[36354]: Deploying daemon osd.9 on smithi077 2024-03-16T10:13:07.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:06 smithi193 ceph-mon[36311]: Deploying daemon osd.9 on smithi077 2024-03-16T10:13:08.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:07 smithi192 ceph-mon[36354]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T10:13:08.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:07 smithi077 ceph-mon[32400]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T10:13:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:07 smithi193 ceph-mon[36311]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T10:13:09.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:09 smithi077 ceph-mon[32400]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:10.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:09 smithi192 ceph-mon[36354]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:10.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:09 smithi193 ceph-mon[36311]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:11.990 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:11 smithi077 ceph-mon[32400]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:11.990 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:11.990 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:11.990 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:13:12.055 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:11 smithi193 ceph-mon[36311]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:12.055 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:12.055 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:12.055 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:13:12.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:11 smithi192 ceph-mon[36354]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:12.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:12.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:12.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:13:13.906 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:13 smithi077 ceph-mon[32400]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:14.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:13 smithi192 ceph-mon[36354]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:14.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:13 smithi193 ceph-mon[36311]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:15.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.118+0000 7fa6277fe700 1 -- 172.21.15.77:0/2667945834 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa630062710 con 0x7fa61c0a2890 2024-03-16T10:13:15.120 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 9 on host 'smithi077' 2024-03-16T10:13:15.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.120+0000 7fa6349c1700 1 -- 172.21.15.77:0/2667945834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa61c0a2890 msgr2=0x7fa61c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:13:15.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.120+0000 7fa6349c1700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa61c0a2890 0x7fa61c0a4d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa61800abd0 tx=0x7fa61800a380 comp rx=0 tx=0).stop 2024-03-16T10:13:15.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.120+0000 7fa6349c1700 1 -- 172.21.15.77:0/2667945834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6301005c0 msgr2=0x7fa630102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:13:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.120+0000 7fa6349c1700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6301005c0 0x7fa630102f00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa62000d980 tx=0x7fa62000dc90 comp rx=0 tx=0).stop 2024-03-16T10:13:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.120+0000 7fa6349c1700 1 -- 172.21.15.77:0/2667945834 shutdown_connections 2024-03-16T10:13:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.120+0000 7fa6349c1700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa61c0a2890 0x7fa61c0a4d50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.120+0000 7fa6349c1700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6301005c0 0x7fa630102f00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.120+0000 7fa6349c1700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa630103f40 0x7fa630101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.120+0000 7fa6349c1700 1 --2- 172.21.15.77:0/2667945834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6301035a0 0x7fa630101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.120+0000 7fa6349c1700 1 -- 172.21.15.77:0/2667945834 >> 172.21.15.77:0/2667945834 conn(0x7fa6300fbf40 msgr2=0x7fa6301067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:13:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.121+0000 7fa6349c1700 1 -- 172.21.15.77:0/2667945834 shutdown_connections 2024-03-16T10:13:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:13:15.121+0000 7fa6349c1700 1 -- 172.21.15.77:0/2667945834 wait complete. 2024-03-16T10:13:15.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:15 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:15 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:15 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:15.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:15 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:13:15.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:15 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:15 smithi192 ceph-mon[36354]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:15.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:15 smithi192 ceph-mon[36354]: from='osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T10:13:15.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:15 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:15 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:15 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:13:15.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:15 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:15 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:15 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:15.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:15 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:13:15.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:15 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:15 smithi077 ceph-mon[32400]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:15.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:15 smithi077 ceph-mon[32400]: from='osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T10:13:15.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:15 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:15 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:15 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:13:15.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:15 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:15 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:15 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:15.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:15 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:13:15.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:15 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:15 smithi193 ceph-mon[36311]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:15.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:15 smithi193 ceph-mon[36311]: from='osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T10:13:15.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:15 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:15 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:15.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:15 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:13:15.825 DEBUG:teuthology.orchestra.run.smithi077:osd.9> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.9.service 2024-03-16T10:13:15.828 INFO:tasks.cephadm:Deploying osd.10 on smithi192 with /dev/vg_nvme/lv_1... 2024-03-16T10:13:15.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T10:13:16.075 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:13:16.453 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:16 smithi192 ceph-mon[36354]: from='osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T10:13:16.453 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:16 smithi192 ceph-mon[36354]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T10:13:16.454 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:16 smithi192 ceph-mon[36354]: from='osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T10:13:16.454 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:16 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:13:16.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:16 smithi077 ceph-mon[32400]: from='osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T10:13:16.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:16 smithi077 ceph-mon[32400]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T10:13:16.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:16 smithi077 ceph-mon[32400]: from='osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T10:13:16.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:16 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:13:16.495 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:13:16 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:13:16.346+0000 7f79c0b88700 -1 osd.9 0 waiting for initial osdmap 2024-03-16T10:13:16.495 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:13:16 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:13:16.357+0000 7f79bc97b700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:13:16.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:16 smithi193 ceph-mon[36311]: from='osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T10:13:16.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:16 smithi193 ceph-mon[36311]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T10:13:16.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:16 smithi193 ceph-mon[36311]: from='osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-03-16T10:13:16.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:16 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:13:17.534 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:17 smithi192 ceph-mon[36354]: from='osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T10:13:17.534 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:17 smithi192 ceph-mon[36354]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T10:13:17.534 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:17 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:13:17.534 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:17 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:13:17.534 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:17 smithi192 ceph-mon[36354]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:17.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:17 smithi077 ceph-mon[32400]: from='osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T10:13:17.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:17 smithi077 ceph-mon[32400]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T10:13:17.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:17 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:13:17.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:17 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:13:17.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:17 smithi077 ceph-mon[32400]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:17.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:17 smithi193 ceph-mon[36311]: from='osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-03-16T10:13:17.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:17 smithi193 ceph-mon[36311]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T10:13:17.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:17 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:13:17.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:17 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:13:17.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:17 smithi193 ceph-mon[36311]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-16T10:13:18.113 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T10:13:18.113 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T10:13:18.113 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-16T10:13:18.113 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-16T10:13:18.113 INFO:teuthology.orchestra.run.smithi192.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112115 s, 935 MB/s 2024-03-16T10:13:18.113 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 2024-03-16T10:13:18.113 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-16T10:13:18.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:18 smithi192 ceph-mon[36354]: purged_snaps scrub starts 2024-03-16T10:13:18.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:18 smithi192 ceph-mon[36354]: purged_snaps scrub ok 2024-03-16T10:13:18.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:18 smithi192 ceph-mon[36354]: osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086] boot 2024-03-16T10:13:18.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:18 smithi192 ceph-mon[36354]: osdmap e67: 10 total, 10 up, 10 in 2024-03-16T10:13:18.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:18 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:13:18.729 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:18 smithi077 ceph-mon[32400]: purged_snaps scrub starts 2024-03-16T10:13:18.729 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:18 smithi077 ceph-mon[32400]: purged_snaps scrub ok 2024-03-16T10:13:18.729 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:18 smithi077 ceph-mon[32400]: osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086] boot 2024-03-16T10:13:18.729 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:18 smithi077 ceph-mon[32400]: osdmap e67: 10 total, 10 up, 10 in 2024-03-16T10:13:18.729 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:18 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:13:18.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:18 smithi193 ceph-mon[36311]: purged_snaps scrub starts 2024-03-16T10:13:18.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:18 smithi193 ceph-mon[36311]: purged_snaps scrub ok 2024-03-16T10:13:18.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:18 smithi193 ceph-mon[36311]: osd.9 [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086] boot 2024-03-16T10:13:18.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:18 smithi193 ceph-mon[36311]: osdmap e67: 10 total, 10 up, 10 in 2024-03-16T10:13:18.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:18 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T10:13:18.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_1 2024-03-16T10:13:19.056 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:13:19.438 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: Detected new or changed devices on smithi077 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi077 to 4272M 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:13:19.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:19.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T10:13:19.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:19.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: Detected new or changed devices on smithi077 2024-03-16T10:13:19.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:19.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:19.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:19.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:19.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:19.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:19.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi077 to 4272M 2024-03-16T10:13:19.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:19.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:19.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:13:19.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T10:13:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: Detected new or changed devices on smithi077 2024-03-16T10:13:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi077 to 4272M 2024-03-16T10:13:19.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:19.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:19.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:13:19.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:20.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.772+0000 7ff517961700 1 -- 172.21.15.192:0/3008606265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff510101410 msgr2=0x7ff510101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:13:20.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.772+0000 7ff517961700 1 --2- 172.21.15.192:0/3008606265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff510101410 0x7ff510101890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0099c0 tx=0x7ff50c009cd0 comp rx=0 tx=0).stop 2024-03-16T10:13:20.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.772+0000 7ff517961700 1 -- 172.21.15.192:0/3008606265 shutdown_connections 2024-03-16T10:13:20.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.772+0000 7ff517961700 1 --2- 172.21.15.192:0/3008606265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5101027b0 0x7ff510102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:20.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.772+0000 7ff517961700 1 --2- 172.21.15.192:0/3008606265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff510101dd0 0x7ff510109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:20.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.772+0000 7ff517961700 1 --2- 172.21.15.192:0/3008606265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff510101410 0x7ff510101890 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:20.775 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.772+0000 7ff517961700 1 -- 172.21.15.192:0/3008606265 >> 172.21.15.192:0/3008606265 conn(0x7ff5100fbf40 msgr2=0x7ff5100fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:13:20.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.772+0000 7ff517961700 1 -- 172.21.15.192:0/3008606265 shutdown_connections 2024-03-16T10:13:20.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.772+0000 7ff517961700 1 -- 172.21.15.192:0/3008606265 wait complete. 2024-03-16T10:13:20.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff517961700 1 Processor -- start 2024-03-16T10:13:20.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff517961700 1 -- start start 2024-03-16T10:13:20.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff517961700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff510101410 0x7ff5101051b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:13:20.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff517961700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff510101dd0 0x7ff510103800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:13:20.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff517961700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5101027b0 0x7ff510103d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:13:20.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff517961700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff51010ce30 con 0x7ff510101dd0 2024-03-16T10:13:20.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff517961700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff51010ccb0 con 0x7ff5101027b0 2024-03-16T10:13:20.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff517961700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff51010cfb0 con 0x7ff510101410 2024-03-16T10:13:20.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff515efe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5101027b0 0x7ff510103d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:13:20.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff515efe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5101027b0 0x7ff510103d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:57982/0 (socket says 172.21.15.192:57982) 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff515efe700 1 -- 172.21.15.192:0/2054077895 learned_addr learned my addr 172.21.15.192:0/2054077895 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff5156fd700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff510101410 0x7ff5101051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff514efc700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff510101dd0 0x7ff510103800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff515efe700 1 -- 172.21.15.192:0/2054077895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff510101410 msgr2=0x7ff5101051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff515efe700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff510101410 0x7ff5101051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff515efe700 1 -- 172.21.15.192:0/2054077895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff510101dd0 msgr2=0x7ff510103800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff515efe700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff510101dd0 0x7ff510103800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.773+0000 7ff515efe700 1 -- 172.21.15.192:0/2054077895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5101045f0 con 0x7ff5101027b0 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.774+0000 7ff515efe700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5101027b0 0x7ff510103d40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff50800ada0 tx=0x7ff50800c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.774+0000 7ff5067fc700 1 -- 172.21.15.192:0/2054077895 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff508017020 con 0x7ff5101027b0 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.774+0000 7ff5067fc700 1 -- 172.21.15.192:0/2054077895 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff50800a0d0 con 0x7ff5101027b0 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.774+0000 7ff517961700 1 -- 172.21.15.192:0/2054077895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5101a26d0 con 0x7ff5101027b0 2024-03-16T10:13:20.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.774+0000 7ff5067fc700 1 -- 172.21.15.192:0/2054077895 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff508005480 con 0x7ff5101027b0 2024-03-16T10:13:20.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.774+0000 7ff517961700 1 -- 172.21.15.192:0/2054077895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5101a2c10 con 0x7ff5101027b0 2024-03-16T10:13:20.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.775+0000 7ff517961700 1 -- 172.21.15.192:0/2054077895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff510068f90 con 0x7ff5101027b0 2024-03-16T10:13:20.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.776+0000 7ff5067fc700 1 -- 172.21.15.192:0/2054077895 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff508005620 con 0x7ff5101027b0 2024-03-16T10:13:20.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.777+0000 7ff5067fc700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff4fc0a2890 0x7ff4fc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:13:20.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.777+0000 7ff5067fc700 1 -- 172.21.15.192:0/2054077895 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7ff5080cd9f0 con 0x7ff5101027b0 2024-03-16T10:13:20.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.777+0000 7ff5156fd700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff4fc0a2890 0x7ff4fc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:13:20.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.777+0000 7ff5156fd700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff4fc0a2890 0x7ff4fc0a4d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff500009a10 tx=0x7ff500009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:13:20.782 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.781+0000 7ff5067fc700 1 -- 172.21.15.192:0/2054077895 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff508021020 con 0x7ff5101027b0 2024-03-16T10:13:20.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:20.985+0000 7ff517961700 1 -- 172.21.15.192:0/2054077895 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff5100fe0d0 con 0x7ff4fc0a2890 2024-03-16T10:13:21.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:21 smithi192 ceph-mon[36354]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:21.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:21 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:13:21.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:21 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:13:21.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:21 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:22.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:21 smithi193 ceph-mon[36311]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:22.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:21 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:13:22.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:21 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:13:22.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:21 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:21 smithi077 ceph-mon[32400]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:21 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:13:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:21 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:13:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:21 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:22.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:22 smithi192 ceph-mon[36354]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:13:22.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:22 smithi077 ceph-mon[32400]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:13:23.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:22 smithi193 ceph-mon[36311]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:13:24.114 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:23 smithi192 ceph-mon[36354]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:24.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:23 smithi077 ceph-mon[32400]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:23 smithi193 ceph-mon[36311]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:25.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:24 smithi192 ceph-mon[36354]: from='client.? 172.21.15.192:0/817040197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c77ee11a-014b-40fa-bb49-297f4144eec5"}]: dispatch 2024-03-16T10:13:25.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:24 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c77ee11a-014b-40fa-bb49-297f4144eec5"}]: dispatch 2024-03-16T10:13:25.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:24 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c77ee11a-014b-40fa-bb49-297f4144eec5"}]': finished 2024-03-16T10:13:25.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:24 smithi192 ceph-mon[36354]: osdmap e69: 11 total, 10 up, 11 in 2024-03-16T10:13:25.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:25.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:24 smithi192 ceph-mon[36354]: from='client.? 172.21.15.192:0/4102913809' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:13:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:24 smithi077 ceph-mon[32400]: from='client.? 172.21.15.192:0/817040197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c77ee11a-014b-40fa-bb49-297f4144eec5"}]: dispatch 2024-03-16T10:13:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:24 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c77ee11a-014b-40fa-bb49-297f4144eec5"}]: dispatch 2024-03-16T10:13:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:24 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c77ee11a-014b-40fa-bb49-297f4144eec5"}]': finished 2024-03-16T10:13:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:24 smithi077 ceph-mon[32400]: osdmap e69: 11 total, 10 up, 11 in 2024-03-16T10:13:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:24 smithi077 ceph-mon[32400]: from='client.? 172.21.15.192:0/4102913809' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:13:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:24 smithi193 ceph-mon[36311]: from='client.? 172.21.15.192:0/817040197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c77ee11a-014b-40fa-bb49-297f4144eec5"}]: dispatch 2024-03-16T10:13:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:24 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c77ee11a-014b-40fa-bb49-297f4144eec5"}]: dispatch 2024-03-16T10:13:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:24 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c77ee11a-014b-40fa-bb49-297f4144eec5"}]': finished 2024-03-16T10:13:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:24 smithi193 ceph-mon[36311]: osdmap e69: 11 total, 10 up, 11 in 2024-03-16T10:13:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:24 smithi193 ceph-mon[36311]: from='client.? 172.21.15.192:0/4102913809' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:13:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:25 smithi192 ceph-mon[36354]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:26.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:25 smithi077 ceph-mon[32400]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:25 smithi193 ceph-mon[36311]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:28.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:27 smithi192 ceph-mon[36354]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:27 smithi077 ceph-mon[32400]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:27 smithi193 ceph-mon[36311]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:30.040 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:29 smithi192 ceph-mon[36354]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:30.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:29 smithi077 ceph-mon[32400]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:30.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:29 smithi193 ceph-mon[36311]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:32.092 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:31 smithi192 ceph-mon[36354]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:32.092 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T10:13:32.092 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:32.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:31 smithi077 ceph-mon[32400]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:32.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T10:13:32.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:31 smithi193 ceph-mon[36311]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:32.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T10:13:32.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:32.812 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:32 smithi192 ceph-mon[36354]: Deploying daemon osd.10 on smithi192 2024-03-16T10:13:33.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:32 smithi077 ceph-mon[32400]: Deploying daemon osd.10 on smithi192 2024-03-16T10:13:33.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:32 smithi193 ceph-mon[36311]: Deploying daemon osd.10 on smithi192 2024-03-16T10:13:34.041 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:33 smithi192 ceph-mon[36354]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:34.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:33 smithi077 ceph-mon[32400]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:33 smithi193 ceph-mon[36311]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:36.054 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:35 smithi192 ceph-mon[36354]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:36.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:35 smithi077 ceph-mon[32400]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:35 smithi193 ceph-mon[36311]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:37.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:37 smithi192 ceph-mon[36354]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:37.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:37.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:37.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:13:38.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:37 smithi077 ceph-mon[32400]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:13:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:37 smithi193 ceph-mon[36311]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:38.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:13:40.154 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:39 smithi192 ceph-mon[36354]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:40.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:39 smithi077 ceph-mon[32400]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:40.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:39 smithi193 ceph-mon[36311]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:40.550 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 10 on host 'smithi192' 2024-03-16T10:13:40.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.549+0000 7ff5067fc700 1 -- 172.21.15.192:0/2054077895 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ff5100fe0d0 con 0x7ff4fc0a2890 2024-03-16T10:13:40.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.552+0000 7ff517961700 1 -- 172.21.15.192:0/2054077895 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff4fc0a2890 msgr2=0x7ff4fc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:13:40.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.552+0000 7ff517961700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff4fc0a2890 0x7ff4fc0a4d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff500009a10 tx=0x7ff500009190 comp rx=0 tx=0).stop 2024-03-16T10:13:40.553 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.552+0000 7ff517961700 1 -- 172.21.15.192:0/2054077895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5101027b0 msgr2=0x7ff510103d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:13:40.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.552+0000 7ff517961700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5101027b0 0x7ff510103d40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff50800ada0 tx=0x7ff50800c330 comp rx=0 tx=0).stop 2024-03-16T10:13:40.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.553+0000 7ff517961700 1 -- 172.21.15.192:0/2054077895 shutdown_connections 2024-03-16T10:13:40.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.553+0000 7ff517961700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff4fc0a2890 0x7ff4fc0a4d50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:40.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.553+0000 7ff517961700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5101027b0 0x7ff510103d40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:40.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.553+0000 7ff517961700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff510101410 0x7ff5101051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:40.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.553+0000 7ff517961700 1 --2- 172.21.15.192:0/2054077895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff510101dd0 0x7ff510103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:40.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.553+0000 7ff517961700 1 -- 172.21.15.192:0/2054077895 >> 172.21.15.192:0/2054077895 conn(0x7ff5100fbf40 msgr2=0x7ff5100fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:13:40.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.553+0000 7ff517961700 1 -- 172.21.15.192:0/2054077895 shutdown_connections 2024-03-16T10:13:40.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-16T10:13:40.553+0000 7ff517961700 1 -- 172.21.15.192:0/2054077895 wait complete. 2024-03-16T10:13:41.260 DEBUG:teuthology.orchestra.run.smithi192:osd.10> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.10.service 2024-03-16T10:13:41.262 INFO:tasks.cephadm:Deploying osd.11 on smithi193 with /dev/vg_nvme/lv_1... 2024-03-16T10:13:41.262 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T10:13:41.509 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:13:41.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:41 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:41.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:41 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:41.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:41 smithi193 ceph-mon[36311]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:41.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:41 smithi193 ceph-mon[36311]: from='osd.10 [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T10:13:41.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:41 smithi193 ceph-mon[36311]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T10:13:41.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:41 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:41.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:41 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:41.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:41 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:41.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:41 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:41.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:41 smithi192 ceph-mon[36354]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:41.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:41 smithi192 ceph-mon[36354]: from='osd.10 [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T10:13:41.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:41 smithi192 ceph-mon[36354]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T10:13:41.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:41 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:41.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:41 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:41.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:41 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:41.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:41 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:41.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:41 smithi077 ceph-mon[32400]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:41.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:41 smithi077 ceph-mon[32400]: from='osd.10 [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T10:13:41.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:41 smithi077 ceph-mon[32400]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T10:13:41.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:41 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:41.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:41 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:42.839 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:42 smithi193 ceph-mon[36311]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T10:13:42.840 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:42 smithi193 ceph-mon[36311]: from='osd.10 [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:13:42.840 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:42 smithi193 ceph-mon[36311]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T10:13:42.840 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:42 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:42.840 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:42 smithi193 ceph-mon[36311]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:13:42.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:42 smithi192 ceph-mon[36354]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T10:13:42.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:42 smithi192 ceph-mon[36354]: from='osd.10 [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:13:42.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:42 smithi192 ceph-mon[36354]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T10:13:42.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:42 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:42.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:42 smithi192 ceph-mon[36354]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:13:42.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:13:42 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:13:42.534+0000 7f1bdbfce700 -1 osd.10 0 waiting for initial osdmap 2024-03-16T10:13:42.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:13:42 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:13:42.544+0000 7f1bd7dc1700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:13:42.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:42 smithi077 ceph-mon[32400]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T10:13:42.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:42 smithi077 ceph-mon[32400]: from='osd.10 [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:13:42.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:42 smithi077 ceph-mon[32400]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T10:13:42.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:42 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:42.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:42 smithi077 ceph-mon[32400]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-16T10:13:43.559 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T10:13:43.559 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T10:13:43.559 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-16T10:13:43.559 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-16T10:13:43.559 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127529 s, 822 MB/s 2024-03-16T10:13:43.559 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-16T10:13:43.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:43 smithi192 ceph-mon[36354]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-16T10:13:43.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:43 smithi192 ceph-mon[36354]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T10:13:43.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:43 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:43.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:43 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:43.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:43 smithi192 ceph-mon[36354]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:43.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:43 smithi192 ceph-mon[36354]: osd.10 [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377] boot 2024-03-16T10:13:43.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:43 smithi192 ceph-mon[36354]: osdmap e72: 11 total, 11 up, 11 in 2024-03-16T10:13:43.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:43 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:43.861 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:43 smithi193 ceph-mon[36311]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-16T10:13:43.861 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:43 smithi193 ceph-mon[36311]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T10:13:43.861 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:43 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:43.861 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:43 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:43.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:43 smithi193 ceph-mon[36311]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:43.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:43 smithi193 ceph-mon[36311]: osd.10 [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377] boot 2024-03-16T10:13:43.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:43 smithi193 ceph-mon[36311]: osdmap e72: 11 total, 11 up, 11 in 2024-03-16T10:13:43.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:43 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:43.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:43 smithi077 ceph-mon[32400]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-16T10:13:43.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:43 smithi077 ceph-mon[32400]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T10:13:43.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:43 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:43.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:43 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:43.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:43 smithi077 ceph-mon[32400]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-16T10:13:43.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:43 smithi077 ceph-mon[32400]: osd.10 [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377] boot 2024-03-16T10:13:43.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:43 smithi077 ceph-mon[32400]: osdmap e72: 11 total, 11 up, 11 in 2024-03-16T10:13:43.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:43 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T10:13:44.352 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_1 2024-03-16T10:13:44.592 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:13:44.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:44 smithi193 ceph-mon[36311]: purged_snaps scrub starts 2024-03-16T10:13:44.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:44 smithi193 ceph-mon[36311]: purged_snaps scrub ok 2024-03-16T10:13:44.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:44 smithi193 ceph-mon[36311]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T10:13:44.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:44 smithi192 ceph-mon[36354]: purged_snaps scrub starts 2024-03-16T10:13:44.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:44 smithi192 ceph-mon[36354]: purged_snaps scrub ok 2024-03-16T10:13:44.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:44 smithi192 ceph-mon[36354]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T10:13:44.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:44 smithi077 ceph-mon[32400]: purged_snaps scrub starts 2024-03-16T10:13:44.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:44 smithi077 ceph-mon[32400]: purged_snaps scrub ok 2024-03-16T10:13:44.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:44 smithi077 ceph-mon[32400]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T10:13:45.872 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.873 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:45 smithi193 ceph-mon[36311]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T10:13:45.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:13:45.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:45 smithi192 ceph-mon[36354]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T10:13:45.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:13:45.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:45 smithi077 ceph-mon[32400]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T10:13:46.211 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.208+0000 7f66f18cd700 1 -- 172.21.15.193:0/332914801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ec103f40 msgr2=0x7f66ec10a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:13:46.211 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.208+0000 7f66f18cd700 1 --2- 172.21.15.193:0/332914801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ec103f40 0x7f66ec10a7f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f66e400b0a0 tx=0x7f66e400b3b0 comp rx=0 tx=0).stop 2024-03-16T10:13:46.211 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.208+0000 7f66f18cd700 1 -- 172.21.15.193:0/332914801 shutdown_connections 2024-03-16T10:13:46.211 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.208+0000 7f66f18cd700 1 --2- 172.21.15.193:0/332914801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66ec1005c0 0x7f66ec1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:46.211 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.208+0000 7f66f18cd700 1 --2- 172.21.15.193:0/332914801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ec103f40 0x7f66ec10a7f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:46.211 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.208+0000 7f66f18cd700 1 --2- 172.21.15.193:0/332914801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ec1035a0 0x7f66ec103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:46.211 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.208+0000 7f66f18cd700 1 -- 172.21.15.193:0/332914801 >> 172.21.15.193:0/332914801 conn(0x7f66ec0fbf40 msgr2=0x7f66ec0fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:13:46.211 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.208+0000 7f66f18cd700 1 -- 172.21.15.193:0/332914801 shutdown_connections 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.208+0000 7f66f18cd700 1 -- 172.21.15.193:0/332914801 wait complete. 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.208+0000 7f66f18cd700 1 Processor -- start 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66f18cd700 1 -- start start 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66f18cd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ec1005c0 0x7f66ec198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66f18cd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ec1035a0 0x7f66ec1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66f18cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66ec103f40 0x7f66ec19fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66f18cd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec10deb0 con 0x7f66ec103f40 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66f18cd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec10dd30 con 0x7f66ec1035a0 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66f18cd700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec10e030 con 0x7f66ec1005c0 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66eaffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ec1005c0 0x7f66ec198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66eaffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ec1005c0 0x7f66ec198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:36762/0 (socket says 172.21.15.193:36762) 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66eaffd700 1 -- 172.21.15.193:0/2065642466 learned_addr learned my addr 172.21.15.193:0/2065642466 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66ea7fc700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ec1035a0 0x7f66ec1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66eb7fe700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66ec103f40 0x7f66ec19fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66eaffd700 1 -- 172.21.15.193:0/2065642466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ec1035a0 msgr2=0x7f66ec1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66eaffd700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ec1035a0 0x7f66ec1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:46.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66eaffd700 1 -- 172.21.15.193:0/2065642466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66ec103f40 msgr2=0x7f66ec19fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:13:46.213 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66eaffd700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66ec103f40 0x7f66ec19fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:13:46.213 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66eaffd700 1 -- 172.21.15.193:0/2065642466 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec1a00f0 con 0x7f66ec1005c0 2024-03-16T10:13:46.213 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.209+0000 7f66eaffd700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ec1005c0 0x7f66ec198490 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00d9b0 tx=0x7f66dc00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:13:46.213 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.210+0000 7f66f08cb700 1 -- 172.21.15.193:0/2065642466 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc017030 con 0x7f66ec1005c0 2024-03-16T10:13:46.213 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.210+0000 7f66f18cd700 1 -- 172.21.15.193:0/2065642466 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66ec1a03e0 con 0x7f66ec1005c0 2024-03-16T10:13:46.213 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.210+0000 7f66f08cb700 1 -- 172.21.15.193:0/2065642466 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66dc004500 con 0x7f66ec1005c0 2024-03-16T10:13:46.213 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.210+0000 7f66f08cb700 1 -- 172.21.15.193:0/2065642466 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc003970 con 0x7f66ec1005c0 2024-03-16T10:13:46.213 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.210+0000 7f66f18cd700 1 -- 172.21.15.193:0/2065642466 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66ec1a0920 con 0x7f66ec1005c0 2024-03-16T10:13:46.214 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.211+0000 7f66f18cd700 1 -- 172.21.15.193:0/2065642466 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66ec068f90 con 0x7f66ec1005c0 2024-03-16T10:13:46.214 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.212+0000 7f66f08cb700 1 -- 172.21.15.193:0/2065642466 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f66dc01a070 con 0x7f66ec1005c0 2024-03-16T10:13:46.214 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.212+0000 7f66f08cb700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f66d40a2890 0x7f66d40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:13:46.214 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.212+0000 7f66f08cb700 1 -- 172.21.15.193:0/2065642466 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f66dc0cdcc0 con 0x7f66ec1005c0 2024-03-16T10:13:46.214 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.212+0000 7f66ea7fc700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f66d40a2890 0x7f66d40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:13:46.214 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.213+0000 7f66ea7fc700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f66d40a2890 0x7f66d40a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f66e0009c60 tx=0x7f66e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:13:46.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.216+0000 7f66f08cb700 1 -- 172.21.15.193:0/2065642466 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f66dc09b3b0 con 0x7f66ec1005c0 2024-03-16T10:13:46.421 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:13:46.420+0000 7f66f18cd700 1 -- 172.21.15.193:0/2065642466 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f66ec062710 con 0x7f66d40a2890 2024-03-16T10:13:46.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:46 smithi192 ceph-mon[36354]: Detected new or changed devices on smithi192 2024-03-16T10:13:46.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:46 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi192 to 4270M 2024-03-16T10:13:46.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:46 smithi192 ceph-mon[36354]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T10:13:46.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:46 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:13:46.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:46 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:13:46.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:46 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:46.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:46 smithi077 ceph-mon[32400]: Detected new or changed devices on smithi192 2024-03-16T10:13:46.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:46 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi192 to 4270M 2024-03-16T10:13:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:46 smithi077 ceph-mon[32400]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T10:13:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:46 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:13:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:46 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:13:46.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:46 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:47.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:46 smithi193 ceph-mon[36311]: Detected new or changed devices on smithi192 2024-03-16T10:13:47.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:46 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi192 to 4270M 2024-03-16T10:13:47.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:46 smithi193 ceph-mon[36311]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T10:13:47.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:46 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T10:13:47.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:46 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T10:13:47.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:46 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:47.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:47 smithi192 ceph-mon[36354]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:13:47.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:47 smithi192 ceph-mon[36354]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:13:47.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:47 smithi192 ceph-mon[36354]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T10:13:47.979 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:47 smithi193 ceph-mon[36311]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:13:47.979 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:47 smithi193 ceph-mon[36311]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:13:47.979 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:47 smithi193 ceph-mon[36311]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T10:13:47.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:47 smithi077 ceph-mon[32400]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:13:47.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:47 smithi077 ceph-mon[32400]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:13:47.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:47 smithi077 ceph-mon[32400]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T10:13:50.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:49 smithi193 ceph-mon[36311]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-16T10:13:50.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:49 smithi193 ceph-mon[36311]: from='client.? 172.21.15.193:0/2437842694' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fafc7e50-31b9-4bb9-9e17-aee4f80a2e70"}]: dispatch 2024-03-16T10:13:50.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:49 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fafc7e50-31b9-4bb9-9e17-aee4f80a2e70"}]: dispatch 2024-03-16T10:13:50.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:49 smithi193 ceph-mon[36311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fafc7e50-31b9-4bb9-9e17-aee4f80a2e70"}]': finished 2024-03-16T10:13:50.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:49 smithi193 ceph-mon[36311]: osdmap e77: 12 total, 11 up, 12 in 2024-03-16T10:13:50.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:49 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:13:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:49 smithi192 ceph-mon[36354]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-16T10:13:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:49 smithi192 ceph-mon[36354]: from='client.? 172.21.15.193:0/2437842694' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fafc7e50-31b9-4bb9-9e17-aee4f80a2e70"}]: dispatch 2024-03-16T10:13:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:49 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fafc7e50-31b9-4bb9-9e17-aee4f80a2e70"}]: dispatch 2024-03-16T10:13:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:49 smithi192 ceph-mon[36354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fafc7e50-31b9-4bb9-9e17-aee4f80a2e70"}]': finished 2024-03-16T10:13:50.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:49 smithi192 ceph-mon[36354]: osdmap e77: 12 total, 11 up, 12 in 2024-03-16T10:13:50.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:49 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:13:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:49 smithi077 ceph-mon[32400]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-16T10:13:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:49 smithi077 ceph-mon[32400]: from='client.? 172.21.15.193:0/2437842694' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fafc7e50-31b9-4bb9-9e17-aee4f80a2e70"}]: dispatch 2024-03-16T10:13:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:49 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fafc7e50-31b9-4bb9-9e17-aee4f80a2e70"}]: dispatch 2024-03-16T10:13:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:49 smithi077 ceph-mon[32400]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fafc7e50-31b9-4bb9-9e17-aee4f80a2e70"}]': finished 2024-03-16T10:13:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:49 smithi077 ceph-mon[32400]: osdmap e77: 12 total, 11 up, 12 in 2024-03-16T10:13:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:49 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:13:50.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:50 smithi192 ceph-mon[36354]: from='client.? 172.21.15.193:0/2354083931' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:13:51.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:50 smithi077 ceph-mon[32400]: from='client.? 172.21.15.193:0/2354083931' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:13:51.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:50 smithi193 ceph-mon[36311]: from='client.? 172.21.15.193:0/2354083931' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T10:13:52.146 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:51 smithi077 ceph-mon[32400]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-16T10:13:52.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:51 smithi192 ceph-mon[36354]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-16T10:13:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:51 smithi193 ceph-mon[36311]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-16T10:13:54.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:53 smithi193 ceph-mon[36311]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-16T10:13:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:53 smithi192 ceph-mon[36354]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-16T10:13:54.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:53 smithi077 ceph-mon[32400]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-16T10:13:55.957 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:55 smithi193 ceph-mon[36311]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T10:13:56.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:55 smithi192 ceph-mon[36354]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T10:13:56.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:55 smithi077 ceph-mon[32400]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T10:13:57.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T10:13:57.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:56 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:57.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:56 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T10:13:57.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:56 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:57.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T10:13:57.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:56 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:13:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:57 smithi192 ceph-mon[36354]: Deploying daemon osd.11 on smithi193 2024-03-16T10:13:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:57 smithi192 ceph-mon[36354]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:13:58.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:57 smithi077 ceph-mon[32400]: Deploying daemon osd.11 on smithi193 2024-03-16T10:13:58.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:57 smithi077 ceph-mon[32400]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:13:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:57 smithi193 ceph-mon[36311]: Deploying daemon osd.11 on smithi193 2024-03-16T10:13:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:57 smithi193 ceph-mon[36311]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:13:59.973 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:13:59 smithi193 ceph-mon[36311]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:00.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:13:59 smithi192 ceph-mon[36354]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:00.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:13:59 smithi077 ceph-mon[32400]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:02.079 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:01 smithi193 ceph-mon[36311]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:02.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:01 smithi192 ceph-mon[36354]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:01 smithi077 ceph-mon[32400]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:02.884 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:02 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:02.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:02 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:02.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:02 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:03.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:02 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:03.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:02 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:03.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:02 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:03.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:02 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:03.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:02 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:03.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:02 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:04.043 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:03 smithi193 ceph-mon[36311]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:04.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:03 smithi192 ceph-mon[36354]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:04.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:03 smithi077 ceph-mon[32400]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:05.535 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.535+0000 7f66f08cb700 1 -- 172.21.15.193:0/2065642466 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f66ec062710 con 0x7f66d40a2890 2024-03-16T10:14:05.536 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 11 on host 'smithi193' 2024-03-16T10:14:05.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.537+0000 7f66f18cd700 1 -- 172.21.15.193:0/2065642466 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f66d40a2890 msgr2=0x7f66d40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:05.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.537+0000 7f66f18cd700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f66d40a2890 0x7f66d40a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f66e0009c60 tx=0x7f66e0009260 comp rx=0 tx=0).stop 2024-03-16T10:14:05.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.537+0000 7f66f18cd700 1 -- 172.21.15.193:0/2065642466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ec1005c0 msgr2=0x7f66ec198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:05.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.537+0000 7f66f18cd700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ec1005c0 0x7f66ec198490 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00d9b0 tx=0x7f66dc00dcc0 comp rx=0 tx=0).stop 2024-03-16T10:14:05.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.537+0000 7f66f18cd700 1 -- 172.21.15.193:0/2065642466 shutdown_connections 2024-03-16T10:14:05.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.537+0000 7f66f18cd700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f66d40a2890 0x7f66d40a4d50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:05.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.537+0000 7f66f18cd700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66ec103f40 0x7f66ec19fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:05.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.537+0000 7f66f18cd700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ec1005c0 0x7f66ec198490 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:05.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.537+0000 7f66f18cd700 1 --2- 172.21.15.193:0/2065642466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66ec1035a0 0x7f66ec1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:05.538 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.537+0000 7f66f18cd700 1 -- 172.21.15.193:0/2065642466 >> 172.21.15.193:0/2065642466 conn(0x7f66ec0fbf40 msgr2=0x7f66ec106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:05.539 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.538+0000 7f66f18cd700 1 -- 172.21.15.193:0/2065642466 shutdown_connections 2024-03-16T10:14:05.539 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:05.538+0000 7f66f18cd700 1 -- 172.21.15.193:0/2065642466 wait complete. 2024-03-16T10:14:06.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:05 smithi193 ceph-mon[36311]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:06.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:05 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:06.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:05 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:05 smithi192 ceph-mon[36354]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:06.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:05 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:06.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:05 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:06.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:05 smithi077 ceph-mon[32400]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:06.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:05 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:06.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:05 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:06.567 DEBUG:teuthology.orchestra.run.smithi193:osd.11> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.11.service 2024-03-16T10:14:06.569 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-16T10:14:06.570 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-16T10:14:06.790 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:06 smithi193 ceph-mon[36311]: from='osd.11 [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T10:14:06.790 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:06 smithi193 ceph-mon[36311]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T10:14:06.825 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:14:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:06 smithi077 ceph-mon[32400]: from='osd.11 [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T10:14:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:06 smithi077 ceph-mon[32400]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T10:14:07.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:06 smithi192 ceph-mon[36354]: from='osd.11 [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T10:14:07.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:06 smithi192 ceph-mon[36354]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T10:14:08.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:07 smithi192 ceph-mon[36354]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:07 smithi192 ceph-mon[36354]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T10:14:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:07 smithi192 ceph-mon[36354]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T10:14:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:07 smithi192 ceph-mon[36354]: from='osd.11 [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:14:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:07 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:07 smithi192 ceph-mon[36354]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:14:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:07 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:08.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:07 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:08.210 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:07 smithi077 ceph-mon[32400]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:08.210 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:07 smithi077 ceph-mon[32400]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T10:14:08.210 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:07 smithi077 ceph-mon[32400]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T10:14:08.210 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:07 smithi077 ceph-mon[32400]: from='osd.11 [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:14:08.210 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:07 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:08.210 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:07 smithi077 ceph-mon[32400]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:14:08.210 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:07 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:08.210 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:07 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:08.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:07 smithi193 ceph-mon[36311]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:08.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:07 smithi193 ceph-mon[36311]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T10:14:08.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:07 smithi193 ceph-mon[36311]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T10:14:08.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:07 smithi193 ceph-mon[36311]: from='osd.11 [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:14:08.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:07 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:08.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:07 smithi193 ceph-mon[36311]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-16T10:14:08.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:07 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:08.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:07 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:08.503 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:14:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:14:08.241+0000 7f1acc1bf700 -1 osd.11 0 waiting for initial osdmap 2024-03-16T10:14:08.503 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:14:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:14:08.254+0000 7f1ac3faa700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T10:14:08.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.599+0000 7f43c026a700 1 -- 172.21.15.77:0/1433888074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b81011f0 msgr2=0x7f43b8101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:08.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.599+0000 7f43c026a700 1 --2- 172.21.15.77:0/1433888074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b81011f0 0x7f43b8101610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f43b40099c0 tx=0x7f43b4009cd0 comp rx=0 tx=0).stop 2024-03-16T10:14:08.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.599+0000 7f43c026a700 1 -- 172.21.15.77:0/1433888074 shutdown_connections 2024-03-16T10:14:08.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.599+0000 7f43c026a700 1 --2- 172.21.15.77:0/1433888074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b81011f0 0x7f43b8101610 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:08.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.599+0000 7f43c026a700 1 --2- 172.21.15.77:0/1433888074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43b8102d90 0x7f43b8109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:08.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.599+0000 7f43c026a700 1 --2- 172.21.15.77:0/1433888074 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43b81023f0 0x7f43b8102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:08.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.599+0000 7f43c026a700 1 -- 172.21.15.77:0/1433888074 >> 172.21.15.77:0/1433888074 conn(0x7f43b80fc790 msgr2=0x7f43b80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43c026a700 1 -- 172.21.15.77:0/1433888074 shutdown_connections 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43c026a700 1 -- 172.21.15.77:0/1433888074 wait complete. 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43c026a700 1 Processor -- start 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43c026a700 1 -- start start 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43c026a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43b81011f0 0x7f43b81974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43c026a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43b81023f0 0x7f43b8197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43c026a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b8102d90 0x7f43b819ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43c026a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f43b810ce60 con 0x7f43b8102d90 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43c026a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f43b810cce0 con 0x7f43b81011f0 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43c026a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43b810cfe0 con 0x7f43b81023f0 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43be807700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b8102d90 0x7f43b819ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43be807700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b8102d90 0x7f43b819ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48618/0 (socket says 172.21.15.77:48618) 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43be807700 1 -- 172.21.15.77:0/1267729233 learned_addr learned my addr 172.21.15.77:0/1267729233 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.600+0000 7f43bd805700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43b81023f0 0x7f43b8197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:08.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.601+0000 7f43be006700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43b81011f0 0x7f43b81974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.601+0000 7f43be807700 1 -- 172.21.15.77:0/1267729233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43b81023f0 msgr2=0x7f43b8197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.601+0000 7f43be807700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43b81023f0 0x7f43b8197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.601+0000 7f43be807700 1 -- 172.21.15.77:0/1267729233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43b81011f0 msgr2=0x7f43b81974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.601+0000 7f43be807700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43b81011f0 0x7f43b81974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.601+0000 7f43be807700 1 -- 172.21.15.77:0/1267729233 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b819f130 con 0x7f43b8102d90 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.601+0000 7f43be807700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b8102d90 0x7f43b819ea90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f43b0007f10 tx=0x7f43b000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.601+0000 7f43af7fe700 1 -- 172.21.15.77:0/1267729233 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0004bb0 con 0x7f43b8102d90 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.601+0000 7f43af7fe700 1 -- 172.21.15.77:0/1267729233 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43b0004d50 con 0x7f43b8102d90 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.601+0000 7f43af7fe700 1 -- 172.21.15.77:0/1267729233 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b00059f0 con 0x7f43b8102d90 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.601+0000 7f43c026a700 1 -- 172.21.15.77:0/1267729233 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b819f420 con 0x7f43b8102d90 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.601+0000 7f43c026a700 1 -- 172.21.15.77:0/1267729233 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b819f990 con 0x7f43b8102d90 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.602+0000 7f43c026a700 1 -- 172.21.15.77:0/1267729233 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43b8068f90 con 0x7f43b8102d90 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.603+0000 7f43af7fe700 1 -- 172.21.15.77:0/1267729233 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f43b001a020 con 0x7f43b8102d90 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.604+0000 7f43af7fe700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f43a40a2890 0x7f43a40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:08.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.604+0000 7f43be006700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f43a40a2890 0x7f43a40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:08.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.604+0000 7f43af7fe700 1 -- 172.21.15.77:0/1267729233 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f43b00ce260 con 0x7f43b8102d90 2024-03-16T10:14:08.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.604+0000 7f43be006700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f43a40a2890 0x7f43a40a4d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f43b400b580 tx=0x7f43b4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:08.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.607+0000 7f43af7fe700 1 -- 172.21.15.77:0/1267729233 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f43b0010070 con 0x7f43b8102d90 2024-03-16T10:14:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.803+0000 7f43c026a700 1 -- 172.21.15.77:0/1267729233 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f43b8062ee0 con 0x7f43b8102d90 2024-03-16T10:14:08.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.804+0000 7f43af7fe700 1 -- 172.21.15.77:0/1267729233 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f43b009b8c0 con 0x7f43b8102d90 2024-03-16T10:14:08.804 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.806+0000 7f43c026a700 1 -- 172.21.15.77:0/1267729233 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f43a40a2890 msgr2=0x7f43a40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.806+0000 7f43c026a700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f43a40a2890 0x7f43a40a4d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f43b400b580 tx=0x7f43b4005fb0 comp rx=0 tx=0).stop 2024-03-16T10:14:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.806+0000 7f43c026a700 1 -- 172.21.15.77:0/1267729233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b8102d90 msgr2=0x7f43b819ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.806+0000 7f43c026a700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b8102d90 0x7f43b819ea90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f43b0007f10 tx=0x7f43b000d3b0 comp rx=0 tx=0).stop 2024-03-16T10:14:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.806+0000 7f43c026a700 1 -- 172.21.15.77:0/1267729233 shutdown_connections 2024-03-16T10:14:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.806+0000 7f43c026a700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f43a40a2890 0x7f43a40a4d50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.806+0000 7f43c026a700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43b8102d90 0x7f43b819ea90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.806+0000 7f43c026a700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43b81011f0 0x7f43b81974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.806+0000 7f43c026a700 1 --2- 172.21.15.77:0/1267729233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43b81023f0 0x7f43b8197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.806+0000 7f43c026a700 1 -- 172.21.15.77:0/1267729233 >> 172.21.15.77:0/1267729233 conn(0x7f43b80fc790 msgr2=0x7f43b8105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.806+0000 7f43c026a700 1 -- 172.21.15.77:0/1267729233 shutdown_connections 2024-03-16T10:14:08.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:08.806+0000 7f43c026a700 1 -- 172.21.15.77:0/1267729233 wait complete. 2024-03-16T10:14:09.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:08 smithi192 ceph-mon[36354]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-16T10:14:09.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:08 smithi192 ceph-mon[36354]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T10:14:09.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:08 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:09.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:08 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:09.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:08 smithi192 ceph-mon[36354]: from='osd.11 ' entity='osd.11' 2024-03-16T10:14:09.121 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:08 smithi077 ceph-mon[32400]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-16T10:14:09.121 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:08 smithi077 ceph-mon[32400]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T10:14:09.121 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:08 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:09.121 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:08 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:09.121 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:08 smithi077 ceph-mon[32400]: from='osd.11 ' entity='osd.11' 2024-03-16T10:14:09.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:08 smithi193 ceph-mon[36311]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-16T10:14:09.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:08 smithi193 ceph-mon[36311]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T10:14:09.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:08 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:09.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:08 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:09.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:08 smithi193 ceph-mon[36311]: from='osd.11 ' entity='osd.11' 2024-03-16T10:14:09.508 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1710584023,"num_in_osds":12,"osd_in_since":1710584029,"num_remapped_pgs":0} 2024-03-16T10:14:09.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:09 smithi193 ceph-mon[36311]: purged_snaps scrub starts 2024-03-16T10:14:09.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:09 smithi193 ceph-mon[36311]: purged_snaps scrub ok 2024-03-16T10:14:09.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:09 smithi193 ceph-mon[36311]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:09.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:09.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:09 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1267729233' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T10:14:09.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:09 smithi193 ceph-mon[36311]: osd.11 [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113] boot 2024-03-16T10:14:09.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:09 smithi193 ceph-mon[36311]: osdmap e80: 12 total, 12 up, 12 in 2024-03-16T10:14:09.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:10.081 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:09 smithi192 ceph-mon[36354]: purged_snaps scrub starts 2024-03-16T10:14:10.081 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:09 smithi192 ceph-mon[36354]: purged_snaps scrub ok 2024-03-16T10:14:10.081 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:09 smithi192 ceph-mon[36354]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:10.082 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:10.082 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:09 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1267729233' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T10:14:10.082 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:09 smithi192 ceph-mon[36354]: osd.11 [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113] boot 2024-03-16T10:14:10.082 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:09 smithi192 ceph-mon[36354]: osdmap e80: 12 total, 12 up, 12 in 2024-03-16T10:14:10.082 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:09 smithi077 ceph-mon[32400]: purged_snaps scrub starts 2024-03-16T10:14:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:09 smithi077 ceph-mon[32400]: purged_snaps scrub ok 2024-03-16T10:14:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:09 smithi077 ceph-mon[32400]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-16T10:14:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:09 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1267729233' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T10:14:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:09 smithi077 ceph-mon[32400]: osd.11 [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113] boot 2024-03-16T10:14:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:09 smithi077 ceph-mon[32400]: osdmap e80: 12 total, 12 up, 12 in 2024-03-16T10:14:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T10:14:10.510 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-16T10:14:10.760 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:14:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: Detected new or changed devices on smithi193 2024-03-16T10:14:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:14:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:14:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:14:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:14:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi193 to 4270M 2024-03-16T10:14:11.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:11.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:14:11.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:11.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:11.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:14:11.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:11.496 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:11 smithi077 ceph-mon[32400]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T10:14:11.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: Detected new or changed devices on smithi193 2024-03-16T10:14:11.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:14:11.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:14:11.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:14:11.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:14:11.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi193 to 4270M 2024-03-16T10:14:11.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:11.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:14:11.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:11.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:11.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:14:11.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:11.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:11 smithi193 ceph-mon[36311]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: Detected new or changed devices on smithi193 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi193 to 4270M 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:11.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:11.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:14:11.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:11.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:11.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:11 smithi192 ceph-mon[36354]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T10:14:12.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.527+0000 7f30fc836700 1 -- 172.21.15.77:0/51225911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f8101200 msgr2=0x7f30f8101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:12.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.527+0000 7f30fc836700 1 --2- 172.21.15.77:0/51225911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f8101200 0x7f30f8101620 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f30e80099c0 tx=0x7f30e8009cd0 comp rx=0 tx=0).stop 2024-03-16T10:14:12.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.527+0000 7f30fc836700 1 -- 172.21.15.77:0/51225911 shutdown_connections 2024-03-16T10:14:12.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.527+0000 7f30fc836700 1 --2- 172.21.15.77:0/51225911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f8101200 0x7f30f8101620 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:12.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.527+0000 7f30fc836700 1 --2- 172.21.15.77:0/51225911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30f8102da0 0x7f30f8109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.527+0000 7f30fc836700 1 --2- 172.21.15.77:0/51225911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30f8102400 0x7f30f8102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.527+0000 7f30fc836700 1 -- 172.21.15.77:0/51225911 >> 172.21.15.77:0/51225911 conn(0x7f30f80fc7a0 msgr2=0x7f30f80febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.527+0000 7f30fc836700 1 -- 172.21.15.77:0/51225911 shutdown_connections 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.527+0000 7f30fc836700 1 -- 172.21.15.77:0/51225911 wait complete. 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.528+0000 7f30fc836700 1 Processor -- start 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.528+0000 7f30fc836700 1 -- start start 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.528+0000 7f30fc836700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f8101200 0x7f30f8105190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.528+0000 7f30fc836700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30f8102400 0x7f30f81037e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.528+0000 7f30fc836700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30f8102da0 0x7f30f8103d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.528+0000 7f30fc836700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f30f810cdc0 con 0x7f30f8101200 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.528+0000 7f30fc836700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f30f810cc40 con 0x7f30f8102da0 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.528+0000 7f30fc836700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f30f810cf40 con 0x7f30f8102400 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.528+0000 7f30f659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f8101200 0x7f30f8105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.528+0000 7f30f659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f8101200 0x7f30f8105190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44182/0 (socket says 172.21.15.77:44182) 2024-03-16T10:14:12.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.528+0000 7f30f659c700 1 -- 172.21.15.77:0/492081265 learned_addr learned my addr 172.21.15.77:0/492081265 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.528+0000 7f30f5d9b700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30f8102400 0x7f30f81037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30f6d9d700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30f8102da0 0x7f30f8103d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30f659c700 1 -- 172.21.15.77:0/492081265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30f8102400 msgr2=0x7f30f81037e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30f659c700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30f8102400 0x7f30f81037e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30f659c700 1 -- 172.21.15.77:0/492081265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30f8102da0 msgr2=0x7f30f8103d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30f659c700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30f8102da0 0x7f30f8103d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30f659c700 1 -- 172.21.15.77:0/492081265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f81045d0 con 0x7f30f8101200 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30f6d9d700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30f8102da0 0x7f30f8103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30f659c700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f8101200 0x7f30f8105190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f30e800c010 tx=0x7f30e8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30e77fe700 1 -- 172.21.15.77:0/492081265 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8027070 con 0x7f30f8101200 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30e77fe700 1 -- 172.21.15.77:0/492081265 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30e80041d0 con 0x7f30f8101200 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30fc836700 1 -- 172.21.15.77:0/492081265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30f81a2630 con 0x7f30f8101200 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30e77fe700 1 -- 172.21.15.77:0/492081265 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8022e60 con 0x7f30f8101200 2024-03-16T10:14:12.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.529+0000 7f30fc836700 1 -- 172.21.15.77:0/492081265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30f81a2ae0 con 0x7f30f8101200 2024-03-16T10:14:12.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.530+0000 7f30fc836700 1 -- 172.21.15.77:0/492081265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30f8068f90 con 0x7f30f8101200 2024-03-16T10:14:12.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.531+0000 7f30e77fe700 1 -- 172.21.15.77:0/492081265 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f30e802a070 con 0x7f30f8101200 2024-03-16T10:14:12.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.532+0000 7f30e77fe700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f30e00a2890 0x7f30e00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:12.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.532+0000 7f30f5d9b700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f30e00a2890 0x7f30e00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:12.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.532+0000 7f30e77fe700 1 -- 172.21.15.77:0/492081265 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f30e80d0120 con 0x7f30f8101200 2024-03-16T10:14:12.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.532+0000 7f30f5d9b700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f30e00a2890 0x7f30e00a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f30ec005fd0 tx=0x7f30ec005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:12.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.535+0000 7f30e77fe700 1 -- 172.21.15.77:0/492081265 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30e8031070 con 0x7f30f8101200 2024-03-16T10:14:12.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.733+0000 7f30fc836700 1 -- 172.21.15.77:0/492081265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f30f81a2fc0 con 0x7f30f8101200 2024-03-16T10:14:12.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.733+0000 7f30e77fe700 1 -- 172.21.15.77:0/492081265 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f30e8013070 con 0x7f30f8101200 2024-03-16T10:14:12.734 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:12.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.735+0000 7f30fc836700 1 -- 172.21.15.77:0/492081265 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f30e00a2890 msgr2=0x7f30e00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:12.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.735+0000 7f30fc836700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f30e00a2890 0x7f30e00a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f30ec005fd0 tx=0x7f30ec005e30 comp rx=0 tx=0).stop 2024-03-16T10:14:12.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.735+0000 7f30fc836700 1 -- 172.21.15.77:0/492081265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f8101200 msgr2=0x7f30f8105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:12.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.735+0000 7f30fc836700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f8101200 0x7f30f8105190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f30e800c010 tx=0x7f30e8005e80 comp rx=0 tx=0).stop 2024-03-16T10:14:12.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.735+0000 7f30fc836700 1 -- 172.21.15.77:0/492081265 shutdown_connections 2024-03-16T10:14:12.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.735+0000 7f30fc836700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f30e00a2890 0x7f30e00a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:12.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.735+0000 7f30fc836700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30f8101200 0x7f30f8105190 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:12.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.735+0000 7f30fc836700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30f8102da0 0x7f30f8103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:12.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.735+0000 7f30fc836700 1 --2- 172.21.15.77:0/492081265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30f8102400 0x7f30f81037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:12.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.735+0000 7f30fc836700 1 -- 172.21.15.77:0/492081265 >> 172.21.15.77:0/492081265 conn(0x7f30f80fc7a0 msgr2=0x7f30f80fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:12.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.736+0000 7f30fc836700 1 -- 172.21.15.77:0/492081265 shutdown_connections 2024-03-16T10:14:12.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:12.736+0000 7f30fc836700 1 -- 172.21.15.77:0/492081265 wait complete. 2024-03-16T10:14:13.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:12 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/492081265' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T10:14:13.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:12 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/492081265' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T10:14:13.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:12 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/492081265' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T10:14:13.451 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1710584049,"num_in_osds":12,"osd_in_since":1710584029,"num_remapped_pgs":0} 2024-03-16T10:14:13.452 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T10:14:13.688 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:14:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:13 smithi077 ceph-mon[32400]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:14.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:13 smithi192 ceph-mon[36354]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:14.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:13 smithi193 ceph-mon[36311]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:15.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.310+0000 7fd82e503700 1 -- 172.21.15.77:0/22423928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828102550 msgr2=0x7fd828102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:15.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.310+0000 7fd82e503700 1 --2- 172.21.15.77:0/22423928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828102550 0x7fd828102930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd8180099c0 tx=0x7fd818009cd0 comp rx=0 tx=0).stop 2024-03-16T10:14:15.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.310+0000 7fd82e503700 1 -- 172.21.15.77:0/22423928 shutdown_connections 2024-03-16T10:14:15.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.310+0000 7fd82e503700 1 --2- 172.21.15.77:0/22423928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828102550 0x7fd828102930 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:15.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.310+0000 7fd82e503700 1 --2- 172.21.15.77:0/22423928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd828107630 0x7fd82810f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:15.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.310+0000 7fd82e503700 1 --2- 172.21.15.77:0/22423928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd828102f00 0x7fd828106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:15.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.310+0000 7fd82e503700 1 -- 172.21.15.77:0/22423928 >> 172.21.15.77:0/22423928 conn(0x7fd8280fdde0 msgr2=0x7fd828100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:15.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.310+0000 7fd82e503700 1 -- 172.21.15.77:0/22423928 shutdown_connections 2024-03-16T10:14:15.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.310+0000 7fd82e503700 1 -- 172.21.15.77:0/22423928 wait complete. 2024-03-16T10:14:15.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd82e503700 1 Processor -- start 2024-03-16T10:14:15.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd82e503700 1 -- start start 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd82e503700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd828102550 0x7fd82819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd82e503700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828102f00 0x7fd82819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd82e503700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd828107630 0x7fd828198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd82e503700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd828112690 con 0x7fd828102f00 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd82e503700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd828112510 con 0x7fd828102550 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd82e503700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd828112810 con 0x7fd828107630 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd8277fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828102f00 0x7fd82819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd8277fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828102f00 0x7fd82819ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44194/0 (socket says 172.21.15.77:44194) 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd8277fe700 1 -- 172.21.15.77:0/1734493115 learned_addr learned my addr 172.21.15.77:0/1734493115 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd82caa0700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd828107630 0x7fd828198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd827fff700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd828102550 0x7fd82819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd8277fe700 1 -- 172.21.15.77:0/1734493115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd828107630 msgr2=0x7fd828198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd8277fe700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd828107630 0x7fd828198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd8277fe700 1 -- 172.21.15.77:0/1734493115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd828102550 msgr2=0x7fd82819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:15.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd8277fe700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd828102550 0x7fd82819e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:15.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.311+0000 7fd8277fe700 1 -- 172.21.15.77:0/1734493115 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd828199070 con 0x7fd828102f00 2024-03-16T10:14:15.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.312+0000 7fd8277fe700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828102f00 0x7fd82819ec90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00b500 tx=0x7fd81c00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:15.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.312+0000 7fd8257fa700 1 -- 172.21.15.77:0/1734493115 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c015020 con 0x7fd828102f00 2024-03-16T10:14:15.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.312+0000 7fd82e503700 1 -- 172.21.15.77:0/1734493115 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd828199360 con 0x7fd828102f00 2024-03-16T10:14:15.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.312+0000 7fd8257fa700 1 -- 172.21.15.77:0/1734493115 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd81c011d10 con 0x7fd828102f00 2024-03-16T10:14:15.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.312+0000 7fd82e503700 1 -- 172.21.15.77:0/1734493115 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8281a5620 con 0x7fd828102f00 2024-03-16T10:14:15.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.312+0000 7fd8257fa700 1 -- 172.21.15.77:0/1734493115 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c00f2e0 con 0x7fd828102f00 2024-03-16T10:14:15.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.313+0000 7fd82e503700 1 -- 172.21.15.77:0/1734493115 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd814005330 con 0x7fd828102f00 2024-03-16T10:14:15.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.314+0000 7fd8257fa700 1 -- 172.21.15.77:0/1734493115 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd81c018020 con 0x7fd828102f00 2024-03-16T10:14:15.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.315+0000 7fd8257fa700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd8100a2910 0x7fd8100a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:15.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.315+0000 7fd827fff700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd8100a2910 0x7fd8100a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:15.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.315+0000 7fd8257fa700 1 -- 172.21.15.77:0/1734493115 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd81c0cdb50 con 0x7fd828102f00 2024-03-16T10:14:15.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.315+0000 7fd827fff700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd8100a2910 0x7fd8100a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd818006010 tx=0x7fd818009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:15.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.319+0000 7fd8257fa700 1 -- 172.21.15.77:0/1734493115 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd81c01f020 con 0x7fd828102f00 2024-03-16T10:14:15.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.520+0000 7fd82e503700 1 -- 172.21.15.77:0/1734493115 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd8140058f0 con 0x7fd828102f00 2024-03-16T10:14:15.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.521+0000 7fd8257fa700 1 -- 172.21.15.77:0/1734493115 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18432 (secure 0 0 0) 0x7fd81c09b090 con 0x7fd828102f00 2024-03-16T10:14:15.522 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:15.522 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":81,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:14:10.802815+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:14:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.523+0000 7fd82e503700 1 -- 172.21.15.77:0/1734493115 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd8100a2910 msgr2=0x7fd8100a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.523+0000 7fd82e503700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd8100a2910 0x7fd8100a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd818006010 tx=0x7fd818009f10 comp rx=0 tx=0).stop 2024-03-16T10:14:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.523+0000 7fd82e503700 1 -- 172.21.15.77:0/1734493115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828102f00 msgr2=0x7fd82819ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.523+0000 7fd82e503700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828102f00 0x7fd82819ec90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00b500 tx=0x7fd81c00b810 comp rx=0 tx=0).stop 2024-03-16T10:14:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.523+0000 7fd82e503700 1 -- 172.21.15.77:0/1734493115 shutdown_connections 2024-03-16T10:14:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.523+0000 7fd82e503700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd8100a2910 0x7fd8100a4dd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.523+0000 7fd82e503700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd828107630 0x7fd828198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.523+0000 7fd82e503700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd828102550 0x7fd82819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.523+0000 7fd82e503700 1 --2- 172.21.15.77:0/1734493115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828102f00 0x7fd82819ec90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.523+0000 7fd82e503700 1 -- 172.21.15.77:0/1734493115 >> 172.21.15.77:0/1734493115 conn(0x7fd8280fdde0 msgr2=0x7fd828100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.523+0000 7fd82e503700 1 -- 172.21.15.77:0/1734493115 shutdown_connections 2024-03-16T10:14:15.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:15.523+0000 7fd82e503700 1 -- 172.21.15.77:0/1734493115 wait complete. 2024-03-16T10:14:16.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:15 smithi192 ceph-mon[36354]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:16.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:15 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1734493115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:14:16.189 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:15 smithi077 ceph-mon[32400]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:16.189 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:15 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1734493115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:14:16.219 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:10:22.965004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T10:14:16.219 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-16T10:14:16.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:15 smithi193 ceph-mon[36311]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:16.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:15 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1734493115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:14:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:14:17.985 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:17 smithi077 ceph-mon[32400]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:18.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:17 smithi192 ceph-mon[36354]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:18.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.202+0000 7f13fdb7c700 1 -- 172.21.15.77:0/1329961034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13f8101e40 msgr2=0x7f13f8105e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:18.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.202+0000 7f13fdb7c700 1 --2- 172.21.15.77:0/1329961034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13f8101e40 0x7f13f8105e00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0099c0 tx=0x7f13ec009cd0 comp rx=0 tx=0).stop 2024-03-16T10:14:18.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.202+0000 7f13fdb7c700 1 -- 172.21.15.77:0/1329961034 shutdown_connections 2024-03-16T10:14:18.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.202+0000 7f13fdb7c700 1 --2- 172.21.15.77:0/1329961034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f81064e0 0x7f13f810e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:18.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.202+0000 7f13fdb7c700 1 --2- 172.21.15.77:0/1329961034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f13f8101490 0x7f13f8101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:18.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.202+0000 7f13fdb7c700 1 --2- 172.21.15.77:0/1329961034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13f8101e40 0x7f13f8105e00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:18.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.202+0000 7f13fdb7c700 1 -- 172.21.15.77:0/1329961034 >> 172.21.15.77:0/1329961034 conn(0x7f13f8077220 msgr2=0x7f13f8077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:18.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.202+0000 7f13fdb7c700 1 -- 172.21.15.77:0/1329961034 shutdown_connections 2024-03-16T10:14:18.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.202+0000 7f13fdb7c700 1 -- 172.21.15.77:0/1329961034 wait complete. 2024-03-16T10:14:18.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.202+0000 7f13fdb7c700 1 Processor -- start 2024-03-16T10:14:18.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13fdb7c700 1 -- start start 2024-03-16T10:14:18.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13fdb7c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13f8101490 0x7f13f819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:18.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13fdb7c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f8101e40 0x7f13f819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:18.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13fdb7c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f13f81064e0 0x7f13f81989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:18.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13fdb7c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f13f81115a0 con 0x7f13f8101490 2024-03-16T10:14:18.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13fdb7c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f13f8111420 con 0x7f13f8101e40 2024-03-16T10:14:18.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13fdb7c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f13f8111720 con 0x7f13f81064e0 2024-03-16T10:14:18.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13f77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13f8101490 0x7f13f819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:18.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13f77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13f8101490 0x7f13f819e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44218/0 (socket says 172.21.15.77:44218) 2024-03-16T10:14:18.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13f77fe700 1 -- 172.21.15.77:0/2247033086 learned_addr learned my addr 172.21.15.77:0/2247033086 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:18.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13f6ffd700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f8101e40 0x7f13f819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:18.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13f7fff700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f13f81064e0 0x7f13f81989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:18.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13f77fe700 1 -- 172.21.15.77:0/2247033086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f13f81064e0 msgr2=0x7f13f81989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:18.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13f77fe700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f13f81064e0 0x7f13f81989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:18.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13f77fe700 1 -- 172.21.15.77:0/2247033086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f8101e40 msgr2=0x7f13f819ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:18.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13f77fe700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f8101e40 0x7f13f819ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:18.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13f77fe700 1 -- 172.21.15.77:0/2247033086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f8199150 con 0x7f13f8101490 2024-03-16T10:14:18.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.203+0000 7f13f77fe700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13f8101490 0x7f13f819e830 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f13e800ba50 tx=0x7f13e800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:18.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.204+0000 7f13f4ff9700 1 -- 172.21.15.77:0/2247033086 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e8019070 con 0x7f13f8101490 2024-03-16T10:14:18.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.204+0000 7f13fdb7c700 1 -- 172.21.15.77:0/2247033086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13f8199440 con 0x7f13f8101490 2024-03-16T10:14:18.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.204+0000 7f13f4ff9700 1 -- 172.21.15.77:0/2247033086 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13e8014440 con 0x7f13f8101490 2024-03-16T10:14:18.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.204+0000 7f13f4ff9700 1 -- 172.21.15.77:0/2247033086 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e80075d0 con 0x7f13f8101490 2024-03-16T10:14:18.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.204+0000 7f13fdb7c700 1 -- 172.21.15.77:0/2247033086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13f81a5700 con 0x7f13f8101490 2024-03-16T10:14:18.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.205+0000 7f13fdb7c700 1 -- 172.21.15.77:0/2247033086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13f81028e0 con 0x7f13f8101490 2024-03-16T10:14:18.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.206+0000 7f13f4ff9700 1 -- 172.21.15.77:0/2247033086 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f13e801c070 con 0x7f13f8101490 2024-03-16T10:14:18.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.206+0000 7f13f4ff9700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f13e00a2890 0x7f13e00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:18.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.207+0000 7f13f6ffd700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f13e00a2890 0x7f13e00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:18.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.207+0000 7f13f4ff9700 1 -- 172.21.15.77:0/2247033086 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f13e80cda60 con 0x7f13f8101490 2024-03-16T10:14:18.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.207+0000 7f13f6ffd700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f13e00a2890 0x7f13e00a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f13ec000c00 tx=0x7f13ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:18.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.211+0000 7f13f4ff9700 1 -- 172.21.15.77:0/2247033086 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f13e809afa0 con 0x7f13f8101490 2024-03-16T10:14:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:17 smithi193 ceph-mon[36311]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:18.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.406+0000 7f13fdb7c700 1 -- 172.21.15.77:0/2247033086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f13f802db70 con 0x7f13f8101490 2024-03-16T10:14:18.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.407+0000 7f13f4ff9700 1 -- 172.21.15.77:0/2247033086 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f13e80cbe20 con 0x7f13f8101490 2024-03-16T10:14:18.408 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T10:14:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.409+0000 7f13fdb7c700 1 -- 172.21.15.77:0/2247033086 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f13e00a2890 msgr2=0x7f13e00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.409+0000 7f13fdb7c700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f13e00a2890 0x7f13e00a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f13ec000c00 tx=0x7f13ec005fd0 comp rx=0 tx=0).stop 2024-03-16T10:14:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.409+0000 7f13fdb7c700 1 -- 172.21.15.77:0/2247033086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13f8101490 msgr2=0x7f13f819e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:18.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.409+0000 7f13fdb7c700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13f8101490 0x7f13f819e830 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f13e800ba50 tx=0x7f13e800be10 comp rx=0 tx=0).stop 2024-03-16T10:14:18.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.409+0000 7f13fdb7c700 1 -- 172.21.15.77:0/2247033086 shutdown_connections 2024-03-16T10:14:18.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.409+0000 7f13fdb7c700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f13e00a2890 0x7f13e00a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:18.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.409+0000 7f13fdb7c700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13f8101490 0x7f13f819e830 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:18.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.409+0000 7f13fdb7c700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f13f81064e0 0x7f13f81989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:18.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.409+0000 7f13fdb7c700 1 --2- 172.21.15.77:0/2247033086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f8101e40 0x7f13f819ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:18.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.409+0000 7f13fdb7c700 1 -- 172.21.15.77:0/2247033086 >> 172.21.15.77:0/2247033086 conn(0x7f13f8077220 msgr2=0x7f13f806acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:18.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.409+0000 7f13fdb7c700 1 -- 172.21.15.77:0/2247033086 shutdown_connections 2024-03-16T10:14:18.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:18.409+0000 7f13fdb7c700 1 -- 172.21.15.77:0/2247033086 wait complete. 2024-03-16T10:14:18.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:18 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2247033086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:14:19.147 INFO:tasks.cephadm:Adding mds.a on smithi077 2024-03-16T10:14:19.147 INFO:tasks.cephadm:Adding mds.d on smithi077 2024-03-16T10:14:19.147 INFO:tasks.cephadm:Adding mds.g on smithi077 2024-03-16T10:14:19.148 INFO:tasks.cephadm:Adding mds.j on smithi077 2024-03-16T10:14:19.148 INFO:tasks.cephadm:Adding mds.b on smithi192 2024-03-16T10:14:19.148 INFO:tasks.cephadm:Adding mds.e on smithi192 2024-03-16T10:14:19.148 INFO:tasks.cephadm:Adding mds.h on smithi192 2024-03-16T10:14:19.148 INFO:tasks.cephadm:Adding mds.k on smithi192 2024-03-16T10:14:19.148 INFO:tasks.cephadm:Adding mds.c on smithi193 2024-03-16T10:14:19.148 INFO:tasks.cephadm:Adding mds.f on smithi193 2024-03-16T10:14:19.148 INFO:tasks.cephadm:Adding mds.i on smithi193 2024-03-16T10:14:19.148 INFO:tasks.cephadm:Adding mds.l on smithi193 2024-03-16T10:14:19.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi192=b;smithi192=e;smithi192=h;smithi192=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l' 2024-03-16T10:14:19.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:18 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2247033086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:14:19.180 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:18 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2247033086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:14:19.397 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:14:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:19 smithi192 ceph-mon[36354]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:20.211 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:19 smithi077 ceph-mon[32400]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:20.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:19 smithi193 ceph-mon[36311]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:21.073 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.066+0000 7fe4124f5700 1 -- 172.21.15.193:0/3891894859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe40c1034f0 msgr2=0x7fe40c103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.066+0000 7fe4124f5700 1 --2- 172.21.15.193:0/3891894859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe40c1034f0 0x7fe40c103950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe400009990 tx=0x7fe400009ca0 comp rx=0 tx=0).stop 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.066+0000 7fe4124f5700 1 -- 172.21.15.193:0/3891894859 shutdown_connections 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.066+0000 7fe4124f5700 1 --2- 172.21.15.193:0/3891894859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe40c1022f0 0x7fe40c102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.066+0000 7fe4124f5700 1 --2- 172.21.15.193:0/3891894859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe40c103e90 0x7fe40c10a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.066+0000 7fe4124f5700 1 --2- 172.21.15.193:0/3891894859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe40c1034f0 0x7fe40c103950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.066+0000 7fe4124f5700 1 -- 172.21.15.193:0/3891894859 >> 172.21.15.193:0/3891894859 conn(0x7fe40c0fd8b0 msgr2=0x7fe40c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.066+0000 7fe4124f5700 1 -- 172.21.15.193:0/3891894859 shutdown_connections 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.066+0000 7fe4124f5700 1 -- 172.21.15.193:0/3891894859 wait complete. 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe4124f5700 1 Processor -- start 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe4124f5700 1 -- start start 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe4124f5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe40c1022f0 0x7fe40c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe4124f5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe40c1034f0 0x7fe40c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:21.074 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe4124f5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe40c103e90 0x7fe40c19fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe4124f5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe40c10ded0 con 0x7fe40c1034f0 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe4124f5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe40c10dd50 con 0x7fe40c1022f0 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe4124f5700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe40c10e050 con 0x7fe40c103e90 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe410a92700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe40c103e90 0x7fe40c19fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe410a92700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe40c103e90 0x7fe40c19fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:49486/0 (socket says 172.21.15.193:49486) 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe410a92700 1 -- 172.21.15.193:0/3472636643 learned_addr learned my addr 172.21.15.193:0/3472636643 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe40bfff700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe40c1022f0 0x7fe40c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.067+0000 7fe40b7fe700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe40c1034f0 0x7fe40c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.068+0000 7fe410a92700 1 -- 172.21.15.193:0/3472636643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe40c1022f0 msgr2=0x7fe40c198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.068+0000 7fe410a92700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe40c1022f0 0x7fe40c198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.068+0000 7fe410a92700 1 -- 172.21.15.193:0/3472636643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe40c1034f0 msgr2=0x7fe40c198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.068+0000 7fe410a92700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe40c1034f0 0x7fe40c198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.068+0000 7fe410a92700 1 -- 172.21.15.193:0/3472636643 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40c1a0170 con 0x7fe40c103e90 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.068+0000 7fe410a92700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe40c103e90 0x7fe40c19fad0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe404007f10 tx=0x7fe40400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:21.075 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.068+0000 7fe4097fa700 1 -- 172.21.15.193:0/3472636643 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe404004bb0 con 0x7fe40c103e90 2024-03-16T10:14:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.068+0000 7fe4124f5700 1 -- 172.21.15.193:0/3472636643 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe40c1a0460 con 0x7fe40c103e90 2024-03-16T10:14:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.068+0000 7fe4097fa700 1 -- 172.21.15.193:0/3472636643 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe404004d50 con 0x7fe40c103e90 2024-03-16T10:14:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.068+0000 7fe4097fa700 1 -- 172.21.15.193:0/3472636643 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4040059f0 con 0x7fe40c103e90 2024-03-16T10:14:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.068+0000 7fe4124f5700 1 -- 172.21.15.193:0/3472636643 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe40c1a09f0 con 0x7fe40c103e90 2024-03-16T10:14:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.069+0000 7fe4124f5700 1 -- 172.21.15.193:0/3472636643 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe40c068f90 con 0x7fe40c103e90 2024-03-16T10:14:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.071+0000 7fe4097fa700 1 -- 172.21.15.193:0/3472636643 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe40401a020 con 0x7fe40c103e90 2024-03-16T10:14:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.071+0000 7fe4097fa700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3f40a2890 0x7fe3f40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.071+0000 7fe4097fa700 1 -- 172.21.15.193:0/3472636643 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe4040ce260 con 0x7fe40c103e90 2024-03-16T10:14:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.072+0000 7fe40bfff700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3f40a2890 0x7fe3f40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.072+0000 7fe40bfff700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3f40a2890 0x7fe3f40a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe40c103350 tx=0x7fe3fc007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:21.076 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.075+0000 7fe4097fa700 1 -- 172.21.15.193:0/3472636643 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe404010070 con 0x7fe40c103e90 2024-03-16T10:14:21.280 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.280+0000 7fe4124f5700 1 -- 172.21.15.193:0/3472636643 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi192=b;smithi192=e;smithi192=h;smithi192=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}) -- 0x7fe40c0629d0 con 0x7fe3f40a2890 2024-03-16T10:14:21.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.291+0000 7fe4097fa700 1 -- 172.21.15.193:0/3472636643 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fe40c0629d0 con 0x7fe3f40a2890 2024-03-16T10:14:21.291 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mds.all update... 2024-03-16T10:14:21.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.293+0000 7fe4124f5700 1 -- 172.21.15.193:0/3472636643 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3f40a2890 msgr2=0x7fe3f40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.293+0000 7fe4124f5700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3f40a2890 0x7fe3f40a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe40c103350 tx=0x7fe3fc007400 comp rx=0 tx=0).stop 2024-03-16T10:14:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.293+0000 7fe4124f5700 1 -- 172.21.15.193:0/3472636643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe40c103e90 msgr2=0x7fe40c19fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.293+0000 7fe4124f5700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe40c103e90 0x7fe40c19fad0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe404007f10 tx=0x7fe40400d3b0 comp rx=0 tx=0).stop 2024-03-16T10:14:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.293+0000 7fe4124f5700 1 -- 172.21.15.193:0/3472636643 shutdown_connections 2024-03-16T10:14:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.293+0000 7fe4124f5700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3f40a2890 0x7fe3f40a4d50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.293+0000 7fe4124f5700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe40c103e90 0x7fe40c19fad0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.293+0000 7fe4124f5700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe40c1022f0 0x7fe40c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.293+0000 7fe4124f5700 1 --2- 172.21.15.193:0/3472636643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe40c1034f0 0x7fe40c198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.293+0000 7fe4124f5700 1 -- 172.21.15.193:0/3472636643 >> 172.21.15.193:0/3472636643 conn(0x7fe40c0fd8b0 msgr2=0x7fe40c1067f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.293+0000 7fe4124f5700 1 -- 172.21.15.193:0/3472636643 shutdown_connections 2024-03-16T10:14:21.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-16T10:14:21.293+0000 7fe4124f5700 1 -- 172.21.15.193:0/3472636643 wait complete. 2024-03-16T10:14:21.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:21 smithi193 ceph-mon[36311]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:21.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:21 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:21.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:21 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:21.918 DEBUG:teuthology.orchestra.run.smithi077:mds.a> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.a.service 2024-03-16T10:14:21.920 DEBUG:teuthology.orchestra.run.smithi077:mds.d> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.d.service 2024-03-16T10:14:21.964 DEBUG:teuthology.orchestra.run.smithi077:mds.g> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.g.service 2024-03-16T10:14:22.006 DEBUG:teuthology.orchestra.run.smithi077:mds.j> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.j.service 2024-03-16T10:14:22.049 DEBUG:teuthology.orchestra.run.smithi192:mds.b> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.b.service 2024-03-16T10:14:22.051 DEBUG:teuthology.orchestra.run.smithi192:mds.e> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.e.service 2024-03-16T10:14:22.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:21 smithi077 ceph-mon[32400]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:22.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:21 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:22.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:21 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:22.074 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:21 smithi192 ceph-mon[36354]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:22.074 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:21 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:22.074 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:21 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:22.075 DEBUG:teuthology.orchestra.run.smithi192:mds.h> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.h.service 2024-03-16T10:14:22.119 DEBUG:teuthology.orchestra.run.smithi192:mds.k> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.k.service 2024-03-16T10:14:22.161 DEBUG:teuthology.orchestra.run.smithi193:mds.c> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.c.service 2024-03-16T10:14:22.163 DEBUG:teuthology.orchestra.run.smithi193:mds.f> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.f.service 2024-03-16T10:14:22.206 DEBUG:teuthology.orchestra.run.smithi193:mds.i> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.i.service 2024-03-16T10:14:22.248 DEBUG:teuthology.orchestra.run.smithi193:mds.l> sudo journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.l.service 2024-03-16T10:14:22.292 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-16T10:14:22.292 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:14:22.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.403+0000 7f0562e7e640 1 -- 172.21.15.77:0/3577178367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055c100700 msgr2=0x7f055c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:22.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.403+0000 7f0562e7e640 1 --2- 172.21.15.77:0/3577178367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055c100700 0x7f055c100b20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0550009900 tx=0x7f055002bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:22.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 -- 172.21.15.77:0/3577178367 shutdown_connections 2024-03-16T10:14:22.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 --2- 172.21.15.77:0/3577178367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f055c1022b0 0x7f055c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 --2- 172.21.15.77:0/3577178367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f055c1018f0 0x7f055c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 --2- 172.21.15.77:0/3577178367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055c100700 0x7f055c100b20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 -- 172.21.15.77:0/3577178367 >> 172.21.15.77:0/3577178367 conn(0x7f055c0fbc70 msgr2=0x7f055c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:22.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 -- 172.21.15.77:0/3577178367 shutdown_connections 2024-03-16T10:14:22.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 -- 172.21.15.77:0/3577178367 wait complete. 2024-03-16T10:14:22.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 Processor -- start 2024-03-16T10:14:22.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 -- start start 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f055c100700 0x7f055c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f055c1018f0 0x7f055c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055c1022b0 0x7f055c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f055c10c350 con 0x7f055c1022b0 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f055c10c1d0 con 0x7f055c1018f0 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f0562e7e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f055c10c4d0 con 0x7f055c100700 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f056141b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055c1022b0 0x7f055c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f056141b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055c1022b0 0x7f055c19f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43362/0 (socket says 172.21.15.77:43362) 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.404+0000 7f056141b640 1 -- 172.21.15.77:0/551689765 learned_addr learned my addr 172.21.15.77:0/551689765 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f0560c1a640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f055c100700 0x7f055c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f055bfff640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f055c1018f0 0x7f055c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f056141b640 1 -- 172.21.15.77:0/551689765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f055c100700 msgr2=0x7f055c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f056141b640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f055c100700 0x7f055c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f056141b640 1 -- 172.21.15.77:0/551689765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f055c1018f0 msgr2=0x7f055c198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:22.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f056141b640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f055c1018f0 0x7f055c198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f056141b640 1 -- 172.21.15.77:0/551689765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f055c19f740 con 0x7f055c1022b0 2024-03-16T10:14:22.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f056141b640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055c1022b0 0x7f055c19f0d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f055400eaa0 tx=0x7f055400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:22.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f0559ffb640 1 -- 172.21.15.77:0/551689765 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055400cdb0 con 0x7f055c1022b0 2024-03-16T10:14:22.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f0559ffb640 1 -- 172.21.15.77:0/551689765 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0554005020 con 0x7f055c1022b0 2024-03-16T10:14:22.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f0562e7e640 1 -- 172.21.15.77:0/551689765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f055c19fa30 con 0x7f055c1022b0 2024-03-16T10:14:22.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f0559ffb640 1 -- 172.21.15.77:0/551689765 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05540116b0 con 0x7f055c1022b0 2024-03-16T10:14:22.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.405+0000 7f0562e7e640 1 -- 172.21.15.77:0/551689765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f055c19ff30 con 0x7f055c1022b0 2024-03-16T10:14:22.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.406+0000 7f0562e7e640 1 -- 172.21.15.77:0/551689765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f055c1a0220 con 0x7f055c1022b0 2024-03-16T10:14:22.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.407+0000 7f0559ffb640 1 -- 172.21.15.77:0/551689765 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0554005560 con 0x7f055c1022b0 2024-03-16T10:14:22.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.408+0000 7f0559ffb640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f054c0a2800 0x7f054c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:22.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.408+0000 7f0560c1a640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f054c0a2800 0x7f054c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:22.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.408+0000 7f0559ffb640 1 -- 172.21.15.77:0/551689765 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f05540ce820 con 0x7f055c1022b0 2024-03-16T10:14:22.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.408+0000 7f0560c1a640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f054c0a2800 0x7f054c0a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0550000c00 tx=0x7f0550005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:22.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.412+0000 7f0559ffb640 1 -- 172.21.15.77:0/551689765 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0554016030 con 0x7f055c1022b0 2024-03-16T10:14:22.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.596+0000 7f0562e7e640 1 -- 172.21.15.77:0/551689765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f055c1a03f0 con 0x7f055c1022b0 2024-03-16T10:14:22.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.597+0000 7f0559ffb640 1 -- 172.21.15.77:0/551689765 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18432 (secure 0 0 0) 0x7f055409bd60 con 0x7f055c1022b0 2024-03-16T10:14:22.599 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:22.599 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":81,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:14:10.802815+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:14:22.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.599+0000 7f0562e7e640 1 -- 172.21.15.77:0/551689765 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f054c0a2800 msgr2=0x7f054c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:22.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.599+0000 7f0562e7e640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f054c0a2800 0x7f054c0a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0550000c00 tx=0x7f0550005fd0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.599+0000 7f0562e7e640 1 -- 172.21.15.77:0/551689765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055c1022b0 msgr2=0x7f055c19f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:22.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.599+0000 7f0562e7e640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055c1022b0 0x7f055c19f0d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f055400eaa0 tx=0x7f055400ef60 comp rx=0 tx=0).stop 2024-03-16T10:14:22.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.599+0000 7f0562e7e640 1 -- 172.21.15.77:0/551689765 shutdown_connections 2024-03-16T10:14:22.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.599+0000 7f0562e7e640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f055c1022b0 0x7f055c19f0d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.599+0000 7f0562e7e640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f055c1018f0 0x7f055c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.599+0000 7f0562e7e640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f054c0a2800 0x7f054c0a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.599+0000 7f0562e7e640 1 --2- 172.21.15.77:0/551689765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f055c100700 0x7f055c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.599+0000 7f0562e7e640 1 -- 172.21.15.77:0/551689765 >> 172.21.15.77:0/551689765 conn(0x7f055c0fbc70 msgr2=0x7f055c0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:22.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.600+0000 7f0562e7e640 1 -- 172.21.15.77:0/551689765 shutdown_connections 2024-03-16T10:14:22.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.600+0000 7f0562e7e640 1 -- 172.21.15.77:0/551689765 wait complete. 2024-03-16T10:14:22.613 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:10:22.965004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T10:14:22.613 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T10:14:22.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.718+0000 7f4e355f4640 1 -- 172.21.15.77:0/3395857412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e301006e0 msgr2=0x7f4e30100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.718+0000 7f4e355f4640 1 --2- 172.21.15.77:0/3395857412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e301006e0 0x7f4e30100b00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4e20009900 tx=0x7f4e2002bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.718+0000 7f4e355f4640 1 -- 172.21.15.77:0/3395857412 shutdown_connections 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.718+0000 7f4e355f4640 1 --2- 172.21.15.77:0/3395857412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e30102290 0x7f4e30108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.718+0000 7f4e355f4640 1 --2- 172.21.15.77:0/3395857412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e301018d0 0x7f4e30101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.718+0000 7f4e355f4640 1 --2- 172.21.15.77:0/3395857412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e301006e0 0x7f4e30100b00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.719+0000 7f4e355f4640 1 -- 172.21.15.77:0/3395857412 >> 172.21.15.77:0/3395857412 conn(0x7f4e300fbc70 msgr2=0x7f4e300fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.719+0000 7f4e355f4640 1 -- 172.21.15.77:0/3395857412 shutdown_connections 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.719+0000 7f4e355f4640 1 -- 172.21.15.77:0/3395857412 wait complete. 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.719+0000 7f4e355f4640 1 Processor -- start 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.719+0000 7f4e355f4640 1 -- start start 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.719+0000 7f4e355f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e301006e0 0x7f4e3006a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.719+0000 7f4e355f4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e301018d0 0x7f4e3019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.719+0000 7f4e355f4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e30102290 0x7f4e3019ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.719+0000 7f4e355f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3010c3b0 con 0x7f4e301006e0 2024-03-16T10:14:22.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.719+0000 7f4e355f4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3010c230 con 0x7f4e30102290 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.719+0000 7f4e355f4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3010c530 con 0x7f4e301018d0 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.719+0000 7f4e2effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e301006e0 0x7f4e3006a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e2effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e301006e0 0x7f4e3006a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43368/0 (socket says 172.21.15.77:43368) 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e2effd640 1 -- 172.21.15.77:0/3507730806 learned_addr learned my addr 172.21.15.77:0/3507730806 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e2e7fc640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e301018d0 0x7f4e3019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e2f7fe640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e30102290 0x7f4e3019ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e2effd640 1 -- 172.21.15.77:0/3507730806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e301018d0 msgr2=0x7f4e3019c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e2effd640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e301018d0 0x7f4e3019c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e2effd640 1 -- 172.21.15.77:0/3507730806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e30102290 msgr2=0x7f4e3019ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e2effd640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e30102290 0x7f4e3019ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e2effd640 1 -- 172.21.15.77:0/3507730806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3019f700 con 0x7f4e301006e0 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e2f7fe640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e30102290 0x7f4e3019ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e2effd640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e301006e0 0x7f4e3006a9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4e20000c00 tx=0x7f4e20005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e17fff640 1 -- 172.21.15.77:0/3507730806 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e20043020 con 0x7f4e301006e0 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e17fff640 1 -- 172.21.15.77:0/3507730806 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e200054e0 con 0x7f4e301006e0 2024-03-16T10:14:22.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e17fff640 1 -- 172.21.15.77:0/3507730806 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e20003e60 con 0x7f4e301006e0 2024-03-16T10:14:22.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e355f4640 1 -- 172.21.15.77:0/3507730806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e3019f990 con 0x7f4e301006e0 2024-03-16T10:14:22.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.720+0000 7f4e355f4640 1 -- 172.21.15.77:0/3507730806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e3019fee0 con 0x7f4e301006e0 2024-03-16T10:14:22.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.721+0000 7f4e355f4640 1 -- 172.21.15.77:0/3507730806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e301079a0 con 0x7f4e301006e0 2024-03-16T10:14:22.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.722+0000 7f4e17fff640 1 -- 172.21.15.77:0/3507730806 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4e20046020 con 0x7f4e301006e0 2024-03-16T10:14:22.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.723+0000 7f4e17fff640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4e180a2880 0x7f4e180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:22.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.723+0000 7f4e2e7fc640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4e180a2880 0x7f4e180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:22.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.723+0000 7f4e17fff640 1 -- 172.21.15.77:0/3507730806 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4e200f0890 con 0x7f4e301006e0 2024-03-16T10:14:22.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.724+0000 7f4e2e7fc640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4e180a2880 0x7f4e180a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4e24005e10 tx=0x7f4e24005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:22.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.728+0000 7f4e17fff640 1 -- 172.21.15.77:0/3507730806 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4e2004d020 con 0x7f4e301006e0 2024-03-16T10:14:22.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.910+0000 7f4e355f4640 1 -- 172.21.15.77:0/3507730806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4e300623e0 con 0x7f4e301006e0 2024-03-16T10:14:22.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.910+0000 7f4e17fff640 1 -- 172.21.15.77:0/3507730806 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f4e200bddd0 con 0x7f4e301006e0 2024-03-16T10:14:22.911 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T10:14:22.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.913+0000 7f4e355f4640 1 -- 172.21.15.77:0/3507730806 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4e180a2880 msgr2=0x7f4e180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:22.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.913+0000 7f4e355f4640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4e180a2880 0x7f4e180a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4e24005e10 tx=0x7f4e24005da0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.913+0000 7f4e355f4640 1 -- 172.21.15.77:0/3507730806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e301006e0 msgr2=0x7f4e3006a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:22.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.913+0000 7f4e355f4640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e301006e0 0x7f4e3006a9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4e20000c00 tx=0x7f4e20005dd0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.913+0000 7f4e355f4640 1 -- 172.21.15.77:0/3507730806 shutdown_connections 2024-03-16T10:14:22.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.913+0000 7f4e355f4640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e30102290 0x7f4e3019ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.913+0000 7f4e355f4640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4e180a2880 0x7f4e180a4d40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.913+0000 7f4e355f4640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e301018d0 0x7f4e3019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.913+0000 7f4e355f4640 1 --2- 172.21.15.77:0/3507730806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e301006e0 0x7f4e3006a9c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:22.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.913+0000 7f4e355f4640 1 -- 172.21.15.77:0/3507730806 >> 172.21.15.77:0/3507730806 conn(0x7f4e300fbc70 msgr2=0x7f4e30104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:22.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.913+0000 7f4e355f4640 1 -- 172.21.15.77:0/3507730806 shutdown_connections 2024-03-16T10:14:22.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:22.913+0000 7f4e355f4640 1 -- 172.21.15.77:0/3507730806 wait complete. 2024-03-16T10:14:22.924 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:22 smithi077 ceph-mon[32400]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi192=b;smithi192=e;smithi192=h;smithi192=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:14:22.924 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:22 smithi077 ceph-mon[32400]: Saving service mds.all spec with placement smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi192=b;smithi192=e;smithi192=h;smithi192=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-03-16T10:14:22.924 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:22.924 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:14:22.924 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:22.924 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:22.924 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:22.924 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:22.924 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:22 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/551689765' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:14:22.925 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:23.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.026+0000 7fec44962640 1 -- 172.21.15.77:0/3710559265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec401022b0 msgr2=0x7fec40108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:23.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.026+0000 7fec44962640 1 --2- 172.21.15.77:0/3710559265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec401022b0 0x7fec40108b40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fec38009ca0 tx=0x7fec3802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:14:23.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.026+0000 7fec44962640 1 -- 172.21.15.77:0/3710559265 shutdown_connections 2024-03-16T10:14:23.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.026+0000 7fec44962640 1 --2- 172.21.15.77:0/3710559265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec401022b0 0x7fec40108b40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:23.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.026+0000 7fec44962640 1 --2- 172.21.15.77:0/3710559265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec401018f0 0x7fec40101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:23.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.026+0000 7fec44962640 1 --2- 172.21.15.77:0/3710559265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec40100700 0x7fec40100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:23.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.026+0000 7fec44962640 1 -- 172.21.15.77:0/3710559265 >> 172.21.15.77:0/3710559265 conn(0x7fec400fbc70 msgr2=0x7fec400fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:23.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.026+0000 7fec44962640 1 -- 172.21.15.77:0/3710559265 shutdown_connections 2024-03-16T10:14:23.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.026+0000 7fec44962640 1 -- 172.21.15.77:0/3710559265 wait complete. 2024-03-16T10:14:23.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec44962640 1 Processor -- start 2024-03-16T10:14:23.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec44962640 1 -- start start 2024-03-16T10:14:23.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec44962640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec40100700 0x7fec40197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:23.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec44962640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec401018f0 0x7fec40197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec44962640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec401022b0 0x7fec4019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec44962640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fec4010d460 con 0x7fec401022b0 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec44962640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fec4010d2e0 con 0x7fec40100700 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec44962640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fec4010d5e0 con 0x7fec401018f0 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec3ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec401022b0 0x7fec4019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec3ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec401022b0 0x7fec4019f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43378/0 (socket says 172.21.15.77:43378) 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec3ed9d640 1 -- 172.21.15.77:0/1427761863 learned_addr learned my addr 172.21.15.77:0/1427761863 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec3e59c640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec40100700 0x7fec40197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec37fff640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec401018f0 0x7fec40197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec3ed9d640 1 -- 172.21.15.77:0/1427761863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec401018f0 msgr2=0x7fec40197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec3ed9d640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec401018f0 0x7fec40197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec3ed9d640 1 -- 172.21.15.77:0/1427761863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec40100700 msgr2=0x7fec40197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec3ed9d640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec40100700 0x7fec40197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:23.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.027+0000 7fec3ed9d640 1 -- 172.21.15.77:0/1427761863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4019f6c0 con 0x7fec401022b0 2024-03-16T10:14:23.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.028+0000 7fec3ed9d640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec401022b0 0x7fec4019f050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec3802c630 tx=0x7fec38032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:23.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.028+0000 7fec377fe640 1 -- 172.21.15.77:0/1427761863 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec38041020 con 0x7fec401022b0 2024-03-16T10:14:23.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.028+0000 7fec44962640 1 -- 172.21.15.77:0/1427761863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec4019f950 con 0x7fec401022b0 2024-03-16T10:14:23.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.028+0000 7fec377fe640 1 -- 172.21.15.77:0/1427761863 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec38034cd0 con 0x7fec401022b0 2024-03-16T10:14:23.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.028+0000 7fec377fe640 1 -- 172.21.15.77:0/1427761863 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec38039040 con 0x7fec401022b0 2024-03-16T10:14:23.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.028+0000 7fec44962640 1 -- 172.21.15.77:0/1427761863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec4019fe90 con 0x7fec401022b0 2024-03-16T10:14:23.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.029+0000 7fec44962640 1 -- 172.21.15.77:0/1427761863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec401079c0 con 0x7fec401022b0 2024-03-16T10:14:23.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.030+0000 7fec377fe640 1 -- 172.21.15.77:0/1427761863 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fec38044020 con 0x7fec401022b0 2024-03-16T10:14:23.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.030+0000 7fec377fe640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fec2c0a2800 0x7fec2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:23.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.031+0000 7fec3e59c640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fec2c0a2800 0x7fec2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:23.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.031+0000 7fec377fe640 1 -- 172.21.15.77:0/1427761863 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fec380f08d0 con 0x7fec401022b0 2024-03-16T10:14:23.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.031+0000 7fec3e59c640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fec2c0a2800 0x7fec2c0a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fec40101750 tx=0x7fec3000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:23.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.038+0000 7fec377fe640 1 -- 172.21.15.77:0/1427761863 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec3804b020 con 0x7fec401022b0 2024-03-16T10:14:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:22 smithi192 ceph-mon[36354]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi192=b;smithi192=e;smithi192=h;smithi192=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:14:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:22 smithi192 ceph-mon[36354]: Saving service mds.all spec with placement smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi192=b;smithi192=e;smithi192=h;smithi192=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-03-16T10:14:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:14:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:22 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/551689765' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:14:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:22 smithi193 ceph-mon[36311]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi192=b;smithi192=e;smithi192=h;smithi192=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:14:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:22 smithi193 ceph-mon[36311]: Saving service mds.all spec with placement smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi192=b;smithi192=e;smithi192=h;smithi192=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-03-16T10:14:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:14:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:22 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/551689765' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:14:23.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.257+0000 7fec44962640 1 -- 172.21.15.77:0/1427761863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec4002d370 con 0x7fec401022b0 2024-03-16T10:14:23.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.257+0000 7fec377fe640 1 -- 172.21.15.77:0/1427761863 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fec380bde10 con 0x7fec401022b0 2024-03-16T10:14:23.258 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:23.258 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-16T10:14:23.258 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 1 2024-03-16T10:14:23.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.259+0000 7fec44962640 1 -- 172.21.15.77:0/1427761863 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fec2c0a2800 msgr2=0x7fec2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:23.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.259+0000 7fec44962640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fec2c0a2800 0x7fec2c0a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fec40101750 tx=0x7fec3000b390 comp rx=0 tx=0).stop 2024-03-16T10:14:23.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.259+0000 7fec44962640 1 -- 172.21.15.77:0/1427761863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec401022b0 msgr2=0x7fec4019f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:23.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.259+0000 7fec44962640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec401022b0 0x7fec4019f050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec3802c630 tx=0x7fec38032ea0 comp rx=0 tx=0).stop 2024-03-16T10:14:23.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.259+0000 7fec44962640 1 -- 172.21.15.77:0/1427761863 shutdown_connections 2024-03-16T10:14:23.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.259+0000 7fec44962640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec401022b0 0x7fec4019f050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:23.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.259+0000 7fec44962640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fec2c0a2800 0x7fec2c0a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:23.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.259+0000 7fec44962640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec401018f0 0x7fec40197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:23.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.259+0000 7fec44962640 1 --2- 172.21.15.77:0/1427761863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec40100700 0x7fec40197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:23.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.259+0000 7fec44962640 1 -- 172.21.15.77:0/1427761863 >> 172.21.15.77:0/1427761863 conn(0x7fec400fbc70 msgr2=0x7fec40104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:23.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.259+0000 7fec44962640 1 -- 172.21.15.77:0/1427761863 shutdown_connections 2024-03-16T10:14:23.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:23.259+0000 7fec44962640 1 -- 172.21.15.77:0/1427761863 wait complete. 2024-03-16T10:14:24.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:23 smithi192 ceph-mon[36354]: Deploying daemon mds.l on smithi193 2024-03-16T10:14:24.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:23 smithi192 ceph-mon[36354]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:24.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:23 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3507730806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:14:24.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:23 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1427761863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:24.183 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:23 smithi193 ceph-mon[36311]: Deploying daemon mds.l on smithi193 2024-03-16T10:14:24.183 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:23 smithi193 ceph-mon[36311]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:24.183 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:23 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3507730806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:14:24.183 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:23 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1427761863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:24.183 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:14:23 smithi193 systemd[1]: Starting Ceph mds.l for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:14:24.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:23 smithi077 ceph-mon[32400]: Deploying daemon mds.l on smithi193 2024-03-16T10:14:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:23 smithi077 ceph-mon[32400]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:23 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3507730806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:14:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:23 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1427761863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:24.457 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:14:24 smithi193 podman[66544]: 2024-03-16 10:14:24.085254181 +0000 UTC m=+0.017613123 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:14:24.458 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:14:24 smithi193 podman[66544]: 2024-03-16 10:14:24.205014329 +0000 UTC m=+0.137373250 container create c1f47e11a6a321e9e87ae0132b98f347aaa6d218618dbc78cd6bdc8825aa743d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-16T10:14:24.458 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:14:24 smithi193 podman[66544]: 2024-03-16 10:14:24.457582615 +0000 UTC m=+0.389941554 container init c1f47e11a6a321e9e87ae0132b98f347aaa6d218618dbc78cd6bdc8825aa743d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-16T10:14:24.753 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:14:24 smithi193 podman[66544]: 2024-03-16 10:14:24.460367861 +0000 UTC m=+0.392726790 container start c1f47e11a6a321e9e87ae0132b98f347aaa6d218618dbc78cd6bdc8825aa743d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-l, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-16T10:14:24.753 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:14:24 smithi193 ceph-mds[66564]: -- 172.21.15.193:0/3428839108 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a03a571a40 con 0x55a03a544c00 2024-03-16T10:14:24.753 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:14:24 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-l[66560]: starting mds.l at 2024-03-16T10:14:24.753 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:14:24 smithi193 bash[66544]: c1f47e11a6a321e9e87ae0132b98f347aaa6d218618dbc78cd6bdc8825aa743d 2024-03-16T10:14:24.753 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:14:24 smithi193 systemd[1]: Started Ceph mds.l for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:14:25.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:25.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.384+0000 7fc21892f640 1 -- 172.21.15.77:0/2261670749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc214104c10 msgr2=0x7fc214107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:25.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.384+0000 7fc21892f640 1 --2- 172.21.15.77:0/2261670749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc214104c10 0x7fc214107000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc204009900 tx=0x7fc20402bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:25.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.384+0000 7fc21892f640 1 -- 172.21.15.77:0/2261670749 shutdown_connections 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.384+0000 7fc21892f640 1 --2- 172.21.15.77:0/2261670749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc214107540 0x7fc214109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.384+0000 7fc21892f640 1 --2- 172.21.15.77:0/2261670749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc214104c10 0x7fc214107000 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.384+0000 7fc21892f640 1 --2- 172.21.15.77:0/2261670749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2141022e0 0x7fc2141046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.384+0000 7fc21892f640 1 -- 172.21.15.77:0/2261670749 >> 172.21.15.77:0/2261670749 conn(0x7fc2140fbc70 msgr2=0x7fc2140fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.384+0000 7fc21892f640 1 -- 172.21.15.77:0/2261670749 shutdown_connections 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.384+0000 7fc21892f640 1 -- 172.21.15.77:0/2261670749 wait complete. 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.385+0000 7fc21892f640 1 Processor -- start 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.385+0000 7fc21892f640 1 -- start start 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.385+0000 7fc21892f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2141022e0 0x7fc214101c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.385+0000 7fc21892f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc214104c10 0x7fc214100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.385+0000 7fc21892f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc214107540 0x7fc2141007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.385+0000 7fc21892f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc21410d4d0 con 0x7fc214107540 2024-03-16T10:14:25.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.385+0000 7fc21892f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc21410d350 con 0x7fc214104c10 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.385+0000 7fc21892f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc21410d650 con 0x7fc2141022e0 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.385+0000 7fc212d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc214107540 0x7fc2141007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.385+0000 7fc212d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc214107540 0x7fc2141007d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43398/0 (socket says 172.21.15.77:43398) 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.385+0000 7fc212d9d640 1 -- 172.21.15.77:0/1568546024 learned_addr learned my addr 172.21.15.77:0/1568546024 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.385+0000 7fc21259c640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2141022e0 0x7fc214101c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.386+0000 7fc211d9b640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc214104c10 0x7fc214100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.386+0000 7fc212d9d640 1 -- 172.21.15.77:0/1568546024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2141022e0 msgr2=0x7fc214101c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.386+0000 7fc212d9d640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2141022e0 0x7fc214101c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.386+0000 7fc212d9d640 1 -- 172.21.15.77:0/1568546024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc214104c10 msgr2=0x7fc214100290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.386+0000 7fc212d9d640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc214104c10 0x7fc214100290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.386+0000 7fc212d9d640 1 -- 172.21.15.77:0/1568546024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc214101090 con 0x7fc214107540 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.386+0000 7fc212d9d640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc214107540 0x7fc2141007d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00b970 tx=0x7fc20c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.386+0000 7fc2037fe640 1 -- 172.21.15.77:0/1568546024 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c019070 con 0x7fc214107540 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.386+0000 7fc21892f640 1 -- 172.21.15.77:0/1568546024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2141a1d00 con 0x7fc214107540 2024-03-16T10:14:25.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.386+0000 7fc2037fe640 1 -- 172.21.15.77:0/1568546024 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc20c00cb40 con 0x7fc214107540 2024-03-16T10:14:25.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.386+0000 7fc2037fe640 1 -- 172.21.15.77:0/1568546024 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c007650 con 0x7fc214107540 2024-03-16T10:14:25.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.386+0000 7fc21892f640 1 -- 172.21.15.77:0/1568546024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2141a21d0 con 0x7fc214107540 2024-03-16T10:14:25.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.387+0000 7fc21892f640 1 -- 172.21.15.77:0/1568546024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2140fd760 con 0x7fc214107540 2024-03-16T10:14:25.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.388+0000 7fc2037fe640 1 -- 172.21.15.77:0/1568546024 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc20c01c070 con 0x7fc214107540 2024-03-16T10:14:25.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.389+0000 7fc2037fe640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc1fc0a2800 0x7fc1fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:25.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.389+0000 7fc21259c640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc1fc0a2800 0x7fc1fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:25.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.389+0000 7fc2037fe640 1 -- 172.21.15.77:0/1568546024 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc20c0ce030 con 0x7fc214107540 2024-03-16T10:14:25.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.389+0000 7fc21259c640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc1fc0a2800 0x7fc1fc0a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc2080058c0 tx=0x7fc20800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:25.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.393+0000 7fc2037fe640 1 -- 172.21.15.77:0/1568546024 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc20c09af20 con 0x7fc214107540 2024-03-16T10:14:25.562 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:25 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.193:6832/1430295144,v1:172.21.15.193:6833/1430295144] up:boot 2024-03-16T10:14:25.562 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:25 smithi192 ceph-mon[36354]: fsmap 1 up:standby 2024-03-16T10:14:25.562 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:25 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T10:14:25.562 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:25 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:25.562 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:25 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:25.562 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:25 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:25.562 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:25 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:25.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:25 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:25.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:25 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:25.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:25 smithi192 ceph-mon[36354]: Deploying daemon mds.k on smithi192 2024-03-16T10:14:25.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:25 smithi192 ceph-mon[36354]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:25.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.606+0000 7fc21892f640 1 -- 172.21.15.77:0/1568546024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc21402d3b0 con 0x7fc214107540 2024-03-16T10:14:25.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.607+0000 7fc2037fe640 1 -- 172.21.15.77:0/1568546024 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fc20c09efc0 con 0x7fc214107540 2024-03-16T10:14:25.608 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:25.608 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T10:14:25.608 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 2 2024-03-16T10:14:25.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.609+0000 7fc21892f640 1 -- 172.21.15.77:0/1568546024 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc1fc0a2800 msgr2=0x7fc1fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:25.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.609+0000 7fc21892f640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc1fc0a2800 0x7fc1fc0a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc2080058c0 tx=0x7fc20800a2d0 comp rx=0 tx=0).stop 2024-03-16T10:14:25.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.609+0000 7fc21892f640 1 -- 172.21.15.77:0/1568546024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc214107540 msgr2=0x7fc2141007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:25.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.609+0000 7fc21892f640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc214107540 0x7fc2141007d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00b970 tx=0x7fc20c00be30 comp rx=0 tx=0).stop 2024-03-16T10:14:25.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.609+0000 7fc21892f640 1 -- 172.21.15.77:0/1568546024 shutdown_connections 2024-03-16T10:14:25.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.609+0000 7fc21892f640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc214107540 0x7fc2141007d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:25.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.609+0000 7fc21892f640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc214104c10 0x7fc214100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:25.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.609+0000 7fc21892f640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc1fc0a2800 0x7fc1fc0a4cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:25.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.609+0000 7fc21892f640 1 --2- 172.21.15.77:0/1568546024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2141022e0 0x7fc214101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:25.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.609+0000 7fc21892f640 1 -- 172.21.15.77:0/1568546024 >> 172.21.15.77:0/1568546024 conn(0x7fc2140fbc70 msgr2=0x7fc214102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:25.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.609+0000 7fc21892f640 1 -- 172.21.15.77:0/1568546024 shutdown_connections 2024-03-16T10:14:25.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:25.609+0000 7fc21892f640 1 -- 172.21.15.77:0/1568546024 wait complete. 2024-03-16T10:14:25.619 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:25 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.193:6832/1430295144,v1:172.21.15.193:6833/1430295144] up:boot 2024-03-16T10:14:25.619 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:25 smithi077 ceph-mon[32400]: fsmap 1 up:standby 2024-03-16T10:14:25.619 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T10:14:25.619 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:25.619 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:25.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:25.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:25.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:25.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:25.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:25 smithi077 ceph-mon[32400]: Deploying daemon mds.k on smithi192 2024-03-16T10:14:25.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:25 smithi077 ceph-mon[32400]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:25.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:25 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.193:6832/1430295144,v1:172.21.15.193:6833/1430295144] up:boot 2024-03-16T10:14:25.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:25 smithi193 ceph-mon[36311]: fsmap 1 up:standby 2024-03-16T10:14:25.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:25 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T10:14:25.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:25 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:25.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:25 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:25.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:25 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:25.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:25 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:25.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:25 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:25.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:25 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:25.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:25 smithi193 ceph-mon[36311]: Deploying daemon mds.k on smithi192 2024-03-16T10:14:25.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:25 smithi193 ceph-mon[36311]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:26.642 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:26 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1568546024' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:26.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:26 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1568546024' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:27.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:26 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1568546024' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:27.617 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:27 smithi192 ceph-mon[36354]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:27.618 INFO:journalctl@ceph.mds.k.smithi192.stdout:Mar 16 10:14:27 smithi192 bash[66152]: 841d624d452acf9b150a2ff341f04e565c10b63e4a9b8ee3f2b614aa7e2ce0bd 2024-03-16T10:14:27.621 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:27.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.738+0000 7f3eb800b640 1 -- 172.21.15.77:0/256073267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb0100700 msgr2=0x7f3eb0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:27.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.738+0000 7f3eb800b640 1 --2- 172.21.15.77:0/256073267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb0100700 0x7f3eb0100b20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3eac009900 tx=0x7f3eac02bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:27.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.738+0000 7f3eb800b640 1 -- 172.21.15.77:0/256073267 shutdown_connections 2024-03-16T10:14:27.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.738+0000 7f3eb800b640 1 --2- 172.21.15.77:0/256073267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3eb01022b0 0x7f3eb0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:27.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.738+0000 7f3eb800b640 1 --2- 172.21.15.77:0/256073267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3eb01018f0 0x7f3eb0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:27.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.738+0000 7f3eb800b640 1 --2- 172.21.15.77:0/256073267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb0100700 0x7f3eb0100b20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:27.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.738+0000 7f3eb800b640 1 -- 172.21.15.77:0/256073267 >> 172.21.15.77:0/256073267 conn(0x7f3eb00fbc70 msgr2=0x7f3eb00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:27.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.738+0000 7f3eb800b640 1 -- 172.21.15.77:0/256073267 shutdown_connections 2024-03-16T10:14:27.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.738+0000 7f3eb800b640 1 -- 172.21.15.77:0/256073267 wait complete. 2024-03-16T10:14:27.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.739+0000 7f3eb800b640 1 Processor -- start 2024-03-16T10:14:27.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.739+0000 7f3eb800b640 1 -- start start 2024-03-16T10:14:27.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.739+0000 7f3eb800b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3eb0100700 0x7f3eb0104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:27.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.739+0000 7f3eb800b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3eb01018f0 0x7f3eb0102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:27.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.739+0000 7f3eb800b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb01022b0 0x7f3eb0103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:27.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.739+0000 7f3eb800b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb010c370 con 0x7f3eb01022b0 2024-03-16T10:14:27.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.739+0000 7f3eb800b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb010c1f0 con 0x7f3eb01018f0 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.739+0000 7f3eb800b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb010c4f0 con 0x7f3eb0100700 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.739+0000 7f3eb65a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb01022b0 0x7f3eb0103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.739+0000 7f3eb65a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb01022b0 0x7f3eb0103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43424/0 (socket says 172.21.15.77:43424) 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.739+0000 7f3eb65a8640 1 -- 172.21.15.77:0/3567967741 learned_addr learned my addr 172.21.15.77:0/3567967741 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.739+0000 7f3eb55a6640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3eb01018f0 0x7f3eb0102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.740+0000 7f3eb5da7640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3eb0100700 0x7f3eb0104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.740+0000 7f3eb65a8640 1 -- 172.21.15.77:0/3567967741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3eb0100700 msgr2=0x7f3eb0104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.740+0000 7f3eb65a8640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3eb0100700 0x7f3eb0104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.740+0000 7f3eb65a8640 1 -- 172.21.15.77:0/3567967741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3eb01018f0 msgr2=0x7f3eb0102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.740+0000 7f3eb65a8640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3eb01018f0 0x7f3eb0102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.740+0000 7f3eb65a8640 1 -- 172.21.15.77:0/3567967741 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb0103aa0 con 0x7f3eb01022b0 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.740+0000 7f3eb65a8640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb01022b0 0x7f3eb0103210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009fd0 tx=0x7f3ea800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.740+0000 7f3ea6ffd640 1 -- 172.21.15.77:0/3567967741 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8017070 con 0x7f3eb01022b0 2024-03-16T10:14:27.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.740+0000 7f3eb800b640 1 -- 172.21.15.77:0/3567967741 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb01a1cb0 con 0x7f3eb01022b0 2024-03-16T10:14:27.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.740+0000 7f3ea6ffd640 1 -- 172.21.15.77:0/3567967741 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ea8003e50 con 0x7f3eb01022b0 2024-03-16T10:14:27.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.740+0000 7f3ea6ffd640 1 -- 172.21.15.77:0/3567967741 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80106c0 con 0x7f3eb01022b0 2024-03-16T10:14:27.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.740+0000 7f3eb800b640 1 -- 172.21.15.77:0/3567967741 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb01a21b0 con 0x7f3eb01022b0 2024-03-16T10:14:27.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.741+0000 7f3eb800b640 1 -- 172.21.15.77:0/3567967741 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3eb01a23e0 con 0x7f3eb01022b0 2024-03-16T10:14:27.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.742+0000 7f3ea6ffd640 1 -- 172.21.15.77:0/3567967741 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3ea8003970 con 0x7f3eb01022b0 2024-03-16T10:14:27.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.743+0000 7f3ea6ffd640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e9c0a2800 0x7f3e9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:27.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.743+0000 7f3eb5da7640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e9c0a2800 0x7f3e9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:27.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.743+0000 7f3ea6ffd640 1 -- 172.21.15.77:0/3567967741 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3ea80ce2c0 con 0x7f3eb01022b0 2024-03-16T10:14:27.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.743+0000 7f3eb5da7640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e9c0a2800 0x7f3e9c0a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3eac000c00 tx=0x7f3eac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:27.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.747+0000 7f3ea6ffd640 1 -- 172.21.15.77:0/3567967741 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3ea809b110 con 0x7f3eb01022b0 2024-03-16T10:14:27.916 INFO:journalctl@ceph.mds.k.smithi192.stdout:Mar 16 10:14:27 smithi192 systemd[1]: Started Ceph mds.k for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:14:27.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.968+0000 7f3eb800b640 1 -- 172.21.15.77:0/3567967741 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3eb01a26d0 con 0x7f3eb01022b0 2024-03-16T10:14:27.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.968+0000 7f3ea6ffd640 1 -- 172.21.15.77:0/3567967741 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f3ea809f1b0 con 0x7f3eb01022b0 2024-03-16T10:14:27.969 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:27.969 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T10:14:27.970 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 2 2024-03-16T10:14:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.971+0000 7f3eb800b640 1 -- 172.21.15.77:0/3567967741 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e9c0a2800 msgr2=0x7f3e9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.971+0000 7f3eb800b640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e9c0a2800 0x7f3e9c0a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3eac000c00 tx=0x7f3eac005fd0 comp rx=0 tx=0).stop 2024-03-16T10:14:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.971+0000 7f3eb800b640 1 -- 172.21.15.77:0/3567967741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb01022b0 msgr2=0x7f3eb0103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.971+0000 7f3eb800b640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb01022b0 0x7f3eb0103210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009fd0 tx=0x7f3ea800b260 comp rx=0 tx=0).stop 2024-03-16T10:14:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.971+0000 7f3eb800b640 1 -- 172.21.15.77:0/3567967741 shutdown_connections 2024-03-16T10:14:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.971+0000 7f3eb800b640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb01022b0 0x7f3eb0103210 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.971+0000 7f3eb800b640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3eb01018f0 0x7f3eb0102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.971+0000 7f3eb800b640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e9c0a2800 0x7f3e9c0a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.971+0000 7f3eb800b640 1 --2- 172.21.15.77:0/3567967741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3eb0100700 0x7f3eb0104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.971+0000 7f3eb800b640 1 -- 172.21.15.77:0/3567967741 >> 172.21.15.77:0/3567967741 conn(0x7f3eb00fbc70 msgr2=0x7f3eb0068970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.971+0000 7f3eb800b640 1 -- 172.21.15.77:0/3567967741 shutdown_connections 2024-03-16T10:14:27.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:27.971+0000 7f3eb800b640 1 -- 172.21.15.77:0/3567967741 wait complete. 2024-03-16T10:14:27.981 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:27 smithi077 ceph-mon[32400]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:28.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:27 smithi193 ceph-mon[36311]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:28.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:28.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:28.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:28.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:28.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:28.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:28.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:28 smithi077 ceph-mon[32400]: Deploying daemon mds.j on smithi077 2024-03-16T10:14:28.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:28 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3567967741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:28.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:28 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.192:6832/3214007033,v1:172.21.15.192:6833/3214007033] up:boot 2024-03-16T10:14:28.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:28 smithi077 ceph-mon[32400]: fsmap 2 up:standby 2024-03-16T10:14:28.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T10:14:29.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:29.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:29.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:29.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:29.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:29.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:29.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:28 smithi193 ceph-mon[36311]: Deploying daemon mds.j on smithi077 2024-03-16T10:14:29.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:28 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3567967741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:29.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:28 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.192:6832/3214007033,v1:172.21.15.192:6833/3214007033] up:boot 2024-03-16T10:14:29.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:28 smithi193 ceph-mon[36311]: fsmap 2 up:standby 2024-03-16T10:14:29.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T10:14:29.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:28 smithi192 ceph-mon[36354]: Deploying daemon mds.j on smithi077 2024-03-16T10:14:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:28 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3567967741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:28 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.192:6832/3214007033,v1:172.21.15.192:6833/3214007033] up:boot 2024-03-16T10:14:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:28 smithi192 ceph-mon[36354]: fsmap 2 up:standby 2024-03-16T10:14:29.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T10:14:29.935 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:29 smithi192 ceph-mon[36354]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:29.983 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:29.991 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:29 smithi077 ceph-mon[32400]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:29.991 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:14:29 smithi077 systemd[1]: Starting Ceph mds.j for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:14:30.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:29 smithi193 ceph-mon[36311]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:30.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.090+0000 7f00b17df640 1 -- 172.21.15.77:0/3275866047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac101970 msgr2=0x7f00ac101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:30.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.090+0000 7f00b17df640 1 --2- 172.21.15.77:0/3275866047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac101970 0x7f00ac101df0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009900 tx=0x7f00a002bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:30.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.090+0000 7f00b17df640 1 -- 172.21.15.77:0/3275866047 shutdown_connections 2024-03-16T10:14:30.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.090+0000 7f00b17df640 1 --2- 172.21.15.77:0/3275866047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00ac102330 0x7f00ac108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:30.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.090+0000 7f00b17df640 1 --2- 172.21.15.77:0/3275866047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac101970 0x7f00ac101df0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:30.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.090+0000 7f00b17df640 1 --2- 172.21.15.77:0/3275866047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00ac100780 0x7f00ac100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:30.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.090+0000 7f00b17df640 1 -- 172.21.15.77:0/3275866047 >> 172.21.15.77:0/3275866047 conn(0x7f00ac0fbd10 msgr2=0x7f00ac0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:30.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.090+0000 7f00b17df640 1 -- 172.21.15.77:0/3275866047 shutdown_connections 2024-03-16T10:14:30.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.090+0000 7f00b17df640 1 -- 172.21.15.77:0/3275866047 wait complete. 2024-03-16T10:14:30.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00b17df640 1 Processor -- start 2024-03-16T10:14:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00b17df640 1 -- start start 2024-03-16T10:14:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00b17df640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00ac100780 0x7f00ac06a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00b17df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac101970 0x7f00ac19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00b17df640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00ac102330 0x7f00ac19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00b17df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac10c400 con 0x7f00ac101970 2024-03-16T10:14:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00b17df640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac10c280 con 0x7f00ac102330 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00b17df640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac10c580 con 0x7f00ac100780 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00aa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac101970 0x7f00ac19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00aa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac101970 0x7f00ac19c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43430/0 (socket says 172.21.15.77:43430) 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00aa7fc640 1 -- 172.21.15.77:0/1335665276 learned_addr learned my addr 172.21.15.77:0/1335665276 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00ab7fe640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00ac102330 0x7f00ac19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.091+0000 7f00aaffd640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00ac100780 0x7f00ac06a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.092+0000 7f00aa7fc640 1 -- 172.21.15.77:0/1335665276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00ac100780 msgr2=0x7f00ac06a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.092+0000 7f00aa7fc640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00ac100780 0x7f00ac06a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.092+0000 7f00aa7fc640 1 -- 172.21.15.77:0/1335665276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00ac102330 msgr2=0x7f00ac19ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.092+0000 7f00aa7fc640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00ac102330 0x7f00ac19ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.092+0000 7f00aa7fc640 1 -- 172.21.15.77:0/1335665276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00ac19f760 con 0x7f00ac101970 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.092+0000 7f00aa7fc640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac101970 0x7f00ac19c830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009230 tx=0x7f00a0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.092+0000 7f0093fff640 1 -- 172.21.15.77:0/1335665276 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a0041070 con 0x7f00ac101970 2024-03-16T10:14:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.092+0000 7f00b17df640 1 -- 172.21.15.77:0/1335665276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00ac19f990 con 0x7f00ac101970 2024-03-16T10:14:30.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.092+0000 7f0093fff640 1 -- 172.21.15.77:0/1335665276 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00a00054e0 con 0x7f00ac101970 2024-03-16T10:14:30.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.092+0000 7f0093fff640 1 -- 172.21.15.77:0/1335665276 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a00346a0 con 0x7f00ac101970 2024-03-16T10:14:30.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.092+0000 7f00b17df640 1 -- 172.21.15.77:0/1335665276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00ac19fed0 con 0x7f00ac101970 2024-03-16T10:14:30.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.093+0000 7f00b17df640 1 -- 172.21.15.77:0/1335665276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00ac107a40 con 0x7f00ac101970 2024-03-16T10:14:30.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.094+0000 7f0093fff640 1 -- 172.21.15.77:0/1335665276 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f00a0044070 con 0x7f00ac101970 2024-03-16T10:14:30.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.095+0000 7f0093fff640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00940a2800 0x7f00940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:30.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.095+0000 7f00aaffd640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00940a2800 0x7f00940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:30.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.095+0000 7f0093fff640 1 -- 172.21.15.77:0/1335665276 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f00a00f0690 con 0x7f00ac101970 2024-03-16T10:14:30.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.095+0000 7f00aaffd640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00940a2800 0x7f00940a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f00ac1017d0 tx=0x7f009c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:30.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.099+0000 7f0093fff640 1 -- 172.21.15.77:0/1335665276 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f00a00bdbd0 con 0x7f00ac101970 2024-03-16T10:14:30.244 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:14:29 smithi077 podman[70931]: 2024-03-16 10:14:29.893620457 +0000 UTC m=+0.017703414 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:14:30.245 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:14:30 smithi077 podman[70931]: 2024-03-16 10:14:30.007741498 +0000 UTC m=+0.131824437 container create 08b26e4e30121d1eefeb76cd8066227a6491c13922bec06f77d1ea9fcce460e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-j, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-16T10:14:30.245 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:14:30 smithi077 podman[70931]: 2024-03-16 10:14:30.235246249 +0000 UTC m=+0.359329189 container init 08b26e4e30121d1eefeb76cd8066227a6491c13922bec06f77d1ea9fcce460e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-j, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-16T10:14:30.245 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:14:30 smithi077 podman[70931]: 2024-03-16 10:14:30.238102564 +0000 UTC m=+0.362185516 container start 08b26e4e30121d1eefeb76cd8066227a6491c13922bec06f77d1ea9fcce460e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-16T10:14:30.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.339+0000 7f00b17df640 1 -- 172.21.15.77:0/1335665276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00ac02d370 con 0x7f00ac101970 2024-03-16T10:14:30.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.340+0000 7f0093fff640 1 -- 172.21.15.77:0/1335665276 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f00a00bd4d0 con 0x7f00ac101970 2024-03-16T10:14:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:30.340 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T10:14:30.341 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 3 2024-03-16T10:14:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.342+0000 7f00b17df640 1 -- 172.21.15.77:0/1335665276 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00940a2800 msgr2=0x7f00940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.342+0000 7f00b17df640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00940a2800 0x7f00940a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f00ac1017d0 tx=0x7f009c008040 comp rx=0 tx=0).stop 2024-03-16T10:14:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.342+0000 7f00b17df640 1 -- 172.21.15.77:0/1335665276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac101970 msgr2=0x7f00ac19c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.342+0000 7f00b17df640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac101970 0x7f00ac19c830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009230 tx=0x7f00a0004040 comp rx=0 tx=0).stop 2024-03-16T10:14:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.342+0000 7f00b17df640 1 -- 172.21.15.77:0/1335665276 shutdown_connections 2024-03-16T10:14:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.342+0000 7f00b17df640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00ac102330 0x7f00ac19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.342+0000 7f00b17df640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00ac101970 0x7f00ac19c830 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.342+0000 7f00b17df640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00940a2800 0x7f00940a4cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.342+0000 7f00b17df640 1 --2- 172.21.15.77:0/1335665276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00ac100780 0x7f00ac06a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.342+0000 7f00b17df640 1 -- 172.21.15.77:0/1335665276 >> 172.21.15.77:0/1335665276 conn(0x7f00ac0fbd10 msgr2=0x7f00ac104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.342+0000 7f00b17df640 1 -- 172.21.15.77:0/1335665276 shutdown_connections 2024-03-16T10:14:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:30.342+0000 7f00b17df640 1 -- 172.21.15.77:0/1335665276 wait complete. 2024-03-16T10:14:30.677 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:14:30 smithi077 ceph-mds[70989]: -- 172.21.15.77:0/651168187 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557c300d5a40 con 0x557c300a9000 2024-03-16T10:14:30.677 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:14:30 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-j[70985]: starting mds.j at 2024-03-16T10:14:30.678 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:14:30 smithi077 bash[70931]: 08b26e4e30121d1eefeb76cd8066227a6491c13922bec06f77d1ea9fcce460e3 2024-03-16T10:14:30.678 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:14:30 smithi077 systemd[1]: Started Ceph mds.j for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:14:30.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:30 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1335665276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:30.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:30 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:30.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:30 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:30.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:30 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:30.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:30 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:30.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:30 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:30.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:30 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:31.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:30 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1335665276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:31.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:30 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:31.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:30 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:31.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:30 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:31.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:30 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:31.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:30 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:31.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:30 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:31.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:30 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1335665276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:31.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:30 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:31.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:30 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:31.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:30 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:31.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:30 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:31.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:30 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:31.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:30 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:31.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:31 smithi077 ceph-mon[32400]: Deploying daemon mds.d on smithi077 2024-03-16T10:14:31.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:31 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.77:6834/2545278002,v1:172.21.15.77:6835/2545278002] up:boot 2024-03-16T10:14:31.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:31 smithi077 ceph-mon[32400]: fsmap 3 up:standby 2024-03-16T10:14:31.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T10:14:31.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:31 smithi077 ceph-mon[32400]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:32.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:31 smithi193 ceph-mon[36311]: Deploying daemon mds.d on smithi077 2024-03-16T10:14:32.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:31 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.77:6834/2545278002,v1:172.21.15.77:6835/2545278002] up:boot 2024-03-16T10:14:32.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:31 smithi193 ceph-mon[36311]: fsmap 3 up:standby 2024-03-16T10:14:32.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T10:14:32.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:31 smithi193 ceph-mon[36311]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:32.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:31 smithi192 ceph-mon[36354]: Deploying daemon mds.d on smithi077 2024-03-16T10:14:32.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:31 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.77:6834/2545278002,v1:172.21.15.77:6835/2545278002] up:boot 2024-03-16T10:14:32.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:31 smithi192 ceph-mon[36354]: fsmap 3 up:standby 2024-03-16T10:14:32.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T10:14:32.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:31 smithi192 ceph-mon[36354]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:32.354 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:32.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.553+0000 7ff317b52640 1 -- 172.21.15.77:0/776135290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3101029f0 msgr2=0x7ff310102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.553+0000 7ff317b52640 1 --2- 172.21.15.77:0/776135290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3101029f0 0x7ff310102e70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff300009900 tx=0x7ff30002bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.553+0000 7ff317b52640 1 -- 172.21.15.77:0/776135290 shutdown_connections 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.553+0000 7ff317b52640 1 --2- 172.21.15.77:0/776135290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3101033b0 0x7ff310109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.553+0000 7ff317b52640 1 --2- 172.21.15.77:0/776135290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3101029f0 0x7ff310102e70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.553+0000 7ff317b52640 1 --2- 172.21.15.77:0/776135290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff310101800 0x7ff310101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.553+0000 7ff317b52640 1 -- 172.21.15.77:0/776135290 >> 172.21.15.77:0/776135290 conn(0x7ff3100fcd90 msgr2=0x7ff3100ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.553+0000 7ff317b52640 1 -- 172.21.15.77:0/776135290 shutdown_connections 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.554+0000 7ff317b52640 1 -- 172.21.15.77:0/776135290 wait complete. 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.554+0000 7ff317b52640 1 Processor -- start 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.554+0000 7ff317b52640 1 -- start start 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.554+0000 7ff317b52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff310101800 0x7ff310197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.554+0000 7ff317b52640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3101029f0 0x7ff310198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:32.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.554+0000 7ff317b52640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3101033b0 0x7ff31019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.554+0000 7ff317b52640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff31010d460 con 0x7ff310101800 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.554+0000 7ff317b52640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff31010d2e0 con 0x7ff3101033b0 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff317b52640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff31010d5e0 con 0x7ff3101029f0 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff3158ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff310101800 0x7ff310197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff3158ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff310101800 0x7ff310197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60882/0 (socket says 172.21.15.77:60882) 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff3158ee640 1 -- 172.21.15.77:0/191896871 learned_addr learned my addr 172.21.15.77:0/191896871 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff3160ef640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3101033b0 0x7ff31019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff3150ed640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3101029f0 0x7ff310198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff3158ee640 1 -- 172.21.15.77:0/191896871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3101029f0 msgr2=0x7ff310198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff3158ee640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3101029f0 0x7ff310198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff3158ee640 1 -- 172.21.15.77:0/191896871 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3101033b0 msgr2=0x7ff31019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff3158ee640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3101033b0 0x7ff31019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff3158ee640 1 -- 172.21.15.77:0/191896871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff31019f700 con 0x7ff310101800 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff3150ed640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3101029f0 0x7ff310198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:14:32.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff3158ee640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff310101800 0x7ff310197ae0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00c970 tx=0x7ff30c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:32.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff306ffd640 1 -- 172.21.15.77:0/191896871 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c007c10 con 0x7ff310101800 2024-03-16T10:14:32.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff317b52640 1 -- 172.21.15.77:0/191896871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff31019f9f0 con 0x7ff310101800 2024-03-16T10:14:32.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff306ffd640 1 -- 172.21.15.77:0/191896871 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff30c007db0 con 0x7ff310101800 2024-03-16T10:14:32.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff306ffd640 1 -- 172.21.15.77:0/191896871 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c00f940 con 0x7ff310101800 2024-03-16T10:14:32.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.555+0000 7ff317b52640 1 -- 172.21.15.77:0/191896871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff31019ff40 con 0x7ff310101800 2024-03-16T10:14:32.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.556+0000 7ff317b52640 1 -- 172.21.15.77:0/191896871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff310108a50 con 0x7ff310101800 2024-03-16T10:14:32.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.558+0000 7ff306ffd640 1 -- 172.21.15.77:0/191896871 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff30c00fae0 con 0x7ff310101800 2024-03-16T10:14:32.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.559+0000 7ff306ffd640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff2fc0a2800 0x7ff2fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:32.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.559+0000 7ff3150ed640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff2fc0a2800 0x7ff2fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:32.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.559+0000 7ff306ffd640 1 -- 172.21.15.77:0/191896871 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff30c0ced10 con 0x7ff310101800 2024-03-16T10:14:32.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.559+0000 7ff3150ed640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff2fc0a2800 0x7ff2fc0a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff300036040 tx=0x7ff300000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:32.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.562+0000 7ff306ffd640 1 -- 172.21.15.77:0/191896871 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff30c09c370 con 0x7ff310101800 2024-03-16T10:14:32.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.796+0000 7ff317b52640 1 -- 172.21.15.77:0/191896871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff31002d1b0 con 0x7ff310101800 2024-03-16T10:14:32.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.796+0000 7ff306ffd640 1 -- 172.21.15.77:0/191896871 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7ff30c09bc70 con 0x7ff310101800 2024-03-16T10:14:32.797 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:32.797 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T10:14:32.798 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 4 2024-03-16T10:14:32.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.798+0000 7ff317b52640 1 -- 172.21.15.77:0/191896871 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff2fc0a2800 msgr2=0x7ff2fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:32.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.798+0000 7ff317b52640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff2fc0a2800 0x7ff2fc0a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff300036040 tx=0x7ff300000bc0 comp rx=0 tx=0).stop 2024-03-16T10:14:32.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.798+0000 7ff317b52640 1 -- 172.21.15.77:0/191896871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff310101800 msgr2=0x7ff310197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:32.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.798+0000 7ff317b52640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff310101800 0x7ff310197ae0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00c970 tx=0x7ff30c00ce30 comp rx=0 tx=0).stop 2024-03-16T10:14:32.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.798+0000 7ff317b52640 1 -- 172.21.15.77:0/191896871 shutdown_connections 2024-03-16T10:14:32.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.798+0000 7ff317b52640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3101033b0 0x7ff31019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:32.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.798+0000 7ff317b52640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff2fc0a2800 0x7ff2fc0a4cc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:32.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.798+0000 7ff317b52640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3101029f0 0x7ff310198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:32.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.798+0000 7ff317b52640 1 --2- 172.21.15.77:0/191896871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff310101800 0x7ff310197ae0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:32.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.798+0000 7ff317b52640 1 -- 172.21.15.77:0/191896871 >> 172.21.15.77:0/191896871 conn(0x7ff3100fcd90 msgr2=0x7ff310105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:32.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.798+0000 7ff317b52640 1 -- 172.21.15.77:0/191896871 shutdown_connections 2024-03-16T10:14:32.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:32.798+0000 7ff317b52640 1 -- 172.21.15.77:0/191896871 wait complete. 2024-03-16T10:14:32.995 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:14:32 smithi077 podman[71313]: 2024-03-16 10:14:32.937110714 +0000 UTC m=+0.138801962 container create ffb63e7b892a7e0ddd4667a48efff0c3897bb082d80aac64b1af52060afebd3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-16T10:14:33.367 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:14:33 smithi077 podman[71313]: 2024-03-16 10:14:33.172914814 +0000 UTC m=+0.374606062 container init ffb63e7b892a7e0ddd4667a48efff0c3897bb082d80aac64b1af52060afebd3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:14:33.367 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:14:33 smithi077 podman[71313]: 2024-03-16 10:14:33.175600719 +0000 UTC m=+0.377291976 container start ffb63e7b892a7e0ddd4667a48efff0c3897bb082d80aac64b1af52060afebd3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-d, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-16T10:14:33.367 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:14:33 smithi077 ceph-mds[71334]: -- 172.21.15.77:0/2508438147 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5579b1669a40 con 0x5579b163d000 2024-03-16T10:14:33.367 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:14:33 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-d[71330]: starting mds.d at 2024-03-16T10:14:33.367 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:14:33 smithi077 bash[71313]: ffb63e7b892a7e0ddd4667a48efff0c3897bb082d80aac64b1af52060afebd3f 2024-03-16T10:14:33.367 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:14:33 smithi077 systemd[1]: Started Ceph mds.d for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:14:34.115 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:33 smithi192 ceph-mon[36354]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:34.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:33 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/191896871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:34.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:34.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:34.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:34.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:34.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:34.116 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:33 smithi077 ceph-mon[32400]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:33 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/191896871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:33 smithi193 ceph-mon[36311]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:33 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/191896871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:34.814 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:34.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.928+0000 7fca2b64b640 1 -- 172.21.15.77:0/2976810225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca24100710 msgr2=0x7fca24100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.928+0000 7fca2b64b640 1 --2- 172.21.15.77:0/2976810225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca24100710 0x7fca24100b30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fca14009980 tx=0x7fca1402de60 comp rx=0 tx=0).stop 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.928+0000 7fca2b64b640 1 -- 172.21.15.77:0/2976810225 shutdown_connections 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.928+0000 7fca2b64b640 1 --2- 172.21.15.77:0/2976810225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca241022c0 0x7fca24108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.928+0000 7fca2b64b640 1 --2- 172.21.15.77:0/2976810225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca24101900 0x7fca24101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.928+0000 7fca2b64b640 1 --2- 172.21.15.77:0/2976810225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca24100710 0x7fca24100b30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.928+0000 7fca2b64b640 1 -- 172.21.15.77:0/2976810225 >> 172.21.15.77:0/2976810225 conn(0x7fca240fbcc0 msgr2=0x7fca240fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.928+0000 7fca2b64b640 1 -- 172.21.15.77:0/2976810225 shutdown_connections 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.928+0000 7fca2b64b640 1 -- 172.21.15.77:0/2976810225 wait complete. 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca2b64b640 1 Processor -- start 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca2b64b640 1 -- start start 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca2b64b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca24100710 0x7fca241958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca2b64b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca24101900 0x7fca24195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca2b64b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca241022c0 0x7fca2419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:34.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca2b64b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fca2410b290 con 0x7fca241022c0 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca2b64b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fca2410b110 con 0x7fca24101900 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca2b64b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fca2410b410 con 0x7fca24100710 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca29be8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca241022c0 0x7fca2419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca29be8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca241022c0 0x7fca2419ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60928/0 (socket says 172.21.15.77:60928) 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca29be8640 1 -- 172.21.15.77:0/1453727148 learned_addr learned my addr 172.21.15.77:0/1453727148 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca293e7640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca24100710 0x7fca241958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.929+0000 7fca28be6640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca24101900 0x7fca24195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.930+0000 7fca29be8640 1 -- 172.21.15.77:0/1453727148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca24100710 msgr2=0x7fca241958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.930+0000 7fca29be8640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca24100710 0x7fca241958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.930+0000 7fca29be8640 1 -- 172.21.15.77:0/1453727148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca24101900 msgr2=0x7fca24195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.930+0000 7fca29be8640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca24101900 0x7fca24195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.930+0000 7fca29be8640 1 -- 172.21.15.77:0/1453727148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2419d4d0 con 0x7fca241022c0 2024-03-16T10:14:34.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.930+0000 7fca29be8640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca241022c0 0x7fca2419ce60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00cce0 tx=0x7fca1c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:34.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.930+0000 7fca1a7fc640 1 -- 172.21.15.77:0/1453727148 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c00f450 con 0x7fca241022c0 2024-03-16T10:14:34.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.930+0000 7fca1a7fc640 1 -- 172.21.15.77:0/1453727148 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca1c00fad0 con 0x7fca241022c0 2024-03-16T10:14:34.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.930+0000 7fca2b64b640 1 -- 172.21.15.77:0/1453727148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca2419d7c0 con 0x7fca241022c0 2024-03-16T10:14:34.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.930+0000 7fca1a7fc640 1 -- 172.21.15.77:0/1453727148 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c018c10 con 0x7fca241022c0 2024-03-16T10:14:34.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.930+0000 7fca2b64b640 1 -- 172.21.15.77:0/1453727148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca2419dcc0 con 0x7fca241022c0 2024-03-16T10:14:34.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.931+0000 7fca2b64b640 1 -- 172.21.15.77:0/1453727148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca241079d0 con 0x7fca241022c0 2024-03-16T10:14:34.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.932+0000 7fca1a7fc640 1 -- 172.21.15.77:0/1453727148 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fca1c00f5f0 con 0x7fca241022c0 2024-03-16T10:14:34.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.933+0000 7fca1a7fc640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fca100a2800 0x7fca100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:34.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.933+0000 7fca293e7640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fca100a2800 0x7fca100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:34.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.933+0000 7fca1a7fc640 1 -- 172.21.15.77:0/1453727148 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fca1c0cec70 con 0x7fca241022c0 2024-03-16T10:14:34.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.933+0000 7fca293e7640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fca100a2800 0x7fca100a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fca14005ab0 tx=0x7fca14005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:34.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:34.937+0000 7fca1a7fc640 1 -- 172.21.15.77:0/1453727148 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fca1c014030 con 0x7fca241022c0 2024-03-16T10:14:35.107 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:34 smithi192 ceph-mon[36354]: Deploying daemon mds.e on smithi192 2024-03-16T10:14:35.108 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:34 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.77:6836/1581869083,v1:172.21.15.77:6837/1581869083] up:boot 2024-03-16T10:14:35.108 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:34 smithi192 ceph-mon[36354]: fsmap 4 up:standby 2024-03-16T10:14:35.108 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:34 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T10:14:35.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.149+0000 7fca2b64b640 1 -- 172.21.15.77:0/1453727148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca2402d1b0 con 0x7fca241022c0 2024-03-16T10:14:35.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.150+0000 7fca1a7fc640 1 -- 172.21.15.77:0/1453727148 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7fca1c09c220 con 0x7fca241022c0 2024-03-16T10:14:35.151 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:35.151 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T10:14:35.151 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 5 2024-03-16T10:14:35.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.152+0000 7fca2b64b640 1 -- 172.21.15.77:0/1453727148 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fca100a2800 msgr2=0x7fca100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:35.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.152+0000 7fca2b64b640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fca100a2800 0x7fca100a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fca14005ab0 tx=0x7fca14005fd0 comp rx=0 tx=0).stop 2024-03-16T10:14:35.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.152+0000 7fca2b64b640 1 -- 172.21.15.77:0/1453727148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca241022c0 msgr2=0x7fca2419ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:35.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.152+0000 7fca2b64b640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca241022c0 0x7fca2419ce60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00cce0 tx=0x7fca1c00a620 comp rx=0 tx=0).stop 2024-03-16T10:14:35.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.152+0000 7fca2b64b640 1 -- 172.21.15.77:0/1453727148 shutdown_connections 2024-03-16T10:14:35.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.152+0000 7fca2b64b640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca241022c0 0x7fca2419ce60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:35.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.152+0000 7fca2b64b640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca24101900 0x7fca24195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:35.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.152+0000 7fca2b64b640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fca100a2800 0x7fca100a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:35.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.152+0000 7fca2b64b640 1 --2- 172.21.15.77:0/1453727148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca24100710 0x7fca241958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:35.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.152+0000 7fca2b64b640 1 -- 172.21.15.77:0/1453727148 >> 172.21.15.77:0/1453727148 conn(0x7fca240fbcc0 msgr2=0x7fca24104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:35.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.152+0000 7fca2b64b640 1 -- 172.21.15.77:0/1453727148 shutdown_connections 2024-03-16T10:14:35.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:35.152+0000 7fca2b64b640 1 -- 172.21.15.77:0/1453727148 wait complete. 2024-03-16T10:14:35.163 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:34 smithi077 ceph-mon[32400]: Deploying daemon mds.e on smithi192 2024-03-16T10:14:35.163 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:34 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.77:6836/1581869083,v1:172.21.15.77:6837/1581869083] up:boot 2024-03-16T10:14:35.163 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:34 smithi077 ceph-mon[32400]: fsmap 4 up:standby 2024-03-16T10:14:35.163 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:34 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T10:14:35.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:34 smithi193 ceph-mon[36311]: Deploying daemon mds.e on smithi192 2024-03-16T10:14:35.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:34 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.77:6836/1581869083,v1:172.21.15.77:6837/1581869083] up:boot 2024-03-16T10:14:35.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:34 smithi193 ceph-mon[36311]: fsmap 4 up:standby 2024-03-16T10:14:35.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:34 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T10:14:35.666 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:14:35 smithi192 systemd[1]: Starting Ceph mds.e for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:14:36.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:35 smithi192 ceph-mon[36354]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:36.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:35 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1453727148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:36.071 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:14:35 smithi192 podman[66471]: 2024-03-16 10:14:35.691951007 +0000 UTC m=+0.018227464 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:14:36.071 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:14:35 smithi192 podman[66471]: 2024-03-16 10:14:35.801219312 +0000 UTC m=+0.127495769 container create 05840130053b69d8d94798244f3b2530fa02c4df3058103085f08a2689556fce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:14:36.071 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:14:36 smithi192 podman[66471]: 2024-03-16 10:14:36.07059909 +0000 UTC m=+0.396875549 container init 05840130053b69d8d94798244f3b2530fa02c4df3058103085f08a2689556fce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-e, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-16T10:14:36.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:35 smithi077 ceph-mon[32400]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:36.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:35 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1453727148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:35 smithi193 ceph-mon[36311]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:35 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1453727148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:36.416 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:14:36 smithi192 podman[66471]: 2024-03-16 10:14:36.073204831 +0000 UTC m=+0.399481292 container start 05840130053b69d8d94798244f3b2530fa02c4df3058103085f08a2689556fce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-16T10:14:36.417 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:14:36 smithi192 ceph-mds[66490]: -- 172.21.15.192:0/4027980741 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd5e203a40 con 0x55bd5e1d7000 2024-03-16T10:14:36.417 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:14:36 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-e[66486]: starting mds.e at 2024-03-16T10:14:36.417 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:14:36 smithi192 bash[66471]: 05840130053b69d8d94798244f3b2530fa02c4df3058103085f08a2689556fce 2024-03-16T10:14:36.417 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:14:36 smithi192 systemd[1]: Started Ceph mds.e for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:14:37.165 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.280+0000 7f5ab973b640 1 -- 172.21.15.77:0/2046248005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab4100710 msgr2=0x7f5ab4100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.280+0000 7f5ab973b640 1 --2- 172.21.15.77:0/2046248005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab4100710 0x7f5ab4100b30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4009900 tx=0x7f5aa402bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.280+0000 7f5ab973b640 1 -- 172.21.15.77:0/2046248005 shutdown_connections 2024-03-16T10:14:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.280+0000 7f5ab973b640 1 --2- 172.21.15.77:0/2046248005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ab41022c0 0x7f5ab4108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.280+0000 7f5ab973b640 1 --2- 172.21.15.77:0/2046248005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4101900 0x7f5ab4101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.280+0000 7f5ab973b640 1 --2- 172.21.15.77:0/2046248005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab4100710 0x7f5ab4100b30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.280+0000 7f5ab973b640 1 -- 172.21.15.77:0/2046248005 >> 172.21.15.77:0/2046248005 conn(0x7f5ab40fbcc0 msgr2=0x7f5ab40fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.280+0000 7f5ab973b640 1 -- 172.21.15.77:0/2046248005 shutdown_connections 2024-03-16T10:14:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.281+0000 7f5ab973b640 1 -- 172.21.15.77:0/2046248005 wait complete. 2024-03-16T10:14:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.281+0000 7f5ab973b640 1 Processor -- start 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.281+0000 7f5ab973b640 1 -- start start 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.281+0000 7f5ab973b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ab4100710 0x7f5ab4197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.281+0000 7f5ab973b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4101900 0x7f5ab4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.281+0000 7f5ab973b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab41022c0 0x7f5ab419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.281+0000 7f5ab973b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab410d470 con 0x7f5ab41022c0 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.281+0000 7f5ab973b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab410d2f0 con 0x7f5ab4100710 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.281+0000 7f5ab973b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab410d5f0 con 0x7f5ab4101900 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab41022c0 0x7f5ab419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab41022c0 0x7f5ab419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60940/0 (socket says 172.21.15.77:60940) 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab37fe640 1 -- 172.21.15.77:0/3955993448 learned_addr learned my addr 172.21.15.77:0/3955993448 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab2ffd640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ab4100710 0x7f5ab4197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab27fc640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4101900 0x7f5ab4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab37fe640 1 -- 172.21.15.77:0/3955993448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4101900 msgr2=0x7f5ab4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab37fe640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4101900 0x7f5ab4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab37fe640 1 -- 172.21.15.77:0/3955993448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ab4100710 msgr2=0x7f5ab4197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:37.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab37fe640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ab4100710 0x7f5ab4197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:37.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab37fe640 1 -- 172.21.15.77:0/3955993448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab419f6d0 con 0x7f5ab41022c0 2024-03-16T10:14:37.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab27fc640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4101900 0x7f5ab4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:14:37.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab37fe640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab41022c0 0x7f5ab419f060 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00cce0 tx=0x7f5aac00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:37.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5a9bfff640 1 -- 172.21.15.77:0/3955993448 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac00f450 con 0x7f5ab41022c0 2024-03-16T10:14:37.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab973b640 1 -- 172.21.15.77:0/3955993448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab419f9c0 con 0x7f5ab41022c0 2024-03-16T10:14:37.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5a9bfff640 1 -- 172.21.15.77:0/3955993448 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aac00fad0 con 0x7f5ab41022c0 2024-03-16T10:14:37.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5a9bfff640 1 -- 172.21.15.77:0/3955993448 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac018c10 con 0x7f5ab41022c0 2024-03-16T10:14:37.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.282+0000 7f5ab973b640 1 -- 172.21.15.77:0/3955993448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab419fec0 con 0x7f5ab41022c0 2024-03-16T10:14:37.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.283+0000 7f5ab973b640 1 -- 172.21.15.77:0/3955993448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab4107a30 con 0x7f5ab41022c0 2024-03-16T10:14:37.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.284+0000 7f5a9bfff640 1 -- 172.21.15.77:0/3955993448 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5aac00f5f0 con 0x7f5ab41022c0 2024-03-16T10:14:37.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.285+0000 7f5a9bfff640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5a9c0a2800 0x7f5a9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:37.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.285+0000 7f5a9bfff640 1 -- 172.21.15.77:0/3955993448 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5aac0cecb0 con 0x7f5ab41022c0 2024-03-16T10:14:37.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.285+0000 7f5ab2ffd640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5a9c0a2800 0x7f5a9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:37.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.286+0000 7f5ab2ffd640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5a9c0a2800 0x7f5a9c0a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4000c00 tx=0x7f5aa4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:37.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.289+0000 7f5a9bfff640 1 -- 172.21.15.77:0/3955993448 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5aac014030 con 0x7f5ab41022c0 2024-03-16T10:14:37.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:37 smithi077 ceph-mon[32400]: Deploying daemon mds.f on smithi193 2024-03-16T10:14:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:37 smithi077 ceph-mon[32400]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:37 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.192:6834/3288720506,v1:172.21.15.192:6835/3288720506] up:boot 2024-03-16T10:14:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:37 smithi077 ceph-mon[32400]: fsmap 5 up:standby 2024-03-16T10:14:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:37 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T10:14:37.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:37.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:37 smithi193 ceph-mon[36311]: Deploying daemon mds.f on smithi193 2024-03-16T10:14:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:37 smithi193 ceph-mon[36311]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:37 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.192:6834/3288720506,v1:172.21.15.192:6835/3288720506] up:boot 2024-03-16T10:14:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:37 smithi193 ceph-mon[36311]: fsmap 5 up:standby 2024-03-16T10:14:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:37 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T10:14:37.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.503+0000 7f5ab973b640 1 -- 172.21.15.77:0/3955993448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ab40623e0 con 0x7f5ab41022c0 2024-03-16T10:14:37.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.503+0000 7f5a9bfff640 1 -- 172.21.15.77:0/3955993448 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7f5aac09ce20 con 0x7f5ab41022c0 2024-03-16T10:14:37.504 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:37.504 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-16T10:14:37.505 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 6 2024-03-16T10:14:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.505+0000 7f5ab973b640 1 -- 172.21.15.77:0/3955993448 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5a9c0a2800 msgr2=0x7f5a9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.505+0000 7f5ab973b640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5a9c0a2800 0x7f5a9c0a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4000c00 tx=0x7f5aa4005fd0 comp rx=0 tx=0).stop 2024-03-16T10:14:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.505+0000 7f5ab973b640 1 -- 172.21.15.77:0/3955993448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab41022c0 msgr2=0x7f5ab419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.505+0000 7f5ab973b640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab41022c0 0x7f5ab419f060 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00cce0 tx=0x7f5aac00a620 comp rx=0 tx=0).stop 2024-03-16T10:14:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.505+0000 7f5ab973b640 1 -- 172.21.15.77:0/3955993448 shutdown_connections 2024-03-16T10:14:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.505+0000 7f5ab973b640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ab41022c0 0x7f5ab419f060 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.505+0000 7f5ab973b640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5a9c0a2800 0x7f5a9c0a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.505+0000 7f5ab973b640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4101900 0x7f5ab4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.505+0000 7f5ab973b640 1 --2- 172.21.15.77:0/3955993448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ab4100710 0x7f5ab4197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.505+0000 7f5ab973b640 1 -- 172.21.15.77:0/3955993448 >> 172.21.15.77:0/3955993448 conn(0x7f5ab40fbcc0 msgr2=0x7f5ab4104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.505+0000 7f5ab973b640 1 -- 172.21.15.77:0/3955993448 shutdown_connections 2024-03-16T10:14:37.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:37.505+0000 7f5ab973b640 1 -- 172.21.15.77:0/3955993448 wait complete. 2024-03-16T10:14:37.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:37.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:37.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:37.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:37.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:37.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:37.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:37 smithi192 ceph-mon[36354]: Deploying daemon mds.f on smithi193 2024-03-16T10:14:37.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:37 smithi192 ceph-mon[36354]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:37.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:37 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.192:6834/3288720506,v1:172.21.15.192:6835/3288720506] up:boot 2024-03-16T10:14:37.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:37 smithi192 ceph-mon[36354]: fsmap 5 up:standby 2024-03-16T10:14:37.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:37 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T10:14:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:38 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3955993448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:38.503 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:14:38 smithi193 systemd[1]: Starting Ceph mds.f for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:14:38.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:38 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3955993448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:38.744 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:38 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3955993448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:38.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:14:38 smithi193 podman[66855]: 2024-03-16 10:14:38.534898716 +0000 UTC m=+0.018181219 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:14:38.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:14:38 smithi193 podman[66855]: 2024-03-16 10:14:38.667292473 +0000 UTC m=+0.150574966 container create f91bda9fc43cc9cab5d245d6ed80a398a480fa7bdee115cc63cba6fb9664cee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-16T10:14:38.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:14:38 smithi193 podman[66855]: 2024-03-16 10:14:38.894983089 +0000 UTC m=+0.378265598 container init f91bda9fc43cc9cab5d245d6ed80a398a480fa7bdee115cc63cba6fb9664cee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-16T10:14:39.253 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:14:38 smithi193 podman[66855]: 2024-03-16 10:14:38.89798573 +0000 UTC m=+0.381268216 container start f91bda9fc43cc9cab5d245d6ed80a398a480fa7bdee115cc63cba6fb9664cee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-16T10:14:39.253 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:14:38 smithi193 ceph-mds[66874]: -- 172.21.15.193:0/2871500105 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d120775a40 con 0x55d120749000 2024-03-16T10:14:39.253 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:14:38 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f[66870]: starting mds.f at 2024-03-16T10:14:39.253 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:14:38 smithi193 bash[66855]: f91bda9fc43cc9cab5d245d6ed80a398a480fa7bdee115cc63cba6fb9664cee2 2024-03-16T10:14:39.253 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:14:38 smithi193 systemd[1]: Started Ceph mds.f for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:14:39.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:39.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:39 smithi077 ceph-mon[32400]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:39.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:39 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:39.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:39 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:39.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:39 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:39.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:39 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:39.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:39 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:39.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:39 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:39.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:39 smithi192 ceph-mon[36354]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:39.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:39 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:39.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:39 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:39.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:39 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:39.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:39 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:39.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:39 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:39.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:39 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:39.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.631+0000 7fa511263640 1 -- 172.21.15.77:0/2124819386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa50c1018f0 msgr2=0x7fa50c101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:39.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.631+0000 7fa511263640 1 --2- 172.21.15.77:0/2124819386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa50c1018f0 0x7fa50c101d70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa500009900 tx=0x7fa50002bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:39.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.631+0000 7fa511263640 1 -- 172.21.15.77:0/2124819386 shutdown_connections 2024-03-16T10:14:39.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.631+0000 7fa511263640 1 --2- 172.21.15.77:0/2124819386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c1022b0 0x7fa50c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:39.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.631+0000 7fa511263640 1 --2- 172.21.15.77:0/2124819386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa50c1018f0 0x7fa50c101d70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:39.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.631+0000 7fa511263640 1 --2- 172.21.15.77:0/2124819386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa50c100700 0x7fa50c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:39.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.631+0000 7fa511263640 1 -- 172.21.15.77:0/2124819386 >> 172.21.15.77:0/2124819386 conn(0x7fa50c0fbc70 msgr2=0x7fa50c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:39.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.632+0000 7fa511263640 1 -- 172.21.15.77:0/2124819386 shutdown_connections 2024-03-16T10:14:39.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.632+0000 7fa511263640 1 -- 172.21.15.77:0/2124819386 wait complete. 2024-03-16T10:14:39.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.632+0000 7fa511263640 1 Processor -- start 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.632+0000 7fa511263640 1 -- start start 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.632+0000 7fa511263640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa50c100700 0x7fa50c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.632+0000 7fa511263640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa50c1018f0 0x7fa50c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.632+0000 7fa511263640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c1022b0 0x7fa50c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.632+0000 7fa511263640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c10c380 con 0x7fa50c1018f0 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.632+0000 7fa511263640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c10c200 con 0x7fa50c1022b0 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa511263640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c10c500 con 0x7fa50c100700 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa50a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa50c1018f0 0x7fa50c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa50a7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa50c1018f0 0x7fa50c198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60950/0 (socket says 172.21.15.77:60950) 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa50a7fc640 1 -- 172.21.15.77:0/2643876090 learned_addr learned my addr 172.21.15.77:0/2643876090 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa50affd640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa50c100700 0x7fa50c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa50b7fe640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c1022b0 0x7fa50c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa50a7fc640 1 -- 172.21.15.77:0/2643876090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa50c100700 msgr2=0x7fa50c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa50a7fc640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa50c100700 0x7fa50c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa50a7fc640 1 -- 172.21.15.77:0/2643876090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c1022b0 msgr2=0x7fa50c19f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:39.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa50a7fc640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c1022b0 0x7fa50c19f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa50a7fc640 1 -- 172.21.15.77:0/2643876090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50c19f740 con 0x7fa50c1018f0 2024-03-16T10:14:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa50b7fe640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c1022b0 0x7fa50c19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:14:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa50a7fc640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa50c1018f0 0x7fa50c198060 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa500009230 tx=0x7fa500005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa4f3fff640 1 -- 172.21.15.77:0/2643876090 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa500041070 con 0x7fa50c1018f0 2024-03-16T10:14:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa4f3fff640 1 -- 172.21.15.77:0/2643876090 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5000044d0 con 0x7fa50c1018f0 2024-03-16T10:14:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa4f3fff640 1 -- 172.21.15.77:0/2643876090 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa500034440 con 0x7fa50c1018f0 2024-03-16T10:14:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa511263640 1 -- 172.21.15.77:0/2643876090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50c19f9d0 con 0x7fa50c1018f0 2024-03-16T10:14:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.633+0000 7fa511263640 1 -- 172.21.15.77:0/2643876090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa50c19fec0 con 0x7fa50c1018f0 2024-03-16T10:14:39.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.634+0000 7fa511263640 1 -- 172.21.15.77:0/2643876090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa50c1a00f0 con 0x7fa50c1018f0 2024-03-16T10:14:39.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.635+0000 7fa4f3fff640 1 -- 172.21.15.77:0/2643876090 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa500044070 con 0x7fa50c1018f0 2024-03-16T10:14:39.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.636+0000 7fa4f3fff640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4f40a2800 0x7fa4f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:39.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.636+0000 7fa50affd640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4f40a2800 0x7fa4f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:39.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.636+0000 7fa4f3fff640 1 -- 172.21.15.77:0/2643876090 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa5000f02d0 con 0x7fa50c1018f0 2024-03-16T10:14:39.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.636+0000 7fa50affd640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4f40a2800 0x7fa4f40a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa50c101750 tx=0x7fa4fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:39.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.641+0000 7fa4f3fff640 1 -- 172.21.15.77:0/2643876090 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa5000bd810 con 0x7fa50c1018f0 2024-03-16T10:14:39.702 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:39 smithi193 ceph-mon[36311]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:39.702 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:39 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:39.702 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:39 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:39.702 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:39 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:39.702 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:39 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:39.702 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:39 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:39.702 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:39 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:39.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.852+0000 7fa511263640 1 -- 172.21.15.77:0/2643876090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa50c1a02c0 con 0x7fa50c1018f0 2024-03-16T10:14:39.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.852+0000 7fa4f3fff640 1 -- 172.21.15.77:0/2643876090 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7fa5000bd110 con 0x7fa50c1018f0 2024-03-16T10:14:39.853 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:39.853 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-16T10:14:39.854 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 7 2024-03-16T10:14:39.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.854+0000 7fa511263640 1 -- 172.21.15.77:0/2643876090 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4f40a2800 msgr2=0x7fa4f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:39.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.854+0000 7fa511263640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4f40a2800 0x7fa4f40a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa50c101750 tx=0x7fa4fc005f90 comp rx=0 tx=0).stop 2024-03-16T10:14:39.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.854+0000 7fa511263640 1 -- 172.21.15.77:0/2643876090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa50c1018f0 msgr2=0x7fa50c198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:39.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.854+0000 7fa511263640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa50c1018f0 0x7fa50c198060 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa500009230 tx=0x7fa500005d50 comp rx=0 tx=0).stop 2024-03-16T10:14:39.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.854+0000 7fa511263640 1 -- 172.21.15.77:0/2643876090 shutdown_connections 2024-03-16T10:14:39.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.854+0000 7fa511263640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa50c1022b0 0x7fa50c19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:39.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.854+0000 7fa511263640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa50c1018f0 0x7fa50c198060 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:39.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.854+0000 7fa511263640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4f40a2800 0x7fa4f40a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:39.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.854+0000 7fa511263640 1 --2- 172.21.15.77:0/2643876090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa50c100700 0x7fa50c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:39.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.854+0000 7fa511263640 1 -- 172.21.15.77:0/2643876090 >> 172.21.15.77:0/2643876090 conn(0x7fa50c0fbc70 msgr2=0x7fa50c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:39.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.855+0000 7fa511263640 1 -- 172.21.15.77:0/2643876090 shutdown_connections 2024-03-16T10:14:39.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:39.855+0000 7fa511263640 1 -- 172.21.15.77:0/2643876090 wait complete. 2024-03-16T10:14:40.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:40 smithi192 ceph-mon[36354]: Deploying daemon mds.c on smithi193 2024-03-16T10:14:40.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:40 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.193:6834/3768363028,v1:172.21.15.193:6835/3768363028] up:boot 2024-03-16T10:14:40.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:40 smithi192 ceph-mon[36354]: fsmap 6 up:standby 2024-03-16T10:14:40.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:40 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T10:14:40.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:40 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2643876090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:40.707 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:40 smithi193 ceph-mon[36311]: Deploying daemon mds.c on smithi193 2024-03-16T10:14:40.707 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:40 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.193:6834/3768363028,v1:172.21.15.193:6835/3768363028] up:boot 2024-03-16T10:14:40.707 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:40 smithi193 ceph-mon[36311]: fsmap 6 up:standby 2024-03-16T10:14:40.707 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:40 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T10:14:40.707 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:40 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2643876090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:40.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:40 smithi077 ceph-mon[32400]: Deploying daemon mds.c on smithi193 2024-03-16T10:14:40.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:40 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.193:6834/3768363028,v1:172.21.15.193:6835/3768363028] up:boot 2024-03-16T10:14:40.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:40 smithi077 ceph-mon[32400]: fsmap 6 up:standby 2024-03-16T10:14:40.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:40 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T10:14:40.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:40 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2643876090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:41.447 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:41 smithi193 ceph-mon[36311]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:41.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:41 smithi192 ceph-mon[36354]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:41.700 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:14:41 smithi193 podman[67167]: 2024-03-16 10:14:41.67719159 +0000 UTC m=+0.344754273 container start b02aaf5c0f0890b18d7e43862355b6180771997e89e128a78eb555065eb613a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-03-16T10:14:41.700 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:14:41 smithi193 ceph-mds[67186]: -- 172.21.15.193:0/1813509488 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d760f7a40 con 0x555d760cb000 2024-03-16T10:14:41.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:41 smithi077 ceph-mon[32400]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:41.871 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.983+0000 7f344f234640 1 -- 172.21.15.77:0/544561256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34481029f0 msgr2=0x7f3448102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.983+0000 7f344f234640 1 --2- 172.21.15.77:0/544561256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34481029f0 0x7f3448102e70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3434009900 tx=0x7f343402bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.983+0000 7f344f234640 1 -- 172.21.15.77:0/544561256 shutdown_connections 2024-03-16T10:14:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.983+0000 7f344f234640 1 --2- 172.21.15.77:0/544561256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34481033b0 0x7f3448109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.983+0000 7f344f234640 1 --2- 172.21.15.77:0/544561256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34481029f0 0x7f3448102e70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.983+0000 7f344f234640 1 --2- 172.21.15.77:0/544561256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3448101800 0x7f3448101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.983+0000 7f344f234640 1 -- 172.21.15.77:0/544561256 >> 172.21.15.77:0/544561256 conn(0x7f34480fcd90 msgr2=0x7f34480ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.983+0000 7f344f234640 1 -- 172.21.15.77:0/544561256 shutdown_connections 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.983+0000 7f344f234640 1 -- 172.21.15.77:0/544561256 wait complete. 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344f234640 1 Processor -- start 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344f234640 1 -- start start 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344f234640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3448101800 0x7f3448197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344f234640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34481029f0 0x7f3448198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344f234640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34481033b0 0x7f344819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344f234640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f344810d480 con 0x7f34481033b0 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344f234640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f344810d300 con 0x7f3448101800 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344f234640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f344810d600 con 0x7f34481029f0 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344d7d1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34481033b0 0x7f344819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344d7d1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34481033b0 0x7f344819f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56348/0 (socket says 172.21.15.77:56348) 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344d7d1640 1 -- 172.21.15.77:0/1634621686 learned_addr learned my addr 172.21.15.77:0/1634621686 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344cfd0640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3448101800 0x7f3448197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f3447fff640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34481029f0 0x7f3448198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344d7d1640 1 -- 172.21.15.77:0/1634621686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34481029f0 msgr2=0x7f3448198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344d7d1640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34481029f0 0x7f3448198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344d7d1640 1 -- 172.21.15.77:0/1634621686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3448101800 msgr2=0x7f3448197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344d7d1640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3448101800 0x7f3448197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344d7d1640 1 -- 172.21.15.77:0/1634621686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344819f720 con 0x7f34481033b0 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.984+0000 7f344d7d1640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34481033b0 0x7f344819f0b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f344000eaa0 tx=0x7f344000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.985+0000 7f3445ffb640 1 -- 172.21.15.77:0/1634621686 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344000cdb0 con 0x7f34481033b0 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.985+0000 7f3445ffb640 1 -- 172.21.15.77:0/1634621686 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3440005020 con 0x7f34481033b0 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.985+0000 7f344f234640 1 -- 172.21.15.77:0/1634621686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344819fa10 con 0x7f34481033b0 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.985+0000 7f3445ffb640 1 -- 172.21.15.77:0/1634621686 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34400116b0 con 0x7f34481033b0 2024-03-16T10:14:41.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.985+0000 7f344f234640 1 -- 172.21.15.77:0/1634621686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f344819ff30 con 0x7f34481033b0 2024-03-16T10:14:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.986+0000 7f344f234640 1 -- 172.21.15.77:0/1634621686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3448108b20 con 0x7f34481033b0 2024-03-16T10:14:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.987+0000 7f3445ffb640 1 -- 172.21.15.77:0/1634621686 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3440005560 con 0x7f34481033b0 2024-03-16T10:14:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.987+0000 7f3445ffb640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f34380a2800 0x7f34380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.988+0000 7f344cfd0640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f34380a2800 0x7f34380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:41.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.988+0000 7f3445ffb640 1 -- 172.21.15.77:0/1634621686 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f34400ce710 con 0x7f34481033b0 2024-03-16T10:14:41.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.988+0000 7f344cfd0640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f34380a2800 0x7f34380a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f343c005fd0 tx=0x7f343c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:41.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:41.992+0000 7f3445ffb640 1 -- 172.21.15.77:0/1634621686 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3440016030 con 0x7f34481033b0 2024-03-16T10:14:42.002 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:14:41 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-c[67182]: starting mds.c at 2024-03-16T10:14:42.003 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:14:41 smithi193 bash[67167]: b02aaf5c0f0890b18d7e43862355b6180771997e89e128a78eb555065eb613a1 2024-03-16T10:14:42.003 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:14:41 smithi193 systemd[1]: Started Ceph mds.c for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:14:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.203+0000 7f344f234640 1 -- 172.21.15.77:0/1634621686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34480623e0 con 0x7f34481033b0 2024-03-16T10:14:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.203+0000 7f3445ffb640 1 -- 172.21.15.77:0/1634621686 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7f344009b600 con 0x7f34481033b0 2024-03-16T10:14:42.204 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:42.204 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-16T10:14:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 7 2024-03-16T10:14:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.205+0000 7f344f234640 1 -- 172.21.15.77:0/1634621686 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f34380a2800 msgr2=0x7f34380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.205+0000 7f344f234640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f34380a2800 0x7f34380a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f343c005fd0 tx=0x7f343c00b410 comp rx=0 tx=0).stop 2024-03-16T10:14:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.205+0000 7f344f234640 1 -- 172.21.15.77:0/1634621686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34481033b0 msgr2=0x7f344819f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.205+0000 7f344f234640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34481033b0 0x7f344819f0b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f344000eaa0 tx=0x7f344000ef60 comp rx=0 tx=0).stop 2024-03-16T10:14:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.205+0000 7f344f234640 1 -- 172.21.15.77:0/1634621686 shutdown_connections 2024-03-16T10:14:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.205+0000 7f344f234640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34481033b0 0x7f344819f0b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.205+0000 7f344f234640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f34380a2800 0x7f34380a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.205+0000 7f344f234640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34481029f0 0x7f3448198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.205+0000 7f344f234640 1 --2- 172.21.15.77:0/1634621686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3448101800 0x7f3448197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.205+0000 7f344f234640 1 -- 172.21.15.77:0/1634621686 >> 172.21.15.77:0/1634621686 conn(0x7f34480fcd90 msgr2=0x7f3448105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.205+0000 7f344f234640 1 -- 172.21.15.77:0/1634621686 shutdown_connections 2024-03-16T10:14:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:42.205+0000 7f344f234640 1 -- 172.21.15.77:0/1634621686 wait complete. 2024-03-16T10:14:43.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:42 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:43.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:42 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:43.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:42 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:43.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:42 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:43.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:42 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:43.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:42 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:43.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:42 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1634621686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:43.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:42 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] up:boot 2024-03-16T10:14:43.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:42 smithi192 ceph-mon[36354]: fsmap 7 up:standby 2024-03-16T10:14:43.168 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:42 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T10:14:43.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:42 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:43.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:42 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:43.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:42 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:43.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:42 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:43.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:42 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:43.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:42 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:43.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:42 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1634621686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:43.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:42 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] up:boot 2024-03-16T10:14:43.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:42 smithi077 ceph-mon[32400]: fsmap 7 up:standby 2024-03-16T10:14:43.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:42 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T10:14:43.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:42 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:43.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:42 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:43.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:42 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:43.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:42 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:43.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:42 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:43.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:42 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:43.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:42 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1634621686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:43.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:42 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] up:boot 2024-03-16T10:14:43.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:42 smithi193 ceph-mon[36311]: fsmap 7 up:standby 2024-03-16T10:14:43.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:42 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T10:14:44.098 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:43 smithi192 ceph-mon[36354]: Deploying daemon mds.h on smithi192 2024-03-16T10:14:44.098 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:43 smithi192 ceph-mon[36354]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:44.154 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:43 smithi193 ceph-mon[36311]: Deploying daemon mds.h on smithi192 2024-03-16T10:14:44.154 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:43 smithi193 ceph-mon[36311]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:44.218 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:44.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:43 smithi077 ceph-mon[32400]: Deploying daemon mds.h on smithi192 2024-03-16T10:14:44.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:43 smithi077 ceph-mon[32400]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.330+0000 7f59680ea640 1 -- 172.21.15.77:0/1437527467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5960100090 msgr2=0x7f59601024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.330+0000 7f59680ea640 1 --2- 172.21.15.77:0/1437527467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5960100090 0x7f59601024c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f595c009900 tx=0x7f595c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 -- 172.21.15.77:0/1437527467 shutdown_connections 2024-03-16T10:14:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 --2- 172.21.15.77:0/1437527467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59601053b0 0x7f5960105830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 --2- 172.21.15.77:0/1437527467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5960102a00 0x7f5960104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 --2- 172.21.15.77:0/1437527467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5960100090 0x7f59601024c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 -- 172.21.15.77:0/1437527467 >> 172.21.15.77:0/1437527467 conn(0x7f59600fbc90 msgr2=0x7f59600fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 -- 172.21.15.77:0/1437527467 shutdown_connections 2024-03-16T10:14:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 -- 172.21.15.77:0/1437527467 wait complete. 2024-03-16T10:14:44.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 Processor -- start 2024-03-16T10:14:44.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 -- start start 2024-03-16T10:14:44.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5960100090 0x7f596019e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:44.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5960102a00 0x7f596019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:44.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59601053b0 0x7f59601a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:44.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5960113a70 con 0x7f5960102a00 2024-03-16T10:14:44.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f59601138f0 con 0x7f59601053b0 2024-03-16T10:14:44.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f59680ea640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5960113bf0 con 0x7f5960100090 2024-03-16T10:14:44.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f5965685640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5960102a00 0x7f596019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:44.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.331+0000 7f5965685640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5960102a00 0x7f596019e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56364/0 (socket says 172.21.15.77:56364) 2024-03-16T10:14:44.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5965685640 1 -- 172.21.15.77:0/619056166 learned_addr learned my addr 172.21.15.77:0/619056166 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:44.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5965e86640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5960100090 0x7f596019e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:44.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5966687640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59601053b0 0x7f59601a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:44.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5965685640 1 -- 172.21.15.77:0/619056166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5960100090 msgr2=0x7f596019e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:44.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5965685640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5960100090 0x7f596019e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:44.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5965685640 1 -- 172.21.15.77:0/619056166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59601053b0 msgr2=0x7f59601a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:44.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5965685640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59601053b0 0x7f59601a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:44.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5965685640 1 -- 172.21.15.77:0/619056166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59601a5d50 con 0x7f5960102a00 2024-03-16T10:14:44.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5966687640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59601053b0 0x7f59601a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:14:44.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5965e86640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5960100090 0x7f596019e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:14:44.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5965685640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5960102a00 0x7f596019e6c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f595000d950 tx=0x7f595000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:44.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5956ffd640 1 -- 172.21.15.77:0/619056166 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950017070 con 0x7f5960102a00 2024-03-16T10:14:44.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5956ffd640 1 -- 172.21.15.77:0/619056166 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5950009c30 con 0x7f5960102a00 2024-03-16T10:14:44.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f59680ea640 1 -- 172.21.15.77:0/619056166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59601a5fe0 con 0x7f5960102a00 2024-03-16T10:14:44.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f5956ffd640 1 -- 172.21.15.77:0/619056166 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595000f6b0 con 0x7f5960102a00 2024-03-16T10:14:44.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.332+0000 7f59680ea640 1 -- 172.21.15.77:0/619056166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59601a6500 con 0x7f5960102a00 2024-03-16T10:14:44.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.333+0000 7f59680ea640 1 -- 172.21.15.77:0/619056166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59600fd780 con 0x7f5960102a00 2024-03-16T10:14:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.334+0000 7f5956ffd640 1 -- 172.21.15.77:0/619056166 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f59500109d0 con 0x7f5960102a00 2024-03-16T10:14:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.335+0000 7f5956ffd640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f594c0a2800 0x7f594c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.335+0000 7f5965e86640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f594c0a2800 0x7f594c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.335+0000 7f5956ffd640 1 -- 172.21.15.77:0/619056166 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f59500ce340 con 0x7f5960102a00 2024-03-16T10:14:44.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.335+0000 7f5965e86640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f594c0a2800 0x7f594c0a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f595c006010 tx=0x7f595c02c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:44.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.339+0000 7f5956ffd640 1 -- 172.21.15.77:0/619056166 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5950009de0 con 0x7f5960102a00 2024-03-16T10:14:44.417 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:14:44 smithi192 podman[66779]: 2024-03-16 10:14:44.340777359 +0000 UTC m=+0.143629277 container create cf66993ac3c766aee293cd2c6bf97bb533f648edff0eb828faaa538b741a20e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-h, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-16T10:14:44.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.553+0000 7f59680ea640 1 -- 172.21.15.77:0/619056166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f596002d5b0 con 0x7f5960102a00 2024-03-16T10:14:44.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.554+0000 7f5956ffd640 1 -- 172.21.15.77:0/619056166 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6358 (secure 0 0 0) 0x7f595009b7e0 con 0x7f5960102a00 2024-03-16T10:14:44.555 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:44.555 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-16T10:14:44.555 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2024-03-16T10:14:44.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.556+0000 7f59680ea640 1 -- 172.21.15.77:0/619056166 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f594c0a2800 msgr2=0x7f594c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:44.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.556+0000 7f59680ea640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f594c0a2800 0x7f594c0a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f595c006010 tx=0x7f595c02c8a0 comp rx=0 tx=0).stop 2024-03-16T10:14:44.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.556+0000 7f59680ea640 1 -- 172.21.15.77:0/619056166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5960102a00 msgr2=0x7f596019e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:44.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.556+0000 7f59680ea640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5960102a00 0x7f596019e6c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f595000d950 tx=0x7f595000de10 comp rx=0 tx=0).stop 2024-03-16T10:14:44.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.556+0000 7f59680ea640 1 -- 172.21.15.77:0/619056166 shutdown_connections 2024-03-16T10:14:44.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.556+0000 7f59680ea640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59601053b0 0x7f59601a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:44.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.556+0000 7f59680ea640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5960102a00 0x7f596019e6c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:44.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.556+0000 7f59680ea640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f594c0a2800 0x7f594c0a4cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:44.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.556+0000 7f59680ea640 1 --2- 172.21.15.77:0/619056166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5960100090 0x7f596019e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:44.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.556+0000 7f59680ea640 1 -- 172.21.15.77:0/619056166 >> 172.21.15.77:0/619056166 conn(0x7f59600fbc90 msgr2=0x7f59601033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:44.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.556+0000 7f59680ea640 1 -- 172.21.15.77:0/619056166 shutdown_connections 2024-03-16T10:14:44.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:44.556+0000 7f59680ea640 1 -- 172.21.15.77:0/619056166 wait complete. 2024-03-16T10:14:44.735 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:14:44 smithi192 podman[66779]: 2024-03-16 10:14:44.622704523 +0000 UTC m=+0.425556431 container init cf66993ac3c766aee293cd2c6bf97bb533f648edff0eb828faaa538b741a20e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-h, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:14:44.735 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:14:44 smithi192 podman[66779]: 2024-03-16 10:14:44.625444406 +0000 UTC m=+0.428296320 container start cf66993ac3c766aee293cd2c6bf97bb533f648edff0eb828faaa538b741a20e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-16T10:14:44.735 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:14:44 smithi192 ceph-mds[66798]: -- 172.21.15.192:0/123814431 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556e5befba40 con 0x556e5bece800 2024-03-16T10:14:44.735 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:14:44 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-h[66794]: starting mds.h at 2024-03-16T10:14:45.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:44 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/619056166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:45.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:44 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:45.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:44 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:45.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:44 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:45.086 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:44 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:45.086 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:44 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:45.086 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:44 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:45.166 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:14:44 smithi192 bash[66779]: cf66993ac3c766aee293cd2c6bf97bb533f648edff0eb828faaa538b741a20e5 2024-03-16T10:14:45.166 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:14:44 smithi192 systemd[1]: Started Ceph mds.h for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:14:45.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:44 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/619056166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:45.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:44 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:45.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:44 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:45.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:44 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:45.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:44 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:45.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:44 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:45.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:44 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:45.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:44 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/619056166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:45.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:44 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:45.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:44 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:45.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:44 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:45.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:44 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:45.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:44 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:45.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:44 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:46.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:45 smithi193 ceph-mon[36311]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:45 smithi193 ceph-mon[36311]: Deploying daemon mds.i on smithi193 2024-03-16T10:14:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:45 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] up:boot 2024-03-16T10:14:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:45 smithi193 ceph-mon[36311]: fsmap 8 up:standby 2024-03-16T10:14:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T10:14:46.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:45 smithi192 ceph-mon[36354]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:46.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:45 smithi192 ceph-mon[36354]: Deploying daemon mds.i on smithi193 2024-03-16T10:14:46.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:45 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] up:boot 2024-03-16T10:14:46.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:45 smithi192 ceph-mon[36354]: fsmap 8 up:standby 2024-03-16T10:14:46.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T10:14:46.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:45 smithi077 ceph-mon[32400]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:46.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:45 smithi077 ceph-mon[32400]: Deploying daemon mds.i on smithi193 2024-03-16T10:14:46.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:45 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] up:boot 2024-03-16T10:14:46.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:45 smithi077 ceph-mon[32400]: fsmap 8 up:standby 2024-03-16T10:14:46.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T10:14:46.568 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:46.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.735+0000 7ff1ad359640 1 -- 172.21.15.77:0/23353282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a8104bb0 msgr2=0x7ff1a8106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:46.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.735+0000 7ff1ad359640 1 --2- 172.21.15.77:0/23353282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a8104bb0 0x7ff1a8106fa0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009900 tx=0x7ff19c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:46.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.735+0000 7ff1ad359640 1 -- 172.21.15.77:0/23353282 shutdown_connections 2024-03-16T10:14:46.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.735+0000 7ff1ad359640 1 --2- 172.21.15.77:0/23353282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1a81074e0 0x7ff1a8109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.735+0000 7ff1ad359640 1 --2- 172.21.15.77:0/23353282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a8104bb0 0x7ff1a8106fa0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.735+0000 7ff1ad359640 1 --2- 172.21.15.77:0/23353282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1a8102280 0x7ff1a8104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.735+0000 7ff1ad359640 1 -- 172.21.15.77:0/23353282 >> 172.21.15.77:0/23353282 conn(0x7ff1a80fbc70 msgr2=0x7ff1a80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.735+0000 7ff1ad359640 1 -- 172.21.15.77:0/23353282 shutdown_connections 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.735+0000 7ff1ad359640 1 -- 172.21.15.77:0/23353282 wait complete. 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.736+0000 7ff1ad359640 1 Processor -- start 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.736+0000 7ff1ad359640 1 -- start start 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.736+0000 7ff1ad359640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a8102280 0x7ff1a8197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.736+0000 7ff1ad359640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1a8104bb0 0x7ff1a8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.736+0000 7ff1ad359640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1a81074e0 0x7ff1a819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.736+0000 7ff1ad359640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a810d480 con 0x7ff1a8102280 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.736+0000 7ff1ad359640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a810d300 con 0x7ff1a8104bb0 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.736+0000 7ff1ad359640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a810d600 con 0x7ff1a81074e0 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.736+0000 7ff1a6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a8102280 0x7ff1a8197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.736+0000 7ff1a6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a8102280 0x7ff1a8197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56384/0 (socket says 172.21.15.77:56384) 2024-03-16T10:14:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.736+0000 7ff1a6ffd640 1 -- 172.21.15.77:0/3465755673 learned_addr learned my addr 172.21.15.77:0/3465755673 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff1a77fe640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1a81074e0 0x7ff1a819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff1a67fc640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1a8104bb0 0x7ff1a8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff1a6ffd640 1 -- 172.21.15.77:0/3465755673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1a81074e0 msgr2=0x7ff1a819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff1a6ffd640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1a81074e0 0x7ff1a819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff1a6ffd640 1 -- 172.21.15.77:0/3465755673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1a8104bb0 msgr2=0x7ff1a8198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff1a6ffd640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1a8104bb0 0x7ff1a8198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff1a6ffd640 1 -- 172.21.15.77:0/3465755673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a819f720 con 0x7ff1a8102280 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff1a67fc640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1a8104bb0 0x7ff1a8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff1a6ffd640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a8102280 0x7ff1a8197b00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff19800c890 tx=0x7ff19800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff18ffff640 1 -- 172.21.15.77:0/3465755673 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198004220 con 0x7ff1a8102280 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff18ffff640 1 -- 172.21.15.77:0/3465755673 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1980043c0 con 0x7ff1a8102280 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff1ad359640 1 -- 172.21.15.77:0/3465755673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a819fa10 con 0x7ff1a8102280 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff18ffff640 1 -- 172.21.15.77:0/3465755673 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1980049e0 con 0x7ff1a8102280 2024-03-16T10:14:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.737+0000 7ff1ad359640 1 -- 172.21.15.77:0/3465755673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a819ff60 con 0x7ff1a8102280 2024-03-16T10:14:46.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.738+0000 7ff1ad359640 1 -- 172.21.15.77:0/3465755673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1a80fd760 con 0x7ff1a8102280 2024-03-16T10:14:46.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.739+0000 7ff18ffff640 1 -- 172.21.15.77:0/3465755673 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff1980079b0 con 0x7ff1a8102280 2024-03-16T10:14:46.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.740+0000 7ff18ffff640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff1900a2800 0x7ff1900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:46.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.740+0000 7ff1a67fc640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff1900a2800 0x7ff1900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:46.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.740+0000 7ff18ffff640 1 -- 172.21.15.77:0/3465755673 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff1980ce660 con 0x7ff1a8102280 2024-03-16T10:14:46.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.740+0000 7ff1a67fc640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff1900a2800 0x7ff1900a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff19c006010 tx=0x7ff19c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:46.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.743+0000 7ff18ffff640 1 -- 172.21.15.77:0/3465755673 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff198014030 con 0x7ff1a8102280 2024-03-16T10:14:46.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.957+0000 7ff1ad359640 1 -- 172.21.15.77:0/3465755673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1a802d1b0 con 0x7ff1a8102280 2024-03-16T10:14:46.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.958+0000 7ff18ffff640 1 -- 172.21.15.77:0/3465755673 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7ff19809bba0 con 0x7ff1a8102280 2024-03-16T10:14:46.959 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:46.959 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-16T10:14:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2024-03-16T10:14:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.959+0000 7ff1ad359640 1 -- 172.21.15.77:0/3465755673 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff1900a2800 msgr2=0x7ff1900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.959+0000 7ff1ad359640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff1900a2800 0x7ff1900a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff19c006010 tx=0x7ff19c000bc0 comp rx=0 tx=0).stop 2024-03-16T10:14:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.959+0000 7ff1ad359640 1 -- 172.21.15.77:0/3465755673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a8102280 msgr2=0x7ff1a8197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.960+0000 7ff1ad359640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a8102280 0x7ff1a8197b00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff19800c890 tx=0x7ff19800cd50 comp rx=0 tx=0).stop 2024-03-16T10:14:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.960+0000 7ff1ad359640 1 -- 172.21.15.77:0/3465755673 shutdown_connections 2024-03-16T10:14:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.960+0000 7ff1ad359640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff1900a2800 0x7ff1900a4cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.960+0000 7ff1ad359640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1a81074e0 0x7ff1a819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.960+0000 7ff1ad359640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1a8104bb0 0x7ff1a8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.960+0000 7ff1ad359640 1 --2- 172.21.15.77:0/3465755673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1a8102280 0x7ff1a8197b00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.960+0000 7ff1ad359640 1 -- 172.21.15.77:0/3465755673 >> 172.21.15.77:0/3465755673 conn(0x7ff1a80fbc70 msgr2=0x7ff1a8102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.960+0000 7ff1ad359640 1 -- 172.21.15.77:0/3465755673 shutdown_connections 2024-03-16T10:14:46.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:46.960+0000 7ff1ad359640 1 -- 172.21.15.77:0/3465755673 wait complete. 2024-03-16T10:14:47.164 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:14:46 smithi193 systemd[1]: Starting Ceph mds.i for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:14:47.429 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:14:47 smithi193 podman[67490]: 2024-03-16 10:14:47.06646227 +0000 UTC m=+0.017718562 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:14:47.429 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:14:47 smithi193 podman[67490]: 2024-03-16 10:14:47.184649568 +0000 UTC m=+0.135905903 container create bb895664766a526a7a14f42a960733a0fd09f83cdc6f4580579443c20ad4a1c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-i, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-16T10:14:47.429 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:14:47 smithi193 podman[67490]: 2024-03-16 10:14:47.429037054 +0000 UTC m=+0.380293346 container init bb895664766a526a7a14f42a960733a0fd09f83cdc6f4580579443c20ad4a1c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-i, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-16T10:14:47.753 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:14:47 smithi193 podman[67490]: 2024-03-16 10:14:47.431978375 +0000 UTC m=+0.383234675 container start bb895664766a526a7a14f42a960733a0fd09f83cdc6f4580579443c20ad4a1c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-i, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:14:47.753 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:14:47 smithi193 ceph-mds[67509]: -- 172.21.15.193:0/4289876721 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555b1a8d5a40 con 0x555b1a8a8c00 2024-03-16T10:14:47.753 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:14:47 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-i[67505]: starting mds.i at 2024-03-16T10:14:47.753 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:14:47 smithi193 bash[67490]: bb895664766a526a7a14f42a960733a0fd09f83cdc6f4580579443c20ad4a1c4 2024-03-16T10:14:47.753 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:14:47 smithi193 systemd[1]: Started Ceph mds.i for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:14:48.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:47 smithi192 ceph-mon[36354]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:48.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:47 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3465755673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:48.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:47 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:48.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:47 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:48.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:47 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:48.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:47 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:48.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:47 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:48.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:47 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:48.242 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:47 smithi077 ceph-mon[32400]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:48.242 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:47 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3465755673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:48.242 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:48.243 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:48.243 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:48.243 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:48.243 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:48.243 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:47 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:47 smithi193 ceph-mon[36311]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:47 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3465755673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:47 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:48.972 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.079+0000 7f9a41110640 1 -- 172.21.15.77:0/982643465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a3c100700 msgr2=0x7f9a3c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.079+0000 7f9a41110640 1 --2- 172.21.15.77:0/982643465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a3c100700 0x7f9a3c100b20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c009ca0 tx=0x7f9a2c02c0f0 comp rx=0 tx=0).stop 2024-03-16T10:14:49.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.079+0000 7f9a41110640 1 -- 172.21.15.77:0/982643465 shutdown_connections 2024-03-16T10:14:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.079+0000 7f9a41110640 1 --2- 172.21.15.77:0/982643465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a3c1022b0 0x7f9a3c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.079+0000 7f9a41110640 1 --2- 172.21.15.77:0/982643465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a3c1018f0 0x7f9a3c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.079+0000 7f9a41110640 1 --2- 172.21.15.77:0/982643465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a3c100700 0x7f9a3c100b20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.079+0000 7f9a41110640 1 -- 172.21.15.77:0/982643465 >> 172.21.15.77:0/982643465 conn(0x7f9a3c0fbc70 msgr2=0x7f9a3c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.079+0000 7f9a41110640 1 -- 172.21.15.77:0/982643465 shutdown_connections 2024-03-16T10:14:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.079+0000 7f9a41110640 1 -- 172.21.15.77:0/982643465 wait complete. 2024-03-16T10:14:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a41110640 1 Processor -- start 2024-03-16T10:14:49.081 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a41110640 1 -- start start 2024-03-16T10:14:49.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a41110640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a3c100700 0x7f9a3c0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:49.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a41110640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a3c1018f0 0x7f9a3c19c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:49.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a41110640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a3c1022b0 0x7f9a3c19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a41110640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9a3c10c3d0 con 0x7f9a3c100700 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a41110640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f9a3c10c250 con 0x7f9a3c1018f0 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a41110640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9a3c10c550 con 0x7f9a3c1022b0 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a3ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a3c100700 0x7f9a3c0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a3ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a3c100700 0x7f9a3c0694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56398/0 (socket says 172.21.15.77:56398) 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a3ad9d640 1 -- 172.21.15.77:0/3459316736 learned_addr learned my addr 172.21.15.77:0/3459316736 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a3a59c640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a3c1018f0 0x7f9a3c19c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.080+0000 7f9a3b59e640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a3c1022b0 0x7f9a3c19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.081+0000 7f9a3ad9d640 1 -- 172.21.15.77:0/3459316736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a3c1022b0 msgr2=0x7f9a3c19ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.081+0000 7f9a3ad9d640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a3c1022b0 0x7f9a3c19ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.081+0000 7f9a3ad9d640 1 -- 172.21.15.77:0/3459316736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a3c1018f0 msgr2=0x7f9a3c19c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.081+0000 7f9a3ad9d640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a3c1018f0 0x7f9a3c19c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:49.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.081+0000 7f9a3ad9d640 1 -- 172.21.15.77:0/3459316736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a3c19f830 con 0x7f9a3c100700 2024-03-16T10:14:49.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.081+0000 7f9a3a59c640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a3c1018f0 0x7f9a3c19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:14:49.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.081+0000 7f9a3ad9d640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a3c100700 0x7f9a3c0694b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0324c0 tx=0x7f9a2c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:49.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.081+0000 7f9a23fff640 1 -- 172.21.15.77:0/3459316736 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c039040 con 0x7f9a3c100700 2024-03-16T10:14:49.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.081+0000 7f9a23fff640 1 -- 172.21.15.77:0/3459316736 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a2c044070 con 0x7f9a3c100700 2024-03-16T10:14:49.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.081+0000 7f9a41110640 1 -- 172.21.15.77:0/3459316736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a3c19fa60 con 0x7f9a3c100700 2024-03-16T10:14:49.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.081+0000 7f9a23fff640 1 -- 172.21.15.77:0/3459316736 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c0042a0 con 0x7f9a3c100700 2024-03-16T10:14:49.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.081+0000 7f9a41110640 1 -- 172.21.15.77:0/3459316736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a3c19ff50 con 0x7f9a3c100700 2024-03-16T10:14:49.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.082+0000 7f9a41110640 1 -- 172.21.15.77:0/3459316736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a3c107aa0 con 0x7f9a3c100700 2024-03-16T10:14:49.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.083+0000 7f9a23fff640 1 -- 172.21.15.77:0/3459316736 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9a2c003970 con 0x7f9a3c100700 2024-03-16T10:14:49.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.083+0000 7f9a23fff640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9a240a2800 0x7f9a240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:49.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.084+0000 7f9a3a59c640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9a240a2800 0x7f9a240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:49.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.084+0000 7f9a23fff640 1 -- 172.21.15.77:0/3459316736 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9a2c0f0b50 con 0x7f9a3c100700 2024-03-16T10:14:49.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.084+0000 7f9a3a59c640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9a240a2800 0x7f9a240a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c069220 tx=0x7f9a30008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:49.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.088+0000 7f9a23fff640 1 -- 172.21.15.77:0/3459316736 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a2c0be120 con 0x7f9a3c100700 2024-03-16T10:14:49.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:48 smithi192 ceph-mon[36354]: Deploying daemon mds.g on smithi077 2024-03-16T10:14:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:48 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] up:boot 2024-03-16T10:14:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:48 smithi192 ceph-mon[36354]: fsmap 9 up:standby 2024-03-16T10:14:49.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:48 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T10:14:49.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:48 smithi077 ceph-mon[32400]: Deploying daemon mds.g on smithi077 2024-03-16T10:14:49.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:48 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] up:boot 2024-03-16T10:14:49.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:48 smithi077 ceph-mon[32400]: fsmap 9 up:standby 2024-03-16T10:14:49.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:48 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T10:14:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:48 smithi193 ceph-mon[36311]: Deploying daemon mds.g on smithi077 2024-03-16T10:14:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:48 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] up:boot 2024-03-16T10:14:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:48 smithi193 ceph-mon[36311]: fsmap 9 up:standby 2024-03-16T10:14:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:48 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T10:14:49.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.308+0000 7f9a41110640 1 -- 172.21.15.77:0/3459316736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a3c02d400 con 0x7f9a3c100700 2024-03-16T10:14:49.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.308+0000 7f9a23fff640 1 -- 172.21.15.77:0/3459316736 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7f9a2c0bda20 con 0x7f9a3c100700 2024-03-16T10:14:49.309 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:49.309 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T10:14:49.310 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 10 2024-03-16T10:14:49.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.311+0000 7f9a41110640 1 -- 172.21.15.77:0/3459316736 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9a240a2800 msgr2=0x7f9a240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.311+0000 7f9a41110640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9a240a2800 0x7f9a240a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c069220 tx=0x7f9a30008040 comp rx=0 tx=0).stop 2024-03-16T10:14:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.311+0000 7f9a41110640 1 -- 172.21.15.77:0/3459316736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a3c100700 msgr2=0x7f9a3c0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.311+0000 7f9a41110640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a3c100700 0x7f9a3c0694b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0324c0 tx=0x7f9a2c032f20 comp rx=0 tx=0).stop 2024-03-16T10:14:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.311+0000 7f9a41110640 1 -- 172.21.15.77:0/3459316736 shutdown_connections 2024-03-16T10:14:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.311+0000 7f9a41110640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9a240a2800 0x7f9a240a4cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.311+0000 7f9a41110640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a3c1022b0 0x7f9a3c19ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.311+0000 7f9a41110640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a3c1018f0 0x7f9a3c19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.311+0000 7f9a41110640 1 --2- 172.21.15.77:0/3459316736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a3c100700 0x7f9a3c0694b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.311+0000 7f9a41110640 1 -- 172.21.15.77:0/3459316736 >> 172.21.15.77:0/3459316736 conn(0x7f9a3c0fbc70 msgr2=0x7f9a3c104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.311+0000 7f9a41110640 1 -- 172.21.15.77:0/3459316736 shutdown_connections 2024-03-16T10:14:49.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:49.311+0000 7f9a41110640 1 -- 172.21.15.77:0/3459316736 wait complete. 2024-03-16T10:14:49.839 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:14:49 smithi077 systemd[1]: Starting Ceph mds.g for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:14:49.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:49 smithi192 ceph-mon[36354]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:49.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:49 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3459316736' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:50.104 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:49 smithi077 ceph-mon[32400]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:50.104 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:49 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3459316736' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:50.104 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:14:49 smithi077 podman[71910]: 2024-03-16 10:14:49.844903418 +0000 UTC m=+0.018275984 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:14:50.104 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:14:49 smithi077 podman[71910]: 2024-03-16 10:14:49.95878441 +0000 UTC m=+0.132156978 container create 367a20f0727ea66891733d90982f79474afb1613a3e8aa56541cc864cc7deb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-g, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-03-16T10:14:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:49 smithi193 ceph-mon[36311]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:50.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:49 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3459316736' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:50.495 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:14:50 smithi077 podman[71910]: 2024-03-16 10:14:50.219763989 +0000 UTC m=+0.393136554 container init 367a20f0727ea66891733d90982f79474afb1613a3e8aa56541cc864cc7deb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-g, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-03-16T10:14:50.495 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:14:50 smithi077 podman[71910]: 2024-03-16 10:14:50.222479842 +0000 UTC m=+0.395852417 container start 367a20f0727ea66891733d90982f79474afb1613a3e8aa56541cc864cc7deb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-g, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-16T10:14:50.495 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:14:50 smithi077 ceph-mds[71929]: -- 172.21.15.77:0/2182730219 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cf05d63a40 con 0x55cf05d36c00 2024-03-16T10:14:50.495 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:14:50 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-g[71925]: starting mds.g at 2024-03-16T10:14:50.495 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:14:50 smithi077 bash[71910]: 367a20f0727ea66891733d90982f79474afb1613a3e8aa56541cc864cc7deb5c 2024-03-16T10:14:50.495 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:14:50 smithi077 systemd[1]: Started Ceph mds.g for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:14:51.326 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:51.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.434+0000 7f6a0720b640 1 -- 172.21.15.77:0/2209308619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a00100700 msgr2=0x7f6a00100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:51.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.434+0000 7f6a0720b640 1 --2- 172.21.15.77:0/2209308619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a00100700 0x7f6a00100b20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009900 tx=0x7f69f402bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:51.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.434+0000 7f6a0720b640 1 -- 172.21.15.77:0/2209308619 shutdown_connections 2024-03-16T10:14:51.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.434+0000 7f6a0720b640 1 --2- 172.21.15.77:0/2209308619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a001022b0 0x7f6a00108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:51.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.434+0000 7f6a0720b640 1 --2- 172.21.15.77:0/2209308619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a001018f0 0x7f6a00101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:51.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.434+0000 7f6a0720b640 1 --2- 172.21.15.77:0/2209308619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a00100700 0x7f6a00100b20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:51.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.434+0000 7f6a0720b640 1 -- 172.21.15.77:0/2209308619 >> 172.21.15.77:0/2209308619 conn(0x7f6a000fbc70 msgr2=0x7f6a000fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:51.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.434+0000 7f6a0720b640 1 -- 172.21.15.77:0/2209308619 shutdown_connections 2024-03-16T10:14:51.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.434+0000 7f6a0720b640 1 -- 172.21.15.77:0/2209308619 wait complete. 2024-03-16T10:14:51.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a0720b640 1 Processor -- start 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a0720b640 1 -- start start 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a0720b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00100700 0x7f6a00196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a0720b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a001018f0 0x7f6a00196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a0720b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a001022b0 0x7f6a0019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a0720b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0010c350 con 0x7f6a001022b0 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a0720b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0010c1d0 con 0x7f6a001018f0 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a0720b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0010c4d0 con 0x7f6a00100700 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a057a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a001022b0 0x7f6a0019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a057a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a001022b0 0x7f6a0019e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56436/0 (socket says 172.21.15.77:56436) 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a057a8640 1 -- 172.21.15.77:0/760481897 learned_addr learned my addr 172.21.15.77:0/760481897 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f69fffff640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a001018f0 0x7f6a00196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a04fa7640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00100700 0x7f6a00196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a057a8640 1 -- 172.21.15.77:0/760481897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00100700 msgr2=0x7f6a00196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:51.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a057a8640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00100700 0x7f6a00196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:51.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a057a8640 1 -- 172.21.15.77:0/760481897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a001018f0 msgr2=0x7f6a00196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:51.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a057a8640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a001018f0 0x7f6a00196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:51.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.435+0000 7f6a057a8640 1 -- 172.21.15.77:0/760481897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0019e670 con 0x7f6a001022b0 2024-03-16T10:14:51.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.436+0000 7f6a057a8640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a001022b0 0x7f6a0019e000 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f69f800cad0 tx=0x7f69f800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:51.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.436+0000 7f69fdffb640 1 -- 172.21.15.77:0/760481897 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f800ae20 con 0x7f6a001022b0 2024-03-16T10:14:51.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.436+0000 7f6a0720b640 1 -- 172.21.15.77:0/760481897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a0019e960 con 0x7f6a001022b0 2024-03-16T10:14:51.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.436+0000 7f69fdffb640 1 -- 172.21.15.77:0/760481897 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69f8005020 con 0x7f6a001022b0 2024-03-16T10:14:51.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.436+0000 7f69fdffb640 1 -- 172.21.15.77:0/760481897 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f800f6f0 con 0x7f6a001022b0 2024-03-16T10:14:51.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.436+0000 7f6a0720b640 1 -- 172.21.15.77:0/760481897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a0019eeb0 con 0x7f6a001022b0 2024-03-16T10:14:51.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.437+0000 7f6a0720b640 1 -- 172.21.15.77:0/760481897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a00107aa0 con 0x7f6a001022b0 2024-03-16T10:14:51.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.438+0000 7f69fdffb640 1 -- 172.21.15.77:0/760481897 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f69f80051d0 con 0x7f6a001022b0 2024-03-16T10:14:51.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.439+0000 7f69fdffb640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f69f00a2800 0x7f69f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:51.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.439+0000 7f6a04fa7640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f69f00a2800 0x7f69f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:51.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.439+0000 7f69fdffb640 1 -- 172.21.15.77:0/760481897 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f69f80ceaa0 con 0x7f6a001022b0 2024-03-16T10:14:51.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.439+0000 7f6a04fa7640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f69f00a2800 0x7f69f00a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f69f402c7b0 tx=0x7f69f4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:51.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.442+0000 7f69fdffb640 1 -- 172.21.15.77:0/760481897 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f69f8014030 con 0x7f6a001022b0 2024-03-16T10:14:51.597 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:51 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:51.597 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:51 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:51.597 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:51 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:51.597 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:51 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:51.597 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:51 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:51.597 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:51 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:51.597 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:51 smithi077 ceph-mon[32400]: Deploying daemon mds.a on smithi077 2024-03-16T10:14:51.597 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:51 smithi077 ceph-mon[32400]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:51.597 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:51 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.77:6838/3943375297,v1:172.21.15.77:6839/3943375297] up:boot 2024-03-16T10:14:51.597 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:51 smithi077 ceph-mon[32400]: fsmap 10 up:standby 2024-03-16T10:14:51.598 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:51 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T10:14:51.666 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.665+0000 7f6a0720b640 1 -- 172.21.15.77:0/760481897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a0002d1b0 con 0x7f6a001022b0 2024-03-16T10:14:51.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:51 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:51.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:51 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:51.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:51 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:51.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:51 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:51.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:51 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:51.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:51 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:51.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:51 smithi192 ceph-mon[36354]: Deploying daemon mds.a on smithi077 2024-03-16T10:14:51.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:51 smithi192 ceph-mon[36354]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:51.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:51 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.77:6838/3943375297,v1:172.21.15.77:6839/3943375297] up:boot 2024-03-16T10:14:51.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:51 smithi192 ceph-mon[36354]: fsmap 10 up:standby 2024-03-16T10:14:51.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:51 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T10:14:51.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.666+0000 7f69fdffb640 1 -- 172.21.15.77:0/760481897 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8839 (secure 0 0 0) 0x7f69f809bfe0 con 0x7f6a001022b0 2024-03-16T10:14:51.668 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:51.668 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14646,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T10:14:51.668 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 11 2024-03-16T10:14:51.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.669+0000 7f6a0720b640 1 -- 172.21.15.77:0/760481897 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f69f00a2800 msgr2=0x7f69f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:51.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.669+0000 7f6a0720b640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f69f00a2800 0x7f69f00a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f69f402c7b0 tx=0x7f69f4000bc0 comp rx=0 tx=0).stop 2024-03-16T10:14:51.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.669+0000 7f6a0720b640 1 -- 172.21.15.77:0/760481897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a001022b0 msgr2=0x7f6a0019e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:51.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.669+0000 7f6a0720b640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a001022b0 0x7f6a0019e000 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f69f800cad0 tx=0x7f69f800cf90 comp rx=0 tx=0).stop 2024-03-16T10:14:51.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.670+0000 7f6a0720b640 1 -- 172.21.15.77:0/760481897 shutdown_connections 2024-03-16T10:14:51.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.670+0000 7f6a0720b640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a001022b0 0x7f6a0019e000 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:51.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.670+0000 7f6a0720b640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a001018f0 0x7f6a00196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:51.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.670+0000 7f6a0720b640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f69f00a2800 0x7f69f00a4cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:51.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.670+0000 7f6a0720b640 1 --2- 172.21.15.77:0/760481897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a00100700 0x7f6a00196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:51.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.670+0000 7f6a0720b640 1 -- 172.21.15.77:0/760481897 >> 172.21.15.77:0/760481897 conn(0x7f6a000fbc70 msgr2=0x7f6a00104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:51.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.670+0000 7f6a0720b640 1 -- 172.21.15.77:0/760481897 shutdown_connections 2024-03-16T10:14:51.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:51.670+0000 7f6a0720b640 1 -- 172.21.15.77:0/760481897 wait complete. 2024-03-16T10:14:51.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:51 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:51.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:51 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:51.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:51 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:51.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:51 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:51.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:51 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:51.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:51 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:51.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:51 smithi193 ceph-mon[36311]: Deploying daemon mds.a on smithi077 2024-03-16T10:14:51.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:51 smithi193 ceph-mon[36311]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:51.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:51 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.77:6838/3943375297,v1:172.21.15.77:6839/3943375297] up:boot 2024-03-16T10:14:51.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:51 smithi193 ceph-mon[36311]: fsmap 10 up:standby 2024-03-16T10:14:51.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:51 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T10:14:52.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:52 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/760481897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:52.737 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:52 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/760481897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:52.738 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:14:52 smithi077 systemd[1]: Starting Ceph mds.a for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:14:52.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:52 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/760481897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:52.994 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:14:52 smithi077 podman[72274]: 2024-03-16 10:14:52.639889657 +0000 UTC m=+0.017623960 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:14:52.995 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:14:52 smithi077 podman[72274]: 2024-03-16 10:14:52.780224605 +0000 UTC m=+0.157958908 container create 2c643f6991bd5e3d168773ab9669ba48f8c506a4f8d822c71bcbffefb34e754a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-16T10:14:53.417 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:14:53 smithi077 podman[72274]: 2024-03-16 10:14:53.024947683 +0000 UTC m=+0.402681980 container init 2c643f6991bd5e3d168773ab9669ba48f8c506a4f8d822c71bcbffefb34e754a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-16T10:14:53.418 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:14:53 smithi077 podman[72274]: 2024-03-16 10:14:53.027518978 +0000 UTC m=+0.405253283 container start 2c643f6991bd5e3d168773ab9669ba48f8c506a4f8d822c71bcbffefb34e754a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True) 2024-03-16T10:14:53.418 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:14:53 smithi077 ceph-mds[72293]: -- 172.21.15.77:0/3102383470 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55908f03fa40 con 0x55908f012800 2024-03-16T10:14:53.418 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:14:53 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a[72289]: starting mds.a at 2024-03-16T10:14:53.418 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:14:53 smithi077 bash[72274]: 2c643f6991bd5e3d168773ab9669ba48f8c506a4f8d822c71bcbffefb34e754a 2024-03-16T10:14:53.418 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:14:53 smithi077 systemd[1]: Started Ceph mds.a for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:14:53.685 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:53.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:53 smithi077 ceph-mon[32400]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:53.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:53.714 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:53.714 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:53.714 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:53.714 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:53.714 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:53 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:53.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:53 smithi193 ceph-mon[36311]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:53.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:53.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:53.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:53.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:53.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:53.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:53 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:53.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.794+0000 7fcf96169640 1 -- 172.21.15.77:0/2286174864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf90102320 msgr2=0x7fcf90104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:53.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.794+0000 7fcf96169640 1 --2- 172.21.15.77:0/2286174864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf90102320 0x7fcf90104710 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009900 tx=0x7fcf8002bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:53.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.794+0000 7fcf96169640 1 -- 172.21.15.77:0/2286174864 shutdown_connections 2024-03-16T10:14:53.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.794+0000 7fcf96169640 1 --2- 172.21.15.77:0/2286174864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf90107580 0x7fcf90109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:53.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.794+0000 7fcf96169640 1 --2- 172.21.15.77:0/2286174864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf90104c50 0x7fcf90107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:53.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.794+0000 7fcf96169640 1 --2- 172.21.15.77:0/2286174864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf90102320 0x7fcf90104710 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:53.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.794+0000 7fcf96169640 1 -- 172.21.15.77:0/2286174864 >> 172.21.15.77:0/2286174864 conn(0x7fcf900fbc70 msgr2=0x7fcf900fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:53.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.794+0000 7fcf96169640 1 -- 172.21.15.77:0/2286174864 shutdown_connections 2024-03-16T10:14:53.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.794+0000 7fcf96169640 1 -- 172.21.15.77:0/2286174864 wait complete. 2024-03-16T10:14:53.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.794+0000 7fcf96169640 1 Processor -- start 2024-03-16T10:14:53.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.794+0000 7fcf96169640 1 -- start start 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf96169640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf90102320 0x7fcf90197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf96169640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf90104c50 0x7fcf901980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf96169640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf90107580 0x7fcf9019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf96169640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9010d540 con 0x7fcf90102320 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf96169640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9010d3c0 con 0x7fcf90107580 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf96169640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9010d6c0 con 0x7fcf90104c50 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf90102320 0x7fcf90197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf90102320 0x7fcf90197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38846/0 (socket says 172.21.15.77:38846) 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8f7fe640 1 -- 172.21.15.77:0/3448083192 learned_addr learned my addr 172.21.15.77:0/3448083192 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8ffff640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf90107580 0x7fcf9019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8f7fe640 1 -- 172.21.15.77:0/3448083192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf90104c50 msgr2=0x7fcf901980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8f7fe640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf90104c50 0x7fcf901980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8f7fe640 1 -- 172.21.15.77:0/3448083192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf90107580 msgr2=0x7fcf9019f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:53.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8f7fe640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf90107580 0x7fcf9019f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8f7fe640 1 -- 172.21.15.77:0/3448083192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf9019f7b0 con 0x7fcf90102320 2024-03-16T10:14:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8f7fe640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf90102320 0x7fcf90197b90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcf80000c00 tx=0x7fcf80005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8cff9640 1 -- 172.21.15.77:0/3448083192 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf80043020 con 0x7fcf90102320 2024-03-16T10:14:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf96169640 1 -- 172.21.15.77:0/3448083192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf9019fa40 con 0x7fcf90102320 2024-03-16T10:14:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8cff9640 1 -- 172.21.15.77:0/3448083192 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf800054e0 con 0x7fcf90102320 2024-03-16T10:14:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf8cff9640 1 -- 172.21.15.77:0/3448083192 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf80003e60 con 0x7fcf90102320 2024-03-16T10:14:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.795+0000 7fcf96169640 1 -- 172.21.15.77:0/3448083192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf9019ff90 con 0x7fcf90102320 2024-03-16T10:14:53.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.796+0000 7fcf96169640 1 -- 172.21.15.77:0/3448083192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf900fd760 con 0x7fcf90102320 2024-03-16T10:14:53.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.797+0000 7fcf8cff9640 1 -- 172.21.15.77:0/3448083192 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcf80046020 con 0x7fcf90102320 2024-03-16T10:14:53.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.798+0000 7fcf8cff9640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf780a2800 0x7fcf780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:53.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.798+0000 7fcf8effd640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf780a2800 0x7fcf780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:53.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.798+0000 7fcf8cff9640 1 -- 172.21.15.77:0/3448083192 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcf800f0c10 con 0x7fcf90102320 2024-03-16T10:14:53.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.798+0000 7fcf8effd640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf780a2800 0x7fcf780a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcf901990a0 tx=0x7fcf84006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:53.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:53.802+0000 7fcf8cff9640 1 -- 172.21.15.77:0/3448083192 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcf8004d020 con 0x7fcf90102320 2024-03-16T10:14:53.804 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:53 smithi192 ceph-mon[36354]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:53.804 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:53.804 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:53.804 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:53.804 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T10:14:53.804 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T10:14:53.804 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:53 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:54.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.015+0000 7fcf96169640 1 -- 172.21.15.77:0/3448083192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf9002d400 con 0x7fcf90102320 2024-03-16T10:14:54.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.016+0000 7fcf8cff9640 1 -- 172.21.15.77:0/3448083192 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7fcf800be150 con 0x7fcf90102320 2024-03-16T10:14:54.016 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:54.017 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14646,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T10:14:54.017 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 12 2024-03-16T10:14:54.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.017+0000 7fcf96169640 1 -- 172.21.15.77:0/3448083192 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf780a2800 msgr2=0x7fcf780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:54.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.017+0000 7fcf96169640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf780a2800 0x7fcf780a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcf901990a0 tx=0x7fcf84006c90 comp rx=0 tx=0).stop 2024-03-16T10:14:54.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.018+0000 7fcf96169640 1 -- 172.21.15.77:0/3448083192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf90102320 msgr2=0x7fcf90197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:54.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.018+0000 7fcf96169640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf90102320 0x7fcf90197b90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcf80000c00 tx=0x7fcf80005dd0 comp rx=0 tx=0).stop 2024-03-16T10:14:54.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.018+0000 7fcf96169640 1 -- 172.21.15.77:0/3448083192 shutdown_connections 2024-03-16T10:14:54.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.018+0000 7fcf96169640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf90107580 0x7fcf9019f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:54.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.018+0000 7fcf96169640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf780a2800 0x7fcf780a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:54.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.018+0000 7fcf96169640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf90104c50 0x7fcf901980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:54.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.018+0000 7fcf96169640 1 --2- 172.21.15.77:0/3448083192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf90102320 0x7fcf90197b90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:54.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.018+0000 7fcf96169640 1 -- 172.21.15.77:0/3448083192 >> 172.21.15.77:0/3448083192 conn(0x7fcf900fbc70 msgr2=0x7fcf90102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:54.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.018+0000 7fcf96169640 1 -- 172.21.15.77:0/3448083192 shutdown_connections 2024-03-16T10:14:54.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:54.018+0000 7fcf96169640 1 -- 172.21.15.77:0/3448083192 wait complete. 2024-03-16T10:14:54.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:54 smithi077 ceph-mon[32400]: Deploying daemon mds.b on smithi192 2024-03-16T10:14:54.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:54 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] up:boot 2024-03-16T10:14:54.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:54 smithi077 ceph-mon[32400]: fsmap 11 up:standby 2024-03-16T10:14:54.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:54 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T10:14:54.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:54 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3448083192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:54.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:54 smithi193 ceph-mon[36311]: Deploying daemon mds.b on smithi192 2024-03-16T10:14:54.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:54 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] up:boot 2024-03-16T10:14:54.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:54 smithi193 ceph-mon[36311]: fsmap 11 up:standby 2024-03-16T10:14:54.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:54 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T10:14:54.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:54 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3448083192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:54.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:54 smithi192 ceph-mon[36354]: Deploying daemon mds.b on smithi192 2024-03-16T10:14:54.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:54 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] up:boot 2024-03-16T10:14:54.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:54 smithi192 ceph-mon[36354]: fsmap 11 up:standby 2024-03-16T10:14:54.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:54 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T10:14:54.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:54 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3448083192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:55.744 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:55 smithi077 ceph-mon[32400]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:55.750 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:55 smithi192 ceph-mon[36354]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:55.750 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:14:55 smithi192 systemd[1]: Starting Ceph mds.b for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:14:55.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:55 smithi193 ceph-mon[36311]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:56.007 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:14:55 smithi192 podman[67087]: 2024-03-16 10:14:55.652818426 +0000 UTC m=+0.017798994 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:14:56.008 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:14:55 smithi192 podman[67087]: 2024-03-16 10:14:55.763155734 +0000 UTC m=+0.128136288 container create ca4655a331f9ccf6f9ede3fb3a655e2f9f98ee23e0ef71aa95308657ddc1ca6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-16T10:14:56.008 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:14:56 smithi192 podman[67087]: 2024-03-16 10:14:56.007668699 +0000 UTC m=+0.372649253 container init ca4655a331f9ccf6f9ede3fb3a655e2f9f98ee23e0ef71aa95308657ddc1ca6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:14:56.030 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:56.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.144+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1513009151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cbc1049e0 msgr2=0x7f5cbc106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.144+0000 7f5cc1f9f640 1 --2- 172.21.15.77:0/1513009151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cbc1049e0 0x7f5cbc106dd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009900 tx=0x7f5cb002bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.144+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1513009151 shutdown_connections 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.144+0000 7f5cc1f9f640 1 --2- 172.21.15.77:0/1513009151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cbc107310 0x7f5cbc109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.144+0000 7f5cc1f9f640 1 --2- 172.21.15.77:0/1513009151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cbc1049e0 0x7f5cbc106dd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.144+0000 7f5cc1f9f640 1 --2- 172.21.15.77:0/1513009151 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cbc1020b0 0x7f5cbc1044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.144+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1513009151 >> 172.21.15.77:0/1513009151 conn(0x7f5cbc0fbac0 msgr2=0x7f5cbc0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.145+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1513009151 shutdown_connections 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.145+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1513009151 wait complete. 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.145+0000 7f5cc1f9f640 1 Processor -- start 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.145+0000 7f5cc1f9f640 1 -- start start 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.145+0000 7f5cc1f9f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cbc1020b0 0x7f5cbc1978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.145+0000 7f5cc1f9f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cbc1049e0 0x7f5cbc197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.145+0000 7f5cc1f9f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cbc107310 0x7f5cbc19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.145+0000 7f5cc1f9f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5cbc10d290 con 0x7f5cbc107310 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.145+0000 7f5cc1f9f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f5cbc10d110 con 0x7f5cbc1049e0 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.145+0000 7f5cc1f9f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5cbc10d410 con 0x7f5cbc1020b0 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cc179e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cbc107310 0x7f5cbc19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cc179e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cbc107310 0x7f5cbc19ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38864/0 (socket says 172.21.15.77:38864) 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cc179e640 1 -- 172.21.15.77:0/1104188935 learned_addr learned my addr 172.21.15.77:0/1104188935 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cbbfff640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cbc1049e0 0x7f5cbc197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cc0f9d640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cbc1020b0 0x7f5cbc1978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cc179e640 1 -- 172.21.15.77:0/1104188935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cbc1020b0 msgr2=0x7f5cbc1978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cc179e640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cbc1020b0 0x7f5cbc1978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cc179e640 1 -- 172.21.15.77:0/1104188935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cbc1049e0 msgr2=0x7f5cbc197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cc179e640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cbc1049e0 0x7f5cbc197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cc179e640 1 -- 172.21.15.77:0/1104188935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cbc19f4c0 con 0x7f5cbc107310 2024-03-16T10:14:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cc179e640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cbc107310 0x7f5cbc19ee50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400cce0 tx=0x7f5cb400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:56.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cb9ffb640 1 -- 172.21.15.77:0/1104188935 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb400f450 con 0x7f5cbc107310 2024-03-16T10:14:56.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cb9ffb640 1 -- 172.21.15.77:0/1104188935 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cb400fad0 con 0x7f5cbc107310 2024-03-16T10:14:56.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1104188935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cbc19f7b0 con 0x7f5cbc107310 2024-03-16T10:14:56.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cb9ffb640 1 -- 172.21.15.77:0/1104188935 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4018c10 con 0x7f5cbc107310 2024-03-16T10:14:56.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.146+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1104188935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cbc19fc80 con 0x7f5cbc107310 2024-03-16T10:14:56.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.147+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1104188935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cbc0fd5b0 con 0x7f5cbc107310 2024-03-16T10:14:56.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.148+0000 7f5cb9ffb640 1 -- 172.21.15.77:0/1104188935 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5cb400f5f0 con 0x7f5cbc107310 2024-03-16T10:14:56.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.149+0000 7f5cb9ffb640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ca40a2800 0x7f5ca40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:56.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.149+0000 7f5cc0f9d640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ca40a2800 0x7f5ca40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:56.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.149+0000 7f5cb9ffb640 1 -- 172.21.15.77:0/1104188935 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5cb40cecc0 con 0x7f5cbc107310 2024-03-16T10:14:56.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.150+0000 7f5cc0f9d640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ca40a2800 0x7f5ca40a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5cac009b60 tx=0x7f5cac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:56.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.154+0000 7f5cb9ffb640 1 -- 172.21.15.77:0/1104188935 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5cb4014030 con 0x7f5cbc107310 2024-03-16T10:14:56.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.376+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1104188935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5cbc0623e0 con 0x7f5cbc107310 2024-03-16T10:14:56.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.377+0000 7f5cb9ffb640 1 -- 172.21.15.77:0/1104188935 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7f5cb409c200 con 0x7f5cbc107310 2024-03-16T10:14:56.378 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:56.378 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14646,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T10:14:56.378 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 12 2024-03-16T10:14:56.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.379+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1104188935 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ca40a2800 msgr2=0x7f5ca40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:56.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.379+0000 7f5cc1f9f640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ca40a2800 0x7f5ca40a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5cac009b60 tx=0x7f5cac005f90 comp rx=0 tx=0).stop 2024-03-16T10:14:56.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.380+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1104188935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cbc107310 msgr2=0x7f5cbc19ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:56.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.380+0000 7f5cc1f9f640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cbc107310 0x7f5cbc19ee50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400cce0 tx=0x7f5cb400a620 comp rx=0 tx=0).stop 2024-03-16T10:14:56.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.380+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1104188935 shutdown_connections 2024-03-16T10:14:56.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.380+0000 7f5cc1f9f640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cbc107310 0x7f5cbc19ee50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:56.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.380+0000 7f5cc1f9f640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cbc1049e0 0x7f5cbc197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:56.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.380+0000 7f5cc1f9f640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ca40a2800 0x7f5ca40a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:56.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.380+0000 7f5cc1f9f640 1 --2- 172.21.15.77:0/1104188935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cbc1020b0 0x7f5cbc1978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:56.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.380+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1104188935 >> 172.21.15.77:0/1104188935 conn(0x7f5cbc0fbac0 msgr2=0x7f5cbc1029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:56.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.380+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1104188935 shutdown_connections 2024-03-16T10:14:56.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:56.380+0000 7f5cc1f9f640 1 -- 172.21.15.77:0/1104188935 wait complete. 2024-03-16T10:14:56.416 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:14:56 smithi192 podman[67087]: 2024-03-16 10:14:56.010393411 +0000 UTC m=+0.375373975 container start ca4655a331f9ccf6f9ede3fb3a655e2f9f98ee23e0ef71aa95308657ddc1ca6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-16T10:14:56.416 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:14:56 smithi192 ceph-mds[67105]: -- 172.21.15.192:0/3303087163 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5639da4e3a40 con 0x5639da4b7000 2024-03-16T10:14:56.416 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:14:56 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b[67101]: starting mds.b at 2024-03-16T10:14:56.416 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:14:56 smithi192 bash[67087]: ca4655a331f9ccf6f9ede3fb3a655e2f9f98ee23e0ef71aa95308657ddc1ca6a 2024-03-16T10:14:56.417 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:14:56 smithi192 systemd[1]: Started Ceph mds.b for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:14:57.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:57.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:57 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1104188935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:57.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:57 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] up:boot 2024-03-16T10:14:57.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:57 smithi192 ceph-mon[36354]: fsmap 12 up:standby 2024-03-16T10:14:57.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:57 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T10:14:57.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:57 smithi192 ceph-mon[36354]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:57 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1104188935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:57 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] up:boot 2024-03-16T10:14:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:57 smithi077 ceph-mon[32400]: fsmap 12 up:standby 2024-03-16T10:14:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:57 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T10:14:57.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:57 smithi077 ceph-mon[32400]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:57.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:57.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:57.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:57 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1104188935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:57.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:57 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] up:boot 2024-03-16T10:14:57.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:57 smithi193 ceph-mon[36311]: fsmap 12 up:standby 2024-03-16T10:14:57.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:57 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T10:14:57.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:57 smithi193 ceph-mon[36311]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:58.393 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:14:58.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.497+0000 7f5acb0de640 1 -- 172.21.15.77:0/272719059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ac4100260 msgr2=0x7f5ac4109e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:58.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.497+0000 7f5acb0de640 1 --2- 172.21.15.77:0/272719059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ac4100260 0x7f5ac4109e10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009ca0 tx=0x7f5abc02c0f0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.497+0000 7f5acb0de640 1 -- 172.21.15.77:0/272719059 shutdown_connections 2024-03-16T10:14:58.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.497+0000 7f5acb0de640 1 --2- 172.21.15.77:0/272719059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ac4100260 0x7f5ac4109e10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.497+0000 7f5acb0de640 1 --2- 172.21.15.77:0/272719059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ac40ff8a0 0x7f5ac40ffd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.497+0000 7f5acb0de640 1 --2- 172.21.15.77:0/272719059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ac4102e30 0x7f5ac4103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.497+0000 7f5acb0de640 1 -- 172.21.15.77:0/272719059 >> 172.21.15.77:0/272719059 conn(0x7f5ac40fb400 msgr2=0x7f5ac40fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.498+0000 7f5acb0de640 1 -- 172.21.15.77:0/272719059 shutdown_connections 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.498+0000 7f5acb0de640 1 -- 172.21.15.77:0/272719059 wait complete. 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.498+0000 7f5acb0de640 1 Processor -- start 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.498+0000 7f5acb0de640 1 -- start start 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5acb0de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ac40ff8a0 0x7f5ac4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5acb0de640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ac4100260 0x7f5ac4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5acb0de640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ac4102e30 0x7f5ac419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5acb0de640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac410d480 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5acb0de640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac410d300 con 0x7f5ac4100260 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5acb0de640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac410d600 con 0x7f5ac4102e30 2024-03-16T10:14:58.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac8e7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ac40ff8a0 0x7f5ac4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:58.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac8e7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ac40ff8a0 0x7f5ac4197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38874/0 (socket says 172.21.15.77:38874) 2024-03-16T10:14:58.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac8e7a640 1 -- 172.21.15.77:0/4103388882 learned_addr learned my addr 172.21.15.77:0/4103388882 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:58.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac3fff640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ac4100260 0x7f5ac4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:58.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac967b640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ac4102e30 0x7f5ac419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:58.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac8e7a640 1 -- 172.21.15.77:0/4103388882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ac4102e30 msgr2=0x7f5ac419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:58.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac8e7a640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ac4102e30 0x7f5ac419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac8e7a640 1 -- 172.21.15.77:0/4103388882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ac4100260 msgr2=0x7f5ac4197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:58.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac8e7a640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ac4100260 0x7f5ac4197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac8e7a640 1 -- 172.21.15.77:0/4103388882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac419f6b0 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac967b640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ac4102e30 0x7f5ac419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:14:58.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac8e7a640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ac40ff8a0 0x7f5ac4197a90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800d950 tx=0x7f5ab800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:58.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac1ffb640 1 -- 172.21.15.77:0/4103388882 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8017070 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac1ffb640 1 -- 172.21.15.77:0/4103388882 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ab8009c30 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5acb0de640 1 -- 172.21.15.77:0/4103388882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac419f9a0 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5ac1ffb640 1 -- 172.21.15.77:0/4103388882 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab800f600 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.499+0000 7f5acb0de640 1 -- 172.21.15.77:0/4103388882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac419fef0 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.500+0000 7f5acb0de640 1 -- 172.21.15.77:0/4103388882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac4108c20 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.501+0000 7f5ac1ffb640 1 -- 172.21.15.77:0/4103388882 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5ab801a070 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.502+0000 7f5ac1ffb640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ab40a2880 0x7f5ab40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:58.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.502+0000 7f5ac3fff640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ab40a2880 0x7f5ab40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:58.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.502+0000 7f5ac1ffb640 1 -- 172.21.15.77:0/4103388882 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5ab80cdf80 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.503+0000 7f5ac3fff640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ab40a2880 0x7f5ab40a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4198fa0 tx=0x7f5ab000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:58.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.507+0000 7f5ac1ffb640 1 -- 172.21.15.77:0/4103388882 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ab8009de0 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.730+0000 7f5acb0de640 1 -- 172.21.15.77:0/4103388882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ac402d440 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.731+0000 7f5ac1ffb640 1 -- 172.21.15.77:0/4103388882 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10491 (secure 0 0 0) 0x7f5ab809b4c0 con 0x7f5ac40ff8a0 2024-03-16T10:14:58.732 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:58.732 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14646,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-16T10:14:58.732 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-03-16T10:14:58.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.733+0000 7f5acb0de640 1 -- 172.21.15.77:0/4103388882 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ab40a2880 msgr2=0x7f5ab40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:58.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.733+0000 7f5acb0de640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ab40a2880 0x7f5ab40a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4198fa0 tx=0x7f5ab000b390 comp rx=0 tx=0).stop 2024-03-16T10:14:58.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.733+0000 7f5acb0de640 1 -- 172.21.15.77:0/4103388882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ac40ff8a0 msgr2=0x7f5ac4197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:58.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.733+0000 7f5acb0de640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ac40ff8a0 0x7f5ac4197a90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800d950 tx=0x7f5ab800de10 comp rx=0 tx=0).stop 2024-03-16T10:14:58.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.733+0000 7f5acb0de640 1 -- 172.21.15.77:0/4103388882 shutdown_connections 2024-03-16T10:14:58.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.733+0000 7f5acb0de640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ab40a2880 0x7f5ab40a4d40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.733+0000 7f5acb0de640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ac4102e30 0x7f5ac419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.733+0000 7f5acb0de640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ac4100260 0x7f5ac4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.733+0000 7f5acb0de640 1 --2- 172.21.15.77:0/4103388882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ac40ff8a0 0x7f5ac4197a90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.733+0000 7f5acb0de640 1 -- 172.21.15.77:0/4103388882 >> 172.21.15.77:0/4103388882 conn(0x7f5ac40fb400 msgr2=0x7f5ac4105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:58.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.733+0000 7f5acb0de640 1 -- 172.21.15.77:0/4103388882 shutdown_connections 2024-03-16T10:14:58.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.733+0000 7f5acb0de640 1 -- 172.21.15.77:0/4103388882 wait complete. 2024-03-16T10:14:58.745 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:14:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 -- 172.21.15.77:0/677654497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b41018f0 msgr2=0x7f79b4101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 --2- 172.21.15.77:0/677654497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b41018f0 0x7f79b4101d70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009900 tx=0x7f79a402bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 -- 172.21.15.77:0/677654497 shutdown_connections 2024-03-16T10:14:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 --2- 172.21.15.77:0/677654497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79b41022b0 0x7f79b4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 --2- 172.21.15.77:0/677654497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b41018f0 0x7f79b4101d70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 --2- 172.21.15.77:0/677654497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79b4100700 0x7f79b4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 -- 172.21.15.77:0/677654497 >> 172.21.15.77:0/677654497 conn(0x7f79b40fbc70 msgr2=0x7f79b40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 -- 172.21.15.77:0/677654497 shutdown_connections 2024-03-16T10:14:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 -- 172.21.15.77:0/677654497 wait complete. 2024-03-16T10:14:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 Processor -- start 2024-03-16T10:14:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 -- start start 2024-03-16T10:14:58.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79b4100700 0x7f79b4197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b41018f0 0x7f79b4198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79b41022b0 0x7f79b419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f79b410c360 con 0x7f79b41018f0 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f79b410c1e0 con 0x7f79b41022b0 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.849+0000 7f79bba0b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f79b410c4e0 con 0x7f79b4100700 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79b8fa6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b41018f0 0x7f79b4198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79b8fa6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b41018f0 0x7f79b4198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38890/0 (socket says 172.21.15.77:38890) 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79b8fa6640 1 -- 172.21.15.77:0/2118589169 learned_addr learned my addr 172.21.15.77:0/2118589169 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79b9fa8640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79b41022b0 0x7f79b419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79b97a7640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79b4100700 0x7f79b4197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79b8fa6640 1 -- 172.21.15.77:0/2118589169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79b4100700 msgr2=0x7f79b4197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79b8fa6640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79b4100700 0x7f79b4197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79b8fa6640 1 -- 172.21.15.77:0/2118589169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79b41022b0 msgr2=0x7f79b419f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79b8fa6640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79b41022b0 0x7f79b419f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79b8fa6640 1 -- 172.21.15.77:0/2118589169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b419f750 con 0x7f79b41018f0 2024-03-16T10:14:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79b97a7640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79b4100700 0x7f79b4197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:14:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79b8fa6640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b41018f0 0x7f79b4198070 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009230 tx=0x7f79a4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79aa7fc640 1 -- 172.21.15.77:0/2118589169 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a4041070 con 0x7f79b41018f0 2024-03-16T10:14:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79bba0b640 1 -- 172.21.15.77:0/2118589169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b419f9e0 con 0x7f79b41018f0 2024-03-16T10:14:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79aa7fc640 1 -- 172.21.15.77:0/2118589169 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79a40054e0 con 0x7f79b41018f0 2024-03-16T10:14:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79aa7fc640 1 -- 172.21.15.77:0/2118589169 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a4034440 con 0x7f79b41018f0 2024-03-16T10:14:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.850+0000 7f79bba0b640 1 -- 172.21.15.77:0/2118589169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b419ff20 con 0x7f79b41018f0 2024-03-16T10:14:58.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.851+0000 7f79bba0b640 1 -- 172.21.15.77:0/2118589169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b41a00f0 con 0x7f79b41018f0 2024-03-16T10:14:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.853+0000 7f79aa7fc640 1 -- 172.21.15.77:0/2118589169 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f79a4044070 con 0x7f79b41018f0 2024-03-16T10:14:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.853+0000 7f79aa7fc640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f79a00a2800 0x7f79a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.853+0000 7f79b97a7640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f79a00a2800 0x7f79a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.853+0000 7f79aa7fc640 1 -- 172.21.15.77:0/2118589169 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f79a40f06a0 con 0x7f79b41018f0 2024-03-16T10:14:58.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.854+0000 7f79b97a7640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f79a00a2800 0x7f79a00a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f79b0005d00 tx=0x7f79b0005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:58.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:58.857+0000 7f79aa7fc640 1 -- 172.21.15.77:0/2118589169 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f79a40bdbe0 con 0x7f79b41018f0 2024-03-16T10:14:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:14:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:58.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:58.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:58.996 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:58 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/4103388882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:59.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.041+0000 7f79bba0b640 1 -- 172.21.15.77:0/2118589169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79b41a0480 con 0x7f79b41018f0 2024-03-16T10:14:59.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.042+0000 7f79aa7fc640 1 -- 172.21.15.77:0/2118589169 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18432 (secure 0 0 0) 0x7f79a40bd4e0 con 0x7f79b41018f0 2024-03-16T10:14:59.043 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:59.043 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":81,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:14:10.802815+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:14:59.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.044+0000 7f79bba0b640 1 -- 172.21.15.77:0/2118589169 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f79a00a2800 msgr2=0x7f79a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.044+0000 7f79bba0b640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f79a00a2800 0x7f79a00a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f79b0005d00 tx=0x7f79b0005c90 comp rx=0 tx=0).stop 2024-03-16T10:14:59.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.044+0000 7f79bba0b640 1 -- 172.21.15.77:0/2118589169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b41018f0 msgr2=0x7f79b4198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.044+0000 7f79bba0b640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b41018f0 0x7f79b4198070 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009230 tx=0x7f79a4005dd0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.044+0000 7f79bba0b640 1 -- 172.21.15.77:0/2118589169 shutdown_connections 2024-03-16T10:14:59.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.044+0000 7f79bba0b640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79b41022b0 0x7f79b419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.044+0000 7f79bba0b640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b41018f0 0x7f79b4198070 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.044+0000 7f79bba0b640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f79a00a2800 0x7f79a00a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.044+0000 7f79bba0b640 1 --2- 172.21.15.77:0/2118589169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79b4100700 0x7f79b4197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.044+0000 7f79bba0b640 1 -- 172.21.15.77:0/2118589169 >> 172.21.15.77:0/2118589169 conn(0x7f79b40fbc70 msgr2=0x7f79b40fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:59.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.044+0000 7f79bba0b640 1 -- 172.21.15.77:0/2118589169 shutdown_connections 2024-03-16T10:14:59.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.044+0000 7f79bba0b640 1 -- 172.21.15.77:0/2118589169 wait complete. 2024-03-16T10:14:59.056 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:10:22.965004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T10:14:59.056 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T10:14:59.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 -- 172.21.15.77:0/561026883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782c1000d0 msgr2=0x7f782c1004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 --2- 172.21.15.77:0/561026883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782c1000d0 0x7f782c1004f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7828009900 tx=0x7f782802bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:59.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 -- 172.21.15.77:0/561026883 shutdown_connections 2024-03-16T10:14:59.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 --2- 172.21.15.77:0/561026883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f782c1013d0 0x7f782c101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 --2- 172.21.15.77:0/561026883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f782c100a30 0x7f782c100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 --2- 172.21.15.77:0/561026883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782c1000d0 0x7f782c1004f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 -- 172.21.15.77:0/561026883 >> 172.21.15.77:0/561026883 conn(0x7f782c0fbc90 msgr2=0x7f782c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 -- 172.21.15.77:0/561026883 shutdown_connections 2024-03-16T10:14:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 -- 172.21.15.77:0/561026883 wait complete. 2024-03-16T10:14:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 Processor -- start 2024-03-16T10:14:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 -- start start 2024-03-16T10:14:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782c1000d0 0x7f782c199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f782c100a30 0x7f782c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.168+0000 7f7834153640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f782c1013d0 0x7f782c1a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7834153640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f782c10f660 con 0x7f782c1000d0 2024-03-16T10:14:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7834153640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f782c10f4e0 con 0x7f782c1013d0 2024-03-16T10:14:59.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7834153640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f782c10f7e0 con 0x7f782c100a30 2024-03-16T10:14:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7831eef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782c1000d0 0x7f782c199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7831eef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782c1000d0 0x7f782c199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38902/0 (socket says 172.21.15.77:38902) 2024-03-16T10:14:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7831eef640 1 -- 172.21.15.77:0/2529906616 learned_addr learned my addr 172.21.15.77:0/2529906616 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f78326f0640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f782c1013d0 0x7f782c1a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f78316ee640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f782c100a30 0x7f782c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7831eef640 1 -- 172.21.15.77:0/2529906616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f782c100a30 msgr2=0x7f782c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7831eef640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f782c100a30 0x7f782c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7831eef640 1 -- 172.21.15.77:0/2529906616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f782c1013d0 msgr2=0x7f782c1a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7831eef640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f782c1013d0 0x7f782c1a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7831eef640 1 -- 172.21.15.77:0/2529906616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782c1a1960 con 0x7f782c1000d0 2024-03-16T10:14:59.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7831eef640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782c1000d0 0x7f782c199d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7828009230 tx=0x7f7828005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7822ffd640 1 -- 172.21.15.77:0/2529906616 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828041070 con 0x7f782c1000d0 2024-03-16T10:14:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7834153640 1 -- 172.21.15.77:0/2529906616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f782c1a1bf0 con 0x7f782c1000d0 2024-03-16T10:14:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7822ffd640 1 -- 172.21.15.77:0/2529906616 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78280044d0 con 0x7f782c1000d0 2024-03-16T10:14:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7822ffd640 1 -- 172.21.15.77:0/2529906616 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828034440 con 0x7f782c1000d0 2024-03-16T10:14:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.169+0000 7f7834153640 1 -- 172.21.15.77:0/2529906616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f782c1a20e0 con 0x7f782c1000d0 2024-03-16T10:14:59.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.170+0000 7f7834153640 1 -- 172.21.15.77:0/2529906616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f782c10b010 con 0x7f782c1000d0 2024-03-16T10:14:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.171+0000 7f7822ffd640 1 -- 172.21.15.77:0/2529906616 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7828044070 con 0x7f782c1000d0 2024-03-16T10:14:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.172+0000 7f7822ffd640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f78180a2800 0x7f78180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.172+0000 7f78316ee640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f78180a2800 0x7f78180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.172+0000 7f7822ffd640 1 -- 172.21.15.77:0/2529906616 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f78280f0520 con 0x7f782c1000d0 2024-03-16T10:14:59.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.173+0000 7f78316ee640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f78180a2800 0x7f78180a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f781c009bd0 tx=0x7f781c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:59.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.176+0000 7f7822ffd640 1 -- 172.21.15.77:0/2529906616 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f78280bd410 con 0x7f782c1000d0 2024-03-16T10:14:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:14:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:59.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:59.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:58 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/4103388882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:59.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.360+0000 7f7834153640 1 -- 172.21.15.77:0/2529906616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f782c02d340 con 0x7f782c1000d0 2024-03-16T10:14:59.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.360+0000 7f7822ffd640 1 -- 172.21.15.77:0/2529906616 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f78280c14b0 con 0x7f782c1000d0 2024-03-16T10:14:59.361 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T10:14:59.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.362+0000 7f7834153640 1 -- 172.21.15.77:0/2529906616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f78180a2800 msgr2=0x7f78180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.362+0000 7f7834153640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f78180a2800 0x7f78180a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f781c009bd0 tx=0x7f781c005f90 comp rx=0 tx=0).stop 2024-03-16T10:14:59.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.362+0000 7f7834153640 1 -- 172.21.15.77:0/2529906616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782c1000d0 msgr2=0x7f782c199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.362+0000 7f7834153640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782c1000d0 0x7f782c199d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7828009230 tx=0x7f7828005d50 comp rx=0 tx=0).stop 2024-03-16T10:14:59.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.362+0000 7f7834153640 1 -- 172.21.15.77:0/2529906616 shutdown_connections 2024-03-16T10:14:59.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.362+0000 7f7834153640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f782c1013d0 0x7f782c1a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.362+0000 7f7834153640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f78180a2800 0x7f78180a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.362+0000 7f7834153640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f782c100a30 0x7f782c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.362+0000 7f7834153640 1 --2- 172.21.15.77:0/2529906616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782c1000d0 0x7f782c199d50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.362+0000 7f7834153640 1 -- 172.21.15.77:0/2529906616 >> 172.21.15.77:0/2529906616 conn(0x7f782c0fbc90 msgr2=0x7f782c0fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:59.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.362+0000 7f7834153640 1 -- 172.21.15.77:0/2529906616 shutdown_connections 2024-03-16T10:14:59.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.362+0000 7f7834153640 1 -- 172.21.15.77:0/2529906616 wait complete. 2024-03-16T10:14:59.374 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-16T10:14:59.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:14:59.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:14:59.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:14:59.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:14:59.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:14:59.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:58 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/4103388882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:14:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 -- 172.21.15.77:0/614395224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb8101910 msgr2=0x7f6eb8101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 --2- 172.21.15.77:0/614395224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb8101910 0x7f6eb8101d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8009900 tx=0x7f6ea802bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 -- 172.21.15.77:0/614395224 shutdown_connections 2024-03-16T10:14:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 --2- 172.21.15.77:0/614395224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6eb81022d0 0x7f6eb8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 --2- 172.21.15.77:0/614395224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb8101910 0x7f6eb8101d90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 --2- 172.21.15.77:0/614395224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb8100720 0x7f6eb8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 -- 172.21.15.77:0/614395224 >> 172.21.15.77:0/614395224 conn(0x7f6eb80fbc90 msgr2=0x7f6eb80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 -- 172.21.15.77:0/614395224 shutdown_connections 2024-03-16T10:14:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 -- 172.21.15.77:0/614395224 wait complete. 2024-03-16T10:14:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 Processor -- start 2024-03-16T10:14:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 -- start start 2024-03-16T10:14:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6eb8100720 0x7f6eb806a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:59.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb8101910 0x7f6eb819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:59.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb81022d0 0x7f6eb819eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:59.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb810c3f0 con 0x7f6eb81022d0 2024-03-16T10:14:59.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb810c270 con 0x7f6eb8101910 2024-03-16T10:14:59.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.478+0000 7f6ec0661640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb810c570 con 0x7f6eb8100720 2024-03-16T10:14:59.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ebebfe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb81022d0 0x7f6eb819eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:59.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ebebfe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb81022d0 0x7f6eb819eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38918/0 (socket says 172.21.15.77:38918) 2024-03-16T10:14:59.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ebdbfc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb8101910 0x7f6eb819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:59.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ebebfe640 1 -- 172.21.15.77:0/3070910842 learned_addr learned my addr 172.21.15.77:0/3070910842 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:59.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ebe3fd640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6eb8100720 0x7f6eb806a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:59.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ebdbfc640 1 -- 172.21.15.77:0/3070910842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6eb8100720 msgr2=0x7f6eb806a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ebdbfc640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6eb8100720 0x7f6eb806a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ebdbfc640 1 -- 172.21.15.77:0/3070910842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb81022d0 msgr2=0x7f6eb819eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ebdbfc640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb81022d0 0x7f6eb819eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ebdbfc640 1 -- 172.21.15.77:0/3070910842 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb819f760 con 0x7f6eb8101910 2024-03-16T10:14:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ebe3fd640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6eb8100720 0x7f6eb806a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:14:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ebdbfc640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb8101910 0x7f6eb819c860 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6ea802c170 tx=0x7f6ea80049c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6eaf7fe640 1 -- 172.21.15.77:0/3070910842 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8043020 con 0x7f6eb8101910 2024-03-16T10:14:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ec0661640 1 -- 172.21.15.77:0/3070910842 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb819f9f0 con 0x7f6eb8101910 2024-03-16T10:14:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6eaf7fe640 1 -- 172.21.15.77:0/3070910842 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea80054e0 con 0x7f6eb8101910 2024-03-16T10:14:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6ec0661640 1 -- 172.21.15.77:0/3070910842 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb819ff60 con 0x7f6eb8101910 2024-03-16T10:14:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.479+0000 7f6eaf7fe640 1 -- 172.21.15.77:0/3070910842 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8003d10 con 0x7f6eb8101910 2024-03-16T10:14:59.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.480+0000 7f6ec0661640 1 -- 172.21.15.77:0/3070910842 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb8107a40 con 0x7f6eb8101910 2024-03-16T10:14:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.481+0000 7f6eaf7fe640 1 -- 172.21.15.77:0/3070910842 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6ea8046020 con 0x7f6eb8101910 2024-03-16T10:14:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.482+0000 7f6eaf7fe640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6ea40a2880 0x7f6ea40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.482+0000 7f6ebe3fd640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6ea40a2880 0x7f6ea40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:59.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.482+0000 7f6eaf7fe640 1 -- 172.21.15.77:0/3070910842 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6ea80f0ba0 con 0x7f6eb8101910 2024-03-16T10:14:59.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.483+0000 7f6ebe3fd640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6ea40a2880 0x7f6ea40a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4005fd0 tx=0x7f6eb400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:59.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.486+0000 7f6eaf7fe640 1 -- 172.21.15.77:0/3070910842 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6ea80f0e20 con 0x7f6eb8101910 2024-03-16T10:14:59.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.672+0000 7f6ec0661640 1 -- 172.21.15.77:0/3070910842 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6eb802d3b0 con 0x7f6eb8101910 2024-03-16T10:14:59.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.672+0000 7f6eaf7fe640 1 -- 172.21.15.77:0/3070910842 <== mon.2 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f6ea804d020 con 0x7f6eb8101910 2024-03-16T10:14:59.673 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:14:59.674 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2024-03-16T10:14:59.674 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:14:59.674 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2024-03-16T10:14:59.674 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2024-03-16T10:14:59.674 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:14:59.674 INFO:teuthology.orchestra.run.smithi077.stdout:] 2024-03-16T10:14:59.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.674+0000 7f6ec0661640 1 -- 172.21.15.77:0/3070910842 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6ea40a2880 msgr2=0x7f6ea40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.674+0000 7f6ec0661640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6ea40a2880 0x7f6ea40a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4005fd0 tx=0x7f6eb400b410 comp rx=0 tx=0).stop 2024-03-16T10:14:59.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.674+0000 7f6ec0661640 1 -- 172.21.15.77:0/3070910842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb8101910 msgr2=0x7f6eb819c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.674+0000 7f6ec0661640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb8101910 0x7f6eb819c860 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6ea802c170 tx=0x7f6ea80049c0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.674+0000 7f6ec0661640 1 -- 172.21.15.77:0/3070910842 shutdown_connections 2024-03-16T10:14:59.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.674+0000 7f6ec0661640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb81022d0 0x7f6eb819eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.674+0000 7f6ec0661640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb8101910 0x7f6eb819c860 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.674+0000 7f6ec0661640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6ea40a2880 0x7f6ea40a4d40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.674+0000 7f6ec0661640 1 --2- 172.21.15.77:0/3070910842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6eb8100720 0x7f6eb806a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.675+0000 7f6ec0661640 1 -- 172.21.15.77:0/3070910842 >> 172.21.15.77:0/3070910842 conn(0x7f6eb80fbc90 msgr2=0x7f6eb8104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:59.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.675+0000 7f6ec0661640 1 -- 172.21.15.77:0/3070910842 shutdown_connections 2024-03-16T10:14:59.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.675+0000 7f6ec0661640 1 -- 172.21.15.77:0/3070910842 wait complete. 2024-03-16T10:14:59.686 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-16T10:14:59.686 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-16T10:14:59.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.792+0000 7f832cac6640 1 -- 172.21.15.77:0/1498837358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8328100210 msgr2=0x7f8328109dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.792+0000 7f832cac6640 1 --2- 172.21.15.77:0/1498837358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8328100210 0x7f8328109dc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8318009900 tx=0x7f831802bc30 comp rx=0 tx=0).stop 2024-03-16T10:14:59.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.792+0000 7f832cac6640 1 -- 172.21.15.77:0/1498837358 shutdown_connections 2024-03-16T10:14:59.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.792+0000 7f832cac6640 1 --2- 172.21.15.77:0/1498837358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8328100210 0x7f8328109dc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.792+0000 7f832cac6640 1 --2- 172.21.15.77:0/1498837358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83280ff850 0x7f83280ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.792+0000 7f832cac6640 1 --2- 172.21.15.77:0/1498837358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8328102de0 0x7f8328103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.792+0000 7f832cac6640 1 -- 172.21.15.77:0/1498837358 >> 172.21.15.77:0/1498837358 conn(0x7f83280fb410 msgr2=0x7f83280fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:14:59.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.792+0000 7f832cac6640 1 -- 172.21.15.77:0/1498837358 shutdown_connections 2024-03-16T10:14:59.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.792+0000 7f832cac6640 1 -- 172.21.15.77:0/1498837358 wait complete. 2024-03-16T10:14:59.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.793+0000 7f832cac6640 1 Processor -- start 2024-03-16T10:14:59.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.793+0000 7f832cac6640 1 -- start start 2024-03-16T10:14:59.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.793+0000 7f832cac6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83280ff850 0x7f83281024e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:59.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.793+0000 7f832cac6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8328100210 0x7f8328100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:59.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.793+0000 7f832cac6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8328102de0 0x7f8328101060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:59.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.793+0000 7f832cac6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f832810d490 con 0x7f8328100210 2024-03-16T10:14:59.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.793+0000 7f832cac6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f832810d310 con 0x7f83280ff850 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.793+0000 7f832cac6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f832810d610 con 0x7f8328102de0 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.793+0000 7f8325d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8328100210 0x7f8328100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.793+0000 7f8325d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8328100210 0x7f8328100b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38938/0 (socket says 172.21.15.77:38938) 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.793+0000 7f8325d9b640 1 -- 172.21.15.77:0/2739996785 learned_addr learned my addr 172.21.15.77:0/2739996785 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.794+0000 7f832659c640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83280ff850 0x7f83281024e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.794+0000 7f8325d9b640 1 -- 172.21.15.77:0/2739996785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8328102de0 msgr2=0x7f8328101060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.794+0000 7f8325d9b640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8328102de0 0x7f8328101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.794+0000 7f8325d9b640 1 -- 172.21.15.77:0/2739996785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83280ff850 msgr2=0x7f83281024e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.794+0000 7f8325d9b640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83280ff850 0x7f83281024e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.794+0000 7f8325d9b640 1 -- 172.21.15.77:0/2739996785 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83281018f0 con 0x7f8328100210 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.794+0000 7f8325d9b640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8328100210 0x7f8328100b20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f831000d770 tx=0x7f831000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.794+0000 7f831f7fe640 1 -- 172.21.15.77:0/2739996785 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310017070 con 0x7f8328100210 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.794+0000 7f832cac6640 1 -- 172.21.15.77:0/2739996785 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83281a1bf0 con 0x7f8328100210 2024-03-16T10:14:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.794+0000 7f831f7fe640 1 -- 172.21.15.77:0/2739996785 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8310005500 con 0x7f8328100210 2024-03-16T10:14:59.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.794+0000 7f831f7fe640 1 -- 172.21.15.77:0/2739996785 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310004e00 con 0x7f8328100210 2024-03-16T10:14:59.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.794+0000 7f832cac6640 1 -- 172.21.15.77:0/2739996785 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83281a2140 con 0x7f8328100210 2024-03-16T10:14:59.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.795+0000 7f832cac6640 1 -- 172.21.15.77:0/2739996785 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8328108ca0 con 0x7f8328100210 2024-03-16T10:14:59.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.796+0000 7f831f7fe640 1 -- 172.21.15.77:0/2739996785 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8310005000 con 0x7f8328100210 2024-03-16T10:14:59.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.796+0000 7f831f7fe640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f83140a2800 0x7f83140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:14:59.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.797+0000 7f832659c640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f83140a2800 0x7f83140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:14:59.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.797+0000 7f831f7fe640 1 -- 172.21.15.77:0/2739996785 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f83100ce0b0 con 0x7f8328100210 2024-03-16T10:14:59.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.797+0000 7f832659c640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f83140a2800 0x7f83140a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8328069b50 tx=0x7f8320009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:14:59.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.801+0000 7f831f7fe640 1 -- 172.21.15.77:0/2739996785 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f831009b5f0 con 0x7f8328100210 2024-03-16T10:14:59.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:14:59.990+0000 7f832cac6640 1 -- 172.21.15.77:0/2739996785 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f832802d250 con 0x7f8328100210 2024-03-16T10:15:00.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:59 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2118589169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:00.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:59 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2529906616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:15:00.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:14:59 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3070910842' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T10:15:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:59 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2118589169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:59 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2529906616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:15:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:14:59 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3070910842' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T10:15:00.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:59 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2118589169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:00.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:59 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2529906616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:15:00.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:14:59 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3070910842' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T10:15:00.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.985+0000 7f831f7fe640 1 -- 172.21.15.77:0/2739996785 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f831009aef0 con 0x7f8328100210 2024-03-16T10:15:00.986 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2024-03-16T10:15:00.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.987+0000 7f832cac6640 1 -- 172.21.15.77:0/2739996785 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f83140a2800 msgr2=0x7f83140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:00.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.987+0000 7f832cac6640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f83140a2800 0x7f83140a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8328069b50 tx=0x7f8320009870 comp rx=0 tx=0).stop 2024-03-16T10:15:00.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.987+0000 7f832cac6640 1 -- 172.21.15.77:0/2739996785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8328100210 msgr2=0x7f8328100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:00.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.987+0000 7f832cac6640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8328100210 0x7f8328100b20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f831000d770 tx=0x7f831000dc30 comp rx=0 tx=0).stop 2024-03-16T10:15:00.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.987+0000 7f832cac6640 1 -- 172.21.15.77:0/2739996785 shutdown_connections 2024-03-16T10:15:00.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.987+0000 7f832cac6640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f83140a2800 0x7f83140a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:00.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.987+0000 7f832cac6640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8328102de0 0x7f8328101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:00.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.987+0000 7f832cac6640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8328100210 0x7f8328100b20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:00.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.987+0000 7f832cac6640 1 --2- 172.21.15.77:0/2739996785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83280ff850 0x7f83281024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:00.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.987+0000 7f832cac6640 1 -- 172.21.15.77:0/2739996785 >> 172.21.15.77:0/2739996785 conn(0x7f83280fb410 msgr2=0x7f8328105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:00.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.987+0000 7f832cac6640 1 -- 172.21.15.77:0/2739996785 shutdown_connections 2024-03-16T10:15:00.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:00.988+0000 7f832cac6640 1 -- 172.21.15.77:0/2739996785 wait complete. 2024-03-16T10:15:01.004 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-16T10:15:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.114+0000 7fe59e6cc640 1 -- 172.21.15.77:0/2510491826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe598102290 msgr2=0x7fe598108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.114+0000 7fe59e6cc640 1 --2- 172.21.15.77:0/2510491826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe598102290 0x7fe598108b20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe590009ca0 tx=0x7fe59002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:15:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 -- 172.21.15.77:0/2510491826 shutdown_connections 2024-03-16T10:15:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 --2- 172.21.15.77:0/2510491826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe598102290 0x7fe598108b20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 --2- 172.21.15.77:0/2510491826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5981018d0 0x7fe598101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 --2- 172.21.15.77:0/2510491826 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5981006e0 0x7fe598100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 -- 172.21.15.77:0/2510491826 >> 172.21.15.77:0/2510491826 conn(0x7fe5980fbc70 msgr2=0x7fe5980fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 -- 172.21.15.77:0/2510491826 shutdown_connections 2024-03-16T10:15:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 -- 172.21.15.77:0/2510491826 wait complete. 2024-03-16T10:15:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 Processor -- start 2024-03-16T10:15:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 -- start start 2024-03-16T10:15:01.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5981006e0 0x7fe598197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5981018d0 0x7fe598198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe598102290 0x7fe59819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe59810d480 con 0x7fe598102290 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe59810d300 con 0x7fe5981006e0 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.115+0000 7fe59e6cc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe59810d600 con 0x7fe5981018d0 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe59cc69640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe598102290 0x7fe59819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe59cc69640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe598102290 0x7fe59819f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38952/0 (socket says 172.21.15.77:38952) 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe59cc69640 1 -- 172.21.15.77:0/1591433033 learned_addr learned my addr 172.21.15.77:0/1591433033 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe5977fe640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5981018d0 0x7fe598198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe597fff640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5981006e0 0x7fe598197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe59cc69640 1 -- 172.21.15.77:0/1591433033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5981018d0 msgr2=0x7fe598198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe59cc69640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5981018d0 0x7fe598198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe59cc69640 1 -- 172.21.15.77:0/1591433033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5981006e0 msgr2=0x7fe598197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe59cc69640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5981006e0 0x7fe598197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:01.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe59cc69640 1 -- 172.21.15.77:0/1591433033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe59819f6f0 con 0x7fe598102290 2024-03-16T10:15:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe59cc69640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe598102290 0x7fe59819f080 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe59002c630 tx=0x7fe590032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe5957fa640 1 -- 172.21.15.77:0/1591433033 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe590041020 con 0x7fe598102290 2024-03-16T10:15:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe59e6cc640 1 -- 172.21.15.77:0/1591433033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe59819f980 con 0x7fe598102290 2024-03-16T10:15:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe5957fa640 1 -- 172.21.15.77:0/1591433033 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe590034cd0 con 0x7fe598102290 2024-03-16T10:15:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe5957fa640 1 -- 172.21.15.77:0/1591433033 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe590039040 con 0x7fe598102290 2024-03-16T10:15:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.116+0000 7fe59e6cc640 1 -- 172.21.15.77:0/1591433033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe59819fec0 con 0x7fe598102290 2024-03-16T10:15:01.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.117+0000 7fe59e6cc640 1 -- 172.21.15.77:0/1591433033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe598107a00 con 0x7fe598102290 2024-03-16T10:15:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.119+0000 7fe5957fa640 1 -- 172.21.15.77:0/1591433033 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe590044020 con 0x7fe598102290 2024-03-16T10:15:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.119+0000 7fe5957fa640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5800a2800 0x7fe5800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.119+0000 7fe597fff640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5800a2800 0x7fe5800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.120+0000 7fe5957fa640 1 -- 172.21.15.77:0/1591433033 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7fe5900f0a40 con 0x7fe598102290 2024-03-16T10:15:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.120+0000 7fe597fff640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5800a2800 0x7fe5800a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe588005fd0 tx=0x7fe58800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.123+0000 7fe5957fa640 1 -- 172.21.15.77:0/1591433033 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe59004b020 con 0x7fe598102290 2024-03-16T10:15:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:00 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2739996785' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T10:15:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:15:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:15:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:00 smithi077 ceph-mon[32400]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:15:01.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:00 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2739996785' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T10:15:01.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:15:01.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:15:01.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:00 smithi193 ceph-mon[36311]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:15:01.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.299+0000 7fe59e6cc640 1 -- 172.21.15.77:0/1591433033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe59802d1b0 con 0x7fe598102290 2024-03-16T10:15:01.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:00 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2739996785' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T10:15:01.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:15:01.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:15:01.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:15:01.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:00 smithi192 ceph-mon[36354]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:15:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.994+0000 7fe5957fa640 1 -- 172.21.15.77:0/1591433033 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7fe5900bde10 con 0x7fe598102290 2024-03-16T10:15:01.995 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2024-03-16T10:15:01.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.996+0000 7fe59e6cc640 1 -- 172.21.15.77:0/1591433033 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5800a2800 msgr2=0x7fe5800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:01.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.996+0000 7fe59e6cc640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5800a2800 0x7fe5800a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe588005fd0 tx=0x7fe58800b390 comp rx=0 tx=0).stop 2024-03-16T10:15:01.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.996+0000 7fe59e6cc640 1 -- 172.21.15.77:0/1591433033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe598102290 msgr2=0x7fe59819f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:01.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.996+0000 7fe59e6cc640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe598102290 0x7fe59819f080 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe59002c630 tx=0x7fe590032ea0 comp rx=0 tx=0).stop 2024-03-16T10:15:01.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.996+0000 7fe59e6cc640 1 -- 172.21.15.77:0/1591433033 shutdown_connections 2024-03-16T10:15:01.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.996+0000 7fe59e6cc640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe598102290 0x7fe59819f080 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:01.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.996+0000 7fe59e6cc640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5800a2800 0x7fe5800a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:01.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.996+0000 7fe59e6cc640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5981018d0 0x7fe598198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:01.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.996+0000 7fe59e6cc640 1 --2- 172.21.15.77:0/1591433033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5981006e0 0x7fe598197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:01.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.996+0000 7fe59e6cc640 1 -- 172.21.15.77:0/1591433033 >> 172.21.15.77:0/1591433033 conn(0x7fe5980fbc70 msgr2=0x7fe598104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:01.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.996+0000 7fe59e6cc640 1 -- 172.21.15.77:0/1591433033 shutdown_connections 2024-03-16T10:15:01.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:01.996+0000 7fe59e6cc640 1 -- 172.21.15.77:0/1591433033 wait complete. 2024-03-16T10:15:02.016 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-16T10:15:02.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.121+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2693873989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0acc1033c0 msgr2=0x7f0acc109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:02.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.121+0000 7f0ad3bc6640 1 --2- 172.21.15.77:0/2693873989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0acc1033c0 0x7f0acc109c50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009900 tx=0x7f0ac402bc30 comp rx=0 tx=0).stop 2024-03-16T10:15:02.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.121+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2693873989 shutdown_connections 2024-03-16T10:15:02.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.121+0000 7f0ad3bc6640 1 --2- 172.21.15.77:0/2693873989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0acc1033c0 0x7f0acc109c50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:02.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.121+0000 7f0ad3bc6640 1 --2- 172.21.15.77:0/2693873989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0acc102a00 0x7f0acc102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:02.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.121+0000 7f0ad3bc6640 1 --2- 172.21.15.77:0/2693873989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0acc101810 0x7f0acc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:02.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.121+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2693873989 >> 172.21.15.77:0/2693873989 conn(0x7f0acc0fcd80 msgr2=0x7f0acc0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:02.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2693873989 shutdown_connections 2024-03-16T10:15:02.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2693873989 wait complete. 2024-03-16T10:15:02.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad3bc6640 1 Processor -- start 2024-03-16T10:15:02.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad3bc6640 1 -- start start 2024-03-16T10:15:02.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad3bc6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0acc101810 0x7f0acc199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad3bc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0acc102a00 0x7f0acc19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad3bc6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0acc1033c0 0x7f0acc1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad3bc6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc10d490 con 0x7f0acc102a00 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad3bc6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc10d310 con 0x7f0acc101810 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad3bc6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc10d610 con 0x7f0acc1033c0 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad1161640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0acc102a00 0x7f0acc19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad1161640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0acc102a00 0x7f0acc19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56510/0 (socket says 172.21.15.77:56510) 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad1161640 1 -- 172.21.15.77:0/2127073666 learned_addr learned my addr 172.21.15.77:0/2127073666 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.122+0000 7f0ad1962640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0acc101810 0x7f0acc199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.123+0000 7f0ad2163640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0acc1033c0 0x7f0acc1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.123+0000 7f0ad1161640 1 -- 172.21.15.77:0/2127073666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0acc1033c0 msgr2=0x7f0acc1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.123+0000 7f0ad1161640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0acc1033c0 0x7f0acc1a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.123+0000 7f0ad1161640 1 -- 172.21.15.77:0/2127073666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0acc101810 msgr2=0x7f0acc199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:02.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.123+0000 7f0ad1161640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0acc101810 0x7f0acc199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:02.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.123+0000 7f0ad1161640 1 -- 172.21.15.77:0/2127073666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0acc1a1900 con 0x7f0acc102a00 2024-03-16T10:15:02.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.123+0000 7f0ad1161640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0acc102a00 0x7f0acc19a230 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00e9f0 tx=0x7f0abc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:02.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.123+0000 7f0ac2ffd640 1 -- 172.21.15.77:0/2127073666 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc00cd80 con 0x7f0acc102a00 2024-03-16T10:15:02.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.123+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2127073666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0acc1a1bf0 con 0x7f0acc102a00 2024-03-16T10:15:02.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.123+0000 7f0ac2ffd640 1 -- 172.21.15.77:0/2127073666 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0abc005500 con 0x7f0acc102a00 2024-03-16T10:15:02.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.123+0000 7f0ac2ffd640 1 -- 172.21.15.77:0/2127073666 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc010690 con 0x7f0acc102a00 2024-03-16T10:15:02.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.123+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2127073666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0acc1a20f0 con 0x7f0acc102a00 2024-03-16T10:15:02.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.124+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2127073666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0acc1a22c0 con 0x7f0acc102a00 2024-03-16T10:15:02.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.125+0000 7f0ac2ffd640 1 -- 172.21.15.77:0/2127073666 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0abc0046c0 con 0x7f0acc102a00 2024-03-16T10:15:02.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.126+0000 7f0ac2ffd640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0ab80a2800 0x7f0ab80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:02.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.126+0000 7f0ad1962640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0ab80a2800 0x7f0ab80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:02.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.126+0000 7f0ac2ffd640 1 -- 172.21.15.77:0/2127073666 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f0abc0cefc0 con 0x7f0acc102a00 2024-03-16T10:15:02.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.126+0000 7f0ad1962640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0ab80a2800 0x7f0ab80a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0acc19b1c0 tx=0x7f0ac8009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:02.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.129+0000 7f0ac2ffd640 1 -- 172.21.15.77:0/2127073666 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0abc053740 con 0x7f0acc102a00 2024-03-16T10:15:02.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:01 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2739996785' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T10:15:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:01 smithi077 ceph-mon[32400]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T10:15:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:01 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1591433033' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T10:15:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:01 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2739996785' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T10:15:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:01 smithi193 ceph-mon[36311]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T10:15:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:01 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1591433033' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T10:15:02.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:02.342+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2127073666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f0acc1a2490 con 0x7f0acc102a00 2024-03-16T10:15:02.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:01 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2739996785' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T10:15:02.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:01 smithi192 ceph-mon[36354]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T10:15:02.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:01 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1591433033' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T10:15:03.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.004+0000 7f0ac2ffd640 1 -- 172.21.15.77:0/2127073666 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f0abc09c210 con 0x7f0acc102a00 2024-03-16T10:15:03.006 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 2 and data pool 3 2024-03-16T10:15:03.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.008+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2127073666 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0ab80a2800 msgr2=0x7f0ab80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.008+0000 7f0ad3bc6640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0ab80a2800 0x7f0ab80a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0acc19b1c0 tx=0x7f0ac8009670 comp rx=0 tx=0).stop 2024-03-16T10:15:03.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.008+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2127073666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0acc102a00 msgr2=0x7f0acc19a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.009+0000 7f0ad3bc6640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0acc102a00 0x7f0acc19a230 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00e9f0 tx=0x7f0abc00edf0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.009+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2127073666 shutdown_connections 2024-03-16T10:15:03.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.009+0000 7f0ad3bc6640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0ab80a2800 0x7f0ab80a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.009+0000 7f0ad3bc6640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0acc1033c0 0x7f0acc1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.009+0000 7f0ad3bc6640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0acc102a00 0x7f0acc19a230 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.009+0000 7f0ad3bc6640 1 --2- 172.21.15.77:0/2127073666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0acc101810 0x7f0acc199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.009+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2127073666 >> 172.21.15.77:0/2127073666 conn(0x7f0acc0fcd80 msgr2=0x7f0acc073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:03.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.010+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2127073666 shutdown_connections 2024-03-16T10:15:03.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.010+0000 7f0ad3bc6640 1 -- 172.21.15.77:0/2127073666 wait complete. 2024-03-16T10:15:03.030 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:15:03.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.133+0000 7f336080c640 1 -- 172.21.15.77:0/3700938958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f335c100fe0 msgr2=0x7f335c101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.133+0000 7f336080c640 1 --2- 172.21.15.77:0/3700938958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f335c100fe0 0x7f335c101460 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f335000aa10 tx=0x7f335002cf70 comp rx=0 tx=0).stop 2024-03-16T10:15:03.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.133+0000 7f336080c640 1 -- 172.21.15.77:0/3700938958 shutdown_connections 2024-03-16T10:15:03.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.133+0000 7f336080c640 1 --2- 172.21.15.77:0/3700938958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f335c1019a0 0x7f335c1082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.133+0000 7f336080c640 1 --2- 172.21.15.77:0/3700938958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f335c100fe0 0x7f335c101460 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.133+0000 7f336080c640 1 --2- 172.21.15.77:0/3700938958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f335c0ffe90 0x7f335c1002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.133+0000 7f336080c640 1 -- 172.21.15.77:0/3700938958 >> 172.21.15.77:0/3700938958 conn(0x7f335c0fb400 msgr2=0x7f335c0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:03.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.133+0000 7f336080c640 1 -- 172.21.15.77:0/3700938958 shutdown_connections 2024-03-16T10:15:03.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.133+0000 7f336080c640 1 -- 172.21.15.77:0/3700938958 wait complete. 2024-03-16T10:15:03.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f336080c640 1 Processor -- start 2024-03-16T10:15:03.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f336080c640 1 -- start start 2024-03-16T10:15:03.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f336080c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f335c0ffe90 0x7f335c195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:03.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f336080c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f335c100fe0 0x7f335c195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:03.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f336080c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f335c1019a0 0x7f335c19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:03.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f336080c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f335c10b2c0 con 0x7f335c0ffe90 2024-03-16T10:15:03.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f336080c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f335c10b140 con 0x7f335c100fe0 2024-03-16T10:15:03.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f336080c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f335c10b440 con 0x7f335c1019a0 2024-03-16T10:15:03.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f335a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f335c0ffe90 0x7f335c195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:03.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f335a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f335c0ffe90 0x7f335c195870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56534/0 (socket says 172.21.15.77:56534) 2024-03-16T10:15:03.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f335a59c640 1 -- 172.21.15.77:0/1586771307 learned_addr learned my addr 172.21.15.77:0/1586771307 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:03.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f3359d9b640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f335c100fe0 0x7f335c195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:03.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f335ad9d640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f335c1019a0 0x7f335c19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:03.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f335a59c640 1 -- 172.21.15.77:0/1586771307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f335c1019a0 msgr2=0x7f335c19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f335a59c640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f335c1019a0 0x7f335c19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f335a59c640 1 -- 172.21.15.77:0/1586771307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f335c100fe0 msgr2=0x7f335c195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f335a59c640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f335c100fe0 0x7f335c195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f335a59c640 1 -- 172.21.15.77:0/1586771307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335c19d490 con 0x7f335c0ffe90 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f335ad9d640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f335c1019a0 0x7f335c19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f3359d9b640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f335c100fe0 0x7f335c195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.134+0000 7f335a59c640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f335c0ffe90 0x7f335c195870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f334c007b50 tx=0x7f334c00e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.135+0000 7f334b7fe640 1 -- 172.21.15.77:0/1586771307 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c007c40 con 0x7f335c0ffe90 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.135+0000 7f334b7fe640 1 -- 172.21.15.77:0/1586771307 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f334c007de0 con 0x7f335c0ffe90 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.135+0000 7f334b7fe640 1 -- 172.21.15.77:0/1586771307 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c00ae60 con 0x7f335c0ffe90 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.135+0000 7f336080c640 1 -- 172.21.15.77:0/1586771307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f335c19d780 con 0x7f335c0ffe90 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.135+0000 7f336080c640 1 -- 172.21.15.77:0/1586771307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f335c19dcf0 con 0x7f335c0ffe90 2024-03-16T10:15:03.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.136+0000 7f336080c640 1 -- 172.21.15.77:0/1586771307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f335c107140 con 0x7f335c0ffe90 2024-03-16T10:15:03.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.137+0000 7f334b7fe640 1 -- 172.21.15.77:0/1586771307 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f334c00a440 con 0x7f335c0ffe90 2024-03-16T10:15:03.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.137+0000 7f334b7fe640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f33440a2880 0x7f33440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:03.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.137+0000 7f3359d9b640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f33440a2880 0x7f33440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:03.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.137+0000 7f334b7fe640 1 -- 172.21.15.77:0/1586771307 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f334c0cfe60 con 0x7f335c0ffe90 2024-03-16T10:15:03.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.138+0000 7f3359d9b640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f33440a2880 0x7f33440a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3350004d40 tx=0x7f3350000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:03.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.142+0000 7f334b7fe640 1 -- 172.21.15.77:0/1586771307 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f334c023060 con 0x7f335c0ffe90 2024-03-16T10:15:03.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.323+0000 7f336080c640 1 -- 172.21.15.77:0/1586771307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f335c062410 con 0x7f335c0ffe90 2024-03-16T10:15:03.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.325+0000 7f334b7fe640 1 -- 172.21.15.77:0/1586771307 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22208 (secure 0 0 0) 0x7f334c09d0e0 con 0x7f335c0ffe90 2024-03-16T10:15:03.326 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:15:03.326 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":84,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:02.344306+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:15:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.327+0000 7f336080c640 1 -- 172.21.15.77:0/1586771307 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f33440a2880 msgr2=0x7f33440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.327+0000 7f336080c640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f33440a2880 0x7f33440a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3350004d40 tx=0x7f3350000bc0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.327+0000 7f336080c640 1 -- 172.21.15.77:0/1586771307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f335c0ffe90 msgr2=0x7f335c195870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.327+0000 7f336080c640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f335c0ffe90 0x7f335c195870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f334c007b50 tx=0x7f334c00e420 comp rx=0 tx=0).stop 2024-03-16T10:15:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.327+0000 7f336080c640 1 -- 172.21.15.77:0/1586771307 shutdown_connections 2024-03-16T10:15:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.327+0000 7f336080c640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f33440a2880 0x7f33440a4d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.327+0000 7f336080c640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f335c1019a0 0x7f335c19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.327+0000 7f336080c640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f335c100fe0 0x7f335c195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.327+0000 7f336080c640 1 --2- 172.21.15.77:0/1586771307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f335c0ffe90 0x7f335c195870 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.327+0000 7f336080c640 1 -- 172.21.15.77:0/1586771307 >> 172.21.15.77:0/1586771307 conn(0x7f335c0fb400 msgr2=0x7f335c104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:03.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.327+0000 7f336080c640 1 -- 172.21.15.77:0/1586771307 shutdown_connections 2024-03-16T10:15:03.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.327+0000 7f336080c640 1 -- 172.21.15.77:0/1586771307 wait complete. 2024-03-16T10:15:03.338 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:03 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1591433033' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T10:15:03.338 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:03 smithi077 ceph-mon[32400]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T10:15:03.338 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:03 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2127073666' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T10:15:03.338 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:03 smithi077 ceph-mon[32400]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T10:15:03.338 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:03 smithi077 ceph-mon[32400]: pgmap v285: 97 pgs: 14 creating+peering, 73 unknown, 10 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:15:03.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:15:03.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:03 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1591433033' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T10:15:03.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:03 smithi192 ceph-mon[36354]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T10:15:03.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:03 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2127073666' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T10:15:03.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:03 smithi192 ceph-mon[36354]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T10:15:03.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:03 smithi192 ceph-mon[36354]: pgmap v285: 97 pgs: 14 creating+peering, 73 unknown, 10 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:15:03.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 -- 172.21.15.77:0/2095373254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90941022b0 msgr2=0x7f9094108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 --2- 172.21.15.77:0/2095373254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90941022b0 0x7f9094108b40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f908c009900 tx=0x7f908c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:15:03.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 -- 172.21.15.77:0/2095373254 shutdown_connections 2024-03-16T10:15:03.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 --2- 172.21.15.77:0/2095373254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90941022b0 0x7f9094108b40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 --2- 172.21.15.77:0/2095373254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90941018f0 0x7f9094101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 --2- 172.21.15.77:0/2095373254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9094100700 0x7f9094100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 -- 172.21.15.77:0/2095373254 >> 172.21.15.77:0/2095373254 conn(0x7f90940fbc70 msgr2=0x7f90940fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:03.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 -- 172.21.15.77:0/2095373254 shutdown_connections 2024-03-16T10:15:03.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 -- 172.21.15.77:0/2095373254 wait complete. 2024-03-16T10:15:03.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 Processor -- start 2024-03-16T10:15:03.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 -- start start 2024-03-16T10:15:03.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9094100700 0x7f90940694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:03.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90941018f0 0x7f909419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:03.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.441+0000 7f9099b69640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90941022b0 0x7f909419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:03.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9099b69640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f909410c3f0 con 0x7f90941018f0 2024-03-16T10:15:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9099b69640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f909410c270 con 0x7f90941022b0 2024-03-16T10:15:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9099b69640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f909410c570 con 0x7f9094100700 2024-03-16T10:15:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9092ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90941018f0 0x7f909419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9092ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90941018f0 0x7f909419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56544/0 (socket says 172.21.15.77:56544) 2024-03-16T10:15:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9092ffd640 1 -- 172.21.15.77:0/898266526 learned_addr learned my addr 172.21.15.77:0/898266526 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9093fff640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90941022b0 0x7f909419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f90937fe640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9094100700 0x7f90940694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9092ffd640 1 -- 172.21.15.77:0/898266526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9094100700 msgr2=0x7f90940694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9092ffd640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9094100700 0x7f90940694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9092ffd640 1 -- 172.21.15.77:0/898266526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90941022b0 msgr2=0x7f909419eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9092ffd640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90941022b0 0x7f909419eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9092ffd640 1 -- 172.21.15.77:0/898266526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909419f7b0 con 0x7f90941018f0 2024-03-16T10:15:03.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9092ffd640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90941018f0 0x7f909419c880 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f908800ebc0 tx=0x7f908800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:03.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9090ff9640 1 -- 172.21.15.77:0/898266526 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9088010630 con 0x7f90941018f0 2024-03-16T10:15:03.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9090ff9640 1 -- 172.21.15.77:0/898266526 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9088010cb0 con 0x7f90941018f0 2024-03-16T10:15:03.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9099b69640 1 -- 172.21.15.77:0/898266526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f909419fa40 con 0x7f90941018f0 2024-03-16T10:15:03.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9090ff9640 1 -- 172.21.15.77:0/898266526 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9088018bd0 con 0x7f90941018f0 2024-03-16T10:15:03.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.442+0000 7f9099b69640 1 -- 172.21.15.77:0/898266526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f909419ff90 con 0x7f90941018f0 2024-03-16T10:15:03.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.443+0000 7f9099b69640 1 -- 172.21.15.77:0/898266526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90941079c0 con 0x7f90941018f0 2024-03-16T10:15:03.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.444+0000 7f9090ff9640 1 -- 172.21.15.77:0/898266526 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f90880107d0 con 0x7f90941018f0 2024-03-16T10:15:03.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.445+0000 7f9090ff9640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f907c0a2800 0x7f907c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:03.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.445+0000 7f90937fe640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f907c0a2800 0x7f907c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:03.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.445+0000 7f9090ff9640 1 -- 172.21.15.77:0/898266526 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f90880cefa0 con 0x7f90941018f0 2024-03-16T10:15:03.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.445+0000 7f90937fe640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f907c0a2800 0x7f907c0a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9094101750 tx=0x7f9084008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:03.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.449+0000 7f9090ff9640 1 -- 172.21.15.77:0/898266526 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f908809c190 con 0x7f90941018f0 2024-03-16T10:15:03.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:03 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1591433033' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T10:15:03.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:03 smithi193 ceph-mon[36311]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T10:15:03.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:03 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2127073666' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T10:15:03.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:03 smithi193 ceph-mon[36311]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T10:15:03.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:03 smithi193 ceph-mon[36311]: pgmap v285: 97 pgs: 14 creating+peering, 73 unknown, 10 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:15:03.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.625+0000 7f9099b69640 1 -- 172.21.15.77:0/898266526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f909402d400 con 0x7f90941018f0 2024-03-16T10:15:03.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.627+0000 7f9090ff9640 1 -- 172.21.15.77:0/898266526 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22208 (secure 0 0 0) 0x7f908809ba90 con 0x7f90941018f0 2024-03-16T10:15:03.628 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:15:03.628 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":84,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:02.344306+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:15:03.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.629+0000 7f9099b69640 1 -- 172.21.15.77:0/898266526 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f907c0a2800 msgr2=0x7f907c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.629+0000 7f9099b69640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f907c0a2800 0x7f907c0a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9094101750 tx=0x7f9084008040 comp rx=0 tx=0).stop 2024-03-16T10:15:03.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.629+0000 7f9099b69640 1 -- 172.21.15.77:0/898266526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90941018f0 msgr2=0x7f909419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.629+0000 7f9099b69640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90941018f0 0x7f909419c880 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f908800ebc0 tx=0x7f908800c6a0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.629+0000 7f9099b69640 1 -- 172.21.15.77:0/898266526 shutdown_connections 2024-03-16T10:15:03.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.629+0000 7f9099b69640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90941022b0 0x7f909419eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.629+0000 7f9099b69640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90941018f0 0x7f909419c880 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.629+0000 7f9099b69640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f907c0a2800 0x7f907c0a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.629+0000 7f9099b69640 1 --2- 172.21.15.77:0/898266526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9094100700 0x7f90940694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.629+0000 7f9099b69640 1 -- 172.21.15.77:0/898266526 >> 172.21.15.77:0/898266526 conn(0x7f90940fbc70 msgr2=0x7f9094104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:03.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.629+0000 7f9099b69640 1 -- 172.21.15.77:0/898266526 shutdown_connections 2024-03-16T10:15:03.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.629+0000 7f9099b69640 1 -- 172.21.15.77:0/898266526 wait complete. 2024-03-16T10:15:03.641 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-16T10:15:03.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.744+0000 7efe26cf8640 1 -- 172.21.15.77:0/3357905120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe201006e0 msgr2=0x7efe20100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.744+0000 7efe26cf8640 1 --2- 172.21.15.77:0/3357905120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe201006e0 0x7efe20100b00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efe14009900 tx=0x7efe1402bc30 comp rx=0 tx=0).stop 2024-03-16T10:15:03.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.744+0000 7efe26cf8640 1 -- 172.21.15.77:0/3357905120 shutdown_connections 2024-03-16T10:15:03.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.744+0000 7efe26cf8640 1 --2- 172.21.15.77:0/3357905120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe20102290 0x7efe20108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.744+0000 7efe26cf8640 1 --2- 172.21.15.77:0/3357905120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe201018d0 0x7efe20101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.744+0000 7efe26cf8640 1 --2- 172.21.15.77:0/3357905120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe201006e0 0x7efe20100b00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.744+0000 7efe26cf8640 1 -- 172.21.15.77:0/3357905120 >> 172.21.15.77:0/3357905120 conn(0x7efe200fbc70 msgr2=0x7efe200fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:03.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.744+0000 7efe26cf8640 1 -- 172.21.15.77:0/3357905120 shutdown_connections 2024-03-16T10:15:03.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.744+0000 7efe26cf8640 1 -- 172.21.15.77:0/3357905120 wait complete. 2024-03-16T10:15:03.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe26cf8640 1 Processor -- start 2024-03-16T10:15:03.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe26cf8640 1 -- start start 2024-03-16T10:15:03.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe26cf8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe201006e0 0x7efe20197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:03.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe26cf8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe201018d0 0x7efe20198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe26cf8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe20102290 0x7efe2019f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe26cf8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efe2010d4c0 con 0x7efe20102290 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe26cf8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7efe2010d340 con 0x7efe201018d0 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe26cf8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efe2010d640 con 0x7efe201006e0 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe25295640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe20102290 0x7efe2019f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe25295640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe20102290 0x7efe2019f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56562/0 (socket says 172.21.15.77:56562) 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe25295640 1 -- 172.21.15.77:0/1688551733 learned_addr learned my addr 172.21.15.77:0/1688551733 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe24a94640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe201006e0 0x7efe20197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe1ffff640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe201018d0 0x7efe20198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe25295640 1 -- 172.21.15.77:0/1688551733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe201006e0 msgr2=0x7efe20197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe25295640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe201006e0 0x7efe20197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe25295640 1 -- 172.21.15.77:0/1688551733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe201018d0 msgr2=0x7efe20198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.745+0000 7efe25295640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe201018d0 0x7efe20198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.746+0000 7efe25295640 1 -- 172.21.15.77:0/1688551733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe2019f7e0 con 0x7efe20102290 2024-03-16T10:15:03.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.746+0000 7efe24a94640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe201006e0 0x7efe20197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:15:03.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.746+0000 7efe25295640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe20102290 0x7efe2019f170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efe1800b970 tx=0x7efe1800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:03.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.746+0000 7efe1dffb640 1 -- 172.21.15.77:0/1688551733 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18019070 con 0x7efe20102290 2024-03-16T10:15:03.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.746+0000 7efe1dffb640 1 -- 172.21.15.77:0/1688551733 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe1800cb40 con 0x7efe20102290 2024-03-16T10:15:03.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.746+0000 7efe26cf8640 1 -- 172.21.15.77:0/1688551733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe2019fad0 con 0x7efe20102290 2024-03-16T10:15:03.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.746+0000 7efe1dffb640 1 -- 172.21.15.77:0/1688551733 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18007650 con 0x7efe20102290 2024-03-16T10:15:03.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.746+0000 7efe26cf8640 1 -- 172.21.15.77:0/1688551733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe201a0070 con 0x7efe20102290 2024-03-16T10:15:03.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.747+0000 7efe26cf8640 1 -- 172.21.15.77:0/1688551733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe20107a80 con 0x7efe20102290 2024-03-16T10:15:03.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.748+0000 7efe1dffb640 1 -- 172.21.15.77:0/1688551733 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efe1801c070 con 0x7efe20102290 2024-03-16T10:15:03.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.748+0000 7efe1dffb640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efe100a2880 0x7efe100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:03.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.748+0000 7efe24a94640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efe100a2880 0x7efe100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:03.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.748+0000 7efe1dffb640 1 -- 172.21.15.77:0/1688551733 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7efe180cddc0 con 0x7efe20102290 2024-03-16T10:15:03.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.749+0000 7efe24a94640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efe100a2880 0x7efe100a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efe1402c7b0 tx=0x7efe14005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:03.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.753+0000 7efe1dffb640 1 -- 172.21.15.77:0/1688551733 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efe1809a960 con 0x7efe20102290 2024-03-16T10:15:03.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:03.967+0000 7efe26cf8640 1 -- 172.21.15.77:0/1688551733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7efe200623e0 con 0x7efe20102290 2024-03-16T10:15:04.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.222+0000 7efe1dffb640 1 -- 172.21.15.77:0/1688551733 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7efe1809ea00 con 0x7efe20102290 2024-03-16T10:15:04.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.225+0000 7efe26cf8640 1 -- 172.21.15.77:0/1688551733 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efe100a2880 msgr2=0x7efe100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:04.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.225+0000 7efe26cf8640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efe100a2880 0x7efe100a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efe1402c7b0 tx=0x7efe14005fd0 comp rx=0 tx=0).stop 2024-03-16T10:15:04.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.225+0000 7efe26cf8640 1 -- 172.21.15.77:0/1688551733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe20102290 msgr2=0x7efe2019f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:04.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.225+0000 7efe26cf8640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe20102290 0x7efe2019f170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efe1800b970 tx=0x7efe1800be30 comp rx=0 tx=0).stop 2024-03-16T10:15:04.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.225+0000 7efe26cf8640 1 -- 172.21.15.77:0/1688551733 shutdown_connections 2024-03-16T10:15:04.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.225+0000 7efe26cf8640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe20102290 0x7efe2019f170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:04.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.225+0000 7efe26cf8640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe201018d0 0x7efe20198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:04.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.225+0000 7efe26cf8640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efe100a2880 0x7efe100a4d40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:04.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.225+0000 7efe26cf8640 1 --2- 172.21.15.77:0/1688551733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe201006e0 0x7efe20197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:04.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.225+0000 7efe26cf8640 1 -- 172.21.15.77:0/1688551733 >> 172.21.15.77:0/1688551733 conn(0x7efe200fbc70 msgr2=0x7efe20104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:04.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.225+0000 7efe26cf8640 1 -- 172.21.15.77:0/1688551733 shutdown_connections 2024-03-16T10:15:04.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.225+0000 7efe26cf8640 1 -- 172.21.15.77:0/1688551733 wait complete. 2024-03-16T10:15:04.237 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-16T10:15:04.238 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-16T10:15:04.262 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:04 smithi077 ceph-mon[32400]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T10:15:04.262 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:04 smithi077 ceph-mon[32400]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T10:15:04.262 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:04 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2127073666' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T10:15:04.262 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:04 smithi077 ceph-mon[32400]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T10:15:04.262 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:04 smithi077 ceph-mon[32400]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T10:15:04.263 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:04 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] up:active 2024-03-16T10:15:04.263 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:04 smithi077 ceph-mon[32400]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T10:15:04.263 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:04 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1586771307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:04.263 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:04 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/898266526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:04.263 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:04 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1688551733' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T10:15:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.343+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2831729446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd10100700 msgr2=0x7fbd10100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.343+0000 7fbd15a0b640 1 --2- 172.21.15.77:0/2831729446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd10100700 0x7fbd10100b20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009900 tx=0x7fbd0402bc30 comp rx=0 tx=0).stop 2024-03-16T10:15:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.343+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2831729446 shutdown_connections 2024-03-16T10:15:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.343+0000 7fbd15a0b640 1 --2- 172.21.15.77:0/2831729446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd101022b0 0x7fbd10108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.343+0000 7fbd15a0b640 1 --2- 172.21.15.77:0/2831729446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd101018f0 0x7fbd10101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.343+0000 7fbd15a0b640 1 --2- 172.21.15.77:0/2831729446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd10100700 0x7fbd10100b20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.343+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2831729446 >> 172.21.15.77:0/2831729446 conn(0x7fbd100fbc70 msgr2=0x7fbd100fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.343+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2831729446 shutdown_connections 2024-03-16T10:15:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.343+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2831729446 wait complete. 2024-03-16T10:15:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.343+0000 7fbd15a0b640 1 Processor -- start 2024-03-16T10:15:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.343+0000 7fbd15a0b640 1 -- start start 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd15a0b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd10100700 0x7fbd1010b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd15a0b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd101018f0 0x7fbd101096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd15a0b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd101022b0 0x7fbd10109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd15a0b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1010d4e0 con 0x7fbd10100700 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd15a0b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1010d360 con 0x7fbd101018f0 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd15a0b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1010d660 con 0x7fbd101022b0 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd0effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd10100700 0x7fbd1010b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd0effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd10100700 0x7fbd1010b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56578/0 (socket says 172.21.15.77:56578) 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd0effd640 1 -- 172.21.15.77:0/2367245108 learned_addr learned my addr 172.21.15.77:0/2367245108 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd0e7fc640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd101018f0 0x7fbd101096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd0f7fe640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd101022b0 0x7fbd10109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd0effd640 1 -- 172.21.15.77:0/2367245108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd101022b0 msgr2=0x7fbd10109c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd0effd640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd101022b0 0x7fbd10109c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd0effd640 1 -- 172.21.15.77:0/2367245108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd101018f0 msgr2=0x7fbd101096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd0effd640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd101018f0 0x7fbd101096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd0effd640 1 -- 172.21.15.77:0/2367245108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1010a490 con 0x7fbd10100700 2024-03-16T10:15:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd0effd640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd10100700 0x7fbd1010b080 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbd04000c00 tx=0x7fbd0402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd14a09640 1 -- 172.21.15.77:0/2367245108 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd04041070 con 0x7fbd10100700 2024-03-16T10:15:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2367245108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd101a1d20 con 0x7fbd10100700 2024-03-16T10:15:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd14a09640 1 -- 172.21.15.77:0/2367245108 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd040044d0 con 0x7fbd10100700 2024-03-16T10:15:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd14a09640 1 -- 172.21.15.77:0/2367245108 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd04034440 con 0x7fbd10100700 2024-03-16T10:15:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.344+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2367245108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd101a21c0 con 0x7fbd10100700 2024-03-16T10:15:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.345+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2367245108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd10107ae0 con 0x7fbd10100700 2024-03-16T10:15:04.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.346+0000 7fbd14a09640 1 -- 172.21.15.77:0/2367245108 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbd04044070 con 0x7fbd10100700 2024-03-16T10:15:04.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.347+0000 7fbd14a09640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbcf80a2800 0x7fbcf80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:04.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.347+0000 7fbd0e7fc640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbcf80a2800 0x7fbcf80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:04.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.347+0000 7fbd14a09640 1 -- 172.21.15.77:0/2367245108 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fbd040f0510 con 0x7fbd10100700 2024-03-16T10:15:04.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.347+0000 7fbd0e7fc640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbcf80a2800 0x7fbcf80a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbd1010adf0 tx=0x7fbd00009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.351+0000 7fbd14a09640 1 -- 172.21.15.77:0/2367245108 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbd040bd700 con 0x7fbd10100700 2024-03-16T10:15:04.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:04 smithi192 ceph-mon[36354]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T10:15:04.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:04 smithi192 ceph-mon[36354]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T10:15:04.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:04 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2127073666' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T10:15:04.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:04 smithi192 ceph-mon[36354]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T10:15:04.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:04 smithi192 ceph-mon[36354]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T10:15:04.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:04 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] up:active 2024-03-16T10:15:04.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:04 smithi192 ceph-mon[36354]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T10:15:04.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:04 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1586771307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:04.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:04 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/898266526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:04.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:04 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1688551733' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T10:15:04.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:04 smithi193 ceph-mon[36311]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T10:15:04.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:04 smithi193 ceph-mon[36311]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T10:15:04.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:04 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2127073666' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T10:15:04.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:04 smithi193 ceph-mon[36311]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T10:15:04.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:04 smithi193 ceph-mon[36311]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T10:15:04.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:04 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] up:active 2024-03-16T10:15:04.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:04 smithi193 ceph-mon[36311]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T10:15:04.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:04 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1586771307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:04.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:04 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/898266526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:04.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:04 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1688551733' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T10:15:04.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:04.566+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2367245108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fbd1002d3d0 con 0x7fbd10100700 2024-03-16T10:15:05.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.229+0000 7fbd14a09640 1 -- 172.21.15.77:0/2367245108 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v17) ==== 105+0+0 (secure 0 0 0) 0x7fbd040bd000 con 0x7fbd10100700 2024-03-16T10:15:05.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.231+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2367245108 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbcf80a2800 msgr2=0x7fbcf80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:05.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.231+0000 7fbd15a0b640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbcf80a2800 0x7fbcf80a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbd1010adf0 tx=0x7fbd00009110 comp rx=0 tx=0).stop 2024-03-16T10:15:05.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.231+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2367245108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd10100700 msgr2=0x7fbd1010b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:05.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.231+0000 7fbd15a0b640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd10100700 0x7fbd1010b080 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbd04000c00 tx=0x7fbd0402c640 comp rx=0 tx=0).stop 2024-03-16T10:15:05.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.231+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2367245108 shutdown_connections 2024-03-16T10:15:05.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.231+0000 7fbd15a0b640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbcf80a2800 0x7fbcf80a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:05.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.231+0000 7fbd15a0b640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd101022b0 0x7fbd10109c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:05.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.231+0000 7fbd15a0b640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd101018f0 0x7fbd101096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:05.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.231+0000 7fbd15a0b640 1 --2- 172.21.15.77:0/2367245108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd10100700 0x7fbd1010b080 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:05.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.231+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2367245108 >> 172.21.15.77:0/2367245108 conn(0x7fbd100fbc70 msgr2=0x7fbd10104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:05.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.231+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2367245108 shutdown_connections 2024-03-16T10:15:05.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.231+0000 7fbd15a0b640 1 -- 172.21.15.77:0/2367245108 wait complete. 2024-03-16T10:15:05.247 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-16T10:15:05.311 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:05 smithi077 ceph-mon[32400]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T10:15:05.312 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:05 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1688551733' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T10:15:05.312 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:05 smithi077 ceph-mon[32400]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T10:15:05.312 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:05 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2367245108' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-16T10:15:05.312 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:05 smithi077 ceph-mon[32400]: pgmap v288: 97 pgs: 18 creating+peering, 47 unknown, 32 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-03-16T10:15:05.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.395+0000 7f45029e8640 1 -- 172.21.15.77:0/1704282218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44fc100f80 msgr2=0x7f44fc103370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:05.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.395+0000 7f45029e8640 1 --2- 172.21.15.77:0/1704282218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44fc100f80 0x7f44fc103370 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009900 tx=0x7f44ec02bc30 comp rx=0 tx=0).stop 2024-03-16T10:15:05.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.395+0000 7f45029e8640 1 -- 172.21.15.77:0/1704282218 shutdown_connections 2024-03-16T10:15:05.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.395+0000 7f45029e8640 1 --2- 172.21.15.77:0/1704282218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44fc1062c0 0x7f44fc108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:05.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.395+0000 7f45029e8640 1 --2- 172.21.15.77:0/1704282218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44fc1038b0 0x7f44fc105ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:05.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.395+0000 7f45029e8640 1 --2- 172.21.15.77:0/1704282218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44fc100f80 0x7f44fc103370 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:05.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.395+0000 7f45029e8640 1 -- 172.21.15.77:0/1704282218 >> 172.21.15.77:0/1704282218 conn(0x7f44fc0fafa0 msgr2=0x7f44fc0fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:05.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.395+0000 7f45029e8640 1 -- 172.21.15.77:0/1704282218 shutdown_connections 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.395+0000 7f45029e8640 1 -- 172.21.15.77:0/1704282218 wait complete. 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.395+0000 7f45029e8640 1 Processor -- start 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.395+0000 7f45029e8640 1 -- start start 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f45029e8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44fc100f80 0x7f44fc100820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f45029e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44fc1038b0 0x7f44fc19c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f45029e8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44fc1062c0 0x7f44fc19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f45029e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc10c3d0 con 0x7f44fc1038b0 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f45029e8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc10c250 con 0x7f44fc1062c0 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f45029e8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc10c550 con 0x7f44fc100f80 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f44fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44fc1038b0 0x7f44fc19c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f44fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44fc1038b0 0x7f44fc19c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56604/0 (socket says 172.21.15.77:56604) 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f44fb7fe640 1 -- 172.21.15.77:0/2429871990 learned_addr learned my addr 172.21.15.77:0/2429871990 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f4500f85640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44fc1062c0 0x7f44fc19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:05.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f44fb7fe640 1 -- 172.21.15.77:0/2429871990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44fc100f80 msgr2=0x7f44fc100820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f44fb7fe640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44fc100f80 0x7f44fc100820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f44fb7fe640 1 -- 172.21.15.77:0/2429871990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44fc1062c0 msgr2=0x7f44fc19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f44fb7fe640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44fc1062c0 0x7f44fc19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f44fb7fe640 1 -- 172.21.15.77:0/2429871990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44fc19f800 con 0x7f44fc1038b0 2024-03-16T10:15:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f44fb7fe640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44fc1038b0 0x7f44fc19c900 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f44e400d950 tx=0x7f44e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f44f97fa640 1 -- 172.21.15.77:0/2429871990 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4017070 con 0x7f44fc1038b0 2024-03-16T10:15:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f44f97fa640 1 -- 172.21.15.77:0/2429871990 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44e4009c30 con 0x7f44fc1038b0 2024-03-16T10:15:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f45029e8640 1 -- 172.21.15.77:0/2429871990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44fc19faf0 con 0x7f44fc1038b0 2024-03-16T10:15:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f44f97fa640 1 -- 172.21.15.77:0/2429871990 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e400f6b0 con 0x7f44fc1038b0 2024-03-16T10:15:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.396+0000 7f45029e8640 1 -- 172.21.15.77:0/2429871990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44fc1a0090 con 0x7f44fc1038b0 2024-03-16T10:15:05.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.397+0000 7f45029e8640 1 -- 172.21.15.77:0/2429871990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44fc0fca90 con 0x7f44fc1038b0 2024-03-16T10:15:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.398+0000 7f44f97fa640 1 -- 172.21.15.77:0/2429871990 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f44e40109d0 con 0x7f44fc1038b0 2024-03-16T10:15:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.399+0000 7f44f97fa640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f44e80a2880 0x7f44e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.399+0000 7f44fbfff640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f44e80a2880 0x7f44e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.399+0000 7f44f97fa640 1 -- 172.21.15.77:0/2429871990 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f44e40ce590 con 0x7f44fc1038b0 2024-03-16T10:15:05.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.400+0000 7f44fbfff640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f44e80a2880 0x7f44e80a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f44ec02c710 tx=0x7f44ec000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:05.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.404+0000 7f44f97fa640 1 -- 172.21.15.77:0/2429871990 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f44e4009de0 con 0x7f44fc1038b0 2024-03-16T10:15:05.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:05 smithi192 ceph-mon[36354]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T10:15:05.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:05 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1688551733' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T10:15:05.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:05 smithi192 ceph-mon[36354]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T10:15:05.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:05 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2367245108' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-16T10:15:05.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:05 smithi192 ceph-mon[36354]: pgmap v288: 97 pgs: 18 creating+peering, 47 unknown, 32 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-03-16T10:15:05.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:05 smithi193 ceph-mon[36311]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T10:15:05.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:05 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1688551733' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T10:15:05.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:05 smithi193 ceph-mon[36311]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T10:15:05.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:05 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2367245108' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-16T10:15:05.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:05 smithi193 ceph-mon[36311]: pgmap v288: 97 pgs: 18 creating+peering, 47 unknown, 32 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-03-16T10:15:05.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:05.618+0000 7f45029e8640 1 -- 172.21.15.77:0/2429871990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f44fc02d3d0 con 0x7f44fc1038b0 2024-03-16T10:15:06.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.236+0000 7f44f97fa640 1 -- 172.21.15.77:0/2429871990 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f44e409b780 con 0x7f44fc1038b0 2024-03-16T10:15:06.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.238+0000 7f45029e8640 1 -- 172.21.15.77:0/2429871990 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f44e80a2880 msgr2=0x7f44e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:06.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.238+0000 7f45029e8640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f44e80a2880 0x7f44e80a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f44ec02c710 tx=0x7f44ec000bc0 comp rx=0 tx=0).stop 2024-03-16T10:15:06.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.239+0000 7f45029e8640 1 -- 172.21.15.77:0/2429871990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44fc1038b0 msgr2=0x7f44fc19c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:06.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.239+0000 7f45029e8640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44fc1038b0 0x7f44fc19c900 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f44e400d950 tx=0x7f44e400de10 comp rx=0 tx=0).stop 2024-03-16T10:15:06.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.239+0000 7f45029e8640 1 -- 172.21.15.77:0/2429871990 shutdown_connections 2024-03-16T10:15:06.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.239+0000 7f45029e8640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44fc1062c0 0x7f44fc19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:06.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.239+0000 7f45029e8640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44fc1038b0 0x7f44fc19c900 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:06.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.239+0000 7f45029e8640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f44e80a2880 0x7f44e80a4d40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:06.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.239+0000 7f45029e8640 1 --2- 172.21.15.77:0/2429871990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44fc100f80 0x7f44fc100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:06.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.239+0000 7f45029e8640 1 -- 172.21.15.77:0/2429871990 >> 172.21.15.77:0/2429871990 conn(0x7f44fc0fafa0 msgr2=0x7f44fc101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:06.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.239+0000 7f45029e8640 1 -- 172.21.15.77:0/2429871990 shutdown_connections 2024-03-16T10:15:06.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.239+0000 7f45029e8640 1 -- 172.21.15.77:0/2429871990 wait complete. 2024-03-16T10:15:06.255 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-16T10:15:06.321 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:06 smithi077 ceph-mon[32400]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-16T10:15:06.321 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:06 smithi077 ceph-mon[32400]: Cluster is now healthy 2024-03-16T10:15:06.321 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:06 smithi077 ceph-mon[32400]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T10:15:06.321 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:06 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2367245108' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-16T10:15:06.321 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:06 smithi077 ceph-mon[32400]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T10:15:06.321 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:06 smithi077 ceph-mon[32400]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T10:15:06.321 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:06 smithi077 ceph-mon[32400]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-16T10:15:06.321 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:06 smithi077 ceph-mon[32400]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-16T10:15:06.322 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:06 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2429871990' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T10:15:06.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.403+0000 7fe334485640 1 -- 172.21.15.77:0/469534335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe32c0ffac0 msgr2=0x7fe32c0ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:06.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.403+0000 7fe334485640 1 --2- 172.21.15.77:0/469534335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe32c0ffac0 0x7fe32c0ffee0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe324009cd0 tx=0x7fe32402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:15:06.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.403+0000 7fe334485640 1 -- 172.21.15.77:0/469534335 shutdown_connections 2024-03-16T10:15:06.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.403+0000 7fe334485640 1 --2- 172.21.15.77:0/469534335 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe32c103450 0x7fe32c109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:06.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.403+0000 7fe334485640 1 --2- 172.21.15.77:0/469534335 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe32c102a90 0x7fe32c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:06.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.403+0000 7fe334485640 1 --2- 172.21.15.77:0/469534335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe32c0ffac0 0x7fe32c0ffee0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:06.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.403+0000 7fe334485640 1 -- 172.21.15.77:0/469534335 >> 172.21.15.77:0/469534335 conn(0x7fe32c0fb3f0 msgr2=0x7fe32c0fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:06.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.403+0000 7fe334485640 1 -- 172.21.15.77:0/469534335 shutdown_connections 2024-03-16T10:15:06.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.403+0000 7fe334485640 1 -- 172.21.15.77:0/469534335 wait complete. 2024-03-16T10:15:06.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe334485640 1 Processor -- start 2024-03-16T10:15:06.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe334485640 1 -- start start 2024-03-16T10:15:06.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe334485640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe32c0ffac0 0x7fe32c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe334485640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe32c102a90 0x7fe32c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe334485640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe32c103450 0x7fe32c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe334485640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c10d490 con 0x7fe32c102a90 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe334485640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c10d310 con 0x7fe32c103450 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe334485640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c10d610 con 0x7fe32c0ffac0 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe331a20640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe32c102a90 0x7fe32c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe331a20640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe32c102a90 0x7fe32c1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56624/0 (socket says 172.21.15.77:56624) 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe331a20640 1 -- 172.21.15.77:0/1457835964 learned_addr learned my addr 172.21.15.77:0/1457835964 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe332221640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe32c0ffac0 0x7fe32c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe332a22640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe32c103450 0x7fe32c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe331a20640 1 -- 172.21.15.77:0/1457835964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe32c0ffac0 msgr2=0x7fe32c197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe331a20640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe32c0ffac0 0x7fe32c197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe331a20640 1 -- 172.21.15.77:0/1457835964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe32c103450 msgr2=0x7fe32c19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.404+0000 7fe331a20640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe32c103450 0x7fe32c19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.405+0000 7fe331a20640 1 -- 172.21.15.77:0/1457835964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32c19f790 con 0x7fe32c102a90 2024-03-16T10:15:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.405+0000 7fe331a20640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe32c102a90 0x7fe32c1980b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe32000e9f0 tx=0x7fe32000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.405+0000 7fe32b7fe640 1 -- 172.21.15.77:0/1457835964 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32000cd80 con 0x7fe32c102a90 2024-03-16T10:15:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.405+0000 7fe334485640 1 -- 172.21.15.77:0/1457835964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe32c19fa80 con 0x7fe32c102a90 2024-03-16T10:15:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.405+0000 7fe32b7fe640 1 -- 172.21.15.77:0/1457835964 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe320005500 con 0x7fe32c102a90 2024-03-16T10:15:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.405+0000 7fe32b7fe640 1 -- 172.21.15.77:0/1457835964 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe320010690 con 0x7fe32c102a90 2024-03-16T10:15:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.405+0000 7fe334485640 1 -- 172.21.15.77:0/1457835964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe32c19ffd0 con 0x7fe32c102a90 2024-03-16T10:15:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.406+0000 7fe334485640 1 -- 172.21.15.77:0/1457835964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe32c108c40 con 0x7fe32c102a90 2024-03-16T10:15:06.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.407+0000 7fe32b7fe640 1 -- 172.21.15.77:0/1457835964 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe3200046c0 con 0x7fe32c102a90 2024-03-16T10:15:06.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.408+0000 7fe32b7fe640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3140a2800 0x7fe3140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:06.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.408+0000 7fe332221640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3140a2800 0x7fe3140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:06.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.408+0000 7fe32b7fe640 1 -- 172.21.15.77:0/1457835964 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fe3200cecf0 con 0x7fe32c102a90 2024-03-16T10:15:06.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.408+0000 7fe332221640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3140a2800 0x7fe3140a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe32402c700 tx=0x7fe324032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:06.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.412+0000 7fe32b7fe640 1 -- 172.21.15.77:0/1457835964 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3200d3070 con 0x7fe32c102a90 2024-03-16T10:15:06.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:06 smithi192 ceph-mon[36354]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-16T10:15:06.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:06 smithi192 ceph-mon[36354]: Cluster is now healthy 2024-03-16T10:15:06.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:06 smithi192 ceph-mon[36354]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T10:15:06.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:06 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2367245108' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-16T10:15:06.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:06 smithi192 ceph-mon[36354]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T10:15:06.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:06 smithi192 ceph-mon[36354]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T10:15:06.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:06 smithi192 ceph-mon[36354]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-16T10:15:06.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:06 smithi192 ceph-mon[36354]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-16T10:15:06.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:06 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2429871990' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T10:15:06.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:06 smithi193 ceph-mon[36311]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-16T10:15:06.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:06 smithi193 ceph-mon[36311]: Cluster is now healthy 2024-03-16T10:15:06.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:06 smithi193 ceph-mon[36311]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T10:15:06.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:06 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2367245108' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-16T10:15:06.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:06 smithi193 ceph-mon[36311]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T10:15:06.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:06 smithi193 ceph-mon[36311]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T10:15:06.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:06 smithi193 ceph-mon[36311]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-16T10:15:06.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:06 smithi193 ceph-mon[36311]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-16T10:15:06.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:06 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2429871990' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T10:15:06.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:06.627+0000 7fe334485640 1 -- 172.21.15.77:0/1457835964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fe32c02d400 con 0x7fe32c102a90 2024-03-16T10:15:06.916 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:15:06 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b[67101]: 2024-03-16T10:15:06.432+0000 7f01bf458700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-16T10:15:06.916 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:15:06 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b[67101]: 2024-03-16T10:15:06.432+0000 7f01bf458700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-16T10:15:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.248+0000 7fe32b7fe640 1 -- 172.21.15.77:0/1457835964 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7fe32009bee0 con 0x7fe32c102a90 2024-03-16T10:15:07.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.252+0000 7fe334485640 1 -- 172.21.15.77:0/1457835964 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3140a2800 msgr2=0x7fe3140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:07.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.252+0000 7fe334485640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3140a2800 0x7fe3140a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe32402c700 tx=0x7fe324032590 comp rx=0 tx=0).stop 2024-03-16T10:15:07.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.252+0000 7fe334485640 1 -- 172.21.15.77:0/1457835964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe32c102a90 msgr2=0x7fe32c1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:07.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.252+0000 7fe334485640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe32c102a90 0x7fe32c1980b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe32000e9f0 tx=0x7fe32000edf0 comp rx=0 tx=0).stop 2024-03-16T10:15:07.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.252+0000 7fe334485640 1 -- 172.21.15.77:0/1457835964 shutdown_connections 2024-03-16T10:15:07.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.252+0000 7fe334485640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe32c103450 0x7fe32c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:07.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.252+0000 7fe334485640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe32c102a90 0x7fe32c1980b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:07.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.252+0000 7fe334485640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3140a2800 0x7fe3140a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:07.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.252+0000 7fe334485640 1 --2- 172.21.15.77:0/1457835964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe32c0ffac0 0x7fe32c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:07.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.252+0000 7fe334485640 1 -- 172.21.15.77:0/1457835964 >> 172.21.15.77:0/1457835964 conn(0x7fe32c0fb3f0 msgr2=0x7fe32c105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:07.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.252+0000 7fe334485640 1 -- 172.21.15.77:0/1457835964 shutdown_connections 2024-03-16T10:15:07.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.252+0000 7fe334485640 1 -- 172.21.15.77:0/1457835964 wait complete. 2024-03-16T10:15:07.269 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-16T10:15:07.270 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-16T10:15:07.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.417+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/3554663975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fe01022b0 msgr2=0x7f1fe0108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:07.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.417+0000 7f1fe6ca1640 1 --2- 172.21.15.77:0/3554663975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fe01022b0 0x7f1fe0108b40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8009ca0 tx=0x7f1fd802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:15:07.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.417+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/3554663975 shutdown_connections 2024-03-16T10:15:07.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.417+0000 7f1fe6ca1640 1 --2- 172.21.15.77:0/3554663975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fe01022b0 0x7f1fe0108b40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:07.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.417+0000 7f1fe6ca1640 1 --2- 172.21.15.77:0/3554663975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fe01018f0 0x7f1fe0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:07.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.417+0000 7f1fe6ca1640 1 --2- 172.21.15.77:0/3554663975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fe0100700 0x7f1fe0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:07.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.417+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/3554663975 >> 172.21.15.77:0/3554663975 conn(0x7f1fe00fbc90 msgr2=0x7f1fe00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:07.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.417+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/3554663975 shutdown_connections 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.417+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/3554663975 wait complete. 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fe6ca1640 1 Processor -- start 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fe6ca1640 1 -- start start 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fe6ca1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fe0100700 0x7f1fe0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fe6ca1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fe01018f0 0x7f1fe0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fe6ca1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fe01022b0 0x7f1fe019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fe6ca1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe010c380 con 0x7f1fe01018f0 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fe6ca1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe010c200 con 0x7f1fe0100700 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fe6ca1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe010c500 con 0x7f1fe01022b0 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fdffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fe01018f0 0x7f1fe0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fdffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fe01018f0 0x7f1fe0198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56648/0 (socket says 172.21.15.77:56648) 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fdffff640 1 -- 172.21.15.77:0/1746535552 learned_addr learned my addr 172.21.15.77:0/1746535552 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:07.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fe523e640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fe01022b0 0x7f1fe019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fe4a3d640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fe0100700 0x7f1fe0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fdffff640 1 -- 172.21.15.77:0/1746535552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fe01022b0 msgr2=0x7f1fe019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fdffff640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fe01022b0 0x7f1fe019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fdffff640 1 -- 172.21.15.77:0/1746535552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fe0100700 msgr2=0x7f1fe0197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fdffff640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fe0100700 0x7f1fe0197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fdffff640 1 -- 172.21.15.77:0/1746535552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe019f710 con 0x7f1fe01018f0 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fe4a3d640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fe0100700 0x7f1fe0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.418+0000 7f1fe523e640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fe01022b0 0x7f1fe019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.419+0000 7f1fdffff640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fe01018f0 0x7f1fe0198030 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400e9f0 tx=0x7f1fd400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.419+0000 7f1fddffb640 1 -- 172.21.15.77:0/1746535552 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd400cd80 con 0x7f1fe01018f0 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.419+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/1746535552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe019fa00 con 0x7f1fe01018f0 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.419+0000 7f1fddffb640 1 -- 172.21.15.77:0/1746535552 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fd4005500 con 0x7f1fe01018f0 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.419+0000 7f1fddffb640 1 -- 172.21.15.77:0/1746535552 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4010690 con 0x7f1fe01018f0 2024-03-16T10:15:07.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.419+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/1746535552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe019ff50 con 0x7f1fe01018f0 2024-03-16T10:15:07.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.420+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/1746535552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe01a01e0 con 0x7f1fe01018f0 2024-03-16T10:15:07.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.421+0000 7f1fddffb640 1 -- 172.21.15.77:0/1746535552 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1fd40046c0 con 0x7f1fe01018f0 2024-03-16T10:15:07.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.421+0000 7f1fddffb640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1fc80a2800 0x7f1fc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:07.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.421+0000 7f1fe4a3d640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1fc80a2800 0x7f1fc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:07.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.421+0000 7f1fddffb640 1 -- 172.21.15.77:0/1746535552 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f1fd40cee10 con 0x7f1fe01018f0 2024-03-16T10:15:07.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.422+0000 7f1fe4a3d640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1fc80a2800 0x7f1fc80a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0005d00 tx=0x7f1fd0005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:07.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.426+0000 7f1fddffb640 1 -- 172.21.15.77:0/1746535552 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1fd409c000 con 0x7f1fe01018f0 2024-03-16T10:15:07.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2429871990' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T10:15:07.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] up:active 2024-03-16T10:15:07.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-mon[32400]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T10:15:07.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-mon[32400]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-16T10:15:07.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-mon[32400]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-16T10:15:07.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-mon[32400]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T10:15:07.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1457835964' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T10:15:07.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-mon[32400]: pgmap v289: 97 pgs: 18 creating+peering, 22 unknown, 57 active+clean; 580 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 3 op/s 2024-03-16T10:15:07.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:07 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2429871990' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T10:15:07.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:07 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] up:active 2024-03-16T10:15:07.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:07 smithi193 ceph-mon[36311]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T10:15:07.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:07 smithi193 ceph-mon[36311]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-16T10:15:07.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:07 smithi193 ceph-mon[36311]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-16T10:15:07.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:07 smithi193 ceph-mon[36311]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T10:15:07.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:07 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1457835964' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T10:15:07.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:07 smithi193 ceph-mon[36311]: pgmap v289: 97 pgs: 18 creating+peering, 22 unknown, 57 active+clean; 580 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 3 op/s 2024-03-16T10:15:07.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:07.625+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/1746535552 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f1fe006a4b0 con 0x7f1fc80a2800 2024-03-16T10:15:07.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:07 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2429871990' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T10:15:07.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:07 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] up:active 2024-03-16T10:15:07.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:07 smithi192 ceph-mon[36354]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T10:15:07.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:07 smithi192 ceph-mon[36354]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-16T10:15:07.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:07 smithi192 ceph-mon[36354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-16T10:15:07.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:07 smithi192 ceph-mon[36354]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T10:15:07.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:07 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1457835964' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T10:15:07.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:07 smithi192 ceph-mon[36354]: pgmap v289: 97 pgs: 18 creating+peering, 22 unknown, 57 active+clean; 580 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 3 op/s 2024-03-16T10:15:07.995 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:15:07.630+0000 7f645e2f7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T10:15:07.995 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:15:07.630+0000 7f645e2f7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T10:15:07.995 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:15:07.630+0000 7f645e2f7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T10:15:07.995 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:15:07.630+0000 7f645e2f7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T10:15:07.995 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:15:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:15:07.630+0000 7f645e2f7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T10:15:08.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1457835964' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T10:15:08.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] up:active 2024-03-16T10:15:08.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-16T10:15:08.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-16T10:15:08.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-03-16T10:15:08.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-03-16T10:15:08.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:08.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] up:active 2024-03-16T10:15:08.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-16T10:15:08.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T10:15:08.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: Cluster is now healthy 2024-03-16T10:15:08.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2024-03-16T10:15:08.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-16T10:15:08.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:08 smithi193 ceph-mon[36311]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1457835964' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] up:active 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] up:active 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: Cluster is now healthy 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2024-03-16T10:15:08.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-16T10:15:08.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:08 smithi192 ceph-mon[36354]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-03-16T10:15:08.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1457835964' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T10:15:08.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] up:active 2024-03-16T10:15:08.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-16T10:15:08.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-16T10:15:08.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-03-16T10:15:08.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-03-16T10:15:08.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:08.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] up:active 2024-03-16T10:15:08.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-16T10:15:08.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T10:15:08.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: Cluster is now healthy 2024-03-16T10:15:08.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2024-03-16T10:15:08.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-16T10:15:08.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:08 smithi077 ceph-mon[32400]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-03-16T10:15:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.234+0000 7f1fddffb640 1 -- 172.21.15.77:0/1746535552 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1fe006a4b0 con 0x7f1fc80a2800 2024-03-16T10:15:09.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.237+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/1746535552 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1fc80a2800 msgr2=0x7f1fc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.237+0000 7f1fe6ca1640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1fc80a2800 0x7f1fc80a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0005d00 tx=0x7f1fd0005c90 comp rx=0 tx=0).stop 2024-03-16T10:15:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.237+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/1746535552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fe01018f0 msgr2=0x7f1fe0198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.237+0000 7f1fe6ca1640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fe01018f0 0x7f1fe0198030 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400e9f0 tx=0x7f1fd400edf0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.238+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/1746535552 shutdown_connections 2024-03-16T10:15:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.238+0000 7f1fe6ca1640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1fc80a2800 0x7f1fc80a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.238+0000 7f1fe6ca1640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fe01022b0 0x7f1fe019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.238+0000 7f1fe6ca1640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fe01018f0 0x7f1fe0198030 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.238+0000 7f1fe6ca1640 1 --2- 172.21.15.77:0/1746535552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fe0100700 0x7f1fe0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.238+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/1746535552 >> 172.21.15.77:0/1746535552 conn(0x7f1fe00fbc90 msgr2=0x7f1fe0069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.238+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/1746535552 shutdown_connections 2024-03-16T10:15:09.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.238+0000 7f1fe6ca1640 1 -- 172.21.15.77:0/1746535552 wait complete. 2024-03-16T10:15:09.253 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-16T10:15:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.365+0000 7f21da868640 1 -- 172.21.15.77:0/335111940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d41018d0 msgr2=0x7f21d4101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.365+0000 7f21da868640 1 --2- 172.21.15.77:0/335111940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d41018d0 0x7f21d4101d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f21c8009900 tx=0x7f21c802bc30 comp rx=0 tx=0).stop 2024-03-16T10:15:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.365+0000 7f21da868640 1 -- 172.21.15.77:0/335111940 shutdown_connections 2024-03-16T10:15:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.365+0000 7f21da868640 1 --2- 172.21.15.77:0/335111940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21d4102290 0x7f21d4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.365+0000 7f21da868640 1 --2- 172.21.15.77:0/335111940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d41018d0 0x7f21d4101d50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.365+0000 7f21da868640 1 --2- 172.21.15.77:0/335111940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21d41006e0 0x7f21d4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.365+0000 7f21da868640 1 -- 172.21.15.77:0/335111940 >> 172.21.15.77:0/335111940 conn(0x7f21d40fbc70 msgr2=0x7f21d40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.365+0000 7f21da868640 1 -- 172.21.15.77:0/335111940 shutdown_connections 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.365+0000 7f21da868640 1 -- 172.21.15.77:0/335111940 wait complete. 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21da868640 1 Processor -- start 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21da868640 1 -- start start 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21da868640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21d41006e0 0x7f21d4199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21da868640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d41018d0 0x7f21d419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21da868640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21d4102290 0x7f21d41a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21da868640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410d480 con 0x7f21d41018d0 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21da868640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410d300 con 0x7f21d4102290 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21da868640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f21d410d600 con 0x7f21d41006e0 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d41018d0 0x7f21d419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d41018d0 0x7f21d419a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56686/0 (socket says 172.21.15.77:56686) 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21d37fe640 1 -- 172.21.15.77:0/1373310048 learned_addr learned my addr 172.21.15.77:0/1373310048 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:09.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21d3fff640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21d41006e0 0x7f21d4199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.366+0000 7f21d8e05640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21d4102290 0x7f21d41a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.367+0000 7f21d37fe640 1 -- 172.21.15.77:0/1373310048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21d41006e0 msgr2=0x7f21d4199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.367+0000 7f21d37fe640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21d41006e0 0x7f21d4199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.367+0000 7f21d37fe640 1 -- 172.21.15.77:0/1373310048 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21d4102290 msgr2=0x7f21d41a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.367+0000 7f21d37fe640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21d4102290 0x7f21d41a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.367+0000 7f21d37fe640 1 -- 172.21.15.77:0/1373310048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d41a1940 con 0x7f21d41018d0 2024-03-16T10:15:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.367+0000 7f21d37fe640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d41018d0 0x7f21d419a270 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f21c8009230 tx=0x7f21c8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.367+0000 7f21d17fa640 1 -- 172.21.15.77:0/1373310048 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8041070 con 0x7f21d41018d0 2024-03-16T10:15:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.367+0000 7f21d17fa640 1 -- 172.21.15.77:0/1373310048 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21c80044d0 con 0x7f21d41018d0 2024-03-16T10:15:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.367+0000 7f21da868640 1 -- 172.21.15.77:0/1373310048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d41a1bd0 con 0x7f21d41018d0 2024-03-16T10:15:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.367+0000 7f21d17fa640 1 -- 172.21.15.77:0/1373310048 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8034440 con 0x7f21d41018d0 2024-03-16T10:15:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.367+0000 7f21da868640 1 -- 172.21.15.77:0/1373310048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d41a2110 con 0x7f21d41018d0 2024-03-16T10:15:09.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.368+0000 7f21da868640 1 -- 172.21.15.77:0/1373310048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d41a22e0 con 0x7f21d41018d0 2024-03-16T10:15:09.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.369+0000 7f21d17fa640 1 -- 172.21.15.77:0/1373310048 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f21c8044070 con 0x7f21d41018d0 2024-03-16T10:15:09.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.370+0000 7f21d17fa640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f21bc0a2880 0x7f21bc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:09.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.370+0000 7f21d3fff640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f21bc0a2880 0x7f21bc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:09.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.370+0000 7f21d17fa640 1 -- 172.21.15.77:0/1373310048 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f21c80f0880 con 0x7f21d41018d0 2024-03-16T10:15:09.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.370+0000 7f21d3fff640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f21bc0a2880 0x7f21bc0a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f21d419b200 tx=0x7f21c400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.374+0000 7f21d17fa640 1 -- 172.21.15.77:0/1373310048 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f21c80bd420 con 0x7f21d41018d0 2024-03-16T10:15:09.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:09 smithi192 ceph-mon[36354]: pgmap v290: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 6.5 KiB/s wr, 23 op/s 2024-03-16T10:15:09.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:09 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] up:active 2024-03-16T10:15:09.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:09 smithi192 ceph-mon[36354]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-16T10:15:09.568 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:09 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:15:09.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.582+0000 7f21da868640 1 -- 172.21.15.77:0/1373310048 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f21d410ac00 con 0x7f21bc0a2880 2024-03-16T10:15:09.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.603+0000 7f21d17fa640 1 -- 172.21.15.77:0/1373310048 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f21d410ac00 con 0x7f21bc0a2880 2024-03-16T10:15:09.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.605+0000 7f21da868640 1 -- 172.21.15.77:0/1373310048 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f21bc0a2880 msgr2=0x7f21bc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.605+0000 7f21da868640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f21bc0a2880 0x7f21bc0a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f21d419b200 tx=0x7f21c400b390 comp rx=0 tx=0).stop 2024-03-16T10:15:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.605+0000 7f21da868640 1 -- 172.21.15.77:0/1373310048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d41018d0 msgr2=0x7f21d419a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.605+0000 7f21da868640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d41018d0 0x7f21d419a270 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f21c8009230 tx=0x7f21c8005d50 comp rx=0 tx=0).stop 2024-03-16T10:15:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.605+0000 7f21da868640 1 -- 172.21.15.77:0/1373310048 shutdown_connections 2024-03-16T10:15:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.605+0000 7f21da868640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21d4102290 0x7f21d41a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.605+0000 7f21da868640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21d41018d0 0x7f21d419a270 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.605+0000 7f21da868640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f21bc0a2880 0x7f21bc0a4d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.605+0000 7f21da868640 1 --2- 172.21.15.77:0/1373310048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21d41006e0 0x7f21d4199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.605+0000 7f21da868640 1 -- 172.21.15.77:0/1373310048 >> 172.21.15.77:0/1373310048 conn(0x7f21d40fbc70 msgr2=0x7f21d41094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.605+0000 7f21da868640 1 -- 172.21.15.77:0/1373310048 shutdown_connections 2024-03-16T10:15:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.605+0000 7f21da868640 1 -- 172.21.15.77:0/1373310048 wait complete. 2024-03-16T10:15:09.615 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:09 smithi077 ceph-mon[32400]: pgmap v290: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 6.5 KiB/s wr, 23 op/s 2024-03-16T10:15:09.615 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:09 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] up:active 2024-03-16T10:15:09.615 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:09 smithi077 ceph-mon[32400]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-16T10:15:09.616 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:09 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:15:09.616 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-16T10:15:09.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.723+0000 7f4435a0c640 1 -- 172.21.15.77:0/1827260659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44301006e0 msgr2=0x7f4430100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:09.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.723+0000 7f4435a0c640 1 --2- 172.21.15.77:0/1827260659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44301006e0 0x7f4430100b00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4420009900 tx=0x7f442002bc30 comp rx=0 tx=0).stop 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.723+0000 7f4435a0c640 1 -- 172.21.15.77:0/1827260659 shutdown_connections 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.723+0000 7f4435a0c640 1 --2- 172.21.15.77:0/1827260659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4430102290 0x7f4430108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.723+0000 7f4435a0c640 1 --2- 172.21.15.77:0/1827260659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44301018d0 0x7f4430101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.723+0000 7f4435a0c640 1 --2- 172.21.15.77:0/1827260659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44301006e0 0x7f4430100b00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.723+0000 7f4435a0c640 1 -- 172.21.15.77:0/1827260659 >> 172.21.15.77:0/1827260659 conn(0x7f44300fbc70 msgr2=0x7f44300fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.723+0000 7f4435a0c640 1 -- 172.21.15.77:0/1827260659 shutdown_connections 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.723+0000 7f4435a0c640 1 -- 172.21.15.77:0/1827260659 wait complete. 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4435a0c640 1 Processor -- start 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4435a0c640 1 -- start start 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4435a0c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44301006e0 0x7f4430195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4435a0c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44301018d0 0x7f4430195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4435a0c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4430102290 0x7f443019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4435a0c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f443010b260 con 0x7f44301018d0 2024-03-16T10:15:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4435a0c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f443010b0e0 con 0x7f4430102290 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4435a0c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f443010b3e0 con 0x7f44301006e0 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4427fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44301018d0 0x7f4430195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4427fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44301018d0 0x7f4430195e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56712/0 (socket says 172.21.15.77:56712) 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4427fff640 1 -- 172.21.15.77:0/1382816073 learned_addr learned my addr 172.21.15.77:0/1382816073 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f442effd640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44301006e0 0x7f4430195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f442f7fe640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4430102290 0x7f443019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4427fff640 1 -- 172.21.15.77:0/1382816073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44301006e0 msgr2=0x7f4430195940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4427fff640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44301006e0 0x7f4430195940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4427fff640 1 -- 172.21.15.77:0/1382816073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4430102290 msgr2=0x7f443019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4427fff640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4430102290 0x7f443019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4427fff640 1 -- 172.21.15.77:0/1382816073 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443019d560 con 0x7f44301018d0 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.724+0000 7f4427fff640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44301018d0 0x7f4430195e80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4418010ad0 tx=0x7f4418010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.725+0000 7f4434a0a640 1 -- 172.21.15.77:0/1382816073 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441801a020 con 0x7f44301018d0 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.725+0000 7f4434a0a640 1 -- 172.21.15.77:0/1382816073 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f441800ec30 con 0x7f44301018d0 2024-03-16T10:15:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.725+0000 7f4435a0c640 1 -- 172.21.15.77:0/1382816073 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443019d850 con 0x7f44301018d0 2024-03-16T10:15:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.725+0000 7f4434a0a640 1 -- 172.21.15.77:0/1382816073 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418015760 con 0x7f44301018d0 2024-03-16T10:15:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.725+0000 7f4435a0c640 1 -- 172.21.15.77:0/1382816073 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f443019ddf0 con 0x7f44301018d0 2024-03-16T10:15:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.726+0000 7f4435a0c640 1 -- 172.21.15.77:0/1382816073 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4430107aa0 con 0x7f44301018d0 2024-03-16T10:15:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.727+0000 7f4434a0a640 1 -- 172.21.15.77:0/1382816073 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f441801d020 con 0x7f44301018d0 2024-03-16T10:15:09.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.727+0000 7f4434a0a640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f441c0a2880 0x7f441c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:09.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.727+0000 7f442effd640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f441c0a2880 0x7f441c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:09.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.728+0000 7f4434a0a640 1 -- 172.21.15.77:0/1382816073 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f44180d1110 con 0x7f44301018d0 2024-03-16T10:15:09.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.728+0000 7f442effd640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f441c0a2880 0x7f441c0a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4420000c00 tx=0x7f4420005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:09.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.732+0000 7f4434a0a640 1 -- 172.21.15.77:0/1382816073 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4418024020 con 0x7f44301018d0 2024-03-16T10:15:09.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:09 smithi193 ceph-mon[36311]: pgmap v290: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 6.5 KiB/s wr, 23 op/s 2024-03-16T10:15:09.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:09 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] up:active 2024-03-16T10:15:09.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:09 smithi193 ceph-mon[36311]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-16T10:15:09.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:09 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:15:09.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.939+0000 7f4435a0c640 1 -- 172.21.15.77:0/1382816073 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f4430061c20 con 0x7f441c0a2880 2024-03-16T10:15:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.943+0000 7f4434a0a640 1 -- 172.21.15.77:0/1382816073 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4430061c20 con 0x7f441c0a2880 2024-03-16T10:15:09.944 INFO:teuthology.orchestra.run.smithi077.stdout:/volumes/qa/sv_0/424f9112-197c-46f8-aa95-5f1fcc345df4 2024-03-16T10:15:09.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.945+0000 7f4435a0c640 1 -- 172.21.15.77:0/1382816073 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f441c0a2880 msgr2=0x7f441c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:09.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.945+0000 7f4435a0c640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f441c0a2880 0x7f441c0a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4420000c00 tx=0x7f4420005fd0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.945+0000 7f4435a0c640 1 -- 172.21.15.77:0/1382816073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44301018d0 msgr2=0x7f4430195e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.945+0000 7f4435a0c640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44301018d0 0x7f4430195e80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4418010ad0 tx=0x7f4418010f90 comp rx=0 tx=0).stop 2024-03-16T10:15:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.946+0000 7f4435a0c640 1 -- 172.21.15.77:0/1382816073 shutdown_connections 2024-03-16T10:15:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.946+0000 7f4435a0c640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4430102290 0x7f443019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.946+0000 7f4435a0c640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44301018d0 0x7f4430195e80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.946+0000 7f4435a0c640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f441c0a2880 0x7f441c0a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.946+0000 7f4435a0c640 1 --2- 172.21.15.77:0/1382816073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44301006e0 0x7f4430195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.946+0000 7f4435a0c640 1 -- 172.21.15.77:0/1382816073 >> 172.21.15.77:0/1382816073 conn(0x7f44300fbc70 msgr2=0x7f4430104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.946+0000 7f4435a0c640 1 -- 172.21.15.77:0/1382816073 shutdown_connections 2024-03-16T10:15:09.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:09.946+0000 7f4435a0c640 1 -- 172.21.15.77:0/1382816073 wait complete. 2024-03-16T10:15:09.957 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-16T10:15:10.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.066+0000 7f4eb955b640 1 -- 172.21.15.77:0/1970611379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb4101810 msgr2=0x7f4eb4101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.066+0000 7f4eb955b640 1 --2- 172.21.15.77:0/1970611379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb4101810 0x7f4eb4101c30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009900 tx=0x7f4ea402bc30 comp rx=0 tx=0).stop 2024-03-16T10:15:10.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.066+0000 7f4eb955b640 1 -- 172.21.15.77:0/1970611379 shutdown_connections 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.066+0000 7f4eb955b640 1 --2- 172.21.15.77:0/1970611379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb41033c0 0x7f4eb4109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.066+0000 7f4eb955b640 1 --2- 172.21.15.77:0/1970611379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4eb4102a00 0x7f4eb4102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.066+0000 7f4eb955b640 1 --2- 172.21.15.77:0/1970611379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb4101810 0x7f4eb4101c30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.066+0000 7f4eb955b640 1 -- 172.21.15.77:0/1970611379 >> 172.21.15.77:0/1970611379 conn(0x7f4eb40fcd80 msgr2=0x7f4eb40ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.066+0000 7f4eb955b640 1 -- 172.21.15.77:0/1970611379 shutdown_connections 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.066+0000 7f4eb955b640 1 -- 172.21.15.77:0/1970611379 wait complete. 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eb955b640 1 Processor -- start 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eb955b640 1 -- start start 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eb955b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb4101810 0x7f4eb4197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eb955b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4eb4102a00 0x7f4eb4198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eb955b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb41033c0 0x7f4eb419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eb955b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb410d460 con 0x7f4eb41033c0 2024-03-16T10:15:10.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eb955b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb410d2e0 con 0x7f4eb4101810 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eb955b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb410d5e0 con 0x7f4eb4102a00 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eb37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb41033c0 0x7f4eb419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eb37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb41033c0 0x7f4eb419f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56728/0 (socket says 172.21.15.77:56728) 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eb37fe640 1 -- 172.21.15.77:0/943353254 learned_addr learned my addr 172.21.15.77:0/943353254 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eb2ffd640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb4101810 0x7f4eb4197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.067+0000 7f4eaa7fc640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4eb4102a00 0x7f4eb4198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.068+0000 7f4eb37fe640 1 -- 172.21.15.77:0/943353254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4eb4102a00 msgr2=0x7f4eb4198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.068+0000 7f4eb37fe640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4eb4102a00 0x7f4eb4198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.068+0000 7f4eb37fe640 1 -- 172.21.15.77:0/943353254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb4101810 msgr2=0x7f4eb4197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.068+0000 7f4eb37fe640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb4101810 0x7f4eb4197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.068+0000 7f4eb37fe640 1 -- 172.21.15.77:0/943353254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eb419f760 con 0x7f4eb41033c0 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.068+0000 7f4eaa7fc640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4eb4102a00 0x7f4eb4198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.068+0000 7f4eb37fe640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb41033c0 0x7f4eb419f0f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4eac00eb10 tx=0x7f4eac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:10.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.068+0000 7f4eb0ff9640 1 -- 172.21.15.77:0/943353254 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac004b20 con 0x7f4eb41033c0 2024-03-16T10:15:10.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.068+0000 7f4eb0ff9640 1 -- 172.21.15.77:0/943353254 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eac004cc0 con 0x7f4eb41033c0 2024-03-16T10:15:10.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.068+0000 7f4eb955b640 1 -- 172.21.15.77:0/943353254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eb419fa50 con 0x7f4eb41033c0 2024-03-16T10:15:10.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.068+0000 7f4eb0ff9640 1 -- 172.21.15.77:0/943353254 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac010760 con 0x7f4eb41033c0 2024-03-16T10:15:10.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.068+0000 7f4eb955b640 1 -- 172.21.15.77:0/943353254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eb419ffa0 con 0x7f4eb41033c0 2024-03-16T10:15:10.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.069+0000 7f4eb955b640 1 -- 172.21.15.77:0/943353254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eb4108b50 con 0x7f4eb41033c0 2024-03-16T10:15:10.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.070+0000 7f4eb0ff9640 1 -- 172.21.15.77:0/943353254 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4eac005020 con 0x7f4eb41033c0 2024-03-16T10:15:10.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.070+0000 7f4eb0ff9640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4ea00a2880 0x7f4ea00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:10.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.071+0000 7f4eb2ffd640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4ea00a2880 0x7f4ea00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:10.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.071+0000 7f4eb0ff9640 1 -- 172.21.15.77:0/943353254 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4eac0ceea0 con 0x7f4eb41033c0 2024-03-16T10:15:10.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.071+0000 7f4eb2ffd640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4ea00a2880 0x7f4ea00a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4000c00 tx=0x7f4ea4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:10.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.078+0000 7f4eb0ff9640 1 -- 172.21.15.77:0/943353254 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4eac09c090 con 0x7f4eb41033c0 2024-03-16T10:15:10.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.300+0000 7f4eb955b640 1 -- 172.21.15.77:0/943353254 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4eb4061c20 con 0x7f4ea00a2880 2024-03-16T10:15:10.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.324+0000 7f4eb0ff9640 1 -- 172.21.15.77:0/943353254 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4eb4061c20 con 0x7f4ea00a2880 2024-03-16T10:15:10.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.326+0000 7f4eb955b640 1 -- 172.21.15.77:0/943353254 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4ea00a2880 msgr2=0x7f4ea00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.326+0000 7f4eb955b640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4ea00a2880 0x7f4ea00a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4000c00 tx=0x7f4ea4005fd0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.326+0000 7f4eb955b640 1 -- 172.21.15.77:0/943353254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb41033c0 msgr2=0x7f4eb419f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.326+0000 7f4eb955b640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb41033c0 0x7f4eb419f0f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4eac00eb10 tx=0x7f4eac00c2d0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.326+0000 7f4eb955b640 1 -- 172.21.15.77:0/943353254 shutdown_connections 2024-03-16T10:15:10.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.326+0000 7f4eb955b640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4eb41033c0 0x7f4eb419f0f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.326+0000 7f4eb955b640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4ea00a2880 0x7f4ea00a4d40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.326+0000 7f4eb955b640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4eb4102a00 0x7f4eb4198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.326+0000 7f4eb955b640 1 --2- 172.21.15.77:0/943353254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb4101810 0x7f4eb4197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.326+0000 7f4eb955b640 1 -- 172.21.15.77:0/943353254 >> 172.21.15.77:0/943353254 conn(0x7f4eb40fcd80 msgr2=0x7f4eb4105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:10.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.326+0000 7f4eb955b640 1 -- 172.21.15.77:0/943353254 shutdown_connections 2024-03-16T10:15:10.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.326+0000 7f4eb955b640 1 -- 172.21.15.77:0/943353254 wait complete. 2024-03-16T10:15:10.338 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-16T10:15:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.445+0000 7fde754c0640 1 -- 172.21.15.77:0/3910446178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde701074c0 msgr2=0x7fde70109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.445+0000 7fde754c0640 1 --2- 172.21.15.77:0/3910446178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde701074c0 0x7fde70109910 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fde68009ca0 tx=0x7fde6802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.445+0000 7fde754c0640 1 -- 172.21.15.77:0/3910446178 shutdown_connections 2024-03-16T10:15:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.445+0000 7fde754c0640 1 --2- 172.21.15.77:0/3910446178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde701074c0 0x7fde70109910 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.445+0000 7fde754c0640 1 --2- 172.21.15.77:0/3910446178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde70104b90 0x7fde70106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.445+0000 7fde754c0640 1 --2- 172.21.15.77:0/3910446178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde70102260 0x7fde70104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.445+0000 7fde754c0640 1 -- 172.21.15.77:0/3910446178 >> 172.21.15.77:0/3910446178 conn(0x7fde700fbc70 msgr2=0x7fde700fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.445+0000 7fde754c0640 1 -- 172.21.15.77:0/3910446178 shutdown_connections 2024-03-16T10:15:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.445+0000 7fde754c0640 1 -- 172.21.15.77:0/3910446178 wait complete. 2024-03-16T10:15:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.446+0000 7fde754c0640 1 Processor -- start 2024-03-16T10:15:10.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.446+0000 7fde754c0640 1 -- start start 2024-03-16T10:15:10.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde754c0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde70102260 0x7fde70197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde754c0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde70104b90 0x7fde70198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde754c0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde701074c0 0x7fde7019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde754c0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fde7010d480 con 0x7fde70104b90 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde754c0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fde7010d300 con 0x7fde70102260 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde754c0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fde7010d600 con 0x7fde701074c0 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde6e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde70104b90 0x7fde70198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde6e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde70104b90 0x7fde70198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56740/0 (socket says 172.21.15.77:56740) 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde6e7fc640 1 -- 172.21.15.77:0/4219352247 learned_addr learned my addr 172.21.15.77:0/4219352247 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde6effd640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde70102260 0x7fde70197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde6f7fe640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde701074c0 0x7fde7019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde6e7fc640 1 -- 172.21.15.77:0/4219352247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde701074c0 msgr2=0x7fde7019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde6e7fc640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde701074c0 0x7fde7019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde6e7fc640 1 -- 172.21.15.77:0/4219352247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde70102260 msgr2=0x7fde70197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde6e7fc640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde70102260 0x7fde70197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde6e7fc640 1 -- 172.21.15.77:0/4219352247 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde7019f750 con 0x7fde70104b90 2024-03-16T10:15:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde6e7fc640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde70104b90 0x7fde70198070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fde6400e9f0 tx=0x7fde6400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde57fff640 1 -- 172.21.15.77:0/4219352247 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6400ccc0 con 0x7fde70104b90 2024-03-16T10:15:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde57fff640 1 -- 172.21.15.77:0/4219352247 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde64005500 con 0x7fde70104b90 2024-03-16T10:15:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde754c0640 1 -- 172.21.15.77:0/4219352247 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde7019fa40 con 0x7fde70104b90 2024-03-16T10:15:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde57fff640 1 -- 172.21.15.77:0/4219352247 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde64010690 con 0x7fde70104b90 2024-03-16T10:15:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.447+0000 7fde754c0640 1 -- 172.21.15.77:0/4219352247 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde7019ff90 con 0x7fde70104b90 2024-03-16T10:15:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.448+0000 7fde754c0640 1 -- 172.21.15.77:0/4219352247 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde700fd760 con 0x7fde70104b90 2024-03-16T10:15:10.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.449+0000 7fde57fff640 1 -- 172.21.15.77:0/4219352247 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fde640046c0 con 0x7fde70104b90 2024-03-16T10:15:10.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.450+0000 7fde57fff640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fde580a2880 0x7fde580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:10.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.450+0000 7fde6effd640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fde580a2880 0x7fde580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:10.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.450+0000 7fde57fff640 1 -- 172.21.15.77:0/4219352247 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fde640cec10 con 0x7fde70104b90 2024-03-16T10:15:10.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.450+0000 7fde6effd640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fde580a2880 0x7fde580a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde60009bd0 tx=0x7fde60005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:10.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.454+0000 7fde57fff640 1 -- 172.21.15.77:0/4219352247 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fde6409be00 con 0x7fde70104b90 2024-03-16T10:15:10.579 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:10 smithi192 ceph-mon[36354]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:10.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.663+0000 7fde754c0640 1 -- 172.21.15.77:0/4219352247 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fde70061c20 con 0x7fde580a2880 2024-03-16T10:15:10.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.667+0000 7fde57fff640 1 -- 172.21.15.77:0/4219352247 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fde70061c20 con 0x7fde580a2880 2024-03-16T10:15:10.667 INFO:teuthology.orchestra.run.smithi077.stdout:/volumes/qa/sv_1/7eff604f-057e-4c66-8c0e-f7c4c1eee205 2024-03-16T10:15:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.669+0000 7fde754c0640 1 -- 172.21.15.77:0/4219352247 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fde580a2880 msgr2=0x7fde580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.669+0000 7fde754c0640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fde580a2880 0x7fde580a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde60009bd0 tx=0x7fde60005f90 comp rx=0 tx=0).stop 2024-03-16T10:15:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.669+0000 7fde754c0640 1 -- 172.21.15.77:0/4219352247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde70104b90 msgr2=0x7fde70198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.669+0000 7fde754c0640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde70104b90 0x7fde70198070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fde6400e9f0 tx=0x7fde6400eeb0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.669+0000 7fde754c0640 1 -- 172.21.15.77:0/4219352247 shutdown_connections 2024-03-16T10:15:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.669+0000 7fde754c0640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fde580a2880 0x7fde580a4d40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.669+0000 7fde754c0640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde701074c0 0x7fde7019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.669+0000 7fde754c0640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde70104b90 0x7fde70198070 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.669+0000 7fde754c0640 1 --2- 172.21.15.77:0/4219352247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde70102260 0x7fde70197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.669+0000 7fde754c0640 1 -- 172.21.15.77:0/4219352247 >> 172.21.15.77:0/4219352247 conn(0x7fde700fbc70 msgr2=0x7fde70102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:10.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.669+0000 7fde754c0640 1 -- 172.21.15.77:0/4219352247 shutdown_connections 2024-03-16T10:15:10.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.669+0000 7fde754c0640 1 -- 172.21.15.77:0/4219352247 wait complete. 2024-03-16T10:15:10.680 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:10 smithi077 ceph-mon[32400]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:10.681 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:15:10.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:10 smithi193 ceph-mon[36311]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:10.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.784+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2068939365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f00ffae0 msgr2=0x7f00f00fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.784+0000 7f00f7f7e640 1 --2- 172.21.15.77:0/2068939365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f00ffae0 0x7f00f00fff00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009930 tx=0x7f00ec02bc30 comp rx=0 tx=0).stop 2024-03-16T10:15:10.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2068939365 shutdown_connections 2024-03-16T10:15:10.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 --2- 172.21.15.77:0/2068939365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f0103470 0x7f00f0109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 --2- 172.21.15.77:0/2068939365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00f0102ab0 0x7f00f0102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 --2- 172.21.15.77:0/2068939365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f00ffae0 0x7f00f00fff00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2068939365 >> 172.21.15.77:0/2068939365 conn(0x7f00f00fb410 msgr2=0x7f00f00fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2068939365 shutdown_connections 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2068939365 wait complete. 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 Processor -- start 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 -- start start 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f00ffae0 0x7f00f0199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00f0102ab0 0x7f00f019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f0103470 0x7f00f01a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f00f010d4b0 con 0x7f00f00ffae0 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f00f010d330 con 0x7f00f0103470 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f7f7e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f00f010d630 con 0x7f00f0102ab0 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f5d1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f00ffae0 0x7f00f0199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f5d1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f00ffae0 0x7f00f0199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56764/0 (socket says 172.21.15.77:56764) 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f5d1a640 1 -- 172.21.15.77:0/2632807412 learned_addr learned my addr 172.21.15.77:0/2632807412 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f651b640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f0103470 0x7f00f01a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.785+0000 7f00f5519640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00f0102ab0 0x7f00f019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:10.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.786+0000 7f00f5d1a640 1 -- 172.21.15.77:0/2632807412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00f0102ab0 msgr2=0x7f00f019a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.786+0000 7f00f5d1a640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00f0102ab0 0x7f00f019a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.786+0000 7f00f5d1a640 1 -- 172.21.15.77:0/2632807412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f0103470 msgr2=0x7f00f01a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:10.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.786+0000 7f00f5d1a640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f0103470 0x7f00f01a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:10.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.786+0000 7f00f5d1a640 1 -- 172.21.15.77:0/2632807412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f01a19d0 con 0x7f00f00ffae0 2024-03-16T10:15:10.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.786+0000 7f00f5d1a640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f00ffae0 0x7f00f0199dc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f00ec02c7b0 tx=0x7f00ec02c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:10.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.786+0000 7f00e6ffd640 1 -- 172.21.15.77:0/2632807412 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec041070 con 0x7f00f00ffae0 2024-03-16T10:15:10.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.786+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2632807412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f01a1c60 con 0x7f00f00ffae0 2024-03-16T10:15:10.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.786+0000 7f00e6ffd640 1 -- 172.21.15.77:0/2632807412 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00ec0054e0 con 0x7f00f00ffae0 2024-03-16T10:15:10.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.786+0000 7f00e6ffd640 1 -- 172.21.15.77:0/2632807412 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec034440 con 0x7f00f00ffae0 2024-03-16T10:15:10.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.786+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2632807412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f01a2150 con 0x7f00f00ffae0 2024-03-16T10:15:10.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.787+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2632807412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00f01a2320 con 0x7f00f00ffae0 2024-03-16T10:15:10.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.788+0000 7f00e6ffd640 1 -- 172.21.15.77:0/2632807412 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f00ec044070 con 0x7f00f00ffae0 2024-03-16T10:15:10.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.789+0000 7f00e6ffd640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00dc0a2800 0x7f00dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:10.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.789+0000 7f00f5519640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00dc0a2800 0x7f00dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:10.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.789+0000 7f00e6ffd640 1 -- 172.21.15.77:0/2632807412 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f00ec0f0480 con 0x7f00f00ffae0 2024-03-16T10:15:10.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.789+0000 7f00f5519640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00dc0a2800 0x7f00dc0a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009bd0 tx=0x7f00e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:10.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:10.793+0000 7f00e6ffd640 1 -- 172.21.15.77:0/2632807412 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f00ec0bd020 con 0x7f00f00ffae0 2024-03-16T10:15:11.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.041+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2632807412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00f01a24f0 con 0x7f00f00ffae0 2024-03-16T10:15:11.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.042+0000 7f00e6ffd640 1 -- 172.21.15.77:0/2632807412 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11938 (secure 0 0 0) 0x7f00ec0c10c0 con 0x7f00f00ffae0 2024-03-16T10:15:11.043 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:15:11.043 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:15:09.227516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[14658,24455,24482,24454,24475]},"id":1}]} 2024-03-16T10:15:11.043 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-16T10:15:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.044+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2632807412 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00dc0a2800 msgr2=0x7f00dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.044+0000 7f00f7f7e640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00dc0a2800 0x7f00dc0a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009bd0 tx=0x7f00e0005f90 comp rx=0 tx=0).stop 2024-03-16T10:15:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.044+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2632807412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f00ffae0 msgr2=0x7f00f0199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.044+0000 7f00f7f7e640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f00ffae0 0x7f00f0199dc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f00ec02c7b0 tx=0x7f00ec02c6d0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.044+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2632807412 shutdown_connections 2024-03-16T10:15:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.044+0000 7f00f7f7e640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f0103470 0x7f00f01a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.044+0000 7f00f7f7e640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f00dc0a2800 0x7f00dc0a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.044+0000 7f00f7f7e640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00f0102ab0 0x7f00f019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.044+0000 7f00f7f7e640 1 --2- 172.21.15.77:0/2632807412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f00ffae0 0x7f00f0199dc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.044+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2632807412 >> 172.21.15.77:0/2632807412 conn(0x7f00f00fb410 msgr2=0x7f00f01008a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.044+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2632807412 shutdown_connections 2024-03-16T10:15:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.044+0000 7f00f7f7e640 1 -- 172.21.15.77:0/2632807412 wait complete. 2024-03-16T10:15:11.056 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:15:11.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.157+0000 7fc7aac27640 1 -- 172.21.15.77:0/2183755196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a41022b0 msgr2=0x7fc7a4108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.157+0000 7fc7aac27640 1 --2- 172.21.15.77:0/2183755196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a41022b0 0x7fc7a4108b40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009ca0 tx=0x7fc79c02c0f0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.157+0000 7fc7aac27640 1 -- 172.21.15.77:0/2183755196 shutdown_connections 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.157+0000 7fc7aac27640 1 --2- 172.21.15.77:0/2183755196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a41022b0 0x7fc7a4108b40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.157+0000 7fc7aac27640 1 --2- 172.21.15.77:0/2183755196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a41018f0 0x7fc7a4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.157+0000 7fc7aac27640 1 --2- 172.21.15.77:0/2183755196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a4100700 0x7fc7a4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.157+0000 7fc7aac27640 1 -- 172.21.15.77:0/2183755196 >> 172.21.15.77:0/2183755196 conn(0x7fc7a40fbc70 msgr2=0x7fc7a40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.157+0000 7fc7aac27640 1 -- 172.21.15.77:0/2183755196 shutdown_connections 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7aac27640 1 -- 172.21.15.77:0/2183755196 wait complete. 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7aac27640 1 Processor -- start 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7aac27640 1 -- start start 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7aac27640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a4100700 0x7fc7a40694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7aac27640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a41018f0 0x7fc7a419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7aac27640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a41022b0 0x7fc7a419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7aac27640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a410c380 con 0x7fc7a41018f0 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7aac27640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a410c200 con 0x7fc7a41022b0 2024-03-16T10:15:11.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7aac27640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a410c500 con 0x7fc7a4100700 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7a3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a41018f0 0x7fc7a419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7a3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a41018f0 0x7fc7a419c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56782/0 (socket says 172.21.15.77:56782) 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7a89c3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a4100700 0x7fc7a40694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7a3fff640 1 -- 172.21.15.77:0/2056102402 learned_addr learned my addr 172.21.15.77:0/2056102402 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.158+0000 7fc7a91c4640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a41022b0 0x7fc7a419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.159+0000 7fc7a3fff640 1 -- 172.21.15.77:0/2056102402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a4100700 msgr2=0x7fc7a40694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.159+0000 7fc7a3fff640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a4100700 0x7fc7a40694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.159+0000 7fc7a3fff640 1 -- 172.21.15.77:0/2056102402 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a41022b0 msgr2=0x7fc7a419ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.159+0000 7fc7a3fff640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a41022b0 0x7fc7a419ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.159+0000 7fc7a3fff640 1 -- 172.21.15.77:0/2056102402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a419f740 con 0x7fc7a41018f0 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.159+0000 7fc7a3fff640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a41018f0 0x7fc7a419c810 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc79000e9f0 tx=0x7fc79000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.159+0000 7fc7a1ffb640 1 -- 172.21.15.77:0/2056102402 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79000cd80 con 0x7fc7a41018f0 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.159+0000 7fc7a1ffb640 1 -- 172.21.15.77:0/2056102402 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc790005500 con 0x7fc7a41018f0 2024-03-16T10:15:11.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.159+0000 7fc7aac27640 1 -- 172.21.15.77:0/2056102402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7a419f9d0 con 0x7fc7a41018f0 2024-03-16T10:15:11.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.159+0000 7fc7a1ffb640 1 -- 172.21.15.77:0/2056102402 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790010690 con 0x7fc7a41018f0 2024-03-16T10:15:11.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.159+0000 7fc7aac27640 1 -- 172.21.15.77:0/2056102402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7a419ff20 con 0x7fc7a41018f0 2024-03-16T10:15:11.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.160+0000 7fc7aac27640 1 -- 172.21.15.77:0/2056102402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7a4107a20 con 0x7fc7a41018f0 2024-03-16T10:15:11.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.161+0000 7fc7a1ffb640 1 -- 172.21.15.77:0/2056102402 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc7900046c0 con 0x7fc7a41018f0 2024-03-16T10:15:11.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.162+0000 7fc7a1ffb640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7940a2880 0x7fc7940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:11.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.162+0000 7fc7a89c3640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7940a2880 0x7fc7940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:11.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.162+0000 7fc7a1ffb640 1 -- 172.21.15.77:0/2056102402 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fc7900cebe0 con 0x7fc7a41018f0 2024-03-16T10:15:11.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.162+0000 7fc7a89c3640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7940a2880 0x7fc7940a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc798005fd0 tx=0x7fc79800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.167+0000 7fc7a1ffb640 1 -- 172.21.15.77:0/2056102402 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc79009bdd0 con 0x7fc7a41018f0 2024-03-16T10:15:11.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.276+0000 7fc7a1ffb640 1 -- 172.21.15.77:0/2056102402 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc790097d40 con 0x7fc7a41018f0 2024-03-16T10:15:11.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.354+0000 7fc7aac27640 1 -- 172.21.15.77:0/2056102402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7a402d380 con 0x7fc7a41018f0 2024-03-16T10:15:11.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.356+0000 7fc7a1ffb640 1 -- 172.21.15.77:0/2056102402 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22195 (secure 0 0 0) 0x7fc79004fa00 con 0x7fc7a41018f0 2024-03-16T10:15:11.357 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:15:11.357 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":85,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:04.004928+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:15:11.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.358+0000 7fc7aac27640 1 -- 172.21.15.77:0/2056102402 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7940a2880 msgr2=0x7fc7940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.358+0000 7fc7aac27640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7940a2880 0x7fc7940a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc798005fd0 tx=0x7fc79800b410 comp rx=0 tx=0).stop 2024-03-16T10:15:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.358+0000 7fc7aac27640 1 -- 172.21.15.77:0/2056102402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a41018f0 msgr2=0x7fc7a419c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.358+0000 7fc7aac27640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a41018f0 0x7fc7a419c810 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc79000e9f0 tx=0x7fc79000edf0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.358+0000 7fc7aac27640 1 -- 172.21.15.77:0/2056102402 shutdown_connections 2024-03-16T10:15:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.358+0000 7fc7aac27640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a41022b0 0x7fc7a419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.358+0000 7fc7aac27640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7a41018f0 0x7fc7a419c810 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.358+0000 7fc7aac27640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7940a2880 0x7fc7940a4d40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.358+0000 7fc7aac27640 1 --2- 172.21.15.77:0/2056102402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a4100700 0x7fc7a40694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.358+0000 7fc7aac27640 1 -- 172.21.15.77:0/2056102402 >> 172.21.15.77:0/2056102402 conn(0x7fc7a40fbc70 msgr2=0x7fc7a4104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.358+0000 7fc7aac27640 1 -- 172.21.15.77:0/2056102402 shutdown_connections 2024-03-16T10:15:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.358+0000 7fc7aac27640 1 -- 172.21.15.77:0/2056102402 wait complete. 2024-03-16T10:15:11.368 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:11 smithi077 ceph-mon[32400]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:11.368 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:11 smithi077 ceph-mon[32400]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:11.368 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:11 smithi077 ceph-mon[32400]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:11.368 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:11 smithi077 ceph-mon[32400]: pgmap v291: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s rd, 5.5 KiB/s wr, 20 op/s 2024-03-16T10:15:11.369 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:11 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2632807412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:15:11.370 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-16T10:15:11.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T10:15:11.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.473+0000 7f4d86881640 1 -- 172.21.15.77:0/1680518957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d80100700 msgr2=0x7f4d80100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.473+0000 7f4d86881640 1 --2- 172.21.15.77:0/1680518957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d80100700 0x7f4d80100b20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4d70009900 tx=0x7f4d7002bc30 comp rx=0 tx=0).stop 2024-03-16T10:15:11.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.473+0000 7f4d86881640 1 -- 172.21.15.77:0/1680518957 shutdown_connections 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.473+0000 7f4d86881640 1 --2- 172.21.15.77:0/1680518957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d801022b0 0x7f4d80108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.473+0000 7f4d86881640 1 --2- 172.21.15.77:0/1680518957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d801018f0 0x7f4d80101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.473+0000 7f4d86881640 1 --2- 172.21.15.77:0/1680518957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d80100700 0x7f4d80100b20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.473+0000 7f4d86881640 1 -- 172.21.15.77:0/1680518957 >> 172.21.15.77:0/1680518957 conn(0x7f4d800fbc70 msgr2=0x7f4d800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.473+0000 7f4d86881640 1 -- 172.21.15.77:0/1680518957 shutdown_connections 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.473+0000 7f4d86881640 1 -- 172.21.15.77:0/1680518957 wait complete. 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d86881640 1 Processor -- start 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d86881640 1 -- start start 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d86881640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d80100700 0x7f4d80197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d86881640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d801018f0 0x7f4d80198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d86881640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d801022b0 0x7f4d8019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d86881640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8010c380 con 0x7f4d801022b0 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d86881640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8010c200 con 0x7f4d801018f0 2024-03-16T10:15:11.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d86881640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8010c500 con 0x7f4d80100700 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d84e1e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d801022b0 0x7f4d8019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d84e1e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d801022b0 0x7f4d8019f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56794/0 (socket says 172.21.15.77:56794) 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d84e1e640 1 -- 172.21.15.77:0/864991317 learned_addr learned my addr 172.21.15.77:0/864991317 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d7ffff640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d80100700 0x7f4d80197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d7f7fe640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d801018f0 0x7f4d80198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d84e1e640 1 -- 172.21.15.77:0/864991317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d80100700 msgr2=0x7f4d80197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d84e1e640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d80100700 0x7f4d80197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d84e1e640 1 -- 172.21.15.77:0/864991317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d801018f0 msgr2=0x7f4d80198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d84e1e640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d801018f0 0x7f4d80198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d84e1e640 1 -- 172.21.15.77:0/864991317 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8019f760 con 0x7f4d801022b0 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.474+0000 7f4d7ffff640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d80100700 0x7f4d80197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.475+0000 7f4d84e1e640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d801022b0 0x7f4d8019f0f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800eb10 tx=0x7f4d7800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:11.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.475+0000 7f4d7d7fa640 1 -- 172.21.15.77:0/864991317 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78004b20 con 0x7f4d801022b0 2024-03-16T10:15:11.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.475+0000 7f4d7d7fa640 1 -- 172.21.15.77:0/864991317 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d78004cc0 con 0x7f4d801022b0 2024-03-16T10:15:11.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.475+0000 7f4d86881640 1 -- 172.21.15.77:0/864991317 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d8019fa50 con 0x7f4d801022b0 2024-03-16T10:15:11.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.475+0000 7f4d7d7fa640 1 -- 172.21.15.77:0/864991317 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78010760 con 0x7f4d801022b0 2024-03-16T10:15:11.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.475+0000 7f4d86881640 1 -- 172.21.15.77:0/864991317 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d8019fff0 con 0x7f4d801022b0 2024-03-16T10:15:11.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.476+0000 7f4d86881640 1 -- 172.21.15.77:0/864991317 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d801a0280 con 0x7f4d801022b0 2024-03-16T10:15:11.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.477+0000 7f4d7d7fa640 1 -- 172.21.15.77:0/864991317 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4d78005020 con 0x7f4d801022b0 2024-03-16T10:15:11.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.477+0000 7f4d7d7fa640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d680a2910 0x7f4d680a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:11.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.477+0000 7f4d7ffff640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d680a2910 0x7f4d680a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:11.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.478+0000 7f4d7d7fa640 1 -- 172.21.15.77:0/864991317 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4d780cf090 con 0x7f4d801022b0 2024-03-16T10:15:11.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.478+0000 7f4d7ffff640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d680a2910 0x7f4d680a4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4d70000c00 tx=0x7f4d70005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:11.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.482+0000 7f4d7d7fa640 1 -- 172.21.15.77:0/864991317 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4d7809c180 con 0x7f4d801022b0 2024-03-16T10:15:11.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.662+0000 7f4d86881640 1 -- 172.21.15.77:0/864991317 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d801a0450 con 0x7f4d680a2910 2024-03-16T10:15:11.666 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:15:11 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b[67101]: 2024-03-16T10:15:11.432+0000 7f01bf458700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-16T10:15:11.666 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:15:11 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b[67101]: 2024-03-16T10:15:11.432+0000 7f01bf458700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-16T10:15:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:11 smithi192 ceph-mon[36354]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:11 smithi192 ceph-mon[36354]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:11 smithi192 ceph-mon[36354]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:11 smithi192 ceph-mon[36354]: pgmap v291: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s rd, 5.5 KiB/s wr, 20 op/s 2024-03-16T10:15:11.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:11 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2632807412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:15:11.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.670+0000 7f4d7d7fa640 1 -- 172.21.15.77:0/864991317 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258911 (secure 0 0 0) 0x7f4d801a0450 con 0x7f4d680a2910 2024-03-16T10:15:11.674 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:15:11.675 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T10:15:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.673+0000 7f4d86881640 1 -- 172.21.15.77:0/864991317 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d680a2910 msgr2=0x7f4d680a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.673+0000 7f4d86881640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d680a2910 0x7f4d680a4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4d70000c00 tx=0x7f4d70005fd0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.673+0000 7f4d86881640 1 -- 172.21.15.77:0/864991317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d801022b0 msgr2=0x7f4d8019f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.673+0000 7f4d86881640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d801022b0 0x7f4d8019f0f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800eb10 tx=0x7f4d7800c2d0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.673+0000 7f4d86881640 1 -- 172.21.15.77:0/864991317 shutdown_connections 2024-03-16T10:15:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.673+0000 7f4d86881640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d801022b0 0x7f4d8019f0f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.674+0000 7f4d86881640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d801018f0 0x7f4d80198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.674+0000 7f4d86881640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d680a2910 0x7f4d680a4dd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.674+0000 7f4d86881640 1 --2- 172.21.15.77:0/864991317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d80100700 0x7f4d80197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.674+0000 7f4d86881640 1 -- 172.21.15.77:0/864991317 >> 172.21.15.77:0/864991317 conn(0x7f4d800fbc70 msgr2=0x7f4d800fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:11.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.674+0000 7f4d86881640 1 -- 172.21.15.77:0/864991317 shutdown_connections 2024-03-16T10:15:11.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.674+0000 7f4d86881640 1 -- 172.21.15.77:0/864991317 wait complete. 2024-03-16T10:15:11.686 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-03-16T10:15:10.717261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596805,"num_objects":74,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":133,"num_read_kb":116,"num_write":335,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":270,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360276,"kb_used_data":11104,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581548,"statfs":{"total":1151940427776,"available":1151571505152,"internally_reserved":0,"allocated":11370496,"data_stored":7051409,"data_compressed":106950,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6437,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":154,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002167"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.526082+0000","last_change":"2024-03-16T10:15:03.016880+0000","last_active":"2024-03-16T10:15:04.526082+0000","last_peered":"2024-03-16T10:15:04.526082+0000","last_clean":"2024-03-16T10:15:04.526082+0000","last_became_active":"2024-03-16T10:15:03.016513+0000","last_became_peered":"2024-03-16T10:15:03.016513+0000","last_unstale":"2024-03-16T10:15:04.526082+0000","last_undegraded":"2024-03-16T10:15:04.526082+0000","last_fullsized":"2024-03-16T10:15:04.526082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:39.834015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390079+0000","last_change":"2024-03-16T10:15:03.015467+0000","last_active":"2024-03-16T10:15:04.390079+0000","last_peered":"2024-03-16T10:15:04.390079+0000","last_clean":"2024-03-16T10:15:04.390079+0000","last_became_active":"2024-03-16T10:15:03.015152+0000","last_became_peered":"2024-03-16T10:15:03.015152+0000","last_unstale":"2024-03-16T10:15:04.390079+0000","last_undegraded":"2024-03-16T10:15:04.390079+0000","last_fullsized":"2024-03-16T10:15:04.390079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:17:29.988569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.389915+0000","last_change":"2024-03-16T10:15:03.023836+0000","last_active":"2024-03-16T10:15:04.389915+0000","last_peered":"2024-03-16T10:15:04.389915+0000","last_clean":"2024-03-16T10:15:04.389915+0000","last_became_active":"2024-03-16T10:15:03.022774+0000","last_became_peered":"2024-03-16T10:15:03.022774+0000","last_unstale":"2024-03-16T10:15:04.389915+0000","last_undegraded":"2024-03-16T10:15:04.389915+0000","last_fullsized":"2024-03-16T10:15:04.389915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:49.837157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390609+0000","last_change":"2024-03-16T10:15:03.030891+0000","last_active":"2024-03-16T10:15:04.390609+0000","last_peered":"2024-03-16T10:15:04.390609+0000","last_clean":"2024-03-16T10:15:04.390609+0000","last_became_active":"2024-03-16T10:15:03.030278+0000","last_became_peered":"2024-03-16T10:15:03.030278+0000","last_unstale":"2024-03-16T10:15:04.390609+0000","last_undegraded":"2024-03-16T10:15:04.390609+0000","last_fullsized":"2024-03-16T10:15:04.390609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:16.166885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.023143+0000","last_change":"2024-03-16T10:15:03.017610+0000","last_active":"2024-03-16T10:15:04.023143+0000","last_peered":"2024-03-16T10:15:04.023143+0000","last_clean":"2024-03-16T10:15:04.023143+0000","last_became_active":"2024-03-16T10:15:03.017250+0000","last_became_peered":"2024-03-16T10:15:03.017250+0000","last_unstale":"2024-03-16T10:15:04.023143+0000","last_undegraded":"2024-03-16T10:15:04.023143+0000","last_fullsized":"2024-03-16T10:15:04.023143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:07.899425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390917+0000","last_change":"2024-03-16T10:15:03.020720+0000","last_active":"2024-03-16T10:15:04.390917+0000","last_peered":"2024-03-16T10:15:04.390917+0000","last_clean":"2024-03-16T10:15:04.390917+0000","last_became_active":"2024-03-16T10:15:03.020334+0000","last_became_peered":"2024-03-16T10:15:03.020334+0000","last_unstale":"2024-03-16T10:15:04.390917+0000","last_undegraded":"2024-03-16T10:15:04.390917+0000","last_fullsized":"2024-03-16T10:15:04.390917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:42:19.210432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390182+0000","last_change":"2024-03-16T10:15:03.017686+0000","last_active":"2024-03-16T10:15:04.390182+0000","last_peered":"2024-03-16T10:15:04.390182+0000","last_clean":"2024-03-16T10:15:04.390182+0000","last_became_active":"2024-03-16T10:15:03.017262+0000","last_became_peered":"2024-03-16T10:15:03.017262+0000","last_unstale":"2024-03-16T10:15:04.390182+0000","last_undegraded":"2024-03-16T10:15:04.390182+0000","last_fullsized":"2024-03-16T10:15:04.390182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:36:22.270651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022666+0000","last_change":"2024-03-16T10:15:03.020130+0000","last_active":"2024-03-16T10:15:04.022666+0000","last_peered":"2024-03-16T10:15:04.022666+0000","last_clean":"2024-03-16T10:15:04.022666+0000","last_became_active":"2024-03-16T10:15:03.019742+0000","last_became_peered":"2024-03-16T10:15:03.019742+0000","last_unstale":"2024-03-16T10:15:04.022666+0000","last_undegraded":"2024-03-16T10:15:04.022666+0000","last_fullsized":"2024-03-16T10:15:04.022666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:52.450792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013990+0000","last_change":"2024-03-16T10:15:03.027589+0000","last_active":"2024-03-16T10:15:04.013990+0000","last_peered":"2024-03-16T10:15:04.013990+0000","last_clean":"2024-03-16T10:15:04.013990+0000","last_became_active":"2024-03-16T10:15:03.027226+0000","last_became_peered":"2024-03-16T10:15:03.027226+0000","last_unstale":"2024-03-16T10:15:04.013990+0000","last_undegraded":"2024-03-16T10:15:04.013990+0000","last_fullsized":"2024-03-16T10:15:04.013990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:14:35.354629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390491+0000","last_change":"2024-03-16T10:15:03.021163+0000","last_active":"2024-03-16T10:15:04.390491+0000","last_peered":"2024-03-16T10:15:04.390491+0000","last_clean":"2024-03-16T10:15:04.390491+0000","last_became_active":"2024-03-16T10:15:03.020483+0000","last_became_peered":"2024-03-16T10:15:03.020483+0000","last_unstale":"2024-03-16T10:15:04.390491+0000","last_undegraded":"2024-03-16T10:15:04.390491+0000","last_fullsized":"2024-03-16T10:15:04.390491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:01:00.873471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.391277+0000","last_change":"2024-03-16T10:15:03.026111+0000","last_active":"2024-03-16T10:15:04.391277+0000","last_peered":"2024-03-16T10:15:04.391277+0000","last_clean":"2024-03-16T10:15:04.391277+0000","last_became_active":"2024-03-16T10:15:03.025296+0000","last_became_peered":"2024-03-16T10:15:03.025296+0000","last_unstale":"2024-03-16T10:15:04.391277+0000","last_undegraded":"2024-03-16T10:15:04.391277+0000","last_fullsized":"2024-03-16T10:15:04.391277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:33:29.606876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390469+0000","last_change":"2024-03-16T10:15:03.023013+0000","last_active":"2024-03-16T10:15:04.390469+0000","last_peered":"2024-03-16T10:15:04.390469+0000","last_clean":"2024-03-16T10:15:04.390469+0000","last_became_active":"2024-03-16T10:15:03.022251+0000","last_became_peered":"2024-03-16T10:15:03.022251+0000","last_unstale":"2024-03-16T10:15:04.390469+0000","last_undegraded":"2024-03-16T10:15:04.390469+0000","last_fullsized":"2024-03-16T10:15:04.390469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:33.544709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013883+0000","last_change":"2024-03-16T10:15:03.009548+0000","last_active":"2024-03-16T10:15:04.013883+0000","last_peered":"2024-03-16T10:15:04.013883+0000","last_clean":"2024-03-16T10:15:04.013883+0000","last_became_active":"2024-03-16T10:15:03.007675+0000","last_became_peered":"2024-03-16T10:15:03.007675+0000","last_unstale":"2024-03-16T10:15:04.013883+0000","last_undegraded":"2024-03-16T10:15:04.013883+0000","last_fullsized":"2024-03-16T10:15:04.013883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:08:02.561602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.389735+0000","last_change":"2024-03-16T10:15:03.020146+0000","last_active":"2024-03-16T10:15:04.389735+0000","last_peered":"2024-03-16T10:15:04.389735+0000","last_clean":"2024-03-16T10:15:04.389735+0000","last_became_active":"2024-03-16T10:15:03.019823+0000","last_became_peered":"2024-03-16T10:15:03.019823+0000","last_unstale":"2024-03-16T10:15:04.389735+0000","last_undegraded":"2024-03-16T10:15:04.389735+0000","last_fullsized":"2024-03-16T10:15:04.389735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:52.276506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.389692+0000","last_change":"2024-03-16T10:15:03.019150+0000","last_active":"2024-03-16T10:15:04.389692+0000","last_peered":"2024-03-16T10:15:04.389692+0000","last_clean":"2024-03-16T10:15:04.389692+0000","last_became_active":"2024-03-16T10:15:03.018830+0000","last_became_peered":"2024-03-16T10:15:03.018830+0000","last_unstale":"2024-03-16T10:15:04.389692+0000","last_undegraded":"2024-03-16T10:15:04.389692+0000","last_fullsized":"2024-03-16T10:15:04.389692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:46.207187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013630+0000","last_change":"2024-03-16T10:15:03.033321+0000","last_active":"2024-03-16T10:15:04.013630+0000","last_peered":"2024-03-16T10:15:04.013630+0000","last_clean":"2024-03-16T10:15:04.013630+0000","last_became_active":"2024-03-16T10:15:03.032944+0000","last_became_peered":"2024-03-16T10:15:03.032944+0000","last_unstale":"2024-03-16T10:15:04.013630+0000","last_undegraded":"2024-03-16T10:15:04.013630+0000","last_fullsized":"2024-03-16T10:15:04.013630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:14.345488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.526221+0000","last_change":"2024-03-16T10:15:03.019897+0000","last_active":"2024-03-16T10:15:04.526221+0000","last_peered":"2024-03-16T10:15:04.526221+0000","last_clean":"2024-03-16T10:15:04.526221+0000","last_became_active":"2024-03-16T10:15:03.019555+0000","last_became_peered":"2024-03-16T10:15:03.019555+0000","last_unstale":"2024-03-16T10:15:04.526221+0000","last_undegraded":"2024-03-16T10:15:04.526221+0000","last_fullsized":"2024-03-16T10:15:04.526221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:58.307285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013466+0000","last_change":"2024-03-16T10:15:03.017482+0000","last_active":"2024-03-16T10:15:04.013466+0000","last_peered":"2024-03-16T10:15:04.013466+0000","last_clean":"2024-03-16T10:15:04.013466+0000","last_became_active":"2024-03-16T10:15:03.016521+0000","last_became_peered":"2024-03-16T10:15:03.016521+0000","last_unstale":"2024-03-16T10:15:04.013466+0000","last_undegraded":"2024-03-16T10:15:04.013466+0000","last_fullsized":"2024-03-16T10:15:04.013466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:11.889111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.014406+0000","last_change":"2024-03-16T10:15:03.018625+0000","last_active":"2024-03-16T10:15:04.014406+0000","last_peered":"2024-03-16T10:15:04.014406+0000","last_clean":"2024-03-16T10:15:04.014406+0000","last_became_active":"2024-03-16T10:15:03.018313+0000","last_became_peered":"2024-03-16T10:15:03.018313+0000","last_unstale":"2024-03-16T10:15:04.014406+0000","last_undegraded":"2024-03-16T10:15:04.014406+0000","last_fullsized":"2024-03-16T10:15:04.014406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:35:58.533559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:08.230706+0000","last_change":"2024-03-16T10:15:02.016545+0000","last_active":"2024-03-16T10:15:08.230706+0000","last_peered":"2024-03-16T10:15:08.230706+0000","last_clean":"2024-03-16T10:15:08.230706+0000","last_became_active":"2024-03-16T10:15:02.012088+0000","last_became_peered":"2024-03-16T10:15:02.012088+0000","last_unstale":"2024-03-16T10:15:08.230706+0000","last_undegraded":"2024-03-16T10:15:08.230706+0000","last_fullsized":"2024-03-16T10:15:08.230706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:50.633539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022920+0000","last_change":"2024-03-16T10:15:03.026565+0000","last_active":"2024-03-16T10:15:04.022920+0000","last_peered":"2024-03-16T10:15:04.022920+0000","last_clean":"2024-03-16T10:15:04.022920+0000","last_became_active":"2024-03-16T10:15:03.025386+0000","last_became_peered":"2024-03-16T10:15:03.025386+0000","last_unstale":"2024-03-16T10:15:04.022920+0000","last_undegraded":"2024-03-16T10:15:04.022920+0000","last_fullsized":"2024-03-16T10:15:04.022920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:28.831764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022959+0000","last_change":"2024-03-16T10:15:02.018097+0000","last_active":"2024-03-16T10:15:04.022959+0000","last_peered":"2024-03-16T10:15:04.022959+0000","last_clean":"2024-03-16T10:15:04.022959+0000","last_became_active":"2024-03-16T10:15:02.017777+0000","last_became_peered":"2024-03-16T10:15:02.017777+0000","last_unstale":"2024-03-16T10:15:04.022959+0000","last_undegraded":"2024-03-16T10:15:04.022959+0000","last_fullsized":"2024-03-16T10:15:04.022959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:04:58.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390097+0000","last_change":"2024-03-16T10:15:03.023384+0000","last_active":"2024-03-16T10:15:04.390097+0000","last_peered":"2024-03-16T10:15:04.390097+0000","last_clean":"2024-03-16T10:15:04.390097+0000","last_became_active":"2024-03-16T10:15:03.022996+0000","last_became_peered":"2024-03-16T10:15:03.022996+0000","last_unstale":"2024-03-16T10:15:04.390097+0000","last_undegraded":"2024-03-16T10:15:04.390097+0000","last_fullsized":"2024-03-16T10:15:04.390097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:29:20.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.251786+0000","last_change":"2024-03-16T10:15:02.014409+0000","last_active":"2024-03-16T10:15:06.251786+0000","last_peered":"2024-03-16T10:15:06.251786+0000","last_clean":"2024-03-16T10:15:06.251786+0000","last_became_active":"2024-03-16T10:15:02.014015+0000","last_became_peered":"2024-03-16T10:15:02.014015+0000","last_unstale":"2024-03-16T10:15:06.251786+0000","last_undegraded":"2024-03-16T10:15:06.251786+0000","last_fullsized":"2024-03-16T10:15:06.251786+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:25:53.133466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013656+0000","last_change":"2024-03-16T10:15:03.031328+0000","last_active":"2024-03-16T10:15:04.013656+0000","last_peered":"2024-03-16T10:15:04.013656+0000","last_clean":"2024-03-16T10:15:04.013656+0000","last_became_active":"2024-03-16T10:15:03.029787+0000","last_became_peered":"2024-03-16T10:15:03.029787+0000","last_unstale":"2024-03-16T10:15:04.013656+0000","last_undegraded":"2024-03-16T10:15:04.013656+0000","last_fullsized":"2024-03-16T10:15:04.013656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:34.321883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.023123+0000","last_change":"2024-03-16T10:15:02.018032+0000","last_active":"2024-03-16T10:15:04.023123+0000","last_peered":"2024-03-16T10:15:04.023123+0000","last_clean":"2024-03-16T10:15:04.023123+0000","last_became_active":"2024-03-16T10:15:02.017677+0000","last_became_peered":"2024-03-16T10:15:02.017677+0000","last_unstale":"2024-03-16T10:15:04.023123+0000","last_undegraded":"2024-03-16T10:15:04.023123+0000","last_fullsized":"2024-03-16T10:15:04.023123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:58:07.179129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390477+0000","last_change":"2024-03-16T10:15:03.024958+0000","last_active":"2024-03-16T10:15:04.390477+0000","last_peered":"2024-03-16T10:15:04.390477+0000","last_clean":"2024-03-16T10:15:04.390477+0000","last_became_active":"2024-03-16T10:15:03.023964+0000","last_became_peered":"2024-03-16T10:15:03.023964+0000","last_unstale":"2024-03-16T10:15:04.390477+0000","last_undegraded":"2024-03-16T10:15:04.390477+0000","last_fullsized":"2024-03-16T10:15:04.390477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:55:00.953258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.249865+0000","last_change":"2024-03-16T10:15:02.015062+0000","last_active":"2024-03-16T10:15:05.249865+0000","last_peered":"2024-03-16T10:15:05.249865+0000","last_clean":"2024-03-16T10:15:05.249865+0000","last_became_active":"2024-03-16T10:15:02.014714+0000","last_became_peered":"2024-03-16T10:15:02.014714+0000","last_unstale":"2024-03-16T10:15:05.249865+0000","last_undegraded":"2024-03-16T10:15:05.249865+0000","last_fullsized":"2024-03-16T10:15:05.249865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:23:55.313009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022990+0000","last_change":"2024-03-16T10:15:03.013956+0000","last_active":"2024-03-16T10:15:04.022990+0000","last_peered":"2024-03-16T10:15:04.022990+0000","last_clean":"2024-03-16T10:15:04.022990+0000","last_became_active":"2024-03-16T10:15:03.013607+0000","last_became_peered":"2024-03-16T10:15:03.013607+0000","last_unstale":"2024-03-16T10:15:04.022990+0000","last_undegraded":"2024-03-16T10:15:04.022990+0000","last_fullsized":"2024-03-16T10:15:04.022990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:40:55.445395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:07.265129+0000","last_change":"2024-03-16T10:15:02.015651+0000","last_active":"2024-03-16T10:15:07.265129+0000","last_peered":"2024-03-16T10:15:07.265129+0000","last_clean":"2024-03-16T10:15:07.265129+0000","last_became_active":"2024-03-16T10:15:02.015254+0000","last_became_peered":"2024-03-16T10:15:02.015254+0000","last_unstale":"2024-03-16T10:15:07.265129+0000","last_undegraded":"2024-03-16T10:15:07.265129+0000","last_fullsized":"2024-03-16T10:15:07.265129+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:41:59.013606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390537+0000","last_change":"2024-03-16T10:15:03.023074+0000","last_active":"2024-03-16T10:15:04.390537+0000","last_peered":"2024-03-16T10:15:04.390537+0000","last_clean":"2024-03-16T10:15:04.390537+0000","last_became_active":"2024-03-16T10:15:03.022700+0000","last_became_peered":"2024-03-16T10:15:03.022700+0000","last_unstale":"2024-03-16T10:15:04.390537+0000","last_undegraded":"2024-03-16T10:15:04.390537+0000","last_fullsized":"2024-03-16T10:15:04.390537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:22.047929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.251472+0000","last_change":"2024-03-16T10:15:02.014793+0000","last_active":"2024-03-16T10:15:06.251472+0000","last_peered":"2024-03-16T10:15:06.251472+0000","last_clean":"2024-03-16T10:15:06.251472+0000","last_became_active":"2024-03-16T10:15:02.014459+0000","last_became_peered":"2024-03-16T10:15:02.014459+0000","last_unstale":"2024-03-16T10:15:06.251472+0000","last_undegraded":"2024-03-16T10:15:06.251472+0000","last_fullsized":"2024-03-16T10:15:06.251472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:59:45.035045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022608+0000","last_change":"2024-03-16T10:15:03.012713+0000","last_active":"2024-03-16T10:15:04.022608+0000","last_peered":"2024-03-16T10:15:04.022608+0000","last_clean":"2024-03-16T10:15:04.022608+0000","last_became_active":"2024-03-16T10:15:03.012407+0000","last_became_peered":"2024-03-16T10:15:03.012407+0000","last_unstale":"2024-03-16T10:15:04.022608+0000","last_undegraded":"2024-03-16T10:15:04.022608+0000","last_fullsized":"2024-03-16T10:15:04.022608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:23:14.592810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.248418+0000","last_change":"2024-03-16T10:15:02.015582+0000","last_active":"2024-03-16T10:15:05.248418+0000","last_peered":"2024-03-16T10:15:05.248418+0000","last_clean":"2024-03-16T10:15:05.248418+0000","last_became_active":"2024-03-16T10:15:02.015073+0000","last_became_peered":"2024-03-16T10:15:02.015073+0000","last_unstale":"2024-03-16T10:15:05.248418+0000","last_undegraded":"2024-03-16T10:15:05.248418+0000","last_fullsized":"2024-03-16T10:15:05.248418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:22:48.212448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013244+0000","last_change":"2024-03-16T10:15:03.016839+0000","last_active":"2024-03-16T10:15:04.013244+0000","last_peered":"2024-03-16T10:15:04.013244+0000","last_clean":"2024-03-16T10:15:04.013244+0000","last_became_active":"2024-03-16T10:15:03.016463+0000","last_became_peered":"2024-03-16T10:15:03.016463+0000","last_unstale":"2024-03-16T10:15:04.013244+0000","last_undegraded":"2024-03-16T10:15:04.013244+0000","last_fullsized":"2024-03-16T10:15:04.013244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:43:34.259049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:08.224296+0000","last_change":"2024-03-16T10:15:02.017884+0000","last_active":"2024-03-16T10:15:08.224296+0000","last_peered":"2024-03-16T10:15:08.224296+0000","last_clean":"2024-03-16T10:15:08.224296+0000","last_became_active":"2024-03-16T10:15:02.017511+0000","last_became_peered":"2024-03-16T10:15:02.017511+0000","last_unstale":"2024-03-16T10:15:08.224296+0000","last_undegraded":"2024-03-16T10:15:08.224296+0000","last_fullsized":"2024-03-16T10:15:08.224296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:31.281631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013899+0000","last_change":"2024-03-16T10:15:03.032259+0000","last_active":"2024-03-16T10:15:04.013899+0000","last_peered":"2024-03-16T10:15:04.013899+0000","last_clean":"2024-03-16T10:15:04.013899+0000","last_became_active":"2024-03-16T10:15:03.031884+0000","last_became_peered":"2024-03-16T10:15:03.031884+0000","last_unstale":"2024-03-16T10:15:04.013899+0000","last_undegraded":"2024-03-16T10:15:04.013899+0000","last_fullsized":"2024-03-16T10:15:04.013899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:37:48.464051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.247747+0000","last_change":"2024-03-16T10:15:02.015372+0000","last_active":"2024-03-16T10:15:05.247747+0000","last_peered":"2024-03-16T10:15:05.247747+0000","last_clean":"2024-03-16T10:15:05.247747+0000","last_became_active":"2024-03-16T10:15:02.015022+0000","last_became_peered":"2024-03-16T10:15:02.015022+0000","last_unstale":"2024-03-16T10:15:05.247747+0000","last_undegraded":"2024-03-16T10:15:05.247747+0000","last_fullsized":"2024-03-16T10:15:05.247747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:42.651903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390159+0000","last_change":"2024-03-16T10:15:03.018661+0000","last_active":"2024-03-16T10:15:04.390159+0000","last_peered":"2024-03-16T10:15:04.390159+0000","last_clean":"2024-03-16T10:15:04.390159+0000","last_became_active":"2024-03-16T10:15:03.018351+0000","last_became_peered":"2024-03-16T10:15:03.018351+0000","last_unstale":"2024-03-16T10:15:04.390159+0000","last_undegraded":"2024-03-16T10:15:04.390159+0000","last_fullsized":"2024-03-16T10:15:04.390159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:21:33.939811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.248221+0000","last_change":"2024-03-16T10:15:02.014343+0000","last_active":"2024-03-16T10:15:05.248221+0000","last_peered":"2024-03-16T10:15:05.248221+0000","last_clean":"2024-03-16T10:15:05.248221+0000","last_became_active":"2024-03-16T10:15:02.013983+0000","last_became_peered":"2024-03-16T10:15:02.013983+0000","last_unstale":"2024-03-16T10:15:05.248221+0000","last_undegraded":"2024-03-16T10:15:05.248221+0000","last_fullsized":"2024-03-16T10:15:05.248221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:35:04.768079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013766+0000","last_change":"2024-03-16T10:15:03.031055+0000","last_active":"2024-03-16T10:15:04.013766+0000","last_peered":"2024-03-16T10:15:04.013766+0000","last_clean":"2024-03-16T10:15:04.013766+0000","last_became_active":"2024-03-16T10:15:03.029656+0000","last_became_peered":"2024-03-16T10:15:03.029656+0000","last_unstale":"2024-03-16T10:15:04.013766+0000","last_undegraded":"2024-03-16T10:15:04.013766+0000","last_fullsized":"2024-03-16T10:15:04.013766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:27:35.684774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'11","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:08.231618+0000","last_change":"2024-03-16T10:15:02.014752+0000","last_active":"2024-03-16T10:15:08.231618+0000","last_peered":"2024-03-16T10:15:08.231618+0000","last_clean":"2024-03-16T10:15:08.231618+0000","last_became_active":"2024-03-16T10:15:02.013763+0000","last_became_peered":"2024-03-16T10:15:02.013763+0000","last_unstale":"2024-03-16T10:15:08.231618+0000","last_undegraded":"2024-03-16T10:15:08.231618+0000","last_fullsized":"2024-03-16T10:15:08.231618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:12:12.759643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2003,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.023110+0000","last_change":"2024-03-16T10:15:03.018838+0000","last_active":"2024-03-16T10:15:04.023110+0000","last_peered":"2024-03-16T10:15:04.023110+0000","last_clean":"2024-03-16T10:15:04.023110+0000","last_became_active":"2024-03-16T10:15:03.017713+0000","last_became_peered":"2024-03-16T10:15:03.017713+0000","last_unstale":"2024-03-16T10:15:04.023110+0000","last_undegraded":"2024-03-16T10:15:04.023110+0000","last_fullsized":"2024-03-16T10:15:04.023110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:03:56.936729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.251324+0000","last_change":"2024-03-16T10:15:02.016996+0000","last_active":"2024-03-16T10:15:06.251324+0000","last_peered":"2024-03-16T10:15:06.251324+0000","last_clean":"2024-03-16T10:15:06.251324+0000","last_became_active":"2024-03-16T10:15:02.016421+0000","last_became_peered":"2024-03-16T10:15:02.016421+0000","last_unstale":"2024-03-16T10:15:06.251324+0000","last_undegraded":"2024-03-16T10:15:06.251324+0000","last_fullsized":"2024-03-16T10:15:06.251324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:25:09.866730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013704+0000","last_change":"2024-03-16T10:15:03.017842+0000","last_active":"2024-03-16T10:15:04.013704+0000","last_peered":"2024-03-16T10:15:04.013704+0000","last_clean":"2024-03-16T10:15:04.013704+0000","last_became_active":"2024-03-16T10:15:03.015119+0000","last_became_peered":"2024-03-16T10:15:03.015119+0000","last_unstale":"2024-03-16T10:15:04.013704+0000","last_undegraded":"2024-03-16T10:15:04.013704+0000","last_fullsized":"2024-03-16T10:15:04.013704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:14:34.474519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.247417+0000","last_change":"2024-03-16T10:15:02.021407+0000","last_active":"2024-03-16T10:15:05.247417+0000","last_peered":"2024-03-16T10:15:05.247417+0000","last_clean":"2024-03-16T10:15:05.247417+0000","last_became_active":"2024-03-16T10:15:02.020884+0000","last_became_peered":"2024-03-16T10:15:02.020884+0000","last_unstale":"2024-03-16T10:15:05.247417+0000","last_undegraded":"2024-03-16T10:15:05.247417+0000","last_fullsized":"2024-03-16T10:15:05.247417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:24.858281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390941+0000","last_change":"2024-03-16T10:15:03.028607+0000","last_active":"2024-03-16T10:15:04.390941+0000","last_peered":"2024-03-16T10:15:04.390941+0000","last_clean":"2024-03-16T10:15:04.390941+0000","last_became_active":"2024-03-16T10:15:03.028280+0000","last_became_peered":"2024-03-16T10:15:03.028280+0000","last_unstale":"2024-03-16T10:15:04.390941+0000","last_undegraded":"2024-03-16T10:15:04.390941+0000","last_fullsized":"2024-03-16T10:15:04.390941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:39.905921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:07.261392+0000","last_change":"2024-03-16T10:15:02.008558+0000","last_active":"2024-03-16T10:15:07.261392+0000","last_peered":"2024-03-16T10:15:07.261392+0000","last_clean":"2024-03-16T10:15:07.261392+0000","last_became_active":"2024-03-16T10:15:02.008014+0000","last_became_peered":"2024-03-16T10:15:02.008014+0000","last_unstale":"2024-03-16T10:15:07.261392+0000","last_undegraded":"2024-03-16T10:15:07.261392+0000","last_fullsized":"2024-03-16T10:15:07.261392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:40:09.217991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.389791+0000","last_change":"2024-03-16T10:15:03.020847+0000","last_active":"2024-03-16T10:15:04.389791+0000","last_peered":"2024-03-16T10:15:04.389791+0000","last_clean":"2024-03-16T10:15:04.389791+0000","last_became_active":"2024-03-16T10:15:03.020512+0000","last_became_peered":"2024-03-16T10:15:03.020512+0000","last_unstale":"2024-03-16T10:15:04.389791+0000","last_undegraded":"2024-03-16T10:15:04.389791+0000","last_fullsized":"2024-03-16T10:15:04.389791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:04.436195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.253290+0000","last_change":"2024-03-16T10:15:02.010690+0000","last_active":"2024-03-16T10:15:06.253290+0000","last_peered":"2024-03-16T10:15:06.253290+0000","last_clean":"2024-03-16T10:15:06.253290+0000","last_became_active":"2024-03-16T10:15:02.010228+0000","last_became_peered":"2024-03-16T10:15:02.010228+0000","last_unstale":"2024-03-16T10:15:06.253290+0000","last_undegraded":"2024-03-16T10:15:06.253290+0000","last_fullsized":"2024-03-16T10:15:06.253290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:30:07.513225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390326+0000","last_change":"2024-03-16T10:15:03.025257+0000","last_active":"2024-03-16T10:15:04.390326+0000","last_peered":"2024-03-16T10:15:04.390326+0000","last_clean":"2024-03-16T10:15:04.390326+0000","last_became_active":"2024-03-16T10:15:03.024498+0000","last_became_peered":"2024-03-16T10:15:03.024498+0000","last_unstale":"2024-03-16T10:15:04.390326+0000","last_undegraded":"2024-03-16T10:15:04.390326+0000","last_fullsized":"2024-03-16T10:15:04.390326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:27:55.894728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.023168+0000","last_change":"2024-03-16T10:15:02.014136+0000","last_active":"2024-03-16T10:15:04.023168+0000","last_peered":"2024-03-16T10:15:04.023168+0000","last_clean":"2024-03-16T10:15:04.023168+0000","last_became_active":"2024-03-16T10:15:02.013721+0000","last_became_peered":"2024-03-16T10:15:02.013721+0000","last_unstale":"2024-03-16T10:15:04.023168+0000","last_undegraded":"2024-03-16T10:15:04.023168+0000","last_fullsized":"2024-03-16T10:15:04.023168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:28:12.301450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.526001+0000","last_change":"2024-03-16T10:15:03.021431+0000","last_active":"2024-03-16T10:15:04.526001+0000","last_peered":"2024-03-16T10:15:04.526001+0000","last_clean":"2024-03-16T10:15:04.526001+0000","last_became_active":"2024-03-16T10:15:03.021116+0000","last_became_peered":"2024-03-16T10:15:03.021116+0000","last_unstale":"2024-03-16T10:15:04.526001+0000","last_undegraded":"2024-03-16T10:15:04.526001+0000","last_fullsized":"2024-03-16T10:15:04.526001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:52:31.595530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013366+0000","last_change":"2024-03-16T10:15:02.005754+0000","last_active":"2024-03-16T10:15:04.013366+0000","last_peered":"2024-03-16T10:15:04.013366+0000","last_clean":"2024-03-16T10:15:04.013366+0000","last_became_active":"2024-03-16T10:15:02.005344+0000","last_became_peered":"2024-03-16T10:15:02.005344+0000","last_unstale":"2024-03-16T10:15:04.013366+0000","last_undegraded":"2024-03-16T10:15:04.013366+0000","last_fullsized":"2024-03-16T10:15:04.013366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:37:37.918861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390985+0000","last_change":"2024-03-16T10:15:03.012887+0000","last_active":"2024-03-16T10:15:04.390985+0000","last_peered":"2024-03-16T10:15:04.390985+0000","last_clean":"2024-03-16T10:15:04.390985+0000","last_became_active":"2024-03-16T10:15:03.012496+0000","last_became_peered":"2024-03-16T10:15:03.012496+0000","last_unstale":"2024-03-16T10:15:04.390985+0000","last_undegraded":"2024-03-16T10:15:04.390985+0000","last_fullsized":"2024-03-16T10:15:04.390985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:11.111049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390438+0000","last_change":"2024-03-16T10:15:02.014334+0000","last_active":"2024-03-16T10:15:04.390438+0000","last_peered":"2024-03-16T10:15:04.390438+0000","last_clean":"2024-03-16T10:15:04.390438+0000","last_became_active":"2024-03-16T10:15:02.013832+0000","last_became_peered":"2024-03-16T10:15:02.013832+0000","last_unstale":"2024-03-16T10:15:04.390438+0000","last_undegraded":"2024-03-16T10:15:04.390438+0000","last_fullsized":"2024-03-16T10:15:04.390438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:20.391594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390298+0000","last_change":"2024-03-16T10:15:03.021276+0000","last_active":"2024-03-16T10:15:04.390298+0000","last_peered":"2024-03-16T10:15:04.390298+0000","last_clean":"2024-03-16T10:15:04.390298+0000","last_became_active":"2024-03-16T10:15:03.020927+0000","last_became_peered":"2024-03-16T10:15:03.020927+0000","last_unstale":"2024-03-16T10:15:04.390298+0000","last_undegraded":"2024-03-16T10:15:04.390298+0000","last_fullsized":"2024-03-16T10:15:04.390298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:53:54.127506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390252+0000","last_change":"2024-03-16T10:15:03.022083+0000","last_active":"2024-03-16T10:15:04.390252+0000","last_peered":"2024-03-16T10:15:04.390252+0000","last_clean":"2024-03-16T10:15:04.390252+0000","last_became_active":"2024-03-16T10:15:03.021609+0000","last_became_peered":"2024-03-16T10:15:03.021609+0000","last_unstale":"2024-03-16T10:15:04.390252+0000","last_undegraded":"2024-03-16T10:15:04.390252+0000","last_fullsized":"2024-03-16T10:15:04.390252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:53.864402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.252965+0000","last_change":"2024-03-16T10:15:02.015497+0000","last_active":"2024-03-16T10:15:06.252965+0000","last_peered":"2024-03-16T10:15:06.252965+0000","last_clean":"2024-03-16T10:15:06.252965+0000","last_became_active":"2024-03-16T10:15:02.015183+0000","last_became_peered":"2024-03-16T10:15:02.015183+0000","last_unstale":"2024-03-16T10:15:06.252965+0000","last_undegraded":"2024-03-16T10:15:06.252965+0000","last_fullsized":"2024-03-16T10:15:06.252965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:44:05.913030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022947+0000","last_change":"2024-03-16T10:15:03.020046+0000","last_active":"2024-03-16T10:15:04.022947+0000","last_peered":"2024-03-16T10:15:04.022947+0000","last_clean":"2024-03-16T10:15:04.022947+0000","last_became_active":"2024-03-16T10:15:03.019647+0000","last_became_peered":"2024-03-16T10:15:03.019647+0000","last_unstale":"2024-03-16T10:15:04.022947+0000","last_undegraded":"2024-03-16T10:15:04.022947+0000","last_fullsized":"2024-03-16T10:15:04.022947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:18.343495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:07.262840+0000","last_change":"2024-03-16T10:15:02.008560+0000","last_active":"2024-03-16T10:15:07.262840+0000","last_peered":"2024-03-16T10:15:07.262840+0000","last_clean":"2024-03-16T10:15:07.262840+0000","last_became_active":"2024-03-16T10:15:02.008094+0000","last_became_peered":"2024-03-16T10:15:02.008094+0000","last_unstale":"2024-03-16T10:15:07.262840+0000","last_undegraded":"2024-03-16T10:15:07.262840+0000","last_fullsized":"2024-03-16T10:15:07.262840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:21:43.555120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":337,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.389703+0000","last_change":"2024-03-16T10:13:47.211815+0000","last_active":"2024-03-16T10:15:04.389703+0000","last_peered":"2024-03-16T10:15:04.389703+0000","last_clean":"2024-03-16T10:15:04.389703+0000","last_became_active":"2024-03-16T10:13:47.211353+0000","last_became_peered":"2024-03-16T10:13:47.211353+0000","last_unstale":"2024-03-16T10:15:04.389703+0000","last_undegraded":"2024-03-16T10:15:04.389703+0000","last_fullsized":"2024-03-16T10:15:04.389703+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_clean_scrub_stamp":"2024-03-16T10:10:22.965421+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:11:20.607686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013199+0000","last_change":"2024-03-16T10:15:03.020181+0000","last_active":"2024-03-16T10:15:04.013199+0000","last_peered":"2024-03-16T10:15:04.013199+0000","last_clean":"2024-03-16T10:15:04.013199+0000","last_became_active":"2024-03-16T10:15:03.019781+0000","last_became_peered":"2024-03-16T10:15:03.019781+0000","last_unstale":"2024-03-16T10:15:04.013199+0000","last_undegraded":"2024-03-16T10:15:04.013199+0000","last_fullsized":"2024-03-16T10:15:04.013199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:04:49.369484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.251678+0000","last_change":"2024-03-16T10:15:02.014461+0000","last_active":"2024-03-16T10:15:06.251678+0000","last_peered":"2024-03-16T10:15:06.251678+0000","last_clean":"2024-03-16T10:15:06.251678+0000","last_became_active":"2024-03-16T10:15:02.014137+0000","last_became_peered":"2024-03-16T10:15:02.014137+0000","last_unstale":"2024-03-16T10:15:06.251678+0000","last_undegraded":"2024-03-16T10:15:06.251678+0000","last_fullsized":"2024-03-16T10:15:06.251678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:36:41.830893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.023156+0000","last_change":"2024-03-16T10:15:02.019441+0000","last_active":"2024-03-16T10:15:04.023156+0000","last_peered":"2024-03-16T10:15:04.023156+0000","last_clean":"2024-03-16T10:15:04.023156+0000","last_became_active":"2024-03-16T10:15:02.019091+0000","last_became_peered":"2024-03-16T10:15:02.019091+0000","last_unstale":"2024-03-16T10:15:04.023156+0000","last_undegraded":"2024-03-16T10:15:04.023156+0000","last_fullsized":"2024-03-16T10:15:04.023156+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:04:51.758722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013478+0000","last_change":"2024-03-16T10:15:03.012141+0000","last_active":"2024-03-16T10:15:04.013478+0000","last_peered":"2024-03-16T10:15:04.013478+0000","last_clean":"2024-03-16T10:15:04.013478+0000","last_became_active":"2024-03-16T10:15:03.011771+0000","last_became_peered":"2024-03-16T10:15:03.011771+0000","last_unstale":"2024-03-16T10:15:04.013478+0000","last_undegraded":"2024-03-16T10:15:04.013478+0000","last_fullsized":"2024-03-16T10:15:04.013478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:02:28.522530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:07.264691+0000","last_change":"2024-03-16T10:15:02.011898+0000","last_active":"2024-03-16T10:15:07.264691+0000","last_peered":"2024-03-16T10:15:07.264691+0000","last_clean":"2024-03-16T10:15:07.264691+0000","last_became_active":"2024-03-16T10:15:02.011578+0000","last_became_peered":"2024-03-16T10:15:02.011578+0000","last_unstale":"2024-03-16T10:15:07.264691+0000","last_undegraded":"2024-03-16T10:15:07.264691+0000","last_fullsized":"2024-03-16T10:15:07.264691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:27:35.756031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022977+0000","last_change":"2024-03-16T10:15:03.024923+0000","last_active":"2024-03-16T10:15:04.022977+0000","last_peered":"2024-03-16T10:15:04.022977+0000","last_clean":"2024-03-16T10:15:04.022977+0000","last_became_active":"2024-03-16T10:15:03.024577+0000","last_became_peered":"2024-03-16T10:15:03.024577+0000","last_unstale":"2024-03-16T10:15:04.022977+0000","last_undegraded":"2024-03-16T10:15:04.022977+0000","last_fullsized":"2024-03-16T10:15:04.022977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:04.650499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.247188+0000","last_change":"2024-03-16T10:15:02.006842+0000","last_active":"2024-03-16T10:15:05.247188+0000","last_peered":"2024-03-16T10:15:05.247188+0000","last_clean":"2024-03-16T10:15:05.247188+0000","last_became_active":"2024-03-16T10:15:02.006483+0000","last_became_peered":"2024-03-16T10:15:02.006483+0000","last_unstale":"2024-03-16T10:15:05.247188+0000","last_undegraded":"2024-03-16T10:15:05.247188+0000","last_fullsized":"2024-03-16T10:15:05.247188+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:30:55.856007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390026+0000","last_change":"2024-03-16T10:15:03.018691+0000","last_active":"2024-03-16T10:15:04.390026+0000","last_peered":"2024-03-16T10:15:04.390026+0000","last_clean":"2024-03-16T10:15:04.390026+0000","last_became_active":"2024-03-16T10:15:03.018304+0000","last_became_peered":"2024-03-16T10:15:03.018304+0000","last_unstale":"2024-03-16T10:15:04.390026+0000","last_undegraded":"2024-03-16T10:15:04.390026+0000","last_fullsized":"2024-03-16T10:15:04.390026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:15.446759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.248013+0000","last_change":"2024-03-16T10:15:02.008011+0000","last_active":"2024-03-16T10:15:05.248013+0000","last_peered":"2024-03-16T10:15:05.248013+0000","last_clean":"2024-03-16T10:15:05.248013+0000","last_became_active":"2024-03-16T10:15:02.007640+0000","last_became_peered":"2024-03-16T10:15:02.007640+0000","last_unstale":"2024-03-16T10:15:05.248013+0000","last_undegraded":"2024-03-16T10:15:05.248013+0000","last_fullsized":"2024-03-16T10:15:05.248013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:22.894518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022799+0000","last_change":"2024-03-16T10:15:03.017262+0000","last_active":"2024-03-16T10:15:04.022799+0000","last_peered":"2024-03-16T10:15:04.022799+0000","last_clean":"2024-03-16T10:15:04.022799+0000","last_became_active":"2024-03-16T10:15:03.016909+0000","last_became_peered":"2024-03-16T10:15:03.016909+0000","last_unstale":"2024-03-16T10:15:04.022799+0000","last_undegraded":"2024-03-16T10:15:04.022799+0000","last_fullsized":"2024-03-16T10:15:04.022799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:53:11.385236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.247181+0000","last_change":"2024-03-16T10:15:02.011325+0000","last_active":"2024-03-16T10:15:05.247181+0000","last_peered":"2024-03-16T10:15:05.247181+0000","last_clean":"2024-03-16T10:15:05.247181+0000","last_became_active":"2024-03-16T10:15:02.010999+0000","last_became_peered":"2024-03-16T10:15:02.010999+0000","last_unstale":"2024-03-16T10:15:05.247181+0000","last_undegraded":"2024-03-16T10:15:05.247181+0000","last_fullsized":"2024-03-16T10:15:05.247181+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:30:08.137693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.525756+0000","last_change":"2024-03-16T10:15:03.012146+0000","last_active":"2024-03-16T10:15:04.525756+0000","last_peered":"2024-03-16T10:15:04.525756+0000","last_clean":"2024-03-16T10:15:04.525756+0000","last_became_active":"2024-03-16T10:15:03.011790+0000","last_became_peered":"2024-03-16T10:15:03.011790+0000","last_unstale":"2024-03-16T10:15:04.525756+0000","last_undegraded":"2024-03-16T10:15:04.525756+0000","last_fullsized":"2024-03-16T10:15:04.525756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:45:22.642925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.243205+0000","last_change":"2024-03-16T10:15:02.017419+0000","last_active":"2024-03-16T10:15:06.243205+0000","last_peered":"2024-03-16T10:15:06.243205+0000","last_clean":"2024-03-16T10:15:06.243205+0000","last_became_active":"2024-03-16T10:15:02.016707+0000","last_became_peered":"2024-03-16T10:15:02.016707+0000","last_unstale":"2024-03-16T10:15:06.243205+0000","last_undegraded":"2024-03-16T10:15:06.243205+0000","last_fullsized":"2024-03-16T10:15:06.243205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:57:41.783962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022915+0000","last_change":"2024-03-16T10:15:03.021094+0000","last_active":"2024-03-16T10:15:04.022915+0000","last_peered":"2024-03-16T10:15:04.022915+0000","last_clean":"2024-03-16T10:15:04.022915+0000","last_became_active":"2024-03-16T10:15:03.017640+0000","last_became_peered":"2024-03-16T10:15:03.017640+0000","last_unstale":"2024-03-16T10:15:04.022915+0000","last_undegraded":"2024-03-16T10:15:04.022915+0000","last_fullsized":"2024-03-16T10:15:04.022915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:02.030112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.247781+0000","last_change":"2024-03-16T10:15:02.015652+0000","last_active":"2024-03-16T10:15:05.247781+0000","last_peered":"2024-03-16T10:15:05.247781+0000","last_clean":"2024-03-16T10:15:05.247781+0000","last_became_active":"2024-03-16T10:15:02.014046+0000","last_became_peered":"2024-03-16T10:15:02.014046+0000","last_unstale":"2024-03-16T10:15:05.247781+0000","last_undegraded":"2024-03-16T10:15:05.247781+0000","last_fullsized":"2024-03-16T10:15:05.247781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390642+0000","last_change":"2024-03-16T10:15:03.023233+0000","last_active":"2024-03-16T10:15:04.390642+0000","last_peered":"2024-03-16T10:15:04.390642+0000","last_clean":"2024-03-16T10:15:04.390642+0000","last_became_active":"2024-03-16T10:15:03.022906+0000","last_became_peered":"2024-03-16T10:15:03.022906+0000","last_unstale":"2024-03-16T10:15:04.390642+0000","last_undegraded":"2024-03-16T10:15:04.390642+0000","last_fullsized":"2024-03-16T10:15:04.390642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:22.943087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:07.261553+0000","last_change":"2024-03-16T10:15:02.010755+0000","last_active":"2024-03-16T10:15:07.261553+0000","last_peered":"2024-03-16T10:15:07.261553+0000","last_clean":"2024-03-16T10:15:07.261553+0000","last_became_active":"2024-03-16T10:15:02.010422+0000","last_became_peered":"2024-03-16T10:15:02.010422+0000","last_unstale":"2024-03-16T10:15:07.261553+0000","last_undegraded":"2024-03-16T10:15:07.261553+0000","last_fullsized":"2024-03-16T10:15:07.261553+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:17.606346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.391301+0000","last_change":"2024-03-16T10:15:03.025789+0000","last_active":"2024-03-16T10:15:04.391301+0000","last_peered":"2024-03-16T10:15:04.391301+0000","last_clean":"2024-03-16T10:15:04.391301+0000","last_became_active":"2024-03-16T10:15:03.024812+0000","last_became_peered":"2024-03-16T10:15:03.024812+0000","last_unstale":"2024-03-16T10:15:04.391301+0000","last_undegraded":"2024-03-16T10:15:04.391301+0000","last_fullsized":"2024-03-16T10:15:04.391301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:53.041234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390214+0000","last_change":"2024-03-16T10:15:02.020599+0000","last_active":"2024-03-16T10:15:04.390214+0000","last_peered":"2024-03-16T10:15:04.390214+0000","last_clean":"2024-03-16T10:15:04.390214+0000","last_became_active":"2024-03-16T10:15:02.020203+0000","last_became_peered":"2024-03-16T10:15:02.020203+0000","last_unstale":"2024-03-16T10:15:04.390214+0000","last_undegraded":"2024-03-16T10:15:04.390214+0000","last_fullsized":"2024-03-16T10:15:04.390214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:46:33.032310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390824+0000","last_change":"2024-03-16T10:15:03.016223+0000","last_active":"2024-03-16T10:15:04.390824+0000","last_peered":"2024-03-16T10:15:04.390824+0000","last_clean":"2024-03-16T10:15:04.390824+0000","last_became_active":"2024-03-16T10:15:03.015857+0000","last_became_peered":"2024-03-16T10:15:03.015857+0000","last_unstale":"2024-03-16T10:15:04.390824+0000","last_undegraded":"2024-03-16T10:15:04.390824+0000","last_fullsized":"2024-03-16T10:15:04.390824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:40:10.329722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013515+0000","last_change":"2024-03-16T10:15:02.015690+0000","last_active":"2024-03-16T10:15:04.013515+0000","last_peered":"2024-03-16T10:15:04.013515+0000","last_clean":"2024-03-16T10:15:04.013515+0000","last_became_active":"2024-03-16T10:15:02.015375+0000","last_became_peered":"2024-03-16T10:15:02.015375+0000","last_unstale":"2024-03-16T10:15:04.013515+0000","last_undegraded":"2024-03-16T10:15:04.013515+0000","last_fullsized":"2024-03-16T10:15:04.013515+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:39.118343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.391233+0000","last_change":"2024-03-16T10:15:03.012702+0000","last_active":"2024-03-16T10:15:04.391233+0000","last_peered":"2024-03-16T10:15:04.391233+0000","last_clean":"2024-03-16T10:15:04.391233+0000","last_became_active":"2024-03-16T10:15:03.012387+0000","last_became_peered":"2024-03-16T10:15:03.012387+0000","last_unstale":"2024-03-16T10:15:04.391233+0000","last_undegraded":"2024-03-16T10:15:04.391233+0000","last_fullsized":"2024-03-16T10:15:04.391233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:16.938372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.525868+0000","last_change":"2024-03-16T10:15:03.015290+0000","last_active":"2024-03-16T10:15:04.525868+0000","last_peered":"2024-03-16T10:15:04.525868+0000","last_clean":"2024-03-16T10:15:04.525868+0000","last_became_active":"2024-03-16T10:15:03.013846+0000","last_became_peered":"2024-03-16T10:15:03.013846+0000","last_unstale":"2024-03-16T10:15:04.525868+0000","last_undegraded":"2024-03-16T10:15:04.525868+0000","last_fullsized":"2024-03-16T10:15:04.525868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:33.377793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013983+0000","last_change":"2024-03-16T10:15:03.023733+0000","last_active":"2024-03-16T10:15:04.013983+0000","last_peered":"2024-03-16T10:15:04.013983+0000","last_clean":"2024-03-16T10:15:04.013983+0000","last_became_active":"2024-03-16T10:15:03.023409+0000","last_became_peered":"2024-03-16T10:15:03.023409+0000","last_unstale":"2024-03-16T10:15:04.013983+0000","last_undegraded":"2024-03-16T10:15:04.013983+0000","last_fullsized":"2024-03-16T10:15:04.013983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:44.748152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.014242+0000","last_change":"2024-03-16T10:15:03.021254+0000","last_active":"2024-03-16T10:15:04.014242+0000","last_peered":"2024-03-16T10:15:04.014242+0000","last_clean":"2024-03-16T10:15:04.014242+0000","last_became_active":"2024-03-16T10:15:03.020885+0000","last_became_peered":"2024-03-16T10:15:03.020885+0000","last_unstale":"2024-03-16T10:15:04.014242+0000","last_undegraded":"2024-03-16T10:15:04.014242+0000","last_fullsized":"2024-03-16T10:15:04.014242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:38:47.269571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.389821+0000","last_change":"2024-03-16T10:15:03.029277+0000","last_active":"2024-03-16T10:15:04.389821+0000","last_peered":"2024-03-16T10:15:04.389821+0000","last_clean":"2024-03-16T10:15:04.389821+0000","last_became_active":"2024-03-16T10:15:03.028960+0000","last_became_peered":"2024-03-16T10:15:03.028960+0000","last_unstale":"2024-03-16T10:15:04.389821+0000","last_undegraded":"2024-03-16T10:15:04.389821+0000","last_fullsized":"2024-03-16T10:15:04.389821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:50:48.362914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013457+0000","last_change":"2024-03-16T10:15:03.020773+0000","last_active":"2024-03-16T10:15:04.013457+0000","last_peered":"2024-03-16T10:15:04.013457+0000","last_clean":"2024-03-16T10:15:04.013457+0000","last_became_active":"2024-03-16T10:15:03.020450+0000","last_became_peered":"2024-03-16T10:15:03.020450+0000","last_unstale":"2024-03-16T10:15:04.013457+0000","last_undegraded":"2024-03-16T10:15:04.013457+0000","last_fullsized":"2024-03-16T10:15:04.013457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:26:42.005725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.526024+0000","last_change":"2024-03-16T10:15:03.017499+0000","last_active":"2024-03-16T10:15:04.526024+0000","last_peered":"2024-03-16T10:15:04.526024+0000","last_clean":"2024-03-16T10:15:04.526024+0000","last_became_active":"2024-03-16T10:15:03.017085+0000","last_became_peered":"2024-03-16T10:15:03.017085+0000","last_unstale":"2024-03-16T10:15:04.526024+0000","last_undegraded":"2024-03-16T10:15:04.526024+0000","last_fullsized":"2024-03-16T10:15:04.526024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:57:09.592420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.014459+0000","last_change":"2024-03-16T10:15:03.018943+0000","last_active":"2024-03-16T10:15:04.014459+0000","last_peered":"2024-03-16T10:15:04.014459+0000","last_clean":"2024-03-16T10:15:04.014459+0000","last_became_active":"2024-03-16T10:15:03.017519+0000","last_became_peered":"2024-03-16T10:15:03.017519+0000","last_unstale":"2024-03-16T10:15:04.014459+0000","last_undegraded":"2024-03-16T10:15:04.014459+0000","last_fullsized":"2024-03-16T10:15:04.014459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:29.026988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022802+0000","last_change":"2024-03-16T10:15:03.024697+0000","last_active":"2024-03-16T10:15:04.022802+0000","last_peered":"2024-03-16T10:15:04.022802+0000","last_clean":"2024-03-16T10:15:04.022802+0000","last_became_active":"2024-03-16T10:15:03.024305+0000","last_became_peered":"2024-03-16T10:15:03.024305+0000","last_unstale":"2024-03-16T10:15:04.022802+0000","last_undegraded":"2024-03-16T10:15:04.022802+0000","last_fullsized":"2024-03-16T10:15:04.022802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:07:35.160746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390239+0000","last_change":"2024-03-16T10:15:03.017504+0000","last_active":"2024-03-16T10:15:04.390239+0000","last_peered":"2024-03-16T10:15:04.390239+0000","last_clean":"2024-03-16T10:15:04.390239+0000","last_became_active":"2024-03-16T10:15:03.017100+0000","last_became_peered":"2024-03-16T10:15:03.017100+0000","last_unstale":"2024-03-16T10:15:04.390239+0000","last_undegraded":"2024-03-16T10:15:04.390239+0000","last_fullsized":"2024-03-16T10:15:04.390239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:23:17.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022564+0000","last_change":"2024-03-16T10:15:03.024684+0000","last_active":"2024-03-16T10:15:04.022564+0000","last_peered":"2024-03-16T10:15:04.022564+0000","last_clean":"2024-03-16T10:15:04.022564+0000","last_became_active":"2024-03-16T10:15:03.024282+0000","last_became_peered":"2024-03-16T10:15:03.024282+0000","last_unstale":"2024-03-16T10:15:04.022564+0000","last_undegraded":"2024-03-16T10:15:04.022564+0000","last_fullsized":"2024-03-16T10:15:04.022564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:29.595162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013794+0000","last_change":"2024-03-16T10:15:03.025337+0000","last_active":"2024-03-16T10:15:04.013794+0000","last_peered":"2024-03-16T10:15:04.013794+0000","last_clean":"2024-03-16T10:15:04.013794+0000","last_became_active":"2024-03-16T10:15:03.024938+0000","last_became_peered":"2024-03-16T10:15:03.024938+0000","last_unstale":"2024-03-16T10:15:04.013794+0000","last_undegraded":"2024-03-16T10:15:04.013794+0000","last_fullsized":"2024-03-16T10:15:04.013794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:13:15.892354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013930+0000","last_change":"2024-03-16T10:15:03.015468+0000","last_active":"2024-03-16T10:15:04.013930+0000","last_peered":"2024-03-16T10:15:04.013930+0000","last_clean":"2024-03-16T10:15:04.013930+0000","last_became_active":"2024-03-16T10:15:03.015097+0000","last_became_peered":"2024-03-16T10:15:03.015097+0000","last_unstale":"2024-03-16T10:15:04.013930+0000","last_undegraded":"2024-03-16T10:15:04.013930+0000","last_fullsized":"2024-03-16T10:15:04.013930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:46.114346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.525839+0000","last_change":"2024-03-16T10:15:03.020690+0000","last_active":"2024-03-16T10:15:04.525839+0000","last_peered":"2024-03-16T10:15:04.525839+0000","last_clean":"2024-03-16T10:15:04.525839+0000","last_became_active":"2024-03-16T10:15:03.020365+0000","last_became_peered":"2024-03-16T10:15:03.020365+0000","last_unstale":"2024-03-16T10:15:04.525839+0000","last_undegraded":"2024-03-16T10:15:04.525839+0000","last_fullsized":"2024-03-16T10:15:04.525839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:25.671533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280280,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6437,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":154,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":27607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383693,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":440104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1032203,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":439429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070494,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":439147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332130,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":1030093,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":442276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":441090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":441483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":438805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":439187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757292,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":1029071,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":9964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T10:15:11.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T10:15:11.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:11 smithi193 ceph-mon[36311]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:11.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:11 smithi193 ceph-mon[36311]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:11.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:11 smithi193 ceph-mon[36311]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:15:11.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:11 smithi193 ceph-mon[36311]: pgmap v291: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s rd, 5.5 KiB/s wr, 20 op/s 2024-03-16T10:15:11.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:11 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2632807412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:15:11.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.827+0000 7f9c125a9640 1 -- 172.21.15.77:0/1866418577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c0c101970 msgr2=0x7f9c0c101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.827+0000 7f9c125a9640 1 --2- 172.21.15.77:0/1866418577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c0c101970 0x7f9c0c101df0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009900 tx=0x7f9c0002bc30 comp rx=0 tx=0).stop 2024-03-16T10:15:11.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.827+0000 7f9c125a9640 1 -- 172.21.15.77:0/1866418577 shutdown_connections 2024-03-16T10:15:11.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.827+0000 7f9c125a9640 1 --2- 172.21.15.77:0/1866418577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c0c102330 0x7f9c0c108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.827+0000 7f9c125a9640 1 --2- 172.21.15.77:0/1866418577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c0c101970 0x7f9c0c101df0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.827+0000 7f9c125a9640 1 --2- 172.21.15.77:0/1866418577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c0c100780 0x7f9c0c100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.827+0000 7f9c125a9640 1 -- 172.21.15.77:0/1866418577 >> 172.21.15.77:0/1866418577 conn(0x7f9c0c0fbd10 msgr2=0x7f9c0c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.827+0000 7f9c125a9640 1 -- 172.21.15.77:0/1866418577 shutdown_connections 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.827+0000 7f9c125a9640 1 -- 172.21.15.77:0/1866418577 wait complete. 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c125a9640 1 Processor -- start 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c125a9640 1 -- start start 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c125a9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c0c100780 0x7f9c0c1969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c125a9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c0c101970 0x7f9c0c196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c125a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c0c102330 0x7f9c0c19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c125a9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c10c370 con 0x7f9c0c102330 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c125a9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c10c1f0 con 0x7f9c0c100780 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c125a9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c10c4f0 con 0x7f9c0c101970 2024-03-16T10:15:11.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c10b46640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c0c102330 0x7f9c0c19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:11.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c10b46640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c0c102330 0x7f9c0c19df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42336/0 (socket says 172.21.15.77:42336) 2024-03-16T10:15:11.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c10b46640 1 -- 172.21.15.77:0/2429949656 learned_addr learned my addr 172.21.15.77:0/2429949656 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:11.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c0bfff640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c0c100780 0x7f9c0c1969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:11.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c0b7fe640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c0c101970 0x7f9c0c196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:11.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c10b46640 1 -- 172.21.15.77:0/2429949656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c0c101970 msgr2=0x7f9c0c196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c10b46640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c0c101970 0x7f9c0c196f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c10b46640 1 -- 172.21.15.77:0/2429949656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c0c100780 msgr2=0x7f9c0c1969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:11.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.828+0000 7f9c10b46640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c0c100780 0x7f9c0c1969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:11.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.829+0000 7f9c10b46640 1 -- 172.21.15.77:0/2429949656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c0c19e600 con 0x7f9c0c102330 2024-03-16T10:15:11.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.829+0000 7f9c10b46640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c0c102330 0x7f9c0c19df90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400eaa0 tx=0x7f9c0400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:11.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.829+0000 7f9c097fa640 1 -- 172.21.15.77:0/2429949656 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0400cdb0 con 0x7f9c0c102330 2024-03-16T10:15:11.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.829+0000 7f9c125a9640 1 -- 172.21.15.77:0/2429949656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c0c19e8f0 con 0x7f9c0c102330 2024-03-16T10:15:11.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.829+0000 7f9c097fa640 1 -- 172.21.15.77:0/2429949656 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c04005020 con 0x7f9c0c102330 2024-03-16T10:15:11.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.829+0000 7f9c097fa640 1 -- 172.21.15.77:0/2429949656 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c040116b0 con 0x7f9c0c102330 2024-03-16T10:15:11.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.829+0000 7f9c125a9640 1 -- 172.21.15.77:0/2429949656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c0c19edc0 con 0x7f9c0c102330 2024-03-16T10:15:11.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.830+0000 7f9c125a9640 1 -- 172.21.15.77:0/2429949656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c0c107af0 con 0x7f9c0c102330 2024-03-16T10:15:11.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.831+0000 7f9c097fa640 1 -- 172.21.15.77:0/2429949656 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9c04005560 con 0x7f9c0c102330 2024-03-16T10:15:11.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.831+0000 7f9c097fa640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9bf40a2890 0x7f9bf40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:11.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.832+0000 7f9c0bfff640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9bf40a2890 0x7f9bf40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:11.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.832+0000 7f9c097fa640 1 -- 172.21.15.77:0/2429949656 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9c040cf000 con 0x7f9c0c102330 2024-03-16T10:15:11.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.832+0000 7f9c0bfff640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9bf40a2890 0x7f9bf40a4d50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c1017d0 tx=0x7f9bfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:11.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:11.835+0000 7f9c097fa640 1 -- 172.21.15.77:0/2429949656 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9c04016030 con 0x7f9c0c102330 2024-03-16T10:15:12.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.010+0000 7f9c125a9640 1 -- 172.21.15.77:0/2429949656 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9c0c061c20 con 0x7f9bf40a2890 2024-03-16T10:15:12.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.020+0000 7f9c097fa640 1 -- 172.21.15.77:0/2429949656 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258911 (secure 0 0 0) 0x7f9c0c061c20 con 0x7f9bf40a2890 2024-03-16T10:15:12.023 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:15:12.024 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T10:15:12.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.023+0000 7f9c125a9640 1 -- 172.21.15.77:0/2429949656 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9bf40a2890 msgr2=0x7f9bf40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:12.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.023+0000 7f9c125a9640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9bf40a2890 0x7f9bf40a4d50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c1017d0 tx=0x7f9bfc005f90 comp rx=0 tx=0).stop 2024-03-16T10:15:12.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.023+0000 7f9c125a9640 1 -- 172.21.15.77:0/2429949656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c0c102330 msgr2=0x7f9c0c19df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:12.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.023+0000 7f9c125a9640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c0c102330 0x7f9c0c19df90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400eaa0 tx=0x7f9c0400ef60 comp rx=0 tx=0).stop 2024-03-16T10:15:12.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.023+0000 7f9c125a9640 1 -- 172.21.15.77:0/2429949656 shutdown_connections 2024-03-16T10:15:12.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.023+0000 7f9c125a9640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c0c102330 0x7f9c0c19df90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:12.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.023+0000 7f9c125a9640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9bf40a2890 0x7f9bf40a4d50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:12.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.023+0000 7f9c125a9640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c0c101970 0x7f9c0c196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:12.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.023+0000 7f9c125a9640 1 --2- 172.21.15.77:0/2429949656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c0c100780 0x7f9c0c1969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:12.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.023+0000 7f9c125a9640 1 -- 172.21.15.77:0/2429949656 >> 172.21.15.77:0/2429949656 conn(0x7f9c0c0fbd10 msgr2=0x7f9c0c104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:12.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.023+0000 7f9c125a9640 1 -- 172.21.15.77:0/2429949656 shutdown_connections 2024-03-16T10:15:12.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:12.023+0000 7f9c125a9640 1 -- 172.21.15.77:0/2429949656 wait complete. 2024-03-16T10:15:12.034 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-03-16T10:15:10.717261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596805,"num_objects":74,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":133,"num_read_kb":116,"num_write":335,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":270,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360276,"kb_used_data":11104,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581548,"statfs":{"total":1151940427776,"available":1151571505152,"internally_reserved":0,"allocated":11370496,"data_stored":7051409,"data_compressed":106950,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6437,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":154,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002167"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.526082+0000","last_change":"2024-03-16T10:15:03.016880+0000","last_active":"2024-03-16T10:15:04.526082+0000","last_peered":"2024-03-16T10:15:04.526082+0000","last_clean":"2024-03-16T10:15:04.526082+0000","last_became_active":"2024-03-16T10:15:03.016513+0000","last_became_peered":"2024-03-16T10:15:03.016513+0000","last_unstale":"2024-03-16T10:15:04.526082+0000","last_undegraded":"2024-03-16T10:15:04.526082+0000","last_fullsized":"2024-03-16T10:15:04.526082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:39.834015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390079+0000","last_change":"2024-03-16T10:15:03.015467+0000","last_active":"2024-03-16T10:15:04.390079+0000","last_peered":"2024-03-16T10:15:04.390079+0000","last_clean":"2024-03-16T10:15:04.390079+0000","last_became_active":"2024-03-16T10:15:03.015152+0000","last_became_peered":"2024-03-16T10:15:03.015152+0000","last_unstale":"2024-03-16T10:15:04.390079+0000","last_undegraded":"2024-03-16T10:15:04.390079+0000","last_fullsized":"2024-03-16T10:15:04.390079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:17:29.988569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.389915+0000","last_change":"2024-03-16T10:15:03.023836+0000","last_active":"2024-03-16T10:15:04.389915+0000","last_peered":"2024-03-16T10:15:04.389915+0000","last_clean":"2024-03-16T10:15:04.389915+0000","last_became_active":"2024-03-16T10:15:03.022774+0000","last_became_peered":"2024-03-16T10:15:03.022774+0000","last_unstale":"2024-03-16T10:15:04.389915+0000","last_undegraded":"2024-03-16T10:15:04.389915+0000","last_fullsized":"2024-03-16T10:15:04.389915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:49.837157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390609+0000","last_change":"2024-03-16T10:15:03.030891+0000","last_active":"2024-03-16T10:15:04.390609+0000","last_peered":"2024-03-16T10:15:04.390609+0000","last_clean":"2024-03-16T10:15:04.390609+0000","last_became_active":"2024-03-16T10:15:03.030278+0000","last_became_peered":"2024-03-16T10:15:03.030278+0000","last_unstale":"2024-03-16T10:15:04.390609+0000","last_undegraded":"2024-03-16T10:15:04.390609+0000","last_fullsized":"2024-03-16T10:15:04.390609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:16.166885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.023143+0000","last_change":"2024-03-16T10:15:03.017610+0000","last_active":"2024-03-16T10:15:04.023143+0000","last_peered":"2024-03-16T10:15:04.023143+0000","last_clean":"2024-03-16T10:15:04.023143+0000","last_became_active":"2024-03-16T10:15:03.017250+0000","last_became_peered":"2024-03-16T10:15:03.017250+0000","last_unstale":"2024-03-16T10:15:04.023143+0000","last_undegraded":"2024-03-16T10:15:04.023143+0000","last_fullsized":"2024-03-16T10:15:04.023143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:07.899425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390917+0000","last_change":"2024-03-16T10:15:03.020720+0000","last_active":"2024-03-16T10:15:04.390917+0000","last_peered":"2024-03-16T10:15:04.390917+0000","last_clean":"2024-03-16T10:15:04.390917+0000","last_became_active":"2024-03-16T10:15:03.020334+0000","last_became_peered":"2024-03-16T10:15:03.020334+0000","last_unstale":"2024-03-16T10:15:04.390917+0000","last_undegraded":"2024-03-16T10:15:04.390917+0000","last_fullsized":"2024-03-16T10:15:04.390917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:42:19.210432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390182+0000","last_change":"2024-03-16T10:15:03.017686+0000","last_active":"2024-03-16T10:15:04.390182+0000","last_peered":"2024-03-16T10:15:04.390182+0000","last_clean":"2024-03-16T10:15:04.390182+0000","last_became_active":"2024-03-16T10:15:03.017262+0000","last_became_peered":"2024-03-16T10:15:03.017262+0000","last_unstale":"2024-03-16T10:15:04.390182+0000","last_undegraded":"2024-03-16T10:15:04.390182+0000","last_fullsized":"2024-03-16T10:15:04.390182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:36:22.270651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022666+0000","last_change":"2024-03-16T10:15:03.020130+0000","last_active":"2024-03-16T10:15:04.022666+0000","last_peered":"2024-03-16T10:15:04.022666+0000","last_clean":"2024-03-16T10:15:04.022666+0000","last_became_active":"2024-03-16T10:15:03.019742+0000","last_became_peered":"2024-03-16T10:15:03.019742+0000","last_unstale":"2024-03-16T10:15:04.022666+0000","last_undegraded":"2024-03-16T10:15:04.022666+0000","last_fullsized":"2024-03-16T10:15:04.022666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:52.450792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013990+0000","last_change":"2024-03-16T10:15:03.027589+0000","last_active":"2024-03-16T10:15:04.013990+0000","last_peered":"2024-03-16T10:15:04.013990+0000","last_clean":"2024-03-16T10:15:04.013990+0000","last_became_active":"2024-03-16T10:15:03.027226+0000","last_became_peered":"2024-03-16T10:15:03.027226+0000","last_unstale":"2024-03-16T10:15:04.013990+0000","last_undegraded":"2024-03-16T10:15:04.013990+0000","last_fullsized":"2024-03-16T10:15:04.013990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:14:35.354629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390491+0000","last_change":"2024-03-16T10:15:03.021163+0000","last_active":"2024-03-16T10:15:04.390491+0000","last_peered":"2024-03-16T10:15:04.390491+0000","last_clean":"2024-03-16T10:15:04.390491+0000","last_became_active":"2024-03-16T10:15:03.020483+0000","last_became_peered":"2024-03-16T10:15:03.020483+0000","last_unstale":"2024-03-16T10:15:04.390491+0000","last_undegraded":"2024-03-16T10:15:04.390491+0000","last_fullsized":"2024-03-16T10:15:04.390491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:01:00.873471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.391277+0000","last_change":"2024-03-16T10:15:03.026111+0000","last_active":"2024-03-16T10:15:04.391277+0000","last_peered":"2024-03-16T10:15:04.391277+0000","last_clean":"2024-03-16T10:15:04.391277+0000","last_became_active":"2024-03-16T10:15:03.025296+0000","last_became_peered":"2024-03-16T10:15:03.025296+0000","last_unstale":"2024-03-16T10:15:04.391277+0000","last_undegraded":"2024-03-16T10:15:04.391277+0000","last_fullsized":"2024-03-16T10:15:04.391277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:33:29.606876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390469+0000","last_change":"2024-03-16T10:15:03.023013+0000","last_active":"2024-03-16T10:15:04.390469+0000","last_peered":"2024-03-16T10:15:04.390469+0000","last_clean":"2024-03-16T10:15:04.390469+0000","last_became_active":"2024-03-16T10:15:03.022251+0000","last_became_peered":"2024-03-16T10:15:03.022251+0000","last_unstale":"2024-03-16T10:15:04.390469+0000","last_undegraded":"2024-03-16T10:15:04.390469+0000","last_fullsized":"2024-03-16T10:15:04.390469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:33.544709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013883+0000","last_change":"2024-03-16T10:15:03.009548+0000","last_active":"2024-03-16T10:15:04.013883+0000","last_peered":"2024-03-16T10:15:04.013883+0000","last_clean":"2024-03-16T10:15:04.013883+0000","last_became_active":"2024-03-16T10:15:03.007675+0000","last_became_peered":"2024-03-16T10:15:03.007675+0000","last_unstale":"2024-03-16T10:15:04.013883+0000","last_undegraded":"2024-03-16T10:15:04.013883+0000","last_fullsized":"2024-03-16T10:15:04.013883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:08:02.561602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.389735+0000","last_change":"2024-03-16T10:15:03.020146+0000","last_active":"2024-03-16T10:15:04.389735+0000","last_peered":"2024-03-16T10:15:04.389735+0000","last_clean":"2024-03-16T10:15:04.389735+0000","last_became_active":"2024-03-16T10:15:03.019823+0000","last_became_peered":"2024-03-16T10:15:03.019823+0000","last_unstale":"2024-03-16T10:15:04.389735+0000","last_undegraded":"2024-03-16T10:15:04.389735+0000","last_fullsized":"2024-03-16T10:15:04.389735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:52.276506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.389692+0000","last_change":"2024-03-16T10:15:03.019150+0000","last_active":"2024-03-16T10:15:04.389692+0000","last_peered":"2024-03-16T10:15:04.389692+0000","last_clean":"2024-03-16T10:15:04.389692+0000","last_became_active":"2024-03-16T10:15:03.018830+0000","last_became_peered":"2024-03-16T10:15:03.018830+0000","last_unstale":"2024-03-16T10:15:04.389692+0000","last_undegraded":"2024-03-16T10:15:04.389692+0000","last_fullsized":"2024-03-16T10:15:04.389692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:46.207187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013630+0000","last_change":"2024-03-16T10:15:03.033321+0000","last_active":"2024-03-16T10:15:04.013630+0000","last_peered":"2024-03-16T10:15:04.013630+0000","last_clean":"2024-03-16T10:15:04.013630+0000","last_became_active":"2024-03-16T10:15:03.032944+0000","last_became_peered":"2024-03-16T10:15:03.032944+0000","last_unstale":"2024-03-16T10:15:04.013630+0000","last_undegraded":"2024-03-16T10:15:04.013630+0000","last_fullsized":"2024-03-16T10:15:04.013630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:14.345488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.526221+0000","last_change":"2024-03-16T10:15:03.019897+0000","last_active":"2024-03-16T10:15:04.526221+0000","last_peered":"2024-03-16T10:15:04.526221+0000","last_clean":"2024-03-16T10:15:04.526221+0000","last_became_active":"2024-03-16T10:15:03.019555+0000","last_became_peered":"2024-03-16T10:15:03.019555+0000","last_unstale":"2024-03-16T10:15:04.526221+0000","last_undegraded":"2024-03-16T10:15:04.526221+0000","last_fullsized":"2024-03-16T10:15:04.526221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:58.307285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013466+0000","last_change":"2024-03-16T10:15:03.017482+0000","last_active":"2024-03-16T10:15:04.013466+0000","last_peered":"2024-03-16T10:15:04.013466+0000","last_clean":"2024-03-16T10:15:04.013466+0000","last_became_active":"2024-03-16T10:15:03.016521+0000","last_became_peered":"2024-03-16T10:15:03.016521+0000","last_unstale":"2024-03-16T10:15:04.013466+0000","last_undegraded":"2024-03-16T10:15:04.013466+0000","last_fullsized":"2024-03-16T10:15:04.013466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:11.889111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.014406+0000","last_change":"2024-03-16T10:15:03.018625+0000","last_active":"2024-03-16T10:15:04.014406+0000","last_peered":"2024-03-16T10:15:04.014406+0000","last_clean":"2024-03-16T10:15:04.014406+0000","last_became_active":"2024-03-16T10:15:03.018313+0000","last_became_peered":"2024-03-16T10:15:03.018313+0000","last_unstale":"2024-03-16T10:15:04.014406+0000","last_undegraded":"2024-03-16T10:15:04.014406+0000","last_fullsized":"2024-03-16T10:15:04.014406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:35:58.533559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:08.230706+0000","last_change":"2024-03-16T10:15:02.016545+0000","last_active":"2024-03-16T10:15:08.230706+0000","last_peered":"2024-03-16T10:15:08.230706+0000","last_clean":"2024-03-16T10:15:08.230706+0000","last_became_active":"2024-03-16T10:15:02.012088+0000","last_became_peered":"2024-03-16T10:15:02.012088+0000","last_unstale":"2024-03-16T10:15:08.230706+0000","last_undegraded":"2024-03-16T10:15:08.230706+0000","last_fullsized":"2024-03-16T10:15:08.230706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:50.633539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022920+0000","last_change":"2024-03-16T10:15:03.026565+0000","last_active":"2024-03-16T10:15:04.022920+0000","last_peered":"2024-03-16T10:15:04.022920+0000","last_clean":"2024-03-16T10:15:04.022920+0000","last_became_active":"2024-03-16T10:15:03.025386+0000","last_became_peered":"2024-03-16T10:15:03.025386+0000","last_unstale":"2024-03-16T10:15:04.022920+0000","last_undegraded":"2024-03-16T10:15:04.022920+0000","last_fullsized":"2024-03-16T10:15:04.022920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:28.831764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022959+0000","last_change":"2024-03-16T10:15:02.018097+0000","last_active":"2024-03-16T10:15:04.022959+0000","last_peered":"2024-03-16T10:15:04.022959+0000","last_clean":"2024-03-16T10:15:04.022959+0000","last_became_active":"2024-03-16T10:15:02.017777+0000","last_became_peered":"2024-03-16T10:15:02.017777+0000","last_unstale":"2024-03-16T10:15:04.022959+0000","last_undegraded":"2024-03-16T10:15:04.022959+0000","last_fullsized":"2024-03-16T10:15:04.022959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:04:58.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390097+0000","last_change":"2024-03-16T10:15:03.023384+0000","last_active":"2024-03-16T10:15:04.390097+0000","last_peered":"2024-03-16T10:15:04.390097+0000","last_clean":"2024-03-16T10:15:04.390097+0000","last_became_active":"2024-03-16T10:15:03.022996+0000","last_became_peered":"2024-03-16T10:15:03.022996+0000","last_unstale":"2024-03-16T10:15:04.390097+0000","last_undegraded":"2024-03-16T10:15:04.390097+0000","last_fullsized":"2024-03-16T10:15:04.390097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:29:20.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.251786+0000","last_change":"2024-03-16T10:15:02.014409+0000","last_active":"2024-03-16T10:15:06.251786+0000","last_peered":"2024-03-16T10:15:06.251786+0000","last_clean":"2024-03-16T10:15:06.251786+0000","last_became_active":"2024-03-16T10:15:02.014015+0000","last_became_peered":"2024-03-16T10:15:02.014015+0000","last_unstale":"2024-03-16T10:15:06.251786+0000","last_undegraded":"2024-03-16T10:15:06.251786+0000","last_fullsized":"2024-03-16T10:15:06.251786+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:25:53.133466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013656+0000","last_change":"2024-03-16T10:15:03.031328+0000","last_active":"2024-03-16T10:15:04.013656+0000","last_peered":"2024-03-16T10:15:04.013656+0000","last_clean":"2024-03-16T10:15:04.013656+0000","last_became_active":"2024-03-16T10:15:03.029787+0000","last_became_peered":"2024-03-16T10:15:03.029787+0000","last_unstale":"2024-03-16T10:15:04.013656+0000","last_undegraded":"2024-03-16T10:15:04.013656+0000","last_fullsized":"2024-03-16T10:15:04.013656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:34.321883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.023123+0000","last_change":"2024-03-16T10:15:02.018032+0000","last_active":"2024-03-16T10:15:04.023123+0000","last_peered":"2024-03-16T10:15:04.023123+0000","last_clean":"2024-03-16T10:15:04.023123+0000","last_became_active":"2024-03-16T10:15:02.017677+0000","last_became_peered":"2024-03-16T10:15:02.017677+0000","last_unstale":"2024-03-16T10:15:04.023123+0000","last_undegraded":"2024-03-16T10:15:04.023123+0000","last_fullsized":"2024-03-16T10:15:04.023123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:58:07.179129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390477+0000","last_change":"2024-03-16T10:15:03.024958+0000","last_active":"2024-03-16T10:15:04.390477+0000","last_peered":"2024-03-16T10:15:04.390477+0000","last_clean":"2024-03-16T10:15:04.390477+0000","last_became_active":"2024-03-16T10:15:03.023964+0000","last_became_peered":"2024-03-16T10:15:03.023964+0000","last_unstale":"2024-03-16T10:15:04.390477+0000","last_undegraded":"2024-03-16T10:15:04.390477+0000","last_fullsized":"2024-03-16T10:15:04.390477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:55:00.953258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.249865+0000","last_change":"2024-03-16T10:15:02.015062+0000","last_active":"2024-03-16T10:15:05.249865+0000","last_peered":"2024-03-16T10:15:05.249865+0000","last_clean":"2024-03-16T10:15:05.249865+0000","last_became_active":"2024-03-16T10:15:02.014714+0000","last_became_peered":"2024-03-16T10:15:02.014714+0000","last_unstale":"2024-03-16T10:15:05.249865+0000","last_undegraded":"2024-03-16T10:15:05.249865+0000","last_fullsized":"2024-03-16T10:15:05.249865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:23:55.313009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022990+0000","last_change":"2024-03-16T10:15:03.013956+0000","last_active":"2024-03-16T10:15:04.022990+0000","last_peered":"2024-03-16T10:15:04.022990+0000","last_clean":"2024-03-16T10:15:04.022990+0000","last_became_active":"2024-03-16T10:15:03.013607+0000","last_became_peered":"2024-03-16T10:15:03.013607+0000","last_unstale":"2024-03-16T10:15:04.022990+0000","last_undegraded":"2024-03-16T10:15:04.022990+0000","last_fullsized":"2024-03-16T10:15:04.022990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:40:55.445395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:07.265129+0000","last_change":"2024-03-16T10:15:02.015651+0000","last_active":"2024-03-16T10:15:07.265129+0000","last_peered":"2024-03-16T10:15:07.265129+0000","last_clean":"2024-03-16T10:15:07.265129+0000","last_became_active":"2024-03-16T10:15:02.015254+0000","last_became_peered":"2024-03-16T10:15:02.015254+0000","last_unstale":"2024-03-16T10:15:07.265129+0000","last_undegraded":"2024-03-16T10:15:07.265129+0000","last_fullsized":"2024-03-16T10:15:07.265129+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:41:59.013606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390537+0000","last_change":"2024-03-16T10:15:03.023074+0000","last_active":"2024-03-16T10:15:04.390537+0000","last_peered":"2024-03-16T10:15:04.390537+0000","last_clean":"2024-03-16T10:15:04.390537+0000","last_became_active":"2024-03-16T10:15:03.022700+0000","last_became_peered":"2024-03-16T10:15:03.022700+0000","last_unstale":"2024-03-16T10:15:04.390537+0000","last_undegraded":"2024-03-16T10:15:04.390537+0000","last_fullsized":"2024-03-16T10:15:04.390537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:22.047929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.251472+0000","last_change":"2024-03-16T10:15:02.014793+0000","last_active":"2024-03-16T10:15:06.251472+0000","last_peered":"2024-03-16T10:15:06.251472+0000","last_clean":"2024-03-16T10:15:06.251472+0000","last_became_active":"2024-03-16T10:15:02.014459+0000","last_became_peered":"2024-03-16T10:15:02.014459+0000","last_unstale":"2024-03-16T10:15:06.251472+0000","last_undegraded":"2024-03-16T10:15:06.251472+0000","last_fullsized":"2024-03-16T10:15:06.251472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:59:45.035045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022608+0000","last_change":"2024-03-16T10:15:03.012713+0000","last_active":"2024-03-16T10:15:04.022608+0000","last_peered":"2024-03-16T10:15:04.022608+0000","last_clean":"2024-03-16T10:15:04.022608+0000","last_became_active":"2024-03-16T10:15:03.012407+0000","last_became_peered":"2024-03-16T10:15:03.012407+0000","last_unstale":"2024-03-16T10:15:04.022608+0000","last_undegraded":"2024-03-16T10:15:04.022608+0000","last_fullsized":"2024-03-16T10:15:04.022608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:23:14.592810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.248418+0000","last_change":"2024-03-16T10:15:02.015582+0000","last_active":"2024-03-16T10:15:05.248418+0000","last_peered":"2024-03-16T10:15:05.248418+0000","last_clean":"2024-03-16T10:15:05.248418+0000","last_became_active":"2024-03-16T10:15:02.015073+0000","last_became_peered":"2024-03-16T10:15:02.015073+0000","last_unstale":"2024-03-16T10:15:05.248418+0000","last_undegraded":"2024-03-16T10:15:05.248418+0000","last_fullsized":"2024-03-16T10:15:05.248418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:22:48.212448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013244+0000","last_change":"2024-03-16T10:15:03.016839+0000","last_active":"2024-03-16T10:15:04.013244+0000","last_peered":"2024-03-16T10:15:04.013244+0000","last_clean":"2024-03-16T10:15:04.013244+0000","last_became_active":"2024-03-16T10:15:03.016463+0000","last_became_peered":"2024-03-16T10:15:03.016463+0000","last_unstale":"2024-03-16T10:15:04.013244+0000","last_undegraded":"2024-03-16T10:15:04.013244+0000","last_fullsized":"2024-03-16T10:15:04.013244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:43:34.259049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:08.224296+0000","last_change":"2024-03-16T10:15:02.017884+0000","last_active":"2024-03-16T10:15:08.224296+0000","last_peered":"2024-03-16T10:15:08.224296+0000","last_clean":"2024-03-16T10:15:08.224296+0000","last_became_active":"2024-03-16T10:15:02.017511+0000","last_became_peered":"2024-03-16T10:15:02.017511+0000","last_unstale":"2024-03-16T10:15:08.224296+0000","last_undegraded":"2024-03-16T10:15:08.224296+0000","last_fullsized":"2024-03-16T10:15:08.224296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:31.281631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013899+0000","last_change":"2024-03-16T10:15:03.032259+0000","last_active":"2024-03-16T10:15:04.013899+0000","last_peered":"2024-03-16T10:15:04.013899+0000","last_clean":"2024-03-16T10:15:04.013899+0000","last_became_active":"2024-03-16T10:15:03.031884+0000","last_became_peered":"2024-03-16T10:15:03.031884+0000","last_unstale":"2024-03-16T10:15:04.013899+0000","last_undegraded":"2024-03-16T10:15:04.013899+0000","last_fullsized":"2024-03-16T10:15:04.013899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:37:48.464051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.247747+0000","last_change":"2024-03-16T10:15:02.015372+0000","last_active":"2024-03-16T10:15:05.247747+0000","last_peered":"2024-03-16T10:15:05.247747+0000","last_clean":"2024-03-16T10:15:05.247747+0000","last_became_active":"2024-03-16T10:15:02.015022+0000","last_became_peered":"2024-03-16T10:15:02.015022+0000","last_unstale":"2024-03-16T10:15:05.247747+0000","last_undegraded":"2024-03-16T10:15:05.247747+0000","last_fullsized":"2024-03-16T10:15:05.247747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:42.651903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390159+0000","last_change":"2024-03-16T10:15:03.018661+0000","last_active":"2024-03-16T10:15:04.390159+0000","last_peered":"2024-03-16T10:15:04.390159+0000","last_clean":"2024-03-16T10:15:04.390159+0000","last_became_active":"2024-03-16T10:15:03.018351+0000","last_became_peered":"2024-03-16T10:15:03.018351+0000","last_unstale":"2024-03-16T10:15:04.390159+0000","last_undegraded":"2024-03-16T10:15:04.390159+0000","last_fullsized":"2024-03-16T10:15:04.390159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:21:33.939811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.248221+0000","last_change":"2024-03-16T10:15:02.014343+0000","last_active":"2024-03-16T10:15:05.248221+0000","last_peered":"2024-03-16T10:15:05.248221+0000","last_clean":"2024-03-16T10:15:05.248221+0000","last_became_active":"2024-03-16T10:15:02.013983+0000","last_became_peered":"2024-03-16T10:15:02.013983+0000","last_unstale":"2024-03-16T10:15:05.248221+0000","last_undegraded":"2024-03-16T10:15:05.248221+0000","last_fullsized":"2024-03-16T10:15:05.248221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:35:04.768079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013766+0000","last_change":"2024-03-16T10:15:03.031055+0000","last_active":"2024-03-16T10:15:04.013766+0000","last_peered":"2024-03-16T10:15:04.013766+0000","last_clean":"2024-03-16T10:15:04.013766+0000","last_became_active":"2024-03-16T10:15:03.029656+0000","last_became_peered":"2024-03-16T10:15:03.029656+0000","last_unstale":"2024-03-16T10:15:04.013766+0000","last_undegraded":"2024-03-16T10:15:04.013766+0000","last_fullsized":"2024-03-16T10:15:04.013766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:27:35.684774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'11","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:08.231618+0000","last_change":"2024-03-16T10:15:02.014752+0000","last_active":"2024-03-16T10:15:08.231618+0000","last_peered":"2024-03-16T10:15:08.231618+0000","last_clean":"2024-03-16T10:15:08.231618+0000","last_became_active":"2024-03-16T10:15:02.013763+0000","last_became_peered":"2024-03-16T10:15:02.013763+0000","last_unstale":"2024-03-16T10:15:08.231618+0000","last_undegraded":"2024-03-16T10:15:08.231618+0000","last_fullsized":"2024-03-16T10:15:08.231618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:12:12.759643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2003,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":13,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.023110+0000","last_change":"2024-03-16T10:15:03.018838+0000","last_active":"2024-03-16T10:15:04.023110+0000","last_peered":"2024-03-16T10:15:04.023110+0000","last_clean":"2024-03-16T10:15:04.023110+0000","last_became_active":"2024-03-16T10:15:03.017713+0000","last_became_peered":"2024-03-16T10:15:03.017713+0000","last_unstale":"2024-03-16T10:15:04.023110+0000","last_undegraded":"2024-03-16T10:15:04.023110+0000","last_fullsized":"2024-03-16T10:15:04.023110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:03:56.936729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.251324+0000","last_change":"2024-03-16T10:15:02.016996+0000","last_active":"2024-03-16T10:15:06.251324+0000","last_peered":"2024-03-16T10:15:06.251324+0000","last_clean":"2024-03-16T10:15:06.251324+0000","last_became_active":"2024-03-16T10:15:02.016421+0000","last_became_peered":"2024-03-16T10:15:02.016421+0000","last_unstale":"2024-03-16T10:15:06.251324+0000","last_undegraded":"2024-03-16T10:15:06.251324+0000","last_fullsized":"2024-03-16T10:15:06.251324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:25:09.866730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013704+0000","last_change":"2024-03-16T10:15:03.017842+0000","last_active":"2024-03-16T10:15:04.013704+0000","last_peered":"2024-03-16T10:15:04.013704+0000","last_clean":"2024-03-16T10:15:04.013704+0000","last_became_active":"2024-03-16T10:15:03.015119+0000","last_became_peered":"2024-03-16T10:15:03.015119+0000","last_unstale":"2024-03-16T10:15:04.013704+0000","last_undegraded":"2024-03-16T10:15:04.013704+0000","last_fullsized":"2024-03-16T10:15:04.013704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:14:34.474519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.247417+0000","last_change":"2024-03-16T10:15:02.021407+0000","last_active":"2024-03-16T10:15:05.247417+0000","last_peered":"2024-03-16T10:15:05.247417+0000","last_clean":"2024-03-16T10:15:05.247417+0000","last_became_active":"2024-03-16T10:15:02.020884+0000","last_became_peered":"2024-03-16T10:15:02.020884+0000","last_unstale":"2024-03-16T10:15:05.247417+0000","last_undegraded":"2024-03-16T10:15:05.247417+0000","last_fullsized":"2024-03-16T10:15:05.247417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:24.858281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390941+0000","last_change":"2024-03-16T10:15:03.028607+0000","last_active":"2024-03-16T10:15:04.390941+0000","last_peered":"2024-03-16T10:15:04.390941+0000","last_clean":"2024-03-16T10:15:04.390941+0000","last_became_active":"2024-03-16T10:15:03.028280+0000","last_became_peered":"2024-03-16T10:15:03.028280+0000","last_unstale":"2024-03-16T10:15:04.390941+0000","last_undegraded":"2024-03-16T10:15:04.390941+0000","last_fullsized":"2024-03-16T10:15:04.390941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:39.905921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:07.261392+0000","last_change":"2024-03-16T10:15:02.008558+0000","last_active":"2024-03-16T10:15:07.261392+0000","last_peered":"2024-03-16T10:15:07.261392+0000","last_clean":"2024-03-16T10:15:07.261392+0000","last_became_active":"2024-03-16T10:15:02.008014+0000","last_became_peered":"2024-03-16T10:15:02.008014+0000","last_unstale":"2024-03-16T10:15:07.261392+0000","last_undegraded":"2024-03-16T10:15:07.261392+0000","last_fullsized":"2024-03-16T10:15:07.261392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:40:09.217991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.389791+0000","last_change":"2024-03-16T10:15:03.020847+0000","last_active":"2024-03-16T10:15:04.389791+0000","last_peered":"2024-03-16T10:15:04.389791+0000","last_clean":"2024-03-16T10:15:04.389791+0000","last_became_active":"2024-03-16T10:15:03.020512+0000","last_became_peered":"2024-03-16T10:15:03.020512+0000","last_unstale":"2024-03-16T10:15:04.389791+0000","last_undegraded":"2024-03-16T10:15:04.389791+0000","last_fullsized":"2024-03-16T10:15:04.389791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:04.436195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.253290+0000","last_change":"2024-03-16T10:15:02.010690+0000","last_active":"2024-03-16T10:15:06.253290+0000","last_peered":"2024-03-16T10:15:06.253290+0000","last_clean":"2024-03-16T10:15:06.253290+0000","last_became_active":"2024-03-16T10:15:02.010228+0000","last_became_peered":"2024-03-16T10:15:02.010228+0000","last_unstale":"2024-03-16T10:15:06.253290+0000","last_undegraded":"2024-03-16T10:15:06.253290+0000","last_fullsized":"2024-03-16T10:15:06.253290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:30:07.513225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390326+0000","last_change":"2024-03-16T10:15:03.025257+0000","last_active":"2024-03-16T10:15:04.390326+0000","last_peered":"2024-03-16T10:15:04.390326+0000","last_clean":"2024-03-16T10:15:04.390326+0000","last_became_active":"2024-03-16T10:15:03.024498+0000","last_became_peered":"2024-03-16T10:15:03.024498+0000","last_unstale":"2024-03-16T10:15:04.390326+0000","last_undegraded":"2024-03-16T10:15:04.390326+0000","last_fullsized":"2024-03-16T10:15:04.390326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:27:55.894728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.023168+0000","last_change":"2024-03-16T10:15:02.014136+0000","last_active":"2024-03-16T10:15:04.023168+0000","last_peered":"2024-03-16T10:15:04.023168+0000","last_clean":"2024-03-16T10:15:04.023168+0000","last_became_active":"2024-03-16T10:15:02.013721+0000","last_became_peered":"2024-03-16T10:15:02.013721+0000","last_unstale":"2024-03-16T10:15:04.023168+0000","last_undegraded":"2024-03-16T10:15:04.023168+0000","last_fullsized":"2024-03-16T10:15:04.023168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:28:12.301450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.526001+0000","last_change":"2024-03-16T10:15:03.021431+0000","last_active":"2024-03-16T10:15:04.526001+0000","last_peered":"2024-03-16T10:15:04.526001+0000","last_clean":"2024-03-16T10:15:04.526001+0000","last_became_active":"2024-03-16T10:15:03.021116+0000","last_became_peered":"2024-03-16T10:15:03.021116+0000","last_unstale":"2024-03-16T10:15:04.526001+0000","last_undegraded":"2024-03-16T10:15:04.526001+0000","last_fullsized":"2024-03-16T10:15:04.526001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:52:31.595530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013366+0000","last_change":"2024-03-16T10:15:02.005754+0000","last_active":"2024-03-16T10:15:04.013366+0000","last_peered":"2024-03-16T10:15:04.013366+0000","last_clean":"2024-03-16T10:15:04.013366+0000","last_became_active":"2024-03-16T10:15:02.005344+0000","last_became_peered":"2024-03-16T10:15:02.005344+0000","last_unstale":"2024-03-16T10:15:04.013366+0000","last_undegraded":"2024-03-16T10:15:04.013366+0000","last_fullsized":"2024-03-16T10:15:04.013366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:37:37.918861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390985+0000","last_change":"2024-03-16T10:15:03.012887+0000","last_active":"2024-03-16T10:15:04.390985+0000","last_peered":"2024-03-16T10:15:04.390985+0000","last_clean":"2024-03-16T10:15:04.390985+0000","last_became_active":"2024-03-16T10:15:03.012496+0000","last_became_peered":"2024-03-16T10:15:03.012496+0000","last_unstale":"2024-03-16T10:15:04.390985+0000","last_undegraded":"2024-03-16T10:15:04.390985+0000","last_fullsized":"2024-03-16T10:15:04.390985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:11.111049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390438+0000","last_change":"2024-03-16T10:15:02.014334+0000","last_active":"2024-03-16T10:15:04.390438+0000","last_peered":"2024-03-16T10:15:04.390438+0000","last_clean":"2024-03-16T10:15:04.390438+0000","last_became_active":"2024-03-16T10:15:02.013832+0000","last_became_peered":"2024-03-16T10:15:02.013832+0000","last_unstale":"2024-03-16T10:15:04.390438+0000","last_undegraded":"2024-03-16T10:15:04.390438+0000","last_fullsized":"2024-03-16T10:15:04.390438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:20.391594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390298+0000","last_change":"2024-03-16T10:15:03.021276+0000","last_active":"2024-03-16T10:15:04.390298+0000","last_peered":"2024-03-16T10:15:04.390298+0000","last_clean":"2024-03-16T10:15:04.390298+0000","last_became_active":"2024-03-16T10:15:03.020927+0000","last_became_peered":"2024-03-16T10:15:03.020927+0000","last_unstale":"2024-03-16T10:15:04.390298+0000","last_undegraded":"2024-03-16T10:15:04.390298+0000","last_fullsized":"2024-03-16T10:15:04.390298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:53:54.127506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390252+0000","last_change":"2024-03-16T10:15:03.022083+0000","last_active":"2024-03-16T10:15:04.390252+0000","last_peered":"2024-03-16T10:15:04.390252+0000","last_clean":"2024-03-16T10:15:04.390252+0000","last_became_active":"2024-03-16T10:15:03.021609+0000","last_became_peered":"2024-03-16T10:15:03.021609+0000","last_unstale":"2024-03-16T10:15:04.390252+0000","last_undegraded":"2024-03-16T10:15:04.390252+0000","last_fullsized":"2024-03-16T10:15:04.390252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:53.864402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.252965+0000","last_change":"2024-03-16T10:15:02.015497+0000","last_active":"2024-03-16T10:15:06.252965+0000","last_peered":"2024-03-16T10:15:06.252965+0000","last_clean":"2024-03-16T10:15:06.252965+0000","last_became_active":"2024-03-16T10:15:02.015183+0000","last_became_peered":"2024-03-16T10:15:02.015183+0000","last_unstale":"2024-03-16T10:15:06.252965+0000","last_undegraded":"2024-03-16T10:15:06.252965+0000","last_fullsized":"2024-03-16T10:15:06.252965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:44:05.913030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022947+0000","last_change":"2024-03-16T10:15:03.020046+0000","last_active":"2024-03-16T10:15:04.022947+0000","last_peered":"2024-03-16T10:15:04.022947+0000","last_clean":"2024-03-16T10:15:04.022947+0000","last_became_active":"2024-03-16T10:15:03.019647+0000","last_became_peered":"2024-03-16T10:15:03.019647+0000","last_unstale":"2024-03-16T10:15:04.022947+0000","last_undegraded":"2024-03-16T10:15:04.022947+0000","last_fullsized":"2024-03-16T10:15:04.022947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:18.343495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:07.262840+0000","last_change":"2024-03-16T10:15:02.008560+0000","last_active":"2024-03-16T10:15:07.262840+0000","last_peered":"2024-03-16T10:15:07.262840+0000","last_clean":"2024-03-16T10:15:07.262840+0000","last_became_active":"2024-03-16T10:15:02.008094+0000","last_became_peered":"2024-03-16T10:15:02.008094+0000","last_unstale":"2024-03-16T10:15:07.262840+0000","last_undegraded":"2024-03-16T10:15:07.262840+0000","last_fullsized":"2024-03-16T10:15:07.262840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:21:43.555120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":337,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.389703+0000","last_change":"2024-03-16T10:13:47.211815+0000","last_active":"2024-03-16T10:15:04.389703+0000","last_peered":"2024-03-16T10:15:04.389703+0000","last_clean":"2024-03-16T10:15:04.389703+0000","last_became_active":"2024-03-16T10:13:47.211353+0000","last_became_peered":"2024-03-16T10:13:47.211353+0000","last_unstale":"2024-03-16T10:15:04.389703+0000","last_undegraded":"2024-03-16T10:15:04.389703+0000","last_fullsized":"2024-03-16T10:15:04.389703+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_clean_scrub_stamp":"2024-03-16T10:10:22.965421+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:11:20.607686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013199+0000","last_change":"2024-03-16T10:15:03.020181+0000","last_active":"2024-03-16T10:15:04.013199+0000","last_peered":"2024-03-16T10:15:04.013199+0000","last_clean":"2024-03-16T10:15:04.013199+0000","last_became_active":"2024-03-16T10:15:03.019781+0000","last_became_peered":"2024-03-16T10:15:03.019781+0000","last_unstale":"2024-03-16T10:15:04.013199+0000","last_undegraded":"2024-03-16T10:15:04.013199+0000","last_fullsized":"2024-03-16T10:15:04.013199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:04:49.369484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.251678+0000","last_change":"2024-03-16T10:15:02.014461+0000","last_active":"2024-03-16T10:15:06.251678+0000","last_peered":"2024-03-16T10:15:06.251678+0000","last_clean":"2024-03-16T10:15:06.251678+0000","last_became_active":"2024-03-16T10:15:02.014137+0000","last_became_peered":"2024-03-16T10:15:02.014137+0000","last_unstale":"2024-03-16T10:15:06.251678+0000","last_undegraded":"2024-03-16T10:15:06.251678+0000","last_fullsized":"2024-03-16T10:15:06.251678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:36:41.830893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.023156+0000","last_change":"2024-03-16T10:15:02.019441+0000","last_active":"2024-03-16T10:15:04.023156+0000","last_peered":"2024-03-16T10:15:04.023156+0000","last_clean":"2024-03-16T10:15:04.023156+0000","last_became_active":"2024-03-16T10:15:02.019091+0000","last_became_peered":"2024-03-16T10:15:02.019091+0000","last_unstale":"2024-03-16T10:15:04.023156+0000","last_undegraded":"2024-03-16T10:15:04.023156+0000","last_fullsized":"2024-03-16T10:15:04.023156+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:04:51.758722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013478+0000","last_change":"2024-03-16T10:15:03.012141+0000","last_active":"2024-03-16T10:15:04.013478+0000","last_peered":"2024-03-16T10:15:04.013478+0000","last_clean":"2024-03-16T10:15:04.013478+0000","last_became_active":"2024-03-16T10:15:03.011771+0000","last_became_peered":"2024-03-16T10:15:03.011771+0000","last_unstale":"2024-03-16T10:15:04.013478+0000","last_undegraded":"2024-03-16T10:15:04.013478+0000","last_fullsized":"2024-03-16T10:15:04.013478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:02:28.522530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:07.264691+0000","last_change":"2024-03-16T10:15:02.011898+0000","last_active":"2024-03-16T10:15:07.264691+0000","last_peered":"2024-03-16T10:15:07.264691+0000","last_clean":"2024-03-16T10:15:07.264691+0000","last_became_active":"2024-03-16T10:15:02.011578+0000","last_became_peered":"2024-03-16T10:15:02.011578+0000","last_unstale":"2024-03-16T10:15:07.264691+0000","last_undegraded":"2024-03-16T10:15:07.264691+0000","last_fullsized":"2024-03-16T10:15:07.264691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:27:35.756031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022977+0000","last_change":"2024-03-16T10:15:03.024923+0000","last_active":"2024-03-16T10:15:04.022977+0000","last_peered":"2024-03-16T10:15:04.022977+0000","last_clean":"2024-03-16T10:15:04.022977+0000","last_became_active":"2024-03-16T10:15:03.024577+0000","last_became_peered":"2024-03-16T10:15:03.024577+0000","last_unstale":"2024-03-16T10:15:04.022977+0000","last_undegraded":"2024-03-16T10:15:04.022977+0000","last_fullsized":"2024-03-16T10:15:04.022977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:04.650499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.247188+0000","last_change":"2024-03-16T10:15:02.006842+0000","last_active":"2024-03-16T10:15:05.247188+0000","last_peered":"2024-03-16T10:15:05.247188+0000","last_clean":"2024-03-16T10:15:05.247188+0000","last_became_active":"2024-03-16T10:15:02.006483+0000","last_became_peered":"2024-03-16T10:15:02.006483+0000","last_unstale":"2024-03-16T10:15:05.247188+0000","last_undegraded":"2024-03-16T10:15:05.247188+0000","last_fullsized":"2024-03-16T10:15:05.247188+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:30:55.856007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390026+0000","last_change":"2024-03-16T10:15:03.018691+0000","last_active":"2024-03-16T10:15:04.390026+0000","last_peered":"2024-03-16T10:15:04.390026+0000","last_clean":"2024-03-16T10:15:04.390026+0000","last_became_active":"2024-03-16T10:15:03.018304+0000","last_became_peered":"2024-03-16T10:15:03.018304+0000","last_unstale":"2024-03-16T10:15:04.390026+0000","last_undegraded":"2024-03-16T10:15:04.390026+0000","last_fullsized":"2024-03-16T10:15:04.390026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:15.446759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.248013+0000","last_change":"2024-03-16T10:15:02.008011+0000","last_active":"2024-03-16T10:15:05.248013+0000","last_peered":"2024-03-16T10:15:05.248013+0000","last_clean":"2024-03-16T10:15:05.248013+0000","last_became_active":"2024-03-16T10:15:02.007640+0000","last_became_peered":"2024-03-16T10:15:02.007640+0000","last_unstale":"2024-03-16T10:15:05.248013+0000","last_undegraded":"2024-03-16T10:15:05.248013+0000","last_fullsized":"2024-03-16T10:15:05.248013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:15:22.894518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022799+0000","last_change":"2024-03-16T10:15:03.017262+0000","last_active":"2024-03-16T10:15:04.022799+0000","last_peered":"2024-03-16T10:15:04.022799+0000","last_clean":"2024-03-16T10:15:04.022799+0000","last_became_active":"2024-03-16T10:15:03.016909+0000","last_became_peered":"2024-03-16T10:15:03.016909+0000","last_unstale":"2024-03-16T10:15:04.022799+0000","last_undegraded":"2024-03-16T10:15:04.022799+0000","last_fullsized":"2024-03-16T10:15:04.022799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:53:11.385236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.247181+0000","last_change":"2024-03-16T10:15:02.011325+0000","last_active":"2024-03-16T10:15:05.247181+0000","last_peered":"2024-03-16T10:15:05.247181+0000","last_clean":"2024-03-16T10:15:05.247181+0000","last_became_active":"2024-03-16T10:15:02.010999+0000","last_became_peered":"2024-03-16T10:15:02.010999+0000","last_unstale":"2024-03-16T10:15:05.247181+0000","last_undegraded":"2024-03-16T10:15:05.247181+0000","last_fullsized":"2024-03-16T10:15:05.247181+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:30:08.137693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.525756+0000","last_change":"2024-03-16T10:15:03.012146+0000","last_active":"2024-03-16T10:15:04.525756+0000","last_peered":"2024-03-16T10:15:04.525756+0000","last_clean":"2024-03-16T10:15:04.525756+0000","last_became_active":"2024-03-16T10:15:03.011790+0000","last_became_peered":"2024-03-16T10:15:03.011790+0000","last_unstale":"2024-03-16T10:15:04.525756+0000","last_undegraded":"2024-03-16T10:15:04.525756+0000","last_fullsized":"2024-03-16T10:15:04.525756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:45:22.642925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:06.243205+0000","last_change":"2024-03-16T10:15:02.017419+0000","last_active":"2024-03-16T10:15:06.243205+0000","last_peered":"2024-03-16T10:15:06.243205+0000","last_clean":"2024-03-16T10:15:06.243205+0000","last_became_active":"2024-03-16T10:15:02.016707+0000","last_became_peered":"2024-03-16T10:15:02.016707+0000","last_unstale":"2024-03-16T10:15:06.243205+0000","last_undegraded":"2024-03-16T10:15:06.243205+0000","last_fullsized":"2024-03-16T10:15:06.243205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:57:41.783962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022915+0000","last_change":"2024-03-16T10:15:03.021094+0000","last_active":"2024-03-16T10:15:04.022915+0000","last_peered":"2024-03-16T10:15:04.022915+0000","last_clean":"2024-03-16T10:15:04.022915+0000","last_became_active":"2024-03-16T10:15:03.017640+0000","last_became_peered":"2024-03-16T10:15:03.017640+0000","last_unstale":"2024-03-16T10:15:04.022915+0000","last_undegraded":"2024-03-16T10:15:04.022915+0000","last_fullsized":"2024-03-16T10:15:04.022915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:02.030112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:05.247781+0000","last_change":"2024-03-16T10:15:02.015652+0000","last_active":"2024-03-16T10:15:05.247781+0000","last_peered":"2024-03-16T10:15:05.247781+0000","last_clean":"2024-03-16T10:15:05.247781+0000","last_became_active":"2024-03-16T10:15:02.014046+0000","last_became_peered":"2024-03-16T10:15:02.014046+0000","last_unstale":"2024-03-16T10:15:05.247781+0000","last_undegraded":"2024-03-16T10:15:05.247781+0000","last_fullsized":"2024-03-16T10:15:05.247781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390642+0000","last_change":"2024-03-16T10:15:03.023233+0000","last_active":"2024-03-16T10:15:04.390642+0000","last_peered":"2024-03-16T10:15:04.390642+0000","last_clean":"2024-03-16T10:15:04.390642+0000","last_became_active":"2024-03-16T10:15:03.022906+0000","last_became_peered":"2024-03-16T10:15:03.022906+0000","last_unstale":"2024-03-16T10:15:04.390642+0000","last_undegraded":"2024-03-16T10:15:04.390642+0000","last_fullsized":"2024-03-16T10:15:04.390642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:22.943087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:07.261553+0000","last_change":"2024-03-16T10:15:02.010755+0000","last_active":"2024-03-16T10:15:07.261553+0000","last_peered":"2024-03-16T10:15:07.261553+0000","last_clean":"2024-03-16T10:15:07.261553+0000","last_became_active":"2024-03-16T10:15:02.010422+0000","last_became_peered":"2024-03-16T10:15:02.010422+0000","last_unstale":"2024-03-16T10:15:07.261553+0000","last_undegraded":"2024-03-16T10:15:07.261553+0000","last_fullsized":"2024-03-16T10:15:07.261553+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:17.606346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.391301+0000","last_change":"2024-03-16T10:15:03.025789+0000","last_active":"2024-03-16T10:15:04.391301+0000","last_peered":"2024-03-16T10:15:04.391301+0000","last_clean":"2024-03-16T10:15:04.391301+0000","last_became_active":"2024-03-16T10:15:03.024812+0000","last_became_peered":"2024-03-16T10:15:03.024812+0000","last_unstale":"2024-03-16T10:15:04.391301+0000","last_undegraded":"2024-03-16T10:15:04.391301+0000","last_fullsized":"2024-03-16T10:15:04.391301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:53.041234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390214+0000","last_change":"2024-03-16T10:15:02.020599+0000","last_active":"2024-03-16T10:15:04.390214+0000","last_peered":"2024-03-16T10:15:04.390214+0000","last_clean":"2024-03-16T10:15:04.390214+0000","last_became_active":"2024-03-16T10:15:02.020203+0000","last_became_peered":"2024-03-16T10:15:02.020203+0000","last_unstale":"2024-03-16T10:15:04.390214+0000","last_undegraded":"2024-03-16T10:15:04.390214+0000","last_fullsized":"2024-03-16T10:15:04.390214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:46:33.032310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390824+0000","last_change":"2024-03-16T10:15:03.016223+0000","last_active":"2024-03-16T10:15:04.390824+0000","last_peered":"2024-03-16T10:15:04.390824+0000","last_clean":"2024-03-16T10:15:04.390824+0000","last_became_active":"2024-03-16T10:15:03.015857+0000","last_became_peered":"2024-03-16T10:15:03.015857+0000","last_unstale":"2024-03-16T10:15:04.390824+0000","last_undegraded":"2024-03-16T10:15:04.390824+0000","last_fullsized":"2024-03-16T10:15:04.390824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:40:10.329722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013515+0000","last_change":"2024-03-16T10:15:02.015690+0000","last_active":"2024-03-16T10:15:04.013515+0000","last_peered":"2024-03-16T10:15:04.013515+0000","last_clean":"2024-03-16T10:15:04.013515+0000","last_became_active":"2024-03-16T10:15:02.015375+0000","last_became_peered":"2024-03-16T10:15:02.015375+0000","last_unstale":"2024-03-16T10:15:04.013515+0000","last_undegraded":"2024-03-16T10:15:04.013515+0000","last_fullsized":"2024-03-16T10:15:04.013515+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:39.118343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.391233+0000","last_change":"2024-03-16T10:15:03.012702+0000","last_active":"2024-03-16T10:15:04.391233+0000","last_peered":"2024-03-16T10:15:04.391233+0000","last_clean":"2024-03-16T10:15:04.391233+0000","last_became_active":"2024-03-16T10:15:03.012387+0000","last_became_peered":"2024-03-16T10:15:03.012387+0000","last_unstale":"2024-03-16T10:15:04.391233+0000","last_undegraded":"2024-03-16T10:15:04.391233+0000","last_fullsized":"2024-03-16T10:15:04.391233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:16.938372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.525868+0000","last_change":"2024-03-16T10:15:03.015290+0000","last_active":"2024-03-16T10:15:04.525868+0000","last_peered":"2024-03-16T10:15:04.525868+0000","last_clean":"2024-03-16T10:15:04.525868+0000","last_became_active":"2024-03-16T10:15:03.013846+0000","last_became_peered":"2024-03-16T10:15:03.013846+0000","last_unstale":"2024-03-16T10:15:04.525868+0000","last_undegraded":"2024-03-16T10:15:04.525868+0000","last_fullsized":"2024-03-16T10:15:04.525868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:33.377793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013983+0000","last_change":"2024-03-16T10:15:03.023733+0000","last_active":"2024-03-16T10:15:04.013983+0000","last_peered":"2024-03-16T10:15:04.013983+0000","last_clean":"2024-03-16T10:15:04.013983+0000","last_became_active":"2024-03-16T10:15:03.023409+0000","last_became_peered":"2024-03-16T10:15:03.023409+0000","last_unstale":"2024-03-16T10:15:04.013983+0000","last_undegraded":"2024-03-16T10:15:04.013983+0000","last_fullsized":"2024-03-16T10:15:04.013983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:44.748152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.014242+0000","last_change":"2024-03-16T10:15:03.021254+0000","last_active":"2024-03-16T10:15:04.014242+0000","last_peered":"2024-03-16T10:15:04.014242+0000","last_clean":"2024-03-16T10:15:04.014242+0000","last_became_active":"2024-03-16T10:15:03.020885+0000","last_became_peered":"2024-03-16T10:15:03.020885+0000","last_unstale":"2024-03-16T10:15:04.014242+0000","last_undegraded":"2024-03-16T10:15:04.014242+0000","last_fullsized":"2024-03-16T10:15:04.014242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:38:47.269571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.389821+0000","last_change":"2024-03-16T10:15:03.029277+0000","last_active":"2024-03-16T10:15:04.389821+0000","last_peered":"2024-03-16T10:15:04.389821+0000","last_clean":"2024-03-16T10:15:04.389821+0000","last_became_active":"2024-03-16T10:15:03.028960+0000","last_became_peered":"2024-03-16T10:15:03.028960+0000","last_unstale":"2024-03-16T10:15:04.389821+0000","last_undegraded":"2024-03-16T10:15:04.389821+0000","last_fullsized":"2024-03-16T10:15:04.389821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:50:48.362914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013457+0000","last_change":"2024-03-16T10:15:03.020773+0000","last_active":"2024-03-16T10:15:04.013457+0000","last_peered":"2024-03-16T10:15:04.013457+0000","last_clean":"2024-03-16T10:15:04.013457+0000","last_became_active":"2024-03-16T10:15:03.020450+0000","last_became_peered":"2024-03-16T10:15:03.020450+0000","last_unstale":"2024-03-16T10:15:04.013457+0000","last_undegraded":"2024-03-16T10:15:04.013457+0000","last_fullsized":"2024-03-16T10:15:04.013457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:26:42.005725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.526024+0000","last_change":"2024-03-16T10:15:03.017499+0000","last_active":"2024-03-16T10:15:04.526024+0000","last_peered":"2024-03-16T10:15:04.526024+0000","last_clean":"2024-03-16T10:15:04.526024+0000","last_became_active":"2024-03-16T10:15:03.017085+0000","last_became_peered":"2024-03-16T10:15:03.017085+0000","last_unstale":"2024-03-16T10:15:04.526024+0000","last_undegraded":"2024-03-16T10:15:04.526024+0000","last_fullsized":"2024-03-16T10:15:04.526024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:57:09.592420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.014459+0000","last_change":"2024-03-16T10:15:03.018943+0000","last_active":"2024-03-16T10:15:04.014459+0000","last_peered":"2024-03-16T10:15:04.014459+0000","last_clean":"2024-03-16T10:15:04.014459+0000","last_became_active":"2024-03-16T10:15:03.017519+0000","last_became_peered":"2024-03-16T10:15:03.017519+0000","last_unstale":"2024-03-16T10:15:04.014459+0000","last_undegraded":"2024-03-16T10:15:04.014459+0000","last_fullsized":"2024-03-16T10:15:04.014459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:29.026988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022802+0000","last_change":"2024-03-16T10:15:03.024697+0000","last_active":"2024-03-16T10:15:04.022802+0000","last_peered":"2024-03-16T10:15:04.022802+0000","last_clean":"2024-03-16T10:15:04.022802+0000","last_became_active":"2024-03-16T10:15:03.024305+0000","last_became_peered":"2024-03-16T10:15:03.024305+0000","last_unstale":"2024-03-16T10:15:04.022802+0000","last_undegraded":"2024-03-16T10:15:04.022802+0000","last_fullsized":"2024-03-16T10:15:04.022802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:07:35.160746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.390239+0000","last_change":"2024-03-16T10:15:03.017504+0000","last_active":"2024-03-16T10:15:04.390239+0000","last_peered":"2024-03-16T10:15:04.390239+0000","last_clean":"2024-03-16T10:15:04.390239+0000","last_became_active":"2024-03-16T10:15:03.017100+0000","last_became_peered":"2024-03-16T10:15:03.017100+0000","last_unstale":"2024-03-16T10:15:04.390239+0000","last_undegraded":"2024-03-16T10:15:04.390239+0000","last_fullsized":"2024-03-16T10:15:04.390239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:23:17.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.022564+0000","last_change":"2024-03-16T10:15:03.024684+0000","last_active":"2024-03-16T10:15:04.022564+0000","last_peered":"2024-03-16T10:15:04.022564+0000","last_clean":"2024-03-16T10:15:04.022564+0000","last_became_active":"2024-03-16T10:15:03.024282+0000","last_became_peered":"2024-03-16T10:15:03.024282+0000","last_unstale":"2024-03-16T10:15:04.022564+0000","last_undegraded":"2024-03-16T10:15:04.022564+0000","last_fullsized":"2024-03-16T10:15:04.022564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:29.595162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013794+0000","last_change":"2024-03-16T10:15:03.025337+0000","last_active":"2024-03-16T10:15:04.013794+0000","last_peered":"2024-03-16T10:15:04.013794+0000","last_clean":"2024-03-16T10:15:04.013794+0000","last_became_active":"2024-03-16T10:15:03.024938+0000","last_became_peered":"2024-03-16T10:15:03.024938+0000","last_unstale":"2024-03-16T10:15:04.013794+0000","last_undegraded":"2024-03-16T10:15:04.013794+0000","last_fullsized":"2024-03-16T10:15:04.013794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:13:15.892354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.013930+0000","last_change":"2024-03-16T10:15:03.015468+0000","last_active":"2024-03-16T10:15:04.013930+0000","last_peered":"2024-03-16T10:15:04.013930+0000","last_clean":"2024-03-16T10:15:04.013930+0000","last_became_active":"2024-03-16T10:15:03.015097+0000","last_became_peered":"2024-03-16T10:15:03.015097+0000","last_unstale":"2024-03-16T10:15:04.013930+0000","last_undegraded":"2024-03-16T10:15:04.013930+0000","last_fullsized":"2024-03-16T10:15:04.013930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:46.114346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T10:15:04.525839+0000","last_change":"2024-03-16T10:15:03.020690+0000","last_active":"2024-03-16T10:15:04.525839+0000","last_peered":"2024-03-16T10:15:04.525839+0000","last_clean":"2024-03-16T10:15:04.525839+0000","last_became_active":"2024-03-16T10:15:03.020365+0000","last_became_peered":"2024-03-16T10:15:03.020365+0000","last_unstale":"2024-03-16T10:15:04.525839+0000","last_undegraded":"2024-03-16T10:15:04.525839+0000","last_fullsized":"2024-03-16T10:15:04.525839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:25.671533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280280,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6437,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":154,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":27607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383693,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":440104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1032203,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":439429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070494,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":439147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332130,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":1030093,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":442276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":441090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":441483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":438805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":439187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757292,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":1029071,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":9964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T10:15:12.040 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-16T10:15:12.040 INFO:tasks.cephadm:Setting up client nodes... 2024-03-16T10:15:12.040 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-16T10:15:12.317 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:12.439 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:12 smithi077 ceph-mon[32400]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-16T10:15:12.439 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:12 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2056102402' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:12.440 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:12 smithi077 ceph-mon[32400]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:15:12.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:12 smithi192 ceph-mon[36354]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-16T10:15:12.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:12 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2056102402' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:12.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:12 smithi192 ceph-mon[36354]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:15:12.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:12 smithi193 ceph-mon[36311]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-16T10:15:12.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:12 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2056102402' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:12.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:12 smithi193 ceph-mon[36311]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:15:13.599 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:13 smithi077 ceph-mon[32400]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:15:13.599 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:13 smithi077 ceph-mon[32400]: pgmap v292: 97 pgs: 97 active+clean; 585 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6.6 KiB/s wr, 25 op/s 2024-03-16T10:15:13.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:13 smithi192 ceph-mon[36354]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:15:13.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:13 smithi192 ceph-mon[36354]: pgmap v292: 97 pgs: 97 active+clean; 585 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6.6 KiB/s wr, 25 op/s 2024-03-16T10:15:13.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:13 smithi193 ceph-mon[36311]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:15:13.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:13 smithi193 ceph-mon[36311]: pgmap v292: 97 pgs: 97 active+clean; 585 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6.6 KiB/s wr, 25 op/s 2024-03-16T10:15:14.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.012+0000 7fbf1dca3700 1 -- 172.21.15.77:0/4010693759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf18108000 msgr2=0x7fbf1810a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:14.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.012+0000 7fbf1dca3700 1 --2- 172.21.15.77:0/4010693759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf18108000 0x7fbf1810a470 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbf1000b0a0 tx=0x7fbf1000b3b0 comp rx=0 tx=0).stop 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.012+0000 7fbf1dca3700 1 -- 172.21.15.77:0/4010693759 shutdown_connections 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.012+0000 7fbf1dca3700 1 --2- 172.21.15.77:0/4010693759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf18108000 0x7fbf1810a470 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.012+0000 7fbf1dca3700 1 --2- 172.21.15.77:0/4010693759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf18102da0 0x7fbf18105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.012+0000 7fbf1dca3700 1 --2- 172.21.15.77:0/4010693759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf181056d0 0x7fbf18107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.012+0000 7fbf1dca3700 1 -- 172.21.15.77:0/4010693759 >> 172.21.15.77:0/4010693759 conn(0x7fbf180fc7d0 msgr2=0x7fbf180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.012+0000 7fbf1dca3700 1 -- 172.21.15.77:0/4010693759 shutdown_connections 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.012+0000 7fbf1dca3700 1 -- 172.21.15.77:0/4010693759 wait complete. 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf1dca3700 1 Processor -- start 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf1dca3700 1 -- start start 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf1dca3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf18102da0 0x7fbf181983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf1dca3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf181056d0 0x7fbf18198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf1dca3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf18108000 0x7fbf1819f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf1dca3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1810df20 con 0x7fbf18102da0 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf1dca3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1810dda0 con 0x7fbf18108000 2024-03-16T10:15:14.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf1dca3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1810e0a0 con 0x7fbf181056d0 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf177fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf18102da0 0x7fbf181983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf177fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf18102da0 0x7fbf181983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42356/0 (socket says 172.21.15.77:42356) 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf177fe700 1 -- 172.21.15.77:0/1563342285 learned_addr learned my addr 172.21.15.77:0/1563342285 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf0ffff700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf181056d0 0x7fbf18198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf17fff700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf18108000 0x7fbf1819f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf177fe700 1 -- 172.21.15.77:0/1563342285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf181056d0 msgr2=0x7fbf18198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf177fe700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf181056d0 0x7fbf18198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf177fe700 1 -- 172.21.15.77:0/1563342285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf18108000 msgr2=0x7fbf1819f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf177fe700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf18108000 0x7fbf1819f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.013+0000 7fbf177fe700 1 -- 172.21.15.77:0/1563342285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf181a0030 con 0x7fbf18102da0 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.014+0000 7fbf177fe700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf18102da0 0x7fbf181983d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbf0800e9d0 tx=0x7fbf0800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.014+0000 7fbf157fa700 1 -- 172.21.15.77:0/1563342285 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0800cbd0 con 0x7fbf18102da0 2024-03-16T10:15:14.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.014+0000 7fbf157fa700 1 -- 172.21.15.77:0/1563342285 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf0800ee70 con 0x7fbf18102da0 2024-03-16T10:15:14.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.014+0000 7fbf1dca3700 1 -- 172.21.15.77:0/1563342285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf181a0320 con 0x7fbf18102da0 2024-03-16T10:15:14.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.014+0000 7fbf157fa700 1 -- 172.21.15.77:0/1563342285 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf08018660 con 0x7fbf18102da0 2024-03-16T10:15:14.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.014+0000 7fbf1dca3700 1 -- 172.21.15.77:0/1563342285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf181a08b0 con 0x7fbf18102da0 2024-03-16T10:15:14.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.015+0000 7fbf1dca3700 1 -- 172.21.15.77:0/1563342285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf180507e0 con 0x7fbf18102da0 2024-03-16T10:15:14.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.016+0000 7fbf157fa700 1 -- 172.21.15.77:0/1563342285 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbf08018800 con 0x7fbf18102da0 2024-03-16T10:15:14.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.016+0000 7fbf157fa700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf040a2a50 0x7fbf040a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:14.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.017+0000 7fbf0ffff700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf040a2a50 0x7fbf040a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:14.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.017+0000 7fbf157fa700 1 -- 172.21.15.77:0/1563342285 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fbf080cebf0 con 0x7fbf18102da0 2024-03-16T10:15:14.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.017+0000 7fbf0ffff700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf040a2a50 0x7fbf040a4f10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbf00009c60 tx=0x7fbf00009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:14.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.021+0000 7fbf157fa700 1 -- 172.21.15.77:0/1563342285 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbf0809bce0 con 0x7fbf18102da0 2024-03-16T10:15:14.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.307+0000 7fbf1dca3700 1 -- 172.21.15.77:0/1563342285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fbf18062ee0 con 0x7fbf18102da0 2024-03-16T10:15:14.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.309+0000 7fbf157fa700 1 -- 172.21.15.77:0/1563342285 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fbf0800cd70 con 0x7fbf18102da0 2024-03-16T10:15:14.310 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-03-16T10:15:14.310 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAycfVlWERvEhAAqEhdlXrkgOzhjDWwCYa7Yg== 2024-03-16T10:15:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.312+0000 7fbf1dca3700 1 -- 172.21.15.77:0/1563342285 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf040a2a50 msgr2=0x7fbf040a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.312+0000 7fbf1dca3700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf040a2a50 0x7fbf040a4f10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbf00009c60 tx=0x7fbf00009260 comp rx=0 tx=0).stop 2024-03-16T10:15:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.312+0000 7fbf1dca3700 1 -- 172.21.15.77:0/1563342285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf18102da0 msgr2=0x7fbf181983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.312+0000 7fbf1dca3700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf18102da0 0x7fbf181983d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbf0800e9d0 tx=0x7fbf0800ece0 comp rx=0 tx=0).stop 2024-03-16T10:15:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.312+0000 7fbf1dca3700 1 -- 172.21.15.77:0/1563342285 shutdown_connections 2024-03-16T10:15:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.312+0000 7fbf1dca3700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf040a2a50 0x7fbf040a4f10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.312+0000 7fbf1dca3700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf18102da0 0x7fbf181983d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.312+0000 7fbf1dca3700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf18108000 0x7fbf1819f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.312+0000 7fbf1dca3700 1 --2- 172.21.15.77:0/1563342285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf181056d0 0x7fbf18198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.312+0000 7fbf1dca3700 1 -- 172.21.15.77:0/1563342285 >> 172.21.15.77:0/1563342285 conn(0x7fbf180fc7d0 msgr2=0x7fbf181036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:14.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.312+0000 7fbf1dca3700 1 -- 172.21.15.77:0/1563342285 shutdown_connections 2024-03-16T10:15:14.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:14.312+0000 7fbf1dca3700 1 -- 172.21.15.77:0/1563342285 wait complete. 2024-03-16T10:15:14.623 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:14 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1563342285' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T10:15:14.623 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:14 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1563342285' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T10:15:14.654 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:14 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1563342285' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T10:15:14.654 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:14 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1563342285' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T10:15:14.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:14 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1563342285' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T10:15:14.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:14 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1563342285' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T10:15:14.991 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:15:14.991 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-16T10:15:14.992 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-16T10:15:15.037 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-16T10:15:15.037 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-16T10:15:15.037 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-16T10:15:15.342 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:15.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:15 smithi192 ceph-mon[36354]: pgmap v293: 97 pgs: 97 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 10 KiB/s wr, 28 op/s 2024-03-16T10:15:15.667 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:15 smithi193 ceph-mon[36311]: pgmap v293: 97 pgs: 97 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 10 KiB/s wr, 28 op/s 2024-03-16T10:15:15.685 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:15 smithi077 ceph-mon[32400]: pgmap v293: 97 pgs: 97 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 10 KiB/s wr, 28 op/s 2024-03-16T10:15:16.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.970+0000 7f9fa811d700 1 -- 172.21.15.77:0/4079894743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fa0107210 msgr2=0x7f9fa01075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:16.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.970+0000 7f9fa811d700 1 --2- 172.21.15.77:0/4079894743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fa0107210 0x7f9fa01075f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9f90009990 tx=0x7f9f90009ca0 comp rx=0 tx=0).stop 2024-03-16T10:15:16.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.970+0000 7f9fa811d700 1 -- 172.21.15.77:0/4079894743 shutdown_connections 2024-03-16T10:15:16.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.970+0000 7f9fa811d700 1 --2- 172.21.15.77:0/4079894743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fa0107210 0x7f9fa01075f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:16.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.970+0000 7f9fa811d700 1 --2- 172.21.15.77:0/4079894743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa0101b70 0x7f9fa010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:16.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.970+0000 7f9fa811d700 1 --2- 172.21.15.77:0/4079894743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa01011d0 0x7f9fa0101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.970+0000 7f9fa811d700 1 -- 172.21.15.77:0/4079894743 >> 172.21.15.77:0/4079894743 conn(0x7f9fa0078100 msgr2=0x7f9fa0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.970+0000 7f9fa811d700 1 -- 172.21.15.77:0/4079894743 shutdown_connections 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.970+0000 7f9fa811d700 1 -- 172.21.15.77:0/4079894743 wait complete. 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa811d700 1 Processor -- start 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa811d700 1 -- start start 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa811d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fa01011d0 0x7f9fa01998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa811d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa0101b70 0x7f9fa0199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa811d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa0107210 0x7f9fa01a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa811d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa01114a0 con 0x7f9fa01011d0 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa811d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa0111320 con 0x7f9fa0101b70 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa811d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa0111620 con 0x7f9fa0107210 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa5eb9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fa01011d0 0x7f9fa01998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa5eb9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fa01011d0 0x7f9fa01998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42368/0 (socket says 172.21.15.77:42368) 2024-03-16T10:15:16.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa5eb9700 1 -- 172.21.15.77:0/3595457284 learned_addr learned my addr 172.21.15.77:0/3595457284 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:16.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa66ba700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa0107210 0x7f9fa01a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:16.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.971+0000 7f9fa56b8700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa0101b70 0x7f9fa0199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:16.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9fa5eb9700 1 -- 172.21.15.77:0/3595457284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa0107210 msgr2=0x7f9fa01a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:16.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9fa5eb9700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa0107210 0x7f9fa01a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:16.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9fa5eb9700 1 -- 172.21.15.77:0/3595457284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa0101b70 msgr2=0x7f9fa0199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:16.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9fa5eb9700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa0101b70 0x7f9fa0199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:16.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9fa5eb9700 1 -- 172.21.15.77:0/3595457284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa01a4e80 con 0x7f9fa01011d0 2024-03-16T10:15:16.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9fa56b8700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa0101b70 0x7f9fa0199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:16.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9fa66ba700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa0107210 0x7f9fa01a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:16.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9fa5eb9700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fa01011d0 0x7f9fa01998c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9f90005910 tx=0x7f9f9000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:16.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9f96ffd700 1 -- 172.21.15.77:0/3595457284 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90027070 con 0x7f9fa01011d0 2024-03-16T10:15:16.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9f96ffd700 1 -- 172.21.15.77:0/3595457284 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f90004650 con 0x7f9fa01011d0 2024-03-16T10:15:16.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9fa811d700 1 -- 172.21.15.77:0/3595457284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa01a5050 con 0x7f9fa01011d0 2024-03-16T10:15:16.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9f96ffd700 1 -- 172.21.15.77:0/3595457284 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9000f460 con 0x7f9fa01011d0 2024-03-16T10:15:16.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.972+0000 7f9fa811d700 1 -- 172.21.15.77:0/3595457284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa01a54a0 con 0x7f9fa01011d0 2024-03-16T10:15:16.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.973+0000 7f9fa811d700 1 -- 172.21.15.77:0/3595457284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fa0068f90 con 0x7f9fa01011d0 2024-03-16T10:15:16.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.974+0000 7f9f96ffd700 1 -- 172.21.15.77:0/3595457284 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9f9002a070 con 0x7f9fa01011d0 2024-03-16T10:15:16.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.975+0000 7f9f96ffd700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9f8c0a2920 0x7f9f8c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:16.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.975+0000 7f9f96ffd700 1 -- 172.21.15.77:0/3595457284 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9f900cf8c0 con 0x7f9fa01011d0 2024-03-16T10:15:16.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.975+0000 7f9fa56b8700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9f8c0a2920 0x7f9f8c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:16.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.975+0000 7f9fa56b8700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9f8c0a2920 0x7f9f8c0a4de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0078770 tx=0x7f9f9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:16.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:16.979+0000 7f9f96ffd700 1 -- 172.21.15.77:0/3595457284 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9f9005e740 con 0x7f9fa01011d0 2024-03-16T10:15:17.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.240+0000 7f9fa811d700 1 -- 172.21.15.77:0/3595457284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f9fa002db70 con 0x7f9fa01011d0 2024-03-16T10:15:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.245+0000 7f9f96ffd700 1 -- 172.21.15.77:0/3595457284 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262661 (secure 0 0 0) 0x7f9f90013070 con 0x7f9fa01011d0 2024-03-16T10:15:17.249 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:15:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.248+0000 7f9fa811d700 1 -- 172.21.15.77:0/3595457284 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9f8c0a2920 msgr2=0x7f9f8c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.248+0000 7f9fa811d700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9f8c0a2920 0x7f9f8c0a4de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0078770 tx=0x7f9f9c009260 comp rx=0 tx=0).stop 2024-03-16T10:15:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.248+0000 7f9fa811d700 1 -- 172.21.15.77:0/3595457284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fa01011d0 msgr2=0x7f9fa01998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.248+0000 7f9fa811d700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fa01011d0 0x7f9fa01998c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9f90005910 tx=0x7f9f9000bab0 comp rx=0 tx=0).stop 2024-03-16T10:15:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.248+0000 7f9fa811d700 1 -- 172.21.15.77:0/3595457284 shutdown_connections 2024-03-16T10:15:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.248+0000 7f9fa811d700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9f8c0a2920 0x7f9f8c0a4de0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.248+0000 7f9fa811d700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fa01011d0 0x7f9fa01998c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.248+0000 7f9fa811d700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa0107210 0x7f9fa01a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.248+0000 7f9fa811d700 1 --2- 172.21.15.77:0/3595457284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa0101b70 0x7f9fa0199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.248+0000 7f9fa811d700 1 -- 172.21.15.77:0/3595457284 >> 172.21.15.77:0/3595457284 conn(0x7f9fa0078100 msgr2=0x7f9fa00ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.248+0000 7f9fa811d700 1 -- 172.21.15.77:0/3595457284 shutdown_connections 2024-03-16T10:15:17.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:17.248+0000 7f9fa811d700 1 -- 172.21.15.77:0/3595457284 wait complete. 2024-03-16T10:15:17.954 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1427985582},{"type":"v1","addr":"172.21.15.77:6801","nonce":1427985582}]},"active_addr":"172.21.15.77:6801/1427985582","active_change":"2024-03-16T10:07:58.634953+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.77:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":3759057583}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2271041119}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":1908073181}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":3031899593}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2688888223}]}]} 2024-03-16T10:15:17.960 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-16T10:15:17.960 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-16T10:15:17.960 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T10:15:18.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:17 smithi192 ceph-mon[36354]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 9.0 KiB/s wr, 25 op/s 2024-03-16T10:15:18.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:17 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3595457284' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T10:15:18.241 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:18.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:17 smithi077 ceph-mon[32400]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 9.0 KiB/s wr, 25 op/s 2024-03-16T10:15:18.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:17 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3595457284' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T10:15:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:17 smithi193 ceph-mon[36311]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 9.0 KiB/s wr, 25 op/s 2024-03-16T10:15:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:17 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3595457284' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T10:15:19.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.962+0000 7f7a20da7700 1 -- 172.21.15.77:0/535272967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a1c101b70 msgr2=0x7f7a1c10e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:19.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.962+0000 7f7a20da7700 1 --2- 172.21.15.77:0/535272967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a1c101b70 0x7f7a1c10e050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7a1400b0a0 tx=0x7f7a1400b3b0 comp rx=0 tx=0).stop 2024-03-16T10:15:19.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.962+0000 7f7a20da7700 1 -- 172.21.15.77:0/535272967 shutdown_connections 2024-03-16T10:15:19.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.962+0000 7f7a20da7700 1 --2- 172.21.15.77:0/535272967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a1c101b70 0x7f7a1c10e050 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:19.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.962+0000 7f7a20da7700 1 --2- 172.21.15.77:0/535272967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a1c107210 0x7f7a1c1075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:19.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.962+0000 7f7a20da7700 1 --2- 172.21.15.77:0/535272967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a1c1011d0 0x7f7a1c101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:19.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.962+0000 7f7a20da7700 1 -- 172.21.15.77:0/535272967 >> 172.21.15.77:0/535272967 conn(0x7f7a1c078100 msgr2=0x7f7a1c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.962+0000 7f7a20da7700 1 -- 172.21.15.77:0/535272967 shutdown_connections 2024-03-16T10:15:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.962+0000 7f7a20da7700 1 -- 172.21.15.77:0/535272967 wait complete. 2024-03-16T10:15:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a20da7700 1 Processor -- start 2024-03-16T10:15:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a20da7700 1 -- start start 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a20da7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a1c1011d0 0x7f7a1c19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a20da7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a1c101b70 0x7f7a1c19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a20da7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a1c107210 0x7f7a1c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a20da7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c111460 con 0x7f7a1c101b70 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a20da7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c1112e0 con 0x7f7a1c1011d0 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a20da7700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c1115e0 con 0x7f7a1c107210 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a19d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a1c101b70 0x7f7a1c19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a19d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a1c101b70 0x7f7a1c19eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42396/0 (socket says 172.21.15.77:42396) 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a19d9b700 1 -- 172.21.15.77:0/340793752 learned_addr learned my addr 172.21.15.77:0/340793752 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a1a59c700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a1c1011d0 0x7f7a1c19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a1ad9d700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a1c107210 0x7f7a1c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a19d9b700 1 -- 172.21.15.77:0/340793752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a1c107210 msgr2=0x7f7a1c1987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a19d9b700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a1c107210 0x7f7a1c1987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a19d9b700 1 -- 172.21.15.77:0/340793752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a1c1011d0 msgr2=0x7f7a1c19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a19d9b700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a1c1011d0 0x7f7a1c19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a19d9b700 1 -- 172.21.15.77:0/340793752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1c198f80 con 0x7f7a1c101b70 2024-03-16T10:15:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.963+0000 7f7a1a59c700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a1c1011d0 0x7f7a1c19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.964+0000 7f7a19d9b700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a1c101b70 0x7f7a1c19eb60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400e9a0 tx=0x7f7a0400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.964+0000 7f7a137fe700 1 -- 172.21.15.77:0/340793752 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0400ca10 con 0x7f7a1c101b70 2024-03-16T10:15:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.964+0000 7f7a137fe700 1 -- 172.21.15.77:0/340793752 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a0400cbb0 con 0x7f7a1c101b70 2024-03-16T10:15:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.964+0000 7f7a20da7700 1 -- 172.21.15.77:0/340793752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a1c199270 con 0x7f7a1c101b70 2024-03-16T10:15:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.964+0000 7f7a137fe700 1 -- 172.21.15.77:0/340793752 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a04018680 con 0x7f7a1c101b70 2024-03-16T10:15:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.964+0000 7f7a20da7700 1 -- 172.21.15.77:0/340793752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a1c1a5480 con 0x7f7a1c101b70 2024-03-16T10:15:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.965+0000 7f7a20da7700 1 -- 172.21.15.77:0/340793752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a1c1028c0 con 0x7f7a1c101b70 2024-03-16T10:15:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.966+0000 7f7a137fe700 1 -- 172.21.15.77:0/340793752 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7a04010430 con 0x7f7a1c101b70 2024-03-16T10:15:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.967+0000 7f7a137fe700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7a080a29a0 0x7f7a080a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.967+0000 7f7a1a59c700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7a080a29a0 0x7f7a080a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.967+0000 7f7a137fe700 1 -- 172.21.15.77:0/340793752 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7a040cebf0 con 0x7f7a1c101b70 2024-03-16T10:15:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.967+0000 7f7a1a59c700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7a080a29a0 0x7f7a080a4e60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009680 tx=0x7f7a0c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:19.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:19.972+0000 7f7a137fe700 1 -- 172.21.15.77:0/340793752 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a0409bce0 con 0x7f7a1c101b70 2024-03-16T10:15:19.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:19 smithi077 ceph-mon[32400]: pgmap v295: 97 pgs: 97 active+clean; 634 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 KiB/s wr, 29 op/s 2024-03-16T10:15:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:19 smithi192 ceph-mon[36354]: pgmap v295: 97 pgs: 97 active+clean; 634 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 KiB/s wr, 29 op/s 2024-03-16T10:15:20.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.192+0000 7f7a20da7700 1 -- 172.21.15.77:0/340793752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7a1c1a56e0 con 0x7f7a1c101b70 2024-03-16T10:15:20.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.194+0000 7f7a137fe700 1 -- 172.21.15.77:0/340793752 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22195 (secure 0 0 0) 0x7f7a0409b5e0 con 0x7f7a1c101b70 2024-03-16T10:15:20.195 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:15:20.195 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":85,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:04.004928+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:15:20.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.196+0000 7f7a20da7700 1 -- 172.21.15.77:0/340793752 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7a080a29a0 msgr2=0x7f7a080a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:20.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.196+0000 7f7a20da7700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7a080a29a0 0x7f7a080a4e60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009680 tx=0x7f7a0c009260 comp rx=0 tx=0).stop 2024-03-16T10:15:20.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.196+0000 7f7a20da7700 1 -- 172.21.15.77:0/340793752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a1c101b70 msgr2=0x7f7a1c19eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:20.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.196+0000 7f7a20da7700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a1c101b70 0x7f7a1c19eb60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400e9a0 tx=0x7f7a0400ecb0 comp rx=0 tx=0).stop 2024-03-16T10:15:20.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.196+0000 7f7a20da7700 1 -- 172.21.15.77:0/340793752 shutdown_connections 2024-03-16T10:15:20.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.196+0000 7f7a20da7700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7a080a29a0 0x7f7a080a4e60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:20.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.196+0000 7f7a20da7700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a1c107210 0x7f7a1c1987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:20.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.196+0000 7f7a20da7700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a1c1011d0 0x7f7a1c19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:20.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.196+0000 7f7a20da7700 1 --2- 172.21.15.77:0/340793752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a1c101b70 0x7f7a1c19eb60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:20.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.196+0000 7f7a20da7700 1 -- 172.21.15.77:0/340793752 >> 172.21.15.77:0/340793752 conn(0x7f7a1c078100 msgr2=0x7f7a1c100430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:20.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.196+0000 7f7a20da7700 1 -- 172.21.15.77:0/340793752 shutdown_connections 2024-03-16T10:15:20.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:20.196+0000 7f7a20da7700 1 -- 172.21.15.77:0/340793752 wait complete. 2024-03-16T10:15:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:19 smithi193 ceph-mon[36311]: pgmap v295: 97 pgs: 97 active+clean; 634 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 KiB/s wr, 29 op/s 2024-03-16T10:15:20.900 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-16T10:15:20.901 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T10:15:21.154 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:21.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:20 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/340793752' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:21.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:20 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/340793752' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:21.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:20 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/340793752' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:22.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:21 smithi192 ceph-mon[36354]: pgmap v296: 97 pgs: 97 active+clean; 634 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 17 op/s 2024-03-16T10:15:22.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:21 smithi077 ceph-mon[32400]: pgmap v296: 97 pgs: 97 active+clean; 634 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 17 op/s 2024-03-16T10:15:22.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:21 smithi193 ceph-mon[36311]: pgmap v296: 97 pgs: 97 active+clean; 634 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 17 op/s 2024-03-16T10:15:22.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.928+0000 7f7c14c07700 1 -- 172.21.15.77:0/3793530315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c10101b70 msgr2=0x7f7c1010e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:22.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.928+0000 7f7c14c07700 1 --2- 172.21.15.77:0/3793530315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c10101b70 0x7f7c1010e050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800b0a0 tx=0x7f7c0800b3b0 comp rx=0 tx=0).stop 2024-03-16T10:15:22.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.928+0000 7f7c14c07700 1 -- 172.21.15.77:0/3793530315 shutdown_connections 2024-03-16T10:15:22.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.928+0000 7f7c14c07700 1 --2- 172.21.15.77:0/3793530315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c10101b70 0x7f7c1010e050 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:22.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.928+0000 7f7c14c07700 1 --2- 172.21.15.77:0/3793530315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c10107210 0x7f7c101075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:22.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.928+0000 7f7c14c07700 1 --2- 172.21.15.77:0/3793530315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c101011d0 0x7f7c10101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:22.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.929+0000 7f7c14c07700 1 -- 172.21.15.77:0/3793530315 >> 172.21.15.77:0/3793530315 conn(0x7f7c10078100 msgr2=0x7f7c10078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:22.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.929+0000 7f7c14c07700 1 -- 172.21.15.77:0/3793530315 shutdown_connections 2024-03-16T10:15:22.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.929+0000 7f7c14c07700 1 -- 172.21.15.77:0/3793530315 wait complete. 2024-03-16T10:15:22.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.929+0000 7f7c14c07700 1 Processor -- start 2024-03-16T10:15:22.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.929+0000 7f7c14c07700 1 -- start start 2024-03-16T10:15:22.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c14c07700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c101011d0 0x7f7c1019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:22.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c14c07700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c10101b70 0x7f7c1019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:22.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c14c07700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c10107210 0x7f7c101988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:22.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c14c07700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7c101114e0 con 0x7f7c10101b70 2024-03-16T10:15:22.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c14c07700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7c10111360 con 0x7f7c10107210 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c14c07700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7c10111660 con 0x7f7c101011d0 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c0dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c10101b70 0x7f7c1019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c0dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c10101b70 0x7f7c1019ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46892/0 (socket says 172.21.15.77:46892) 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c0dd9b700 1 -- 172.21.15.77:0/2950303248 learned_addr learned my addr 172.21.15.77:0/2950303248 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c0ed9d700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c10107210 0x7f7c101988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c0e59c700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c101011d0 0x7f7c1019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c0dd9b700 1 -- 172.21.15.77:0/2950303248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c101011d0 msgr2=0x7f7c1019e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c0dd9b700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c101011d0 0x7f7c1019e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c0dd9b700 1 -- 172.21.15.77:0/2950303248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c10107210 msgr2=0x7f7c101988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c0dd9b700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c10107210 0x7f7c101988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c0dd9b700 1 -- 172.21.15.77:0/2950303248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c10199000 con 0x7f7c10101b70 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.930+0000 7f7c0dd9b700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c10101b70 0x7f7c1019ec20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400e9a0 tx=0x7f7c0400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.931+0000 7f7bff7fe700 1 -- 172.21.15.77:0/2950303248 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400cbd0 con 0x7f7c10101b70 2024-03-16T10:15:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.931+0000 7f7c14c07700 1 -- 172.21.15.77:0/2950303248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c101992f0 con 0x7f7c10101b70 2024-03-16T10:15:22.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.931+0000 7f7bff7fe700 1 -- 172.21.15.77:0/2950303248 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c0400cd70 con 0x7f7c10101b70 2024-03-16T10:15:22.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.931+0000 7f7bff7fe700 1 -- 172.21.15.77:0/2950303248 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04018750 con 0x7f7c10101b70 2024-03-16T10:15:22.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.931+0000 7f7c14c07700 1 -- 172.21.15.77:0/2950303248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c101a55b0 con 0x7f7c10101b70 2024-03-16T10:15:22.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.932+0000 7f7bfd7fa700 1 -- 172.21.15.77:0/2950303248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf4005330 con 0x7f7c10101b70 2024-03-16T10:15:22.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.933+0000 7f7bff7fe700 1 -- 172.21.15.77:0/2950303248 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7c04010430 con 0x7f7c10101b70 2024-03-16T10:15:22.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.933+0000 7f7bff7fe700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7bf80a2920 0x7f7bf80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:22.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.934+0000 7f7c0e59c700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7bf80a2920 0x7f7bf80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:22.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.934+0000 7f7bff7fe700 1 -- 172.21.15.77:0/2950303248 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7c040cf1c0 con 0x7f7c10101b70 2024-03-16T10:15:22.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.934+0000 7f7c0e59c700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7bf80a2920 0x7f7bf80a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7c101999e0 tx=0x7f7c00009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:22.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:22.937+0000 7f7bff7fe700 1 -- 172.21.15.77:0/2950303248 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7c0409c2b0 con 0x7f7c10101b70 2024-03-16T10:15:23.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.136+0000 7f7bfd7fa700 1 -- 172.21.15.77:0/2950303248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7bf40058f0 con 0x7f7c10101b70 2024-03-16T10:15:23.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.139+0000 7f7bff7fe700 1 -- 172.21.15.77:0/2950303248 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22195 (secure 0 0 0) 0x7f7c0409bbb0 con 0x7f7c10101b70 2024-03-16T10:15:23.140 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:15:23.140 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":85,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:04.004928+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:15:23.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.141+0000 7f7bfd7fa700 1 -- 172.21.15.77:0/2950303248 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7bf80a2920 msgr2=0x7f7bf80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:23.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.141+0000 7f7bfd7fa700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7bf80a2920 0x7f7bf80a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7c101999e0 tx=0x7f7c00009260 comp rx=0 tx=0).stop 2024-03-16T10:15:23.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.141+0000 7f7bfd7fa700 1 -- 172.21.15.77:0/2950303248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c10101b70 msgr2=0x7f7c1019ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:23.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.141+0000 7f7bfd7fa700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c10101b70 0x7f7c1019ec20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400e9a0 tx=0x7f7c0400ecb0 comp rx=0 tx=0).stop 2024-03-16T10:15:23.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.141+0000 7f7bfd7fa700 1 -- 172.21.15.77:0/2950303248 shutdown_connections 2024-03-16T10:15:23.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.141+0000 7f7bfd7fa700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7bf80a2920 0x7f7bf80a4de0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:23.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.141+0000 7f7bfd7fa700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c10107210 0x7f7c101988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:23.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.141+0000 7f7bfd7fa700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c101011d0 0x7f7c1019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:23.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.141+0000 7f7bfd7fa700 1 --2- 172.21.15.77:0/2950303248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c10101b70 0x7f7c1019ec20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:23.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.141+0000 7f7bfd7fa700 1 -- 172.21.15.77:0/2950303248 >> 172.21.15.77:0/2950303248 conn(0x7f7c10078100 msgr2=0x7f7c101004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:23.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.141+0000 7f7bfd7fa700 1 -- 172.21.15.77:0/2950303248 shutdown_connections 2024-03-16T10:15:23.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:23.141+0000 7f7bfd7fa700 1 -- 172.21.15.77:0/2950303248 wait complete. 2024-03-16T10:15:23.820 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-16T10:15:23.820 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-16T10:15:23.820 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-16T10:15:23.821 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-16T10:15:23.821 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-16T10:15:23.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-16T10:15:23.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-16T10:15:23.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-16T10:15:23.823 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-16T10:15:23.823 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-16T10:15:23.824 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-16T10:15:23.824 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-16T10:15:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:23 smithi192 ceph-mon[36354]: pgmap v297: 97 pgs: 97 active+clean; 634 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 19 op/s 2024-03-16T10:15:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:23 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2950303248' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:24.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:23 smithi077 ceph-mon[32400]: pgmap v297: 97 pgs: 97 active+clean; 634 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 19 op/s 2024-03-16T10:15:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:23 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2950303248' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:24.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:23 smithi193 ceph-mon[36311]: pgmap v297: 97 pgs: 97 active+clean; 634 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 19 op/s 2024-03-16T10:15:24.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:23 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2950303248' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:15:24.517 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:25.341 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:25.353 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:25.354 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:26.095 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:25 smithi077 ceph-mon[32400]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 KiB/s wr, 14 op/s 2024-03-16T10:15:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:25 smithi192 ceph-mon[36354]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 KiB/s wr, 14 op/s 2024-03-16T10:15:26.238 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:26.239 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:26.239 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:26.242 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:25 smithi193 ceph-mon[36311]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 KiB/s wr, 14 op/s 2024-03-16T10:15:27.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.872+0000 7fb441ff2700 1 -- 172.21.15.77:0/322552877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb43c06aa70 msgr2=0x7fb43c06ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:27.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.872+0000 7fb441ff2700 1 --2- 172.21.15.77:0/322552877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb43c06aa70 0x7fb43c06ae50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb4300099c0 tx=0x7fb430009cd0 comp rx=0 tx=0).stop 2024-03-16T10:15:27.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.872+0000 7fb441ff2700 1 -- 172.21.15.77:0/322552877 shutdown_connections 2024-03-16T10:15:27.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.872+0000 7fb441ff2700 1 --2- 172.21.15.77:0/322552877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb43c06aa70 0x7fb43c06ae50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:27.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.872+0000 7fb441ff2700 1 --2- 172.21.15.77:0/322552877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb43c1075b0 0x7fb43c10f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:27.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.872+0000 7fb441ff2700 1 --2- 172.21.15.77:0/322552877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb43c06b420 0x7fb43c106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:27.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.872+0000 7fb441ff2700 1 -- 172.21.15.77:0/322552877 >> 172.21.15.77:0/322552877 conn(0x7fb43c077220 msgr2=0x7fb43c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:27.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.872+0000 7fb441ff2700 1 -- 172.21.15.77:0/322552877 shutdown_connections 2024-03-16T10:15:27.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.872+0000 7fb441ff2700 1 -- 172.21.15.77:0/322552877 wait complete. 2024-03-16T10:15:27.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.872+0000 7fb441ff2700 1 Processor -- start 2024-03-16T10:15:27.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb441ff2700 1 -- start start 2024-03-16T10:15:27.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb441ff2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb43c06aa70 0x7fb43c1012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb441ff2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb43c06b420 0x7fb43c101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb441ff2700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb43c1075b0 0x7fb43c101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb441ff2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c1125c0 con 0x7fb43c06b420 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb441ff2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c112440 con 0x7fb43c06aa70 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb441ff2700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c112740 con 0x7fb43c1075b0 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb43affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb43c06b420 0x7fb43c101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb43affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb43c06b420 0x7fb43c101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46910/0 (socket says 172.21.15.77:46910) 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb43affd700 1 -- 172.21.15.77:0/84416034 learned_addr learned my addr 172.21.15.77:0/84416034 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb43b7fe700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb43c06aa70 0x7fb43c1012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb43bfff700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb43c1075b0 0x7fb43c101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb43affd700 1 -- 172.21.15.77:0/84416034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb43c1075b0 msgr2=0x7fb43c101d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb43affd700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb43c1075b0 0x7fb43c101d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb43affd700 1 -- 172.21.15.77:0/84416034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb43c06aa70 msgr2=0x7fb43c1012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb43affd700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb43c06aa70 0x7fb43c1012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb43affd700 1 -- 172.21.15.77:0/84416034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43c1a6fa0 con 0x7fb43c06b420 2024-03-16T10:15:27.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb43b7fe700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb43c06aa70 0x7fb43c1012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.873+0000 7fb43affd700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb43c06b420 0x7fb43c101810 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00e970 tx=0x7fb42c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.874+0000 7fb438ff9700 1 -- 172.21.15.77:0/84416034 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c004a30 con 0x7fb43c06b420 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.874+0000 7fb438ff9700 1 -- 172.21.15.77:0/84416034 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb42c00ccc0 con 0x7fb43c06b420 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.874+0000 7fb441ff2700 1 -- 172.21.15.77:0/84416034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43c1a7290 con 0x7fb43c06b420 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.874+0000 7fb438ff9700 1 -- 172.21.15.77:0/84416034 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c005650 con 0x7fb43c06b420 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.874+0000 7fb441ff2700 1 -- 172.21.15.77:0/84416034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb43c1a77d0 con 0x7fb43c06b420 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.874+0000 7fb441ff2700 1 -- 172.21.15.77:0/84416034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb43c0629a0 con 0x7fb43c06b420 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.876+0000 7fb438ff9700 1 -- 172.21.15.77:0/84416034 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb42c004bd0 con 0x7fb43c06b420 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.877+0000 7fb438ff9700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4240a2920 0x7fb4240a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.877+0000 7fb43b7fe700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4240a2920 0x7fb4240a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.877+0000 7fb438ff9700 1 -- 172.21.15.77:0/84416034 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb42c0cf0c0 con 0x7fb43c06b420 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.877+0000 7fb438ff9700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243] conn(0x7fb4240ac100 0x7fb4240ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.877+0000 7fb43b7fe700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4240a2920 0x7fb4240a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb430000c00 tx=0x7fb430005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.877+0000 7fb438ff9700 1 -- 172.21.15.77:0/84416034 --> [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb42c015e80 con 0x7fb4240ac100 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.877+0000 7fb438ff9700 1 -- 172.21.15.77:0/84416034 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fb42c0cf5a0 con 0x7fb43c06b420 2024-03-16T10:15:27.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.877+0000 7fb43bfff700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243] conn(0x7fb4240ac100 0x7fb4240ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:27.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.878+0000 7fb43bfff700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243] conn(0x7fb4240ac100 0x7fb4240ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:27.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.878+0000 7fb438ff9700 1 -- 172.21.15.77:0/84416034 <== osd.7 v2:172.21.15.192:6816/384726243 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb42c015e80 con 0x7fb4240ac100 2024-03-16T10:15:27.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.895+0000 7fb441ff2700 1 -- 172.21.15.77:0/84416034 --> [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb43c062e20 con 0x7fb4240ac100 2024-03-16T10:15:27.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.895+0000 7fb438ff9700 1 -- 172.21.15.77:0/84416034 <== osd.7 v2:172.21.15.192:6816/384726243 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb43c062e20 con 0x7fb4240ac100 2024-03-16T10:15:27.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.895+0000 7fb441ff2700 1 -- 172.21.15.77:0/84416034 >> [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243] conn(0x7fb4240ac100 msgr2=0x7fb4240ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:27.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.895+0000 7fb441ff2700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243] conn(0x7fb4240ac100 0x7fb4240ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:27.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.895+0000 7fb441ff2700 1 -- 172.21.15.77:0/84416034 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4240a2920 msgr2=0x7fb4240a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:27.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.895+0000 7fb441ff2700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4240a2920 0x7fb4240a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb430000c00 tx=0x7fb430005fb0 comp rx=0 tx=0).stop 2024-03-16T10:15:27.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.895+0000 7fb441ff2700 1 -- 172.21.15.77:0/84416034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb43c06b420 msgr2=0x7fb43c101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:27.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.895+0000 7fb441ff2700 1 --2- 172.21.15.77:0/84416034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb43c06b420 0x7fb43c101810 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00e970 tx=0x7fb42c00ec80 comp rx=0 tx=0).stop 2024-03-16T10:15:27.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.895+0000 7fb43bfff700 1 -- 172.21.15.77:0/84416034 reap_dead start 2024-03-16T10:15:27.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.895+0000 7fb441ff2700 1 -- 172.21.15.77:0/84416034 shutdown_connections 2024-03-16T10:15:27.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.896+0000 7fb441ff2700 1 -- 172.21.15.77:0/84416034 >> 172.21.15.77:0/84416034 conn(0x7fb43c077220 msgr2=0x7fb43c0ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:27.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.896+0000 7fb441ff2700 1 -- 172.21.15.77:0/84416034 shutdown_connections 2024-03-16T10:15:27.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:27.896+0000 7fb441ff2700 1 -- 172.21.15.77:0/84416034 wait complete. 2024-03-16T10:15:28.076 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:27 smithi077 ceph-mon[32400]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-03-16T10:15:28.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:27 smithi192 ceph-mon[36354]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-03-16T10:15:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:27 smithi193 ceph-mon[36311]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-03-16T10:15:28.723 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:28.724 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:28.725 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:29.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.637+0000 7f2737f93700 1 -- 172.21.15.77:0/3674881792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27301075f0 msgr2=0x7f273010f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:29.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.637+0000 7f2737f93700 1 --2- 172.21.15.77:0/3674881792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27301075f0 0x7f273010f1c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f272800b0d0 tx=0x7f272800b3e0 comp rx=0 tx=0).stop 2024-03-16T10:15:29.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.637+0000 7f2737f93700 1 -- 172.21.15.77:0/3674881792 shutdown_connections 2024-03-16T10:15:29.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.637+0000 7f2737f93700 1 --2- 172.21.15.77:0/3674881792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27301075f0 0x7f273010f1c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:29.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.637+0000 7f2737f93700 1 --2- 172.21.15.77:0/3674881792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2730102510 0x7f27301028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:29.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.637+0000 7f2737f93700 1 --2- 172.21.15.77:0/3674881792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2730102ec0 0x7f2730106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:29.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.637+0000 7f2737f93700 1 -- 172.21.15.77:0/3674881792 >> 172.21.15.77:0/3674881792 conn(0x7f27300fdd80 msgr2=0x7f27301001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:29.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.637+0000 7f2737f93700 1 -- 172.21.15.77:0/3674881792 shutdown_connections 2024-03-16T10:15:29.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.637+0000 7f2737f93700 1 -- 172.21.15.77:0/3674881792 wait complete. 2024-03-16T10:15:29.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2737f93700 1 Processor -- start 2024-03-16T10:15:29.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2737f93700 1 -- start start 2024-03-16T10:15:29.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2737f93700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2730102510 0x7f273019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2737f93700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2730102ec0 0x7f273019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2737f93700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27301075f0 0x7f2730198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2737f93700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2730112600 con 0x7f2730102510 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2737f93700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2730112480 con 0x7f27301075f0 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2737f93700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2730112780 con 0x7f2730102ec0 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2735d2f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2730102510 0x7f273019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2735d2f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2730102510 0x7f273019e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46920/0 (socket says 172.21.15.77:46920) 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2735d2f700 1 -- 172.21.15.77:0/3066560487 learned_addr learned my addr 172.21.15.77:0/3066560487 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f273552e700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2730102ec0 0x7f273019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2736530700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27301075f0 0x7f2730198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2735d2f700 1 -- 172.21.15.77:0/3066560487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2730102ec0 msgr2=0x7f273019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2735d2f700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2730102ec0 0x7f273019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2735d2f700 1 -- 172.21.15.77:0/3066560487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27301075f0 msgr2=0x7f2730198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2735d2f700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27301075f0 0x7f2730198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.638+0000 7f2735d2f700 1 -- 172.21.15.77:0/3066560487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2730199040 con 0x7f2730102510 2024-03-16T10:15:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.639+0000 7f2735d2f700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2730102510 0x7f273019e690 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f272000e9a0 tx=0x7f272000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:29.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.639+0000 7f2726ffd700 1 -- 172.21.15.77:0/3066560487 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272000ca10 con 0x7f2730102510 2024-03-16T10:15:29.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.639+0000 7f2726ffd700 1 -- 172.21.15.77:0/3066560487 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f272000cbb0 con 0x7f2730102510 2024-03-16T10:15:29.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.639+0000 7f2737f93700 1 -- 172.21.15.77:0/3066560487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2730199330 con 0x7f2730102510 2024-03-16T10:15:29.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.639+0000 7f2726ffd700 1 -- 172.21.15.77:0/3066560487 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2720018680 con 0x7f2730102510 2024-03-16T10:15:29.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.639+0000 7f2737f93700 1 -- 172.21.15.77:0/3066560487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27301a54f0 con 0x7f2730102510 2024-03-16T10:15:29.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.640+0000 7f2737f93700 1 -- 172.21.15.77:0/3066560487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f27300ff240 con 0x7f2730102510 2024-03-16T10:15:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.641+0000 7f2726ffd700 1 -- 172.21.15.77:0/3066560487 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2720010430 con 0x7f2730102510 2024-03-16T10:15:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.641+0000 7f2726ffd700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f271c0a2920 0x7f271c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.641+0000 7f273552e700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f271c0a2920 0x7f271c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.642+0000 7f2726ffd700 1 -- 172.21.15.77:0/3066560487 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f27200cf480 con 0x7f2730102510 2024-03-16T10:15:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.642+0000 7f2726ffd700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054] conn(0x7f271c0ac100 0x7f271c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.642+0000 7f2726ffd700 1 -- 172.21.15.77:0/3066560487 --> [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f272001bec0 con 0x7f271c0ac100 2024-03-16T10:15:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.642+0000 7f2726ffd700 1 -- 172.21.15.77:0/3066560487 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f27200cf960 con 0x7f2730102510 2024-03-16T10:15:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.642+0000 7f2736530700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054] conn(0x7f271c0ac100 0x7f271c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.642+0000 7f273552e700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f271c0a2920 0x7f271c0a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2730101490 tx=0x7f272c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.642+0000 7f2736530700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054] conn(0x7f271c0ac100 0x7f271c0ae520 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:29.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.643+0000 7f2726ffd700 1 -- 172.21.15.77:0/3066560487 <== osd.5 v2:172.21.15.193:6808/83797054 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f272001bec0 con 0x7f271c0ac100 2024-03-16T10:15:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.659+0000 7f2737f93700 1 -- 172.21.15.77:0/3066560487 --> [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f27301039f0 con 0x7f271c0ac100 2024-03-16T10:15:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.660+0000 7f2726ffd700 1 -- 172.21.15.77:0/3066560487 <== osd.5 v2:172.21.15.193:6808/83797054 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f27301039f0 con 0x7f271c0ac100 2024-03-16T10:15:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.660+0000 7f2737f93700 1 -- 172.21.15.77:0/3066560487 >> [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054] conn(0x7f271c0ac100 msgr2=0x7f271c0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.660+0000 7f2737f93700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054] conn(0x7f271c0ac100 0x7f271c0ae520 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.660+0000 7f2737f93700 1 -- 172.21.15.77:0/3066560487 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f271c0a2920 msgr2=0x7f271c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.660+0000 7f2737f93700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f271c0a2920 0x7f271c0a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2730101490 tx=0x7f272c009260 comp rx=0 tx=0).stop 2024-03-16T10:15:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.660+0000 7f2737f93700 1 -- 172.21.15.77:0/3066560487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2730102510 msgr2=0x7f273019e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.660+0000 7f2737f93700 1 --2- 172.21.15.77:0/3066560487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2730102510 0x7f273019e690 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f272000e9a0 tx=0x7f272000ecb0 comp rx=0 tx=0).stop 2024-03-16T10:15:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.660+0000 7f2736530700 1 -- 172.21.15.77:0/3066560487 reap_dead start 2024-03-16T10:15:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.660+0000 7f2737f93700 1 -- 172.21.15.77:0/3066560487 shutdown_connections 2024-03-16T10:15:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.660+0000 7f2737f93700 1 -- 172.21.15.77:0/3066560487 >> 172.21.15.77:0/3066560487 conn(0x7f27300fdd80 msgr2=0x7f27300fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.660+0000 7f2737f93700 1 -- 172.21.15.77:0/3066560487 shutdown_connections 2024-03-16T10:15:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:29.660+0000 7f2737f93700 1 -- 172.21.15.77:0/3066560487 wait complete. 2024-03-16T10:15:30.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:29 smithi192 ceph-mon[36354]: pgmap v300: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-16T10:15:30.181 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:29 smithi077 ceph-mon[32400]: pgmap v300: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-16T10:15:30.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:29 smithi193 ceph-mon[36311]: pgmap v300: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-16T10:15:31.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:31 smithi077 ceph-mon[32400]: pgmap v301: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 9 op/s 2024-03-16T10:15:32.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:31 smithi192 ceph-mon[36354]: pgmap v301: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 9 op/s 2024-03-16T10:15:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:31 smithi193 ceph-mon[36311]: pgmap v301: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 9 op/s 2024-03-16T10:15:33.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:33 smithi077 ceph-mon[32400]: pgmap v302: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-16T10:15:34.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:33 smithi192 ceph-mon[36354]: pgmap v302: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-16T10:15:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:33 smithi193 ceph-mon[36311]: pgmap v302: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-16T10:15:35.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.924+0000 7fe470255700 1 -- 172.21.15.77:0/361959399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4681075b0 msgr2=0x7fe46810f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:35.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.924+0000 7fe470255700 1 --2- 172.21.15.77:0/361959399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4681075b0 0x7fe46810f180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe46000b0a0 tx=0x7fe46000b3b0 comp rx=0 tx=0).stop 2024-03-16T10:15:35.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.924+0000 7fe470255700 1 -- 172.21.15.77:0/361959399 shutdown_connections 2024-03-16T10:15:35.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.924+0000 7fe470255700 1 --2- 172.21.15.77:0/361959399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4681075b0 0x7fe46810f180 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:35.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.924+0000 7fe470255700 1 --2- 172.21.15.77:0/361959399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe46806aa70 0x7fe46806ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:35.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.924+0000 7fe470255700 1 --2- 172.21.15.77:0/361959399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe46806b420 0x7fe468106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:35.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.924+0000 7fe470255700 1 -- 172.21.15.77:0/361959399 >> 172.21.15.77:0/361959399 conn(0x7fe4680772b0 msgr2=0x7fe4680776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:35.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.925+0000 7fe470255700 1 -- 172.21.15.77:0/361959399 shutdown_connections 2024-03-16T10:15:35.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.925+0000 7fe470255700 1 -- 172.21.15.77:0/361959399 wait complete. 2024-03-16T10:15:35.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.925+0000 7fe470255700 1 Processor -- start 2024-03-16T10:15:35.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.925+0000 7fe470255700 1 -- start start 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.925+0000 7fe470255700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe46806aa70 0x7fe4681012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.925+0000 7fe470255700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe46806b420 0x7fe468101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.925+0000 7fe470255700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4681075b0 0x7fe468101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.925+0000 7fe470255700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe468112590 con 0x7fe46806aa70 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.925+0000 7fe470255700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe468112410 con 0x7fe4681075b0 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.925+0000 7fe470255700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe468112710 con 0x7fe46806b420 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46dff1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe46806aa70 0x7fe4681012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46e7f2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4681075b0 0x7fe468101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46d7f0700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe46806b420 0x7fe468101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46e7f2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4681075b0 0x7fe468101d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:51208/0 (socket says 172.21.15.77:51208) 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46d7f0700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe46806b420 0x7fe468101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:47066/0 (socket says 172.21.15.77:47066) 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46e7f2700 1 -- 172.21.15.77:0/3322823503 learned_addr learned my addr 172.21.15.77:0/3322823503 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46d7f0700 1 -- 172.21.15.77:0/3322823503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4681075b0 msgr2=0x7fe468101d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:35.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46d7f0700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4681075b0 0x7fe468101d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:35.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46d7f0700 1 -- 172.21.15.77:0/3322823503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe46806aa70 msgr2=0x7fe4681012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:35.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46d7f0700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe46806aa70 0x7fe4681012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:35.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46d7f0700 1 -- 172.21.15.77:0/3322823503 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4681a6f00 con 0x7fe46806b420 2024-03-16T10:15:35.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46e7f2700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4681075b0 0x7fe468101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:35.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46dff1700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe46806aa70 0x7fe4681012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:35.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe46d7f0700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe46806b420 0x7fe468101810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe45800d720 tx=0x7fe45800da30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:35.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe45effd700 1 -- 172.21.15.77:0/3322823503 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458014030 con 0x7fe46806b420 2024-03-16T10:15:35.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe470255700 1 -- 172.21.15.77:0/3322823503 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4681a71f0 con 0x7fe46806b420 2024-03-16T10:15:35.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe45effd700 1 -- 172.21.15.77:0/3322823503 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe458004500 con 0x7fe46806b420 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe45effd700 1 -- 172.21.15.77:0/3322823503 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458005480 con 0x7fe46806b420 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.926+0000 7fe470255700 1 -- 172.21.15.77:0/3322823503 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4681a7730 con 0x7fe46806b420 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.927+0000 7fe470255700 1 -- 172.21.15.77:0/3322823503 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe4680629e0 con 0x7fe46806b420 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.929+0000 7fe45effd700 1 -- 172.21.15.77:0/3322823503 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe458005620 con 0x7fe46806b420 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.929+0000 7fe45effd700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe4540a2920 0x7fe4540a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.930+0000 7fe46dff1700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe4540a2920 0x7fe4540a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.930+0000 7fe45effd700 1 -- 172.21.15.77:0/3322823503 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fe4580ce8c0 con 0x7fe46806b420 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.930+0000 7fe45effd700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113] conn(0x7fe4540ac100 0x7fe4540ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.930+0000 7fe45effd700 1 -- 172.21.15.77:0/3322823503 --> [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe458011ec0 con 0x7fe4540ac100 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.930+0000 7fe45effd700 1 -- 172.21.15.77:0/3322823503 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fe4580cec70 con 0x7fe46806b420 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.930+0000 7fe46dff1700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe4540a2920 0x7fe4540a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe464011490 tx=0x7fe464009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.930+0000 7fe46e7f2700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113] conn(0x7fe4540ac100 0x7fe4540ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.931+0000 7fe46e7f2700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113] conn(0x7fe4540ac100 0x7fe4540ae520 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.932+0000 7fe45effd700 1 -- 172.21.15.77:0/3322823503 <== osd.11 v2:172.21.15.193:6824/2760387113 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe458011ec0 con 0x7fe4540ac100 2024-03-16T10:15:35.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.948+0000 7fe470255700 1 -- 172.21.15.77:0/3322823503 --> [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe468062de0 con 0x7fe4540ac100 2024-03-16T10:15:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.949+0000 7fe45effd700 1 -- 172.21.15.77:0/3322823503 <== osd.11 v2:172.21.15.193:6824/2760387113 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe468062de0 con 0x7fe4540ac100 2024-03-16T10:15:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.949+0000 7fe470255700 1 -- 172.21.15.77:0/3322823503 >> [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113] conn(0x7fe4540ac100 msgr2=0x7fe4540ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.949+0000 7fe470255700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113] conn(0x7fe4540ac100 0x7fe4540ae520 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.950+0000 7fe470255700 1 -- 172.21.15.77:0/3322823503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe4540a2920 msgr2=0x7fe4540a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.950+0000 7fe470255700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe4540a2920 0x7fe4540a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe464011490 tx=0x7fe464009260 comp rx=0 tx=0).stop 2024-03-16T10:15:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.950+0000 7fe470255700 1 -- 172.21.15.77:0/3322823503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe46806b420 msgr2=0x7fe468101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.950+0000 7fe470255700 1 --2- 172.21.15.77:0/3322823503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe46806b420 0x7fe468101810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe45800d720 tx=0x7fe45800da30 comp rx=0 tx=0).stop 2024-03-16T10:15:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.950+0000 7fe46e7f2700 1 -- 172.21.15.77:0/3322823503 reap_dead start 2024-03-16T10:15:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.950+0000 7fe470255700 1 -- 172.21.15.77:0/3322823503 shutdown_connections 2024-03-16T10:15:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.950+0000 7fe470255700 1 -- 172.21.15.77:0/3322823503 >> 172.21.15.77:0/3322823503 conn(0x7fe4680772b0 msgr2=0x7fe4680ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.950+0000 7fe470255700 1 -- 172.21.15.77:0/3322823503 shutdown_connections 2024-03-16T10:15:35.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:34.950+0000 7fe470255700 1 -- 172.21.15.77:0/3322823503 wait complete. 2024-03-16T10:15:35.964 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:35 smithi077 ceph-mon[32400]: pgmap v303: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-16T10:15:36.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:35 smithi192 ceph-mon[36354]: pgmap v303: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-16T10:15:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:35 smithi193 ceph-mon[36311]: pgmap v303: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-16T10:15:37.176 INFO:teuthology.orchestra.run.smithi077.stdout:163208757297 2024-03-16T10:15:37.177 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-16T10:15:37.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:37 smithi077 ceph-mon[32400]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-16T10:15:38.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:37 smithi192 ceph-mon[36354]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-16T10:15:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:37 smithi193 ceph-mon[36311]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-16T10:15:38.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 -- 172.21.15.77:0/89191322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7984102320 msgr2=0x7f79841027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:38.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 --2- 172.21.15.77:0/89191322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7984102320 0x7f79841027a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f79800099c0 tx=0x7f7980009cd0 comp rx=0 tx=0).stop 2024-03-16T10:15:38.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 -- 172.21.15.77:0/89191322 shutdown_connections 2024-03-16T10:15:38.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 --2- 172.21.15.77:0/89191322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7984102ce0 0x7f798410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:38.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 --2- 172.21.15.77:0/89191322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7984108300 0x7f79841086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:38.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 --2- 172.21.15.77:0/89191322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7984102320 0x7f79841027a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:38.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 -- 172.21.15.77:0/89191322 >> 172.21.15.77:0/89191322 conn(0x7f7984079400 msgr2=0x7f7984079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:38.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 -- 172.21.15.77:0/89191322 shutdown_connections 2024-03-16T10:15:38.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 -- 172.21.15.77:0/89191322 wait complete. 2024-03-16T10:15:38.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 Processor -- start 2024-03-16T10:15:38.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 -- start start 2024-03-16T10:15:38.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7984102320 0x7f798419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:38.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.541+0000 7f798bd59700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7984102ce0 0x7f798419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:38.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f798bd59700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7984108300 0x7f79841987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:38.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f798bd59700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7984112580 con 0x7f7984102320 2024-03-16T10:15:38.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f798bd59700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7984112400 con 0x7f7984108300 2024-03-16T10:15:38.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f798bd59700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7984112700 con 0x7f7984102ce0 2024-03-16T10:15:38.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f7989af5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7984102320 0x7f798419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f79892f4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7984102ce0 0x7f798419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f7989af5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7984102320 0x7f798419e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36708/0 (socket says 172.21.15.77:36708) 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f7989af5700 1 -- 172.21.15.77:0/3809510946 learned_addr learned my addr 172.21.15.77:0/3809510946 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f798a2f6700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7984108300 0x7f79841987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f79892f4700 1 -- 172.21.15.77:0/3809510946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7984108300 msgr2=0x7f79841987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f79892f4700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7984108300 0x7f79841987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f79892f4700 1 -- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7984102320 msgr2=0x7f798419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f79892f4700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7984102320 0x7f798419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f79892f4700 1 -- 172.21.15.77:0/3809510946 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7984198fc0 con 0x7f7984102ce0 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f7989af5700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7984102320 0x7f798419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f798a2f6700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7984108300 0x7f79841987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f79892f4700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7984102ce0 0x7f798419eba0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f798000b580 tx=0x7f7980003a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f797affd700 1 -- 172.21.15.77:0/3809510946 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7980003680 con 0x7f7984102ce0 2024-03-16T10:15:38.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f798bd59700 1 -- 172.21.15.77:0/3809510946 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7984199250 con 0x7f7984102ce0 2024-03-16T10:15:38.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f797affd700 1 -- 172.21.15.77:0/3809510946 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79800055a0 con 0x7f7984102ce0 2024-03-16T10:15:38.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.542+0000 7f797affd700 1 -- 172.21.15.77:0/3809510946 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798000fca0 con 0x7f7984102ce0 2024-03-16T10:15:38.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.543+0000 7f798bd59700 1 -- 172.21.15.77:0/3809510946 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79841a54c0 con 0x7f7984102ce0 2024-03-16T10:15:38.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.543+0000 7f798bd59700 1 -- 172.21.15.77:0/3809510946 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7984062a60 con 0x7f7984102ce0 2024-03-16T10:15:38.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.545+0000 7f797affd700 1 -- 172.21.15.77:0/3809510946 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f798000f460 con 0x7f7984102ce0 2024-03-16T10:15:38.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.545+0000 7f797affd700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f79700a2ac0 0x7f79700a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:38.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.545+0000 7f7989af5700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f79700a2ac0 0x7f79700a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:38.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.545+0000 7f797affd700 1 -- 172.21.15.77:0/3809510946 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f79800cfae0 con 0x7f7984102ce0 2024-03-16T10:15:38.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.546+0000 7f797affd700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613] conn(0x7f79700ac2a0 0x7f79700ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:38.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.546+0000 7f797affd700 1 -- 172.21.15.77:0/3809510946 --> [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7980003680 con 0x7f79700ac2a0 2024-03-16T10:15:38.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.546+0000 7f797affd700 1 -- 172.21.15.77:0/3809510946 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f79800cffc0 con 0x7f7984102ce0 2024-03-16T10:15:38.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.546+0000 7f798a2f6700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613] conn(0x7f79700ac2a0 0x7f79700ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:38.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.546+0000 7f7989af5700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f79700a2ac0 0x7f79700a4f80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7974011630 tx=0x7f7974009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:38.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.546+0000 7f798a2f6700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613] conn(0x7f79700ac2a0 0x7f79700ae6c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:38.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.547+0000 7f797affd700 1 -- 172.21.15.77:0/3809510946 <== osd.0 v2:172.21.15.77:6802/2169364613 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7980003680 con 0x7f79700ac2a0 2024-03-16T10:15:38.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.563+0000 7f798bd59700 1 -- 172.21.15.77:0/3809510946 --> [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7984062ee0 con 0x7f79700ac2a0 2024-03-16T10:15:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.563+0000 7f797affd700 1 -- 172.21.15.77:0/3809510946 <== osd.0 v2:172.21.15.77:6802/2169364613 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7984062ee0 con 0x7f79700ac2a0 2024-03-16T10:15:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.563+0000 7f798bd59700 1 -- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613] conn(0x7f79700ac2a0 msgr2=0x7f79700ae6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:38.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.563+0000 7f798bd59700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613] conn(0x7f79700ac2a0 0x7f79700ae6c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.563+0000 7f798bd59700 1 -- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f79700a2ac0 msgr2=0x7f79700a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.563+0000 7f798bd59700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f79700a2ac0 0x7f79700a4f80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7974011630 tx=0x7f7974009260 comp rx=0 tx=0).stop 2024-03-16T10:15:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.563+0000 7f798bd59700 1 -- 172.21.15.77:0/3809510946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7984102ce0 msgr2=0x7f798419eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.563+0000 7f798bd59700 1 --2- 172.21.15.77:0/3809510946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7984102ce0 0x7f798419eba0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f798000b580 tx=0x7f7980003a90 comp rx=0 tx=0).stop 2024-03-16T10:15:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.563+0000 7f798a2f6700 1 -- 172.21.15.77:0/3809510946 reap_dead start 2024-03-16T10:15:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.563+0000 7f798bd59700 1 -- 172.21.15.77:0/3809510946 shutdown_connections 2024-03-16T10:15:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.563+0000 7f798bd59700 1 -- 172.21.15.77:0/3809510946 >> 172.21.15.77:0/3809510946 conn(0x7f7984079400 msgr2=0x7f7984100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.563+0000 7f798bd59700 1 -- 172.21.15.77:0/3809510946 shutdown_connections 2024-03-16T10:15:38.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:38.564+0000 7f798bd59700 1 -- 172.21.15.77:0/3809510946 wait complete. 2024-03-16T10:15:39.475 INFO:teuthology.orchestra.run.smithi077.stdout:34359738444 2024-03-16T10:15:39.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-16T10:15:40.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:39 smithi192 ceph-mon[36354]: pgmap v305: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-16T10:15:40.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:39 smithi077 ceph-mon[32400]: pgmap v305: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-16T10:15:40.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:39 smithi193 ceph-mon[36311]: pgmap v305: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-16T10:15:40.375 INFO:teuthology.orchestra.run.smithi077.stdout:219043332135 2024-03-16T10:15:40.376 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-16T10:15:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.729+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/772144821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f88107210 msgr2=0x7f7f881075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:41.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.729+0000 7f7f8fa2a700 1 --2- 172.21.15.77:0/772144821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f88107210 0x7f7f881075f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009990 tx=0x7f7f78009ca0 comp rx=0 tx=0).stop 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.729+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/772144821 shutdown_connections 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.729+0000 7f7f8fa2a700 1 --2- 172.21.15.77:0/772144821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f88107210 0x7f7f881075f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.729+0000 7f7f8fa2a700 1 --2- 172.21.15.77:0/772144821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f88101b70 0x7f7f8810e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.729+0000 7f7f8fa2a700 1 --2- 172.21.15.77:0/772144821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f881011d0 0x7f7f88101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.729+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/772144821 >> 172.21.15.77:0/772144821 conn(0x7f7f88078100 msgr2=0x7f7f88078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.729+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/772144821 shutdown_connections 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.729+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/772144821 wait complete. 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8fa2a700 1 Processor -- start 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8fa2a700 1 -- start start 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8fa2a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f881011d0 0x7f7f88199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8fa2a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f88101b70 0x7f7f88199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8fa2a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f88107210 0x7f7f881a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:41.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8fa2a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7f88111450 con 0x7f7f88107210 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8fa2a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7f881112d0 con 0x7f7f881011d0 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8fa2a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7f881115d0 con 0x7f7f88101b70 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8dfc7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f88107210 0x7f7f881a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8dfc7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f88107210 0x7f7f881a48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59274/0 (socket says 172.21.15.77:59274) 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8dfc7700 1 -- 172.21.15.77:0/3595868486 learned_addr learned my addr 172.21.15.77:0/3595868486 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8d7c6700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f881011d0 0x7f7f88199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.730+0000 7f7f8cfc5700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f88101b70 0x7f7f88199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.731+0000 7f7f8dfc7700 1 -- 172.21.15.77:0/3595868486 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f88101b70 msgr2=0x7f7f88199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.731+0000 7f7f8dfc7700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f88101b70 0x7f7f88199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.731+0000 7f7f8dfc7700 1 -- 172.21.15.77:0/3595868486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f881011d0 msgr2=0x7f7f88199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.731+0000 7f7f8dfc7700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f881011d0 0x7f7f88199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.731+0000 7f7f8dfc7700 1 -- 172.21.15.77:0/3595868486 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f881a4e10 con 0x7f7f88107210 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.731+0000 7f7f8dfc7700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f88107210 0x7f7f881a48d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000ef30 tx=0x7f7f8000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.731+0000 7f7f7e7fc700 1 -- 172.21.15.77:0/3595868486 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f800092e0 con 0x7f7f88107210 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.731+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/3595868486 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f881a4fe0 con 0x7f7f88107210 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.731+0000 7f7f7e7fc700 1 -- 172.21.15.77:0/3595868486 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f80004bb0 con 0x7f7f88107210 2024-03-16T10:15:41.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.731+0000 7f7f7e7fc700 1 -- 172.21.15.77:0/3595868486 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f80004280 con 0x7f7f88107210 2024-03-16T10:15:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.731+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/3595868486 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f881a54f0 con 0x7f7f88107210 2024-03-16T10:15:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.732+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/3595868486 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7f880629e0 con 0x7f7f88107210 2024-03-16T10:15:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.733+0000 7f7f7e7fc700 1 -- 172.21.15.77:0/3595868486 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7f8001b030 con 0x7f7f88107210 2024-03-16T10:15:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.734+0000 7f7f7e7fc700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7f740a2920 0x7f7f740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.734+0000 7f7f8d7c6700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7f740a2920 0x7f7f740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.734+0000 7f7f7e7fc700 1 -- 172.21.15.77:0/3595868486 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7f800ced80 con 0x7f7f88107210 2024-03-16T10:15:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.734+0000 7f7f7e7fc700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423] conn(0x7f7f740ac100 0x7f7f740ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.734+0000 7f7f7e7fc700 1 -- 172.21.15.77:0/3595868486 --> [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f740aebd0 con 0x7f7f740ac100 2024-03-16T10:15:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.734+0000 7f7f7e7fc700 1 -- 172.21.15.77:0/3595868486 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f7f800cf260 con 0x7f7f88107210 2024-03-16T10:15:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.734+0000 7f7f8cfc5700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423] conn(0x7f7f740ac100 0x7f7f740ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.734+0000 7f7f8d7c6700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7f740a2920 0x7f7f740a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7f7800b2e0 tx=0x7f7f78005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.735+0000 7f7f8cfc5700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423] conn(0x7f7f740ac100 0x7f7f740ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:41.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.735+0000 7f7f7e7fc700 1 -- 172.21.15.77:0/3595868486 <== osd.3 v2:172.21.15.77:6810/2887114423 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7f740aebd0 con 0x7f7f740ac100 2024-03-16T10:15:41.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.753+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/3595868486 --> [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7f88062e60 con 0x7f7f740ac100 2024-03-16T10:15:41.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.753+0000 7f7f7e7fc700 1 -- 172.21.15.77:0/3595868486 <== osd.3 v2:172.21.15.77:6810/2887114423 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7f88062e60 con 0x7f7f740ac100 2024-03-16T10:15:41.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.754+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423] conn(0x7f7f740ac100 msgr2=0x7f7f740ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:41.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.754+0000 7f7f8fa2a700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423] conn(0x7f7f740ac100 0x7f7f740ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:41.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.754+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7f740a2920 msgr2=0x7f7f740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:41.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.754+0000 7f7f8fa2a700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7f740a2920 0x7f7f740a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7f7800b2e0 tx=0x7f7f78005960 comp rx=0 tx=0).stop 2024-03-16T10:15:41.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.754+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f88107210 msgr2=0x7f7f881a48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:41.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.754+0000 7f7f8fa2a700 1 --2- 172.21.15.77:0/3595868486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f88107210 0x7f7f881a48d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000ef30 tx=0x7f7f8000c3a0 comp rx=0 tx=0).stop 2024-03-16T10:15:41.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.754+0000 7f7f8dfc7700 1 -- 172.21.15.77:0/3595868486 reap_dead start 2024-03-16T10:15:41.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.754+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/3595868486 shutdown_connections 2024-03-16T10:15:41.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.754+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/3595868486 >> 172.21.15.77:0/3595868486 conn(0x7f7f88078100 msgr2=0x7f7f880ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:41.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.754+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/3595868486 shutdown_connections 2024-03-16T10:15:41.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:41.754+0000 7f7f8fa2a700 1 -- 172.21.15.77:0/3595868486 wait complete. 2024-03-16T10:15:41.944 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:41 smithi077 ceph-mon[32400]: pgmap v306: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T10:15:42.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:41 smithi192 ceph-mon[36354]: pgmap v306: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T10:15:42.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:41 smithi193 ceph-mon[36311]: pgmap v306: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T10:15:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.508+0000 7f76bc390700 1 -- 172.21.15.77:0/1271852375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b4102320 msgr2=0x7f76b41027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.508+0000 7f76bc390700 1 --2- 172.21.15.77:0/1271852375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b4102320 0x7f76b41027a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f76a40099c0 tx=0x7f76a4009cd0 comp rx=0 tx=0).stop 2024-03-16T10:15:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.508+0000 7f76bc390700 1 -- 172.21.15.77:0/1271852375 shutdown_connections 2024-03-16T10:15:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.508+0000 7f76bc390700 1 --2- 172.21.15.77:0/1271852375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76b4102ce0 0x7f76b410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.508+0000 7f76bc390700 1 --2- 172.21.15.77:0/1271852375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76b4108300 0x7f76b41086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.508+0000 7f76bc390700 1 --2- 172.21.15.77:0/1271852375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b4102320 0x7f76b41027a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.508+0000 7f76bc390700 1 -- 172.21.15.77:0/1271852375 >> 172.21.15.77:0/1271852375 conn(0x7f76b4079400 msgr2=0x7f76b4079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.508+0000 7f76bc390700 1 -- 172.21.15.77:0/1271852375 shutdown_connections 2024-03-16T10:15:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.508+0000 7f76bc390700 1 -- 172.21.15.77:0/1271852375 wait complete. 2024-03-16T10:15:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76bc390700 1 Processor -- start 2024-03-16T10:15:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76bc390700 1 -- start start 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76bc390700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76b4102320 0x7f76b419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76bc390700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76b4102ce0 0x7f76b419ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76bc390700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b4108300 0x7f76b41988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76bc390700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f76b41125b0 con 0x7f76b4108300 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76bc390700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f76b4112430 con 0x7f76b4102320 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76bc390700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76b4112730 con 0x7f76b4102ce0 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76ba92d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b4108300 0x7f76b41988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76ba92d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b4108300 0x7f76b41988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59282/0 (socket says 172.21.15.77:59282) 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76ba92d700 1 -- 172.21.15.77:0/86629503 learned_addr learned my addr 172.21.15.77:0/86629503 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76ba12c700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76b4102320 0x7f76b419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76b992b700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76b4102ce0 0x7f76b419ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76ba92d700 1 -- 172.21.15.77:0/86629503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76b4102ce0 msgr2=0x7f76b419ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76ba92d700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76b4102ce0 0x7f76b419ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:42.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76ba92d700 1 -- 172.21.15.77:0/86629503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76b4102320 msgr2=0x7f76b419e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:42.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76ba92d700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76b4102320 0x7f76b419e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:42.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76ba92d700 1 -- 172.21.15.77:0/86629503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b4199050 con 0x7f76b4108300 2024-03-16T10:15:42.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76ba12c700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76b4102320 0x7f76b419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:42.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.509+0000 7f76ba92d700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b4108300 0x7f76b41988f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00ef30 tx=0x7f76ac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:42.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.510+0000 7f76ab7fe700 1 -- 172.21.15.77:0/86629503 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac0092e0 con 0x7f76b4108300 2024-03-16T10:15:42.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.510+0000 7f76ab7fe700 1 -- 172.21.15.77:0/86629503 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ac004bb0 con 0x7f76b4108300 2024-03-16T10:15:42.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.510+0000 7f76ab7fe700 1 -- 172.21.15.77:0/86629503 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac004280 con 0x7f76b4108300 2024-03-16T10:15:42.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.510+0000 7f76bc390700 1 -- 172.21.15.77:0/86629503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76b41992e0 con 0x7f76b4108300 2024-03-16T10:15:42.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.510+0000 7f76bc390700 1 -- 172.21.15.77:0/86629503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76b41a55b0 con 0x7f76b4108300 2024-03-16T10:15:42.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.510+0000 7f76bc390700 1 -- 172.21.15.77:0/86629503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f76b4062a20 con 0x7f76b4108300 2024-03-16T10:15:42.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.512+0000 7f76ab7fe700 1 -- 172.21.15.77:0/86629503 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f76ac01b030 con 0x7f76b4108300 2024-03-16T10:15:42.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.513+0000 7f76ab7fe700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76a00a29a0 0x7f76a00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:42.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.513+0000 7f76ba12c700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76a00a29a0 0x7f76a00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:42.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.513+0000 7f76ab7fe700 1 -- 172.21.15.77:0/86629503 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f76ac0cee10 con 0x7f76b4108300 2024-03-16T10:15:42.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.513+0000 7f76ab7fe700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377] conn(0x7f76a00ac180 0x7f76a00ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:42.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.513+0000 7f76ab7fe700 1 -- 172.21.15.77:0/86629503 --> [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f76ac01fe50 con 0x7f76a00ac180 2024-03-16T10:15:42.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.513+0000 7f76ab7fe700 1 -- 172.21.15.77:0/86629503 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f76ac0cf2f0 con 0x7f76b4108300 2024-03-16T10:15:42.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.513+0000 7f76ba12c700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76a00a29a0 0x7f76a00a4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f76b4100950 tx=0x7f76b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:42.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.513+0000 7f76b992b700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377] conn(0x7f76a00ac180 0x7f76a00ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:42.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.514+0000 7f76b992b700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377] conn(0x7f76a00ac180 0x7f76a00ae5a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:42.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.515+0000 7f76ab7fe700 1 -- 172.21.15.77:0/86629503 <== osd.10 v2:172.21.15.192:6824/946112377 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f76ac01fe50 con 0x7f76a00ac180 2024-03-16T10:15:42.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.534+0000 7f76bc390700 1 -- 172.21.15.77:0/86629503 --> [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f76b4062ef0 con 0x7f76a00ac180 2024-03-16T10:15:42.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.535+0000 7f76ab7fe700 1 -- 172.21.15.77:0/86629503 <== osd.10 v2:172.21.15.192:6824/946112377 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f76b4062ef0 con 0x7f76a00ac180 2024-03-16T10:15:42.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.535+0000 7f76bc390700 1 -- 172.21.15.77:0/86629503 >> [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377] conn(0x7f76a00ac180 msgr2=0x7f76a00ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:42.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.535+0000 7f76bc390700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377] conn(0x7f76a00ac180 0x7f76a00ae5a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:42.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.535+0000 7f76bc390700 1 -- 172.21.15.77:0/86629503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76a00a29a0 msgr2=0x7f76a00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:42.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.535+0000 7f76bc390700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76a00a29a0 0x7f76a00a4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f76b4100950 tx=0x7f76b0009260 comp rx=0 tx=0).stop 2024-03-16T10:15:42.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.535+0000 7f76bc390700 1 -- 172.21.15.77:0/86629503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b4108300 msgr2=0x7f76b41988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:42.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.535+0000 7f76bc390700 1 --2- 172.21.15.77:0/86629503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76b4108300 0x7f76b41988f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00ef30 tx=0x7f76ac00c3a0 comp rx=0 tx=0).stop 2024-03-16T10:15:42.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.535+0000 7f76ba92d700 1 -- 172.21.15.77:0/86629503 reap_dead start 2024-03-16T10:15:42.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.535+0000 7f76bc390700 1 -- 172.21.15.77:0/86629503 shutdown_connections 2024-03-16T10:15:42.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.535+0000 7f76bc390700 1 -- 172.21.15.77:0/86629503 >> 172.21.15.77:0/86629503 conn(0x7f76b4079400 msgr2=0x7f76b4100ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:42.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.535+0000 7f76bc390700 1 -- 172.21.15.77:0/86629503 shutdown_connections 2024-03-16T10:15:42.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:42.535+0000 7f76bc390700 1 -- 172.21.15.77:0/86629503 wait complete. 2024-03-16T10:15:43.068 INFO:teuthology.orchestra.run.smithi077.stdout:107374182461 2024-03-16T10:15:43.069 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-16T10:15:43.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:43 smithi077 ceph-mon[32400]: pgmap v307: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-16T10:15:44.056 INFO:teuthology.orchestra.run.smithi077.stdout:343597383699 2024-03-16T10:15:44.056 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-16T10:15:44.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:43 smithi192 ceph-mon[36354]: pgmap v307: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-16T10:15:44.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:43 smithi193 ceph-mon[36311]: pgmap v307: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-16T10:15:44.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.393+0000 7fb818420700 1 -- 172.21.15.77:0/1372579709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8101008c0 msgr2=0x7fb810100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:44.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.393+0000 7fb818420700 1 --2- 172.21.15.77:0/1372579709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8101008c0 0x7fb810100d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb800009990 tx=0x7fb800009ca0 comp rx=0 tx=0).stop 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.393+0000 7fb818420700 1 -- 172.21.15.77:0/1372579709 shutdown_connections 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.393+0000 7fb818420700 1 --2- 172.21.15.77:0/1372579709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb810101280 0x7fb81010f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.393+0000 7fb818420700 1 --2- 172.21.15.77:0/1372579709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb810107340 0x7fb810107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.393+0000 7fb818420700 1 --2- 172.21.15.77:0/1372579709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8101008c0 0x7fb810100d40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.393+0000 7fb818420700 1 -- 172.21.15.77:0/1372579709 >> 172.21.15.77:0/1372579709 conn(0x7fb8100777d0 msgr2=0x7fb8100febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.393+0000 7fb818420700 1 -- 172.21.15.77:0/1372579709 shutdown_connections 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.393+0000 7fb818420700 1 -- 172.21.15.77:0/1372579709 wait complete. 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb818420700 1 Processor -- start 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb818420700 1 -- start start 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb818420700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8101008c0 0x7fb810101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb818420700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb810101280 0x7fb810101fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb818420700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb810107340 0x7fb810102520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb818420700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb810112720 con 0x7fb810101280 2024-03-16T10:15:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb818420700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb8101125a0 con 0x7fb810107340 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb818420700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb8101128a0 con 0x7fb8101008c0 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb8159bb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb810101280 0x7fb810101fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb8161bc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8101008c0 0x7fb810101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb8159bb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb810101280 0x7fb810101fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59302/0 (socket says 172.21.15.77:59302) 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb8159bb700 1 -- 172.21.15.77:0/3651625406 learned_addr learned my addr 172.21.15.77:0/3651625406 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb8169bd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb810107340 0x7fb810102520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb8159bb700 1 -- 172.21.15.77:0/3651625406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8101008c0 msgr2=0x7fb810101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb8159bb700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8101008c0 0x7fb810101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb8159bb700 1 -- 172.21.15.77:0/3651625406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb810107340 msgr2=0x7fb810102520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb8159bb700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb810107340 0x7fb810102520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb8159bb700 1 -- 172.21.15.77:0/3651625406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8101a70e0 con 0x7fb810101280 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb8161bc700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8101008c0 0x7fb810101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.394+0000 7fb8169bd700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb810107340 0x7fb810102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.395+0000 7fb8159bb700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb810101280 0x7fb810101fe0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb800000c00 tx=0x7fb80000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.395+0000 7fb8077fe700 1 -- 172.21.15.77:0/3651625406 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb800027070 con 0x7fb810101280 2024-03-16T10:15:44.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.395+0000 7fb818420700 1 -- 172.21.15.77:0/3651625406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8101a7370 con 0x7fb810101280 2024-03-16T10:15:44.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.395+0000 7fb8077fe700 1 -- 172.21.15.77:0/3651625406 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb800003d70 con 0x7fb810101280 2024-03-16T10:15:44.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.395+0000 7fb8077fe700 1 -- 172.21.15.77:0/3651625406 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80000f460 con 0x7fb810101280 2024-03-16T10:15:44.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.395+0000 7fb818420700 1 -- 172.21.15.77:0/3651625406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8101a77c0 con 0x7fb810101280 2024-03-16T10:15:44.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.396+0000 7fb818420700 1 -- 172.21.15.77:0/3651625406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb8100629e0 con 0x7fb810101280 2024-03-16T10:15:44.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.397+0000 7fb8077fe700 1 -- 172.21.15.77:0/3651625406 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb80002a070 con 0x7fb810101280 2024-03-16T10:15:44.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.397+0000 7fb8077fe700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb7fc0a2920 0x7fb7fc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:44.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.398+0000 7fb8161bc700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb7fc0a2920 0x7fb7fc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:44.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.398+0000 7fb8077fe700 1 -- 172.21.15.77:0/3651625406 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb8000cfde0 con 0x7fb810101280 2024-03-16T10:15:44.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.398+0000 7fb8077fe700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947] conn(0x7fb7fc0ac100 0x7fb7fc0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:44.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.398+0000 7fb8077fe700 1 -- 172.21.15.77:0/3651625406 --> [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb8000056d0 con 0x7fb7fc0ac100 2024-03-16T10:15:44.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.398+0000 7fb8077fe700 1 -- 172.21.15.77:0/3651625406 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fb8000d02c0 con 0x7fb810101280 2024-03-16T10:15:44.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.398+0000 7fb8169bd700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947] conn(0x7fb7fc0ac100 0x7fb7fc0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:44.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.398+0000 7fb8161bc700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb7fc0a2920 0x7fb7fc0a4de0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009fd0 tx=0x7fb80c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:44.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.398+0000 7fb8169bd700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947] conn(0x7fb7fc0ac100 0x7fb7fc0ae520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:44.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.399+0000 7fb8077fe700 1 -- 172.21.15.77:0/3651625406 <== osd.2 v2:172.21.15.193:6800/2172182947 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb8000056d0 con 0x7fb7fc0ac100 2024-03-16T10:15:44.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.419+0000 7fb818420700 1 -- 172.21.15.77:0/3651625406 --> [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb810068f90 con 0x7fb7fc0ac100 2024-03-16T10:15:44.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.419+0000 7fb8077fe700 1 -- 172.21.15.77:0/3651625406 <== osd.2 v2:172.21.15.193:6800/2172182947 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb810068f90 con 0x7fb7fc0ac100 2024-03-16T10:15:44.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.420+0000 7fb818420700 1 -- 172.21.15.77:0/3651625406 >> [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947] conn(0x7fb7fc0ac100 msgr2=0x7fb7fc0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:44.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.420+0000 7fb818420700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947] conn(0x7fb7fc0ac100 0x7fb7fc0ae520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:44.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.420+0000 7fb818420700 1 -- 172.21.15.77:0/3651625406 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb7fc0a2920 msgr2=0x7fb7fc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:44.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.420+0000 7fb818420700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb7fc0a2920 0x7fb7fc0a4de0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009fd0 tx=0x7fb80c009260 comp rx=0 tx=0).stop 2024-03-16T10:15:44.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.420+0000 7fb818420700 1 -- 172.21.15.77:0/3651625406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb810101280 msgr2=0x7fb810101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:44.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.420+0000 7fb818420700 1 --2- 172.21.15.77:0/3651625406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb810101280 0x7fb810101fe0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb800000c00 tx=0x7fb80000bab0 comp rx=0 tx=0).stop 2024-03-16T10:15:44.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.420+0000 7fb8169bd700 1 -- 172.21.15.77:0/3651625406 reap_dead start 2024-03-16T10:15:44.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.420+0000 7fb818420700 1 -- 172.21.15.77:0/3651625406 shutdown_connections 2024-03-16T10:15:44.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.420+0000 7fb818420700 1 -- 172.21.15.77:0/3651625406 >> 172.21.15.77:0/3651625406 conn(0x7fb8100777d0 msgr2=0x7fb810105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:44.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.420+0000 7fb818420700 1 -- 172.21.15.77:0/3651625406 shutdown_connections 2024-03-16T10:15:44.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:44.420+0000 7fb818420700 1 -- 172.21.15.77:0/3651625406 wait complete. 2024-03-16T10:15:45.087 INFO:teuthology.orchestra.run.smithi077.stdout:77309411396 2024-03-16T10:15:45.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-16T10:15:45.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.280+0000 7f8b5647a700 1 -- 172.21.15.77:0/980354299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50108320 msgr2=0x7f8b50108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:45.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.280+0000 7f8b5647a700 1 --2- 172.21.15.77:0/980354299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50108320 0x7f8b50108700 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8b400099c0 tx=0x7f8b40009cd0 comp rx=0 tx=0).stop 2024-03-16T10:15:45.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.280+0000 7f8b5647a700 1 -- 172.21.15.77:0/980354299 shutdown_connections 2024-03-16T10:15:45.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.280+0000 7f8b5647a700 1 --2- 172.21.15.77:0/980354299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50108320 0x7f8b50108700 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:45.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.280+0000 7f8b5647a700 1 --2- 172.21.15.77:0/980354299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b5006a920 0x7f8b5010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:45.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.280+0000 7f8b5647a700 1 --2- 172.21.15.77:0/980354299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b50069f80 0x7f8b5006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:45.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.280+0000 7f8b5647a700 1 -- 172.21.15.77:0/980354299 >> 172.21.15.77:0/980354299 conn(0x7f8b50078100 msgr2=0x7f8b50078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:45.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.280+0000 7f8b5647a700 1 -- 172.21.15.77:0/980354299 shutdown_connections 2024-03-16T10:15:45.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.281+0000 7f8b5647a700 1 -- 172.21.15.77:0/980354299 wait complete. 2024-03-16T10:15:45.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.281+0000 7f8b5647a700 1 Processor -- start 2024-03-16T10:15:45.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.281+0000 7f8b5647a700 1 -- start start 2024-03-16T10:15:45.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.282+0000 7f8b5647a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b50069f80 0x7f8b50102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:45.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.282+0000 7f8b5647a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b5006a920 0x7f8b501012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:45.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.282+0000 7f8b5647a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50108320 0x7f8b50101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:45.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.282+0000 7f8b5647a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8b50112580 con 0x7f8b50108320 2024-03-16T10:15:45.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.282+0000 7f8b5647a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8b50112400 con 0x7f8b5006a920 2024-03-16T10:15:45.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.282+0000 7f8b5647a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8b50112700 con 0x7f8b50069f80 2024-03-16T10:15:45.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.282+0000 7f8b4f7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b5006a920 0x7f8b501012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:45.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.282+0000 7f8b4f7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b5006a920 0x7f8b501012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:52742/0 (socket says 172.21.15.77:52742) 2024-03-16T10:15:45.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.282+0000 7f8b4f7fe700 1 -- 172.21.15.77:0/3590471726 learned_addr learned my addr 172.21.15.77:0/3590471726 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:45.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.282+0000 7f8b54a17700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50108320 0x7f8b50101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:45.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.283+0000 7f8b4ffff700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b50069f80 0x7f8b50102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:45.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.283+0000 7f8b54a17700 1 -- 172.21.15.77:0/3590471726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b50069f80 msgr2=0x7f8b50102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:45.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.283+0000 7f8b54a17700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b50069f80 0x7f8b50102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:45.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.283+0000 7f8b54a17700 1 -- 172.21.15.77:0/3590471726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b5006a920 msgr2=0x7f8b501012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:45.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.283+0000 7f8b54a17700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b5006a920 0x7f8b501012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:45.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.283+0000 7f8b54a17700 1 -- 172.21.15.77:0/3590471726 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b501ab440 con 0x7f8b50108320 2024-03-16T10:15:45.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.283+0000 7f8b4f7fe700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b5006a920 0x7f8b501012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:45.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.283+0000 7f8b54a17700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50108320 0x7f8b50101900 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800efc0 tx=0x7f8b4800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:45.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.283+0000 7f8b4d7fa700 1 -- 172.21.15.77:0/3590471726 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b480092e0 con 0x7f8b50108320 2024-03-16T10:15:45.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.284+0000 7f8b4d7fa700 1 -- 172.21.15.77:0/3590471726 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b48004bb0 con 0x7f8b50108320 2024-03-16T10:15:45.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.284+0000 7f8b4d7fa700 1 -- 172.21.15.77:0/3590471726 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b48004280 con 0x7f8b50108320 2024-03-16T10:15:45.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.284+0000 7f8b5647a700 1 -- 172.21.15.77:0/3590471726 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b501ab670 con 0x7f8b50108320 2024-03-16T10:15:45.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.284+0000 7f8b5647a700 1 -- 172.21.15.77:0/3590471726 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b501abb30 con 0x7f8b50108320 2024-03-16T10:15:45.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.285+0000 7f8b5647a700 1 -- 172.21.15.77:0/3590471726 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8b500629e0 con 0x7f8b50108320 2024-03-16T10:15:45.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.286+0000 7f8b4d7fa700 1 -- 172.21.15.77:0/3590471726 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8b4801b030 con 0x7f8b50108320 2024-03-16T10:15:45.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.287+0000 7f8b4d7fa700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8b380a2920 0x7f8b380a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:45.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.287+0000 7f8b4ffff700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8b380a2920 0x7f8b380a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:45.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.287+0000 7f8b4d7fa700 1 -- 172.21.15.77:0/3590471726 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f8b480ceee0 con 0x7f8b50108320 2024-03-16T10:15:45.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.287+0000 7f8b4d7fa700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736] conn(0x7f8b380ac100 0x7f8b380ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:45.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.287+0000 7f8b4d7fa700 1 -- 172.21.15.77:0/3590471726 --> [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8b4801fe50 con 0x7f8b380ac100 2024-03-16T10:15:45.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.287+0000 7f8b4d7fa700 1 -- 172.21.15.77:0/3590471726 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f8b480cf3c0 con 0x7f8b50108320 2024-03-16T10:15:45.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.287+0000 7f8b4f7fe700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736] conn(0x7f8b380ac100 0x7f8b380ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:45.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.287+0000 7f8b4ffff700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8b380a2920 0x7f8b380a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8b40000c00 tx=0x7f8b4001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:45.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.288+0000 7f8b4f7fe700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736] conn(0x7f8b380ac100 0x7f8b380ae520 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:45.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.289+0000 7f8b4d7fa700 1 -- 172.21.15.77:0/3590471726 <== osd.6 v2:172.21.15.77:6818/668716736 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8b4801fe50 con 0x7f8b380ac100 2024-03-16T10:15:45.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.317+0000 7f8b5647a700 1 -- 172.21.15.77:0/3590471726 --> [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8b50062e60 con 0x7f8b380ac100 2024-03-16T10:15:45.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.317+0000 7f8b4d7fa700 1 -- 172.21.15.77:0/3590471726 <== osd.6 v2:172.21.15.77:6818/668716736 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8b50062e60 con 0x7f8b380ac100 2024-03-16T10:15:45.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.317+0000 7f8b5647a700 1 -- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736] conn(0x7f8b380ac100 msgr2=0x7f8b380ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:45.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.317+0000 7f8b5647a700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736] conn(0x7f8b380ac100 0x7f8b380ae520 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:45.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.318+0000 7f8b5647a700 1 -- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8b380a2920 msgr2=0x7f8b380a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:45.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.318+0000 7f8b5647a700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8b380a2920 0x7f8b380a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8b40000c00 tx=0x7f8b4001a040 comp rx=0 tx=0).stop 2024-03-16T10:15:45.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.318+0000 7f8b5647a700 1 -- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50108320 msgr2=0x7f8b50101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:45.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.318+0000 7f8b5647a700 1 --2- 172.21.15.77:0/3590471726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8b50108320 0x7f8b50101900 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800efc0 tx=0x7f8b4800c3a0 comp rx=0 tx=0).stop 2024-03-16T10:15:45.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.318+0000 7f8b54a17700 1 -- 172.21.15.77:0/3590471726 reap_dead start 2024-03-16T10:15:45.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.318+0000 7f8b5647a700 1 -- 172.21.15.77:0/3590471726 shutdown_connections 2024-03-16T10:15:45.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.318+0000 7f8b5647a700 1 -- 172.21.15.77:0/3590471726 >> 172.21.15.77:0/3590471726 conn(0x7f8b50078100 msgr2=0x7f8b500ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:45.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.318+0000 7f8b5647a700 1 -- 172.21.15.77:0/3590471726 shutdown_connections 2024-03-16T10:15:45.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:45.318+0000 7f8b5647a700 1 -- 172.21.15.77:0/3590471726 wait complete. 2024-03-16T10:15:45.940 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:45 smithi077 ceph-mon[32400]: pgmap v308: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-16T10:15:46.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:45 smithi192 ceph-mon[36354]: pgmap v308: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-16T10:15:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:45 smithi193 ceph-mon[36311]: pgmap v308: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.495+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/208879886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24108300 msgr2=0x7f5f241086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.495+0000 7f5f2bbf0700 1 --2- 172.21.15.77:0/208879886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24108300 0x7f5f241086e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5f200099c0 tx=0x7f5f20009cd0 comp rx=0 tx=0).stop 2024-03-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.495+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/208879886 shutdown_connections 2024-03-16T10:15:46.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.495+0000 7f5f2bbf0700 1 --2- 172.21.15.77:0/208879886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24108300 0x7f5f241086e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.495+0000 7f5f2bbf0700 1 --2- 172.21.15.77:0/208879886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f24102ce0 0x7f5f2410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.495+0000 7f5f2bbf0700 1 --2- 172.21.15.77:0/208879886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f24102320 0x7f5f241027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.495+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/208879886 >> 172.21.15.77:0/208879886 conn(0x7f5f24079400 msgr2=0x7f5f24079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.495+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/208879886 shutdown_connections 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.495+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/208879886 wait complete. 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.495+0000 7f5f2bbf0700 1 Processor -- start 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.495+0000 7f5f2bbf0700 1 -- start start 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2bbf0700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f24102320 0x7f5f2419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2bbf0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24102ce0 0x7f5f2419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2bbf0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f24108300 0x7f5f241987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2bbf0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241125b0 con 0x7f5f24102ce0 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2bbf0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f5f24112430 con 0x7f5f24108300 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2bbf0700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5f24112730 con 0x7f5f24102320 2024-03-16T10:15:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2918b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24102ce0 0x7f5f2419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2918b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24102ce0 0x7f5f2419eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59346/0 (socket says 172.21.15.77:59346) 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2918b700 1 -- 172.21.15.77:0/10820288 learned_addr learned my addr 172.21.15.77:0/10820288 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2998c700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f24102320 0x7f5f2419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2a18d700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f24108300 0x7f5f241987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2918b700 1 -- 172.21.15.77:0/10820288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f24102320 msgr2=0x7f5f2419e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2918b700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f24102320 0x7f5f2419e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2918b700 1 -- 172.21.15.77:0/10820288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f24108300 msgr2=0x7f5f241987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2918b700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f24108300 0x7f5f241987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2918b700 1 -- 172.21.15.77:0/10820288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f24198f90 con 0x7f5f24102ce0 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2a18d700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f24108300 0x7f5f241987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2918b700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24102ce0 0x7f5f2419eb70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5f1400eb30 tx=0x7f5f1400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f1affd700 1 -- 172.21.15.77:0/10820288 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1400cbd0 con 0x7f5f24102ce0 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f1affd700 1 -- 172.21.15.77:0/10820288 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f14004500 con 0x7f5f24102ce0 2024-03-16T10:15:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.496+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/10820288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f24199280 con 0x7f5f24102ce0 2024-03-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.497+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/10820288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f241a5490 con 0x7f5f24102ce0 2024-03-16T10:15:46.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.497+0000 7f5f1affd700 1 -- 172.21.15.77:0/10820288 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f14010430 con 0x7f5f24102ce0 2024-03-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.497+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/10820288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5f24062a20 con 0x7f5f24102ce0 2024-03-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.499+0000 7f5f1affd700 1 -- 172.21.15.77:0/10820288 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5f14003f00 con 0x7f5f24102ce0 2024-03-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.499+0000 7f5f1affd700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5f100a29a0 0x7f5f100a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.499+0000 7f5f2998c700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5f100a29a0 0x7f5f100a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.499+0000 7f5f1affd700 1 -- 172.21.15.77:0/10820288 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f5f140cef80 con 0x7f5f24102ce0 2024-03-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.500+0000 7f5f1affd700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086] conn(0x7f5f100ac180 0x7f5f100ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.500+0000 7f5f1affd700 1 -- 172.21.15.77:0/10820288 --> [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f14012ec0 con 0x7f5f100ac180 2024-03-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.500+0000 7f5f1affd700 1 -- 172.21.15.77:0/10820288 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f5f140cf460 con 0x7f5f24102ce0 2024-03-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.500+0000 7f5f2a18d700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086] conn(0x7f5f100ac180 0x7f5f100ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.500+0000 7f5f2998c700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5f100a29a0 0x7f5f100a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5f20000c00 tx=0x7f5f20005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.500+0000 7f5f2a18d700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086] conn(0x7f5f100ac180 0x7f5f100ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:46.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.501+0000 7f5f1affd700 1 -- 172.21.15.77:0/10820288 <== osd.9 v2:172.21.15.77:6826/2903071086 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f14012ec0 con 0x7f5f100ac180 2024-03-16T10:15:46.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.517+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/10820288 --> [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5f24062e50 con 0x7f5f100ac180 2024-03-16T10:15:46.518 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.517+0000 7f5f1affd700 1 -- 172.21.15.77:0/10820288 <== osd.9 v2:172.21.15.77:6826/2903071086 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5f24062e50 con 0x7f5f100ac180 2024-03-16T10:15:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.517+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086] conn(0x7f5f100ac180 msgr2=0x7f5f100ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.517+0000 7f5f2bbf0700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086] conn(0x7f5f100ac180 0x7f5f100ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.517+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5f100a29a0 msgr2=0x7f5f100a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.517+0000 7f5f2bbf0700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5f100a29a0 0x7f5f100a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5f20000c00 tx=0x7f5f20005fb0 comp rx=0 tx=0).stop 2024-03-16T10:15:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.517+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24102ce0 msgr2=0x7f5f2419eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:46.519 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.517+0000 7f5f2bbf0700 1 --2- 172.21.15.77:0/10820288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f24102ce0 0x7f5f2419eb70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5f1400eb30 tx=0x7f5f1400ee40 comp rx=0 tx=0).stop 2024-03-16T10:15:46.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.518+0000 7f5f2a18d700 1 -- 172.21.15.77:0/10820288 reap_dead start 2024-03-16T10:15:46.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.518+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/10820288 shutdown_connections 2024-03-16T10:15:46.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.518+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/10820288 >> 172.21.15.77:0/10820288 conn(0x7f5f24079400 msgr2=0x7f5f24100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:46.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.518+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/10820288 shutdown_connections 2024-03-16T10:15:46.520 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.518+0000 7f5f2bbf0700 1 -- 172.21.15.77:0/10820288 wait complete. 2024-03-16T10:15:47.130 INFO:teuthology.orchestra.run.smithi077.stdout:197568495664 2024-03-16T10:15:47.130 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-16T10:15:47.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.296+0000 7fe748f30700 1 -- 172.21.15.77:0/194941252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe744107340 msgr2=0x7fe744107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:47.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.296+0000 7fe748f30700 1 --2- 172.21.15.77:0/194941252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe744107340 0x7fe744107720 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe734009990 tx=0x7fe734009ca0 comp rx=0 tx=0).stop 2024-03-16T10:15:47.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.296+0000 7fe748f30700 1 -- 172.21.15.77:0/194941252 shutdown_connections 2024-03-16T10:15:47.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.296+0000 7fe748f30700 1 --2- 172.21.15.77:0/194941252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe744107340 0x7fe744107720 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:47.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.296+0000 7fe748f30700 1 --2- 172.21.15.77:0/194941252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe744101280 0x7fe74410f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:47.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.296+0000 7fe748f30700 1 --2- 172.21.15.77:0/194941252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7441008c0 0x7fe744100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:47.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.296+0000 7fe748f30700 1 -- 172.21.15.77:0/194941252 >> 172.21.15.77:0/194941252 conn(0x7fe7440777d0 msgr2=0x7fe7440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.296+0000 7fe748f30700 1 -- 172.21.15.77:0/194941252 shutdown_connections 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.296+0000 7fe748f30700 1 -- 172.21.15.77:0/194941252 wait complete. 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe748f30700 1 Processor -- start 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe748f30700 1 -- start start 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe748f30700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7441008c0 0x7fe74419e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe748f30700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe744101280 0x7fe74419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe748f30700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe744107340 0x7fe7441989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe748f30700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe744112700 con 0x7fe744101280 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe748f30700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe744112580 con 0x7fe744107340 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe748f30700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe744112880 con 0x7fe7441008c0 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe741d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe744101280 0x7fe74419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe741d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe744101280 0x7fe74419ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59372/0 (socket says 172.21.15.77:59372) 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe741d9b700 1 -- 172.21.15.77:0/4279787242 learned_addr learned my addr 172.21.15.77:0/4279787242 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:47.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe74259c700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7441008c0 0x7fe74419e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe742d9d700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe744107340 0x7fe7441989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe74259c700 1 -- 172.21.15.77:0/4279787242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe744107340 msgr2=0x7fe7441989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe74259c700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe744107340 0x7fe7441989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.297+0000 7fe74259c700 1 -- 172.21.15.77:0/4279787242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe744101280 msgr2=0x7fe74419ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.298+0000 7fe74259c700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe744101280 0x7fe74419ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.298+0000 7fe74259c700 1 -- 172.21.15.77:0/4279787242 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe744199120 con 0x7fe7441008c0 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.298+0000 7fe742d9d700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe744107340 0x7fe7441989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.298+0000 7fe741d9b700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe744101280 0x7fe74419ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.298+0000 7fe74259c700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7441008c0 0x7fe74419e7b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe7340059d0 tx=0x7fe73400bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.298+0000 7fe7337fe700 1 -- 172.21.15.77:0/4279787242 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734027070 con 0x7fe7441008c0 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.298+0000 7fe7337fe700 1 -- 172.21.15.77:0/4279787242 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe734003d70 con 0x7fe7441008c0 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.298+0000 7fe748f30700 1 -- 172.21.15.77:0/4279787242 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7441993b0 con 0x7fe7441008c0 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.298+0000 7fe7337fe700 1 -- 172.21.15.77:0/4279787242 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734022e30 con 0x7fe7441008c0 2024-03-16T10:15:47.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.298+0000 7fe748f30700 1 -- 172.21.15.77:0/4279787242 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7441019d0 con 0x7fe7441008c0 2024-03-16T10:15:47.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.299+0000 7fe748f30700 1 -- 172.21.15.77:0/4279787242 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe7440629e0 con 0x7fe7441008c0 2024-03-16T10:15:47.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.300+0000 7fe7337fe700 1 -- 172.21.15.77:0/4279787242 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe73402a070 con 0x7fe7441008c0 2024-03-16T10:15:47.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.301+0000 7fe7337fe700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe72c0a29a0 0x7fe72c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:47.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.301+0000 7fe741d9b700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe72c0a29a0 0x7fe72c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:47.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.301+0000 7fe7337fe700 1 -- 172.21.15.77:0/4279787242 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fe7340d0360 con 0x7fe7441008c0 2024-03-16T10:15:47.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.301+0000 7fe7337fe700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545] conn(0x7fe72c0ac180 0x7fe72c0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:47.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.301+0000 7fe7337fe700 1 -- 172.21.15.77:0/4279787242 --> [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe73401de50 con 0x7fe72c0ac180 2024-03-16T10:15:47.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.301+0000 7fe7337fe700 1 -- 172.21.15.77:0/4279787242 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fe7340d0710 con 0x7fe7441008c0 2024-03-16T10:15:47.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.302+0000 7fe741d9b700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe72c0a29a0 0x7fe72c0a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe7440ffe30 tx=0x7fe738009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:47.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.302+0000 7fe742d9d700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545] conn(0x7fe72c0ac180 0x7fe72c0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:47.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.302+0000 7fe742d9d700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545] conn(0x7fe72c0ac180 0x7fe72c0ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:47.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.303+0000 7fe7337fe700 1 -- 172.21.15.77:0/4279787242 <== osd.1 v2:172.21.15.192:6800/4276469545 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe73401de50 con 0x7fe72c0ac180 2024-03-16T10:15:47.321 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:47.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.323+0000 7fe748f30700 1 -- 172.21.15.77:0/4279787242 --> [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe744062de0 con 0x7fe72c0ac180 2024-03-16T10:15:47.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.323+0000 7fe7337fe700 1 -- 172.21.15.77:0/4279787242 <== osd.1 v2:172.21.15.192:6800/4276469545 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe744062de0 con 0x7fe72c0ac180 2024-03-16T10:15:47.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.324+0000 7fe748f30700 1 -- 172.21.15.77:0/4279787242 >> [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545] conn(0x7fe72c0ac180 msgr2=0x7fe72c0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:47.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.324+0000 7fe748f30700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545] conn(0x7fe72c0ac180 0x7fe72c0ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:47.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.324+0000 7fe748f30700 1 -- 172.21.15.77:0/4279787242 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe72c0a29a0 msgr2=0x7fe72c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:47.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.324+0000 7fe748f30700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe72c0a29a0 0x7fe72c0a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe7440ffe30 tx=0x7fe738009260 comp rx=0 tx=0).stop 2024-03-16T10:15:47.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.324+0000 7fe748f30700 1 -- 172.21.15.77:0/4279787242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7441008c0 msgr2=0x7fe74419e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:47.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.324+0000 7fe748f30700 1 --2- 172.21.15.77:0/4279787242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7441008c0 0x7fe74419e7b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe7340059d0 tx=0x7fe73400bab0 comp rx=0 tx=0).stop 2024-03-16T10:15:47.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.324+0000 7fe742d9d700 1 -- 172.21.15.77:0/4279787242 reap_dead start 2024-03-16T10:15:47.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.324+0000 7fe748f30700 1 -- 172.21.15.77:0/4279787242 shutdown_connections 2024-03-16T10:15:47.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.324+0000 7fe748f30700 1 -- 172.21.15.77:0/4279787242 >> 172.21.15.77:0/4279787242 conn(0x7fe7440777d0 msgr2=0x7fe7441053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:47.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.324+0000 7fe748f30700 1 -- 172.21.15.77:0/4279787242 shutdown_connections 2024-03-16T10:15:47.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:47.324+0000 7fe748f30700 1 -- 172.21.15.77:0/4279787242 wait complete. 2024-03-16T10:15:47.882 INFO:teuthology.orchestra.run.smithi077.stdout:309237645338 2024-03-16T10:15:47.883 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-16T10:15:47.906 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:47 smithi077 ceph-mon[32400]: pgmap v309: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-16T10:15:48.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:47 smithi192 ceph-mon[36354]: pgmap v309: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-16T10:15:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:47 smithi193 ceph-mon[36311]: pgmap v309: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-16T10:15:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.401+0000 7f4d3548b700 1 -- 172.21.15.77:0/3887309022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d30106580 msgr2=0x7f4d3010e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.401+0000 7f4d3548b700 1 --2- 172.21.15.77:0/3887309022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d30106580 0x7f4d3010e150 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800b0d0 tx=0x7f4d2800b3e0 comp rx=0 tx=0).stop 2024-03-16T10:15:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.402+0000 7f4d3548b700 1 -- 172.21.15.77:0/3887309022 shutdown_connections 2024-03-16T10:15:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.402+0000 7f4d3548b700 1 --2- 172.21.15.77:0/3887309022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d30106580 0x7f4d3010e150 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.402+0000 7f4d3548b700 1 --2- 172.21.15.77:0/3887309022 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d301014a0 0x7f4d30101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.402+0000 7f4d3548b700 1 --2- 172.21.15.77:0/3887309022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d30101e50 0x7f4d30105ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.402+0000 7f4d3548b700 1 -- 172.21.15.77:0/3887309022 >> 172.21.15.77:0/3887309022 conn(0x7f4d30078300 msgr2=0x7f4d30078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.402+0000 7f4d3548b700 1 -- 172.21.15.77:0/3887309022 shutdown_connections 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.402+0000 7f4d3548b700 1 -- 172.21.15.77:0/3887309022 wait complete. 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.402+0000 7f4d3548b700 1 Processor -- start 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d3548b700 1 -- start start 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d3548b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d301014a0 0x7f4d301999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d3548b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d30101e50 0x7f4d30199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d3548b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d30106580 0x7f4d301a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d3548b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301115e0 con 0x7f4d301014a0 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d3548b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4d30111460 con 0x7f4d30106580 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d3548b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4d30111760 con 0x7f4d30101e50 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d2effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d301014a0 0x7f4d301999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:48.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d2effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d301014a0 0x7f4d301999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59334/0 (socket says 172.21.15.77:59334) 2024-03-16T10:15:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d2effd700 1 -- 172.21.15.77:0/2088869468 learned_addr learned my addr 172.21.15.77:0/2088869468 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d2e7fc700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d30101e50 0x7f4d30199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d2f7fe700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d30106580 0x7f4d301a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d2effd700 1 -- 172.21.15.77:0/2088869468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d30101e50 msgr2=0x7f4d30199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d2effd700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d30101e50 0x7f4d30199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d2effd700 1 -- 172.21.15.77:0/2088869468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d30106580 msgr2=0x7f4d301a4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d2effd700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d30106580 0x7f4d301a4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.403+0000 7f4d2effd700 1 -- 172.21.15.77:0/2088869468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d301a4f60 con 0x7f4d301014a0 2024-03-16T10:15:48.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.404+0000 7f4d2effd700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d301014a0 0x7f4d301999a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000eb30 tx=0x7f4d2000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.404+0000 7f4d17fff700 1 -- 172.21.15.77:0/2088869468 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2000cbd0 con 0x7f4d301014a0 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.404+0000 7f4d3548b700 1 -- 172.21.15.77:0/2088869468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d301a5130 con 0x7f4d301014a0 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.404+0000 7f4d17fff700 1 -- 172.21.15.77:0/2088869468 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d20004500 con 0x7f4d301014a0 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.404+0000 7f4d17fff700 1 -- 172.21.15.77:0/2088869468 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20010430 con 0x7f4d301014a0 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.404+0000 7f4d3548b700 1 -- 172.21.15.77:0/2088869468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d301a56c0 con 0x7f4d301014a0 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.405+0000 7f4d3548b700 1 -- 172.21.15.77:0/2088869468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4d300629e0 con 0x7f4d301014a0 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.406+0000 7f4d17fff700 1 -- 172.21.15.77:0/2088869468 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4d20003f00 con 0x7f4d301014a0 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.406+0000 7f4d17fff700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d180a2920 0x7f4d180a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.407+0000 7f4d2e7fc700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d180a2920 0x7f4d180a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.407+0000 7f4d17fff700 1 -- 172.21.15.77:0/2088869468 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4d200cedc0 con 0x7f4d301014a0 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.407+0000 7f4d17fff700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269] conn(0x7f4d180ac100 0x7f4d180ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.407+0000 7f4d17fff700 1 -- 172.21.15.77:0/2088869468 --> [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d180aebd0 con 0x7f4d180ac100 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.407+0000 7f4d17fff700 1 -- 172.21.15.77:0/2088869468 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f4d200d1510 con 0x7f4d301014a0 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.407+0000 7f4d2f7fe700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269] conn(0x7f4d180ac100 0x7f4d180ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:48.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.407+0000 7f4d2e7fc700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d180a2920 0x7f4d180a4de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4d30107950 tx=0x7f4d24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.407+0000 7f4d2f7fe700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269] conn(0x7f4d180ac100 0x7f4d180ae520 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.408+0000 7f4d17fff700 1 -- 172.21.15.77:0/2088869468 <== osd.8 v2:172.21.15.193:6816/3436040269 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d180aebd0 con 0x7f4d180ac100 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.424+0000 7f4d3548b700 1 -- 172.21.15.77:0/2088869468 --> [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4d30062de0 con 0x7f4d180ac100 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.424+0000 7f4d17fff700 1 -- 172.21.15.77:0/2088869468 <== osd.8 v2:172.21.15.193:6816/3436040269 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4d30062de0 con 0x7f4d180ac100 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.425+0000 7f4d3548b700 1 -- 172.21.15.77:0/2088869468 >> [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269] conn(0x7f4d180ac100 msgr2=0x7f4d180ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.425+0000 7f4d3548b700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269] conn(0x7f4d180ac100 0x7f4d180ae520 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.425+0000 7f4d3548b700 1 -- 172.21.15.77:0/2088869468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d180a2920 msgr2=0x7f4d180a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.425+0000 7f4d3548b700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d180a2920 0x7f4d180a4de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4d30107950 tx=0x7f4d24009260 comp rx=0 tx=0).stop 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.425+0000 7f4d3548b700 1 -- 172.21.15.77:0/2088869468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d301014a0 msgr2=0x7f4d301999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.425+0000 7f4d3548b700 1 --2- 172.21.15.77:0/2088869468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d301014a0 0x7f4d301999a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000eb30 tx=0x7f4d2000ee40 comp rx=0 tx=0).stop 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.425+0000 7f4d2f7fe700 1 -- 172.21.15.77:0/2088869468 reap_dead start 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.425+0000 7f4d3548b700 1 -- 172.21.15.77:0/2088869468 shutdown_connections 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.425+0000 7f4d3548b700 1 -- 172.21.15.77:0/2088869468 >> 172.21.15.77:0/2088869468 conn(0x7f4d30078300 msgr2=0x7f4d300ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:48.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.425+0000 7f4d3548b700 1 -- 172.21.15.77:0/2088869468 shutdown_connections 2024-03-16T10:15:48.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:46.425+0000 7f4d3548b700 1 -- 172.21.15.77:0/2088869468 wait complete. 2024-03-16T10:15:49.261 INFO:teuthology.orchestra.run.smithi077.stdout:287762808865 2024-03-16T10:15:49.261 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-16T10:15:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:253403070502 2024-03-16T10:15:49.383 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-16T10:15:49.968 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:49 smithi077 ceph-mon[32400]: pgmap v310: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-16T10:15:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:49 smithi192 ceph-mon[36354]: pgmap v310: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-16T10:15:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:49 smithi193 ceph-mon[36311]: pgmap v310: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-16T10:15:50.629 INFO:teuthology.orchestra.run.smithi077.stdout:55834574921 2024-03-16T10:15:50.630 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-16T10:15:50.667 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:50.668 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:50.668 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:50.671 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:51.165 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:51.166 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:51.193 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:51.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:51 smithi077 ceph-mon[32400]: pgmap v311: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:15:52.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:51 smithi192 ceph-mon[36354]: pgmap v311: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:15:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:51 smithi193 ceph-mon[36311]: pgmap v311: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:15:54.105 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:53 smithi077 ceph-mon[32400]: pgmap v312: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:15:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:53 smithi192 ceph-mon[36354]: pgmap v312: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:15:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:53 smithi193 ceph-mon[36311]: pgmap v312: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:15:55.646 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:55.647 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:55.647 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:15:55.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:55 smithi077 ceph-mon[32400]: pgmap v313: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-16T10:15:56.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:55 smithi192 ceph-mon[36354]: pgmap v313: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-16T10:15:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:55 smithi193 ceph-mon[36311]: pgmap v313: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-16T10:15:57.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.393+0000 7f893491d700 1 -- 172.21.15.77:0/3824114591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893006a920 msgr2=0x7f893010f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:57.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.393+0000 7f893491d700 1 --2- 172.21.15.77:0/3824114591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893006a920 0x7f893010f160 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f89200099c0 tx=0x7f8920009cd0 comp rx=0 tx=0).stop 2024-03-16T10:15:57.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.393+0000 7f893491d700 1 -- 172.21.15.77:0/3824114591 shutdown_connections 2024-03-16T10:15:57.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.393+0000 7f893491d700 1 --2- 172.21.15.77:0/3824114591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893006a920 0x7f893010f160 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:57.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.393+0000 7f893491d700 1 --2- 172.21.15.77:0/3824114591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8930108320 0x7f8930108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:57.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.393+0000 7f893491d700 1 --2- 172.21.15.77:0/3824114591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8930069f80 0x7f893006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:57.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.393+0000 7f893491d700 1 -- 172.21.15.77:0/3824114591 >> 172.21.15.77:0/3824114591 conn(0x7f8930078100 msgr2=0x7f8930078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.393+0000 7f893491d700 1 -- 172.21.15.77:0/3824114591 shutdown_connections 2024-03-16T10:15:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.393+0000 7f893491d700 1 -- 172.21.15.77:0/3824114591 wait complete. 2024-03-16T10:15:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f893491d700 1 Processor -- start 2024-03-16T10:15:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f893491d700 1 -- start start 2024-03-16T10:15:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f893491d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8930069f80 0x7f89301012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f893491d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893006a920 0x7f89301017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f893491d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8930108320 0x7f8930101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f893491d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f89301125a0 con 0x7f893006a920 2024-03-16T10:15:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f893491d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8930112420 con 0x7f8930108320 2024-03-16T10:15:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f893491d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8930112720 con 0x7f8930069f80 2024-03-16T10:15:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f892dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893006a920 0x7f89301017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f892dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893006a920 0x7f89301017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59350/0 (socket says 172.21.15.77:59350) 2024-03-16T10:15:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f892dd9b700 1 -- 172.21.15.77:0/3480773403 learned_addr learned my addr 172.21.15.77:0/3480773403 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f892e59c700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8930069f80 0x7f89301012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.394+0000 7f892ed9d700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8930108320 0x7f8930101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.395+0000 7f892dd9b700 1 -- 172.21.15.77:0/3480773403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8930069f80 msgr2=0x7f89301012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.395+0000 7f892dd9b700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8930069f80 0x7f89301012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.395+0000 7f892dd9b700 1 -- 172.21.15.77:0/3480773403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8930108320 msgr2=0x7f8930101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.395+0000 7f892dd9b700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8930108320 0x7f8930101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.395+0000 7f892dd9b700 1 -- 172.21.15.77:0/3480773403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89301a6f30 con 0x7f893006a920 2024-03-16T10:15:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.395+0000 7f892dd9b700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893006a920 0x7f89301017e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f891800e9c0 tx=0x7f891800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.395+0000 7f89277fe700 1 -- 172.21.15.77:0/3480773403 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891800cbd0 con 0x7f893006a920 2024-03-16T10:15:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.395+0000 7f893491d700 1 -- 172.21.15.77:0/3480773403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89301a7220 con 0x7f893006a920 2024-03-16T10:15:57.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.395+0000 7f89277fe700 1 -- 172.21.15.77:0/3480773403 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f891800ee60 con 0x7f893006a920 2024-03-16T10:15:57.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.395+0000 7f89277fe700 1 -- 172.21.15.77:0/3480773403 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8918005630 con 0x7f893006a920 2024-03-16T10:15:57.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.395+0000 7f893491d700 1 -- 172.21.15.77:0/3480773403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89301a7710 con 0x7f893006a920 2024-03-16T10:15:57.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.396+0000 7f893491d700 1 -- 172.21.15.77:0/3480773403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8930068f90 con 0x7f893006a920 2024-03-16T10:15:57.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.397+0000 7f89277fe700 1 -- 172.21.15.77:0/3480773403 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8918004750 con 0x7f893006a920 2024-03-16T10:15:57.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.398+0000 7f89277fe700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f891c0a2920 0x7f891c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:57.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.398+0000 7f892e59c700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f891c0a2920 0x7f891c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:57.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.398+0000 7f89277fe700 1 -- 172.21.15.77:0/3480773403 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f89180ceff0 con 0x7f893006a920 2024-03-16T10:15:57.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.399+0000 7f892e59c700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f891c0a2920 0x7f891c0a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f892800f540 tx=0x7f892800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:57.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.402+0000 7f89277fe700 1 -- 172.21.15.77:0/3480773403 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8918053740 con 0x7f893006a920 2024-03-16T10:15:57.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.610+0000 7f893491d700 1 -- 172.21.15.77:0/3480773403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f893002db70 con 0x7f893006a920 2024-03-16T10:15:57.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.610+0000 7f89277fe700 1 -- 172.21.15.77:0/3480773403 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f891809bfe0 con 0x7f893006a920 2024-03-16T10:15:57.611 INFO:teuthology.orchestra.run.smithi077.stdout:34359738448 2024-03-16T10:15:57.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.612+0000 7f893491d700 1 -- 172.21.15.77:0/3480773403 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f891c0a2920 msgr2=0x7f891c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:57.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.612+0000 7f893491d700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f891c0a2920 0x7f891c0a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f892800f540 tx=0x7f892800b040 comp rx=0 tx=0).stop 2024-03-16T10:15:57.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.612+0000 7f893491d700 1 -- 172.21.15.77:0/3480773403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893006a920 msgr2=0x7f89301017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:57.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.612+0000 7f893491d700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893006a920 0x7f89301017e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f891800e9c0 tx=0x7f891800ecd0 comp rx=0 tx=0).stop 2024-03-16T10:15:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.612+0000 7f893491d700 1 -- 172.21.15.77:0/3480773403 shutdown_connections 2024-03-16T10:15:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.612+0000 7f893491d700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f891c0a2920 0x7f891c0a4de0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.612+0000 7f893491d700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8930108320 0x7f8930101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.612+0000 7f893491d700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8930069f80 0x7f89301012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.612+0000 7f893491d700 1 --2- 172.21.15.77:0/3480773403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893006a920 0x7f89301017e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.612+0000 7f893491d700 1 -- 172.21.15.77:0/3480773403 >> 172.21.15.77:0/3480773403 conn(0x7f8930078100 msgr2=0x7f89300ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.612+0000 7f893491d700 1 -- 172.21.15.77:0/3480773403 shutdown_connections 2024-03-16T10:15:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:57.612+0000 7f893491d700 1 -- 172.21.15.77:0/3480773403 wait complete. 2024-03-16T10:15:57.866 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:57 smithi077 ceph-mon[32400]: pgmap v314: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:15:57.866 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:57 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3480773403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T10:15:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:57 smithi192 ceph-mon[36354]: pgmap v314: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:15:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:57 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3480773403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T10:15:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:57 smithi193 ceph-mon[36311]: pgmap v314: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:15:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:57 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3480773403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T10:15:58.873 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-16T10:15:58.873 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-16T10:15:58.873 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8, 5, 9]}]: dispatch 2024-03-16T10:15:58.874 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-03-16T10:15:58.874 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-16T10:15:58.874 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:58 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-03-16T10:15:58.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.900+0000 7f6ba7289700 1 -- 172.21.15.77:0/1270699217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ba0107210 msgr2=0x7f6ba01075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:58.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.900+0000 7f6ba7289700 1 --2- 172.21.15.77:0/1270699217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ba0107210 0x7f6ba01075f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009990 tx=0x7f6b90009ca0 comp rx=0 tx=0).stop 2024-03-16T10:15:58.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.900+0000 7f6ba7289700 1 -- 172.21.15.77:0/1270699217 shutdown_connections 2024-03-16T10:15:58.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.900+0000 7f6ba7289700 1 --2- 172.21.15.77:0/1270699217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ba0107210 0x7f6ba01075f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:58.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.900+0000 7f6ba7289700 1 --2- 172.21.15.77:0/1270699217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ba0101b70 0x7f6ba010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:58.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.900+0000 7f6ba7289700 1 --2- 172.21.15.77:0/1270699217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba01011d0 0x7f6ba0101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:58.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.900+0000 7f6ba7289700 1 -- 172.21.15.77:0/1270699217 >> 172.21.15.77:0/1270699217 conn(0x7f6ba0078100 msgr2=0x7f6ba0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:58.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.900+0000 7f6ba7289700 1 -- 172.21.15.77:0/1270699217 shutdown_connections 2024-03-16T10:15:58.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.900+0000 7f6ba7289700 1 -- 172.21.15.77:0/1270699217 wait complete. 2024-03-16T10:15:58.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba7289700 1 Processor -- start 2024-03-16T10:15:58.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba7289700 1 -- start start 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba7289700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ba01011d0 0x7f6ba01998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba7289700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba0101b70 0x7f6ba0199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba7289700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ba0107210 0x7f6ba01a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba7289700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba0111470 con 0x7f6ba0107210 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba7289700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba01112f0 con 0x7f6ba0101b70 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba7289700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba01115f0 con 0x7f6ba01011d0 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba5826700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ba0107210 0x7f6ba01a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba5025700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ba01011d0 0x7f6ba01998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba5025700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ba01011d0 0x7f6ba01998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44896/0 (socket says 172.21.15.77:44896) 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba5826700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ba0107210 0x7f6ba01a4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59374/0 (socket says 172.21.15.77:59374) 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba5025700 1 -- 172.21.15.77:0/2067173998 learned_addr learned my addr 172.21.15.77:0/2067173998 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba4824700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba0101b70 0x7f6ba0199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:58.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba5826700 1 -- 172.21.15.77:0/2067173998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ba01011d0 msgr2=0x7f6ba01998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:58.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba5826700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ba01011d0 0x7f6ba01998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:58.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba5826700 1 -- 172.21.15.77:0/2067173998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba0101b70 msgr2=0x7f6ba0199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:58.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba5826700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba0101b70 0x7f6ba0199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:58.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.901+0000 7f6ba5826700 1 -- 172.21.15.77:0/2067173998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba01a4e70 con 0x7f6ba0107210 2024-03-16T10:15:58.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.902+0000 7f6ba5826700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ba0107210 0x7f6ba01a4930 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6b98007f10 tx=0x7f6b9800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:58.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.902+0000 7f6b967fc700 1 -- 172.21.15.77:0/2067173998 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b98004db0 con 0x7f6ba0107210 2024-03-16T10:15:58.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.902+0000 7f6b967fc700 1 -- 172.21.15.77:0/2067173998 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b98009e20 con 0x7f6ba0107210 2024-03-16T10:15:58.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.902+0000 7f6b967fc700 1 -- 172.21.15.77:0/2067173998 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b98015400 con 0x7f6ba0107210 2024-03-16T10:15:58.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.902+0000 7f6ba7289700 1 -- 172.21.15.77:0/2067173998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba01a50a0 con 0x7f6ba0107210 2024-03-16T10:15:58.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.902+0000 7f6ba7289700 1 -- 172.21.15.77:0/2067173998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba01a55b0 con 0x7f6ba0107210 2024-03-16T10:15:58.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.903+0000 7f6ba7289700 1 -- 172.21.15.77:0/2067173998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6ba00629e0 con 0x7f6ba0107210 2024-03-16T10:15:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.904+0000 7f6b967fc700 1 -- 172.21.15.77:0/2067173998 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6b9800db20 con 0x7f6ba0107210 2024-03-16T10:15:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.905+0000 7f6b967fc700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b8c0a29a0 0x7f6b8c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.905+0000 7f6ba5025700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b8c0a29a0 0x7f6b8c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.905+0000 7f6b967fc700 1 -- 172.21.15.77:0/2067173998 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8161+0+0 (secure 0 0 0) 0x7f6b980ceef0 con 0x7f6ba0107210 2024-03-16T10:15:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.905+0000 7f6b967fc700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951] conn(0x7f6b8c0ac3e0 0x7f6b8c0ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:15:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.905+0000 7f6b967fc700 1 -- 172.21.15.77:0/2067173998 --> [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6b98020e50 con 0x7f6b8c0ac3e0 2024-03-16T10:15:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.905+0000 7f6b967fc700 1 -- 172.21.15.77:0/2067173998 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f6b980cf3d0 con 0x7f6ba0107210 2024-03-16T10:15:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.905+0000 7f6ba5025700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b8c0a29a0 0x7f6b8c0a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000b5c0 tx=0x7f6b90005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.905+0000 7f6ba4824700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951] conn(0x7f6b8c0ac3e0 0x7f6b8c0ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:15:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.906+0000 7f6ba4824700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951] conn(0x7f6b8c0ac3e0 0x7f6b8c0ae880 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:15:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.906+0000 7f6b967fc700 1 -- 172.21.15.77:0/2067173998 <== osd.4 v2:172.21.15.192:6808/3948604951 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6b98020e50 con 0x7f6b8c0ac3e0 2024-03-16T10:15:58.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.922+0000 7f6ba7289700 1 -- 172.21.15.77:0/2067173998 --> [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6ba0062e60 con 0x7f6b8c0ac3e0 2024-03-16T10:15:58.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.922+0000 7f6b967fc700 1 -- 172.21.15.77:0/2067173998 <== osd.4 v2:172.21.15.192:6808/3948604951 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6ba0062e60 con 0x7f6b8c0ac3e0 2024-03-16T10:15:58.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.923+0000 7f6ba7289700 1 -- 172.21.15.77:0/2067173998 >> [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951] conn(0x7f6b8c0ac3e0 msgr2=0x7f6b8c0ae880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:58.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.923+0000 7f6ba7289700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951] conn(0x7f6b8c0ac3e0 0x7f6b8c0ae880 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:15:58.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.923+0000 7f6ba7289700 1 -- 172.21.15.77:0/2067173998 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b8c0a29a0 msgr2=0x7f6b8c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:58.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.923+0000 7f6ba7289700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b8c0a29a0 0x7f6b8c0a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000b5c0 tx=0x7f6b90005be0 comp rx=0 tx=0).stop 2024-03-16T10:15:58.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.923+0000 7f6ba7289700 1 -- 172.21.15.77:0/2067173998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ba0107210 msgr2=0x7f6ba01a4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:15:58.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.923+0000 7f6ba7289700 1 --2- 172.21.15.77:0/2067173998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ba0107210 0x7f6ba01a4930 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6b98007f10 tx=0x7f6b9800d3b0 comp rx=0 tx=0).stop 2024-03-16T10:15:58.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.923+0000 7f6ba5826700 1 -- 172.21.15.77:0/2067173998 reap_dead start 2024-03-16T10:15:58.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.923+0000 7f6ba7289700 1 -- 172.21.15.77:0/2067173998 shutdown_connections 2024-03-16T10:15:58.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.923+0000 7f6ba7289700 1 -- 172.21.15.77:0/2067173998 >> 172.21.15.77:0/2067173998 conn(0x7f6ba0078100 msgr2=0x7f6ba00fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:15:58.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.923+0000 7f6ba7289700 1 -- 172.21.15.77:0/2067173998 shutdown_connections 2024-03-16T10:15:58.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:15:58.923+0000 7f6ba7289700 1 -- 172.21.15.77:0/2067173998 wait complete. 2024-03-16T10:15:59.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-16T10:15:59.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-16T10:15:59.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8, 5, 9]}]: dispatch 2024-03-16T10:15:59.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-03-16T10:15:59.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-16T10:15:59.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:58 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-03-16T10:15:59.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-16T10:15:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-16T10:15:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8, 5, 9]}]: dispatch 2024-03-16T10:15:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-03-16T10:15:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-16T10:15:59.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:58 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-03-16T10:15:59.883 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:59 smithi077 ceph-mon[32400]: pgmap v315: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-16T10:15:59.883 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-16T10:15:59.883 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-16T10:15:59.883 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8, 5, 9]}]': finished 2024-03-16T10:15:59.883 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-03-16T10:15:59.883 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-16T10:15:59.883 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:59 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-03-16T10:15:59.883 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:15:59 smithi077 ceph-mon[32400]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T10:16:00.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:59 smithi192 ceph-mon[36354]: pgmap v315: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-16T10:16:00.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-16T10:16:00.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-16T10:16:00.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8, 5, 9]}]': finished 2024-03-16T10:16:00.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-03-16T10:16:00.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-16T10:16:00.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:59 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-03-16T10:16:00.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:15:59 smithi192 ceph-mon[36354]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T10:16:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:59 smithi193 ceph-mon[36311]: pgmap v315: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-16T10:16:00.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-16T10:16:00.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-16T10:16:00.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8, 5, 9]}]': finished 2024-03-16T10:16:00.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-03-16T10:16:00.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-16T10:16:00.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:59 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-03-16T10:16:00.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:15:59 smithi193 ceph-mon[36311]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T10:16:00.483 INFO:teuthology.orchestra.run.smithi077.stdout:141733920828 2024-03-16T10:16:00.483 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-16T10:16:01.005 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:00 smithi077 ceph-mon[32400]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T10:16:01.005 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:00 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:16:01.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.138+0000 7fe9f4532700 1 -- 172.21.15.77:0/460831241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec101280 msgr2=0x7fe9ec10f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:01.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.138+0000 7fe9f4532700 1 --2- 172.21.15.77:0/460831241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec101280 0x7fe9ec10f1e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe9e800b0a0 tx=0x7fe9e800b3b0 comp rx=0 tx=0).stop 2024-03-16T10:16:01.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.138+0000 7fe9f4532700 1 -- 172.21.15.77:0/460831241 shutdown_connections 2024-03-16T10:16:01.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.138+0000 7fe9f4532700 1 --2- 172.21.15.77:0/460831241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec101280 0x7fe9ec10f1e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:01.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.138+0000 7fe9f4532700 1 --2- 172.21.15.77:0/460831241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ec107340 0x7fe9ec107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:01.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.138+0000 7fe9f4532700 1 --2- 172.21.15.77:0/460831241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:01.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.138+0000 7fe9f4532700 1 -- 172.21.15.77:0/460831241 >> 172.21.15.77:0/460831241 conn(0x7fe9ec0777d0 msgr2=0x7fe9ec0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:01.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.138+0000 7fe9f4532700 1 -- 172.21.15.77:0/460831241 shutdown_connections 2024-03-16T10:16:01.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.138+0000 7fe9f4532700 1 -- 172.21.15.77:0/460831241 wait complete. 2024-03-16T10:16:01.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f4532700 1 Processor -- start 2024-03-16T10:16:01.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f4532700 1 -- start start 2024-03-16T10:16:01.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f4532700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec19e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:01.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f4532700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ec101280 0x7fe9ec19ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f4532700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec107340 0x7fe9ec198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f4532700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec112720 con 0x7fe9ec107340 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f4532700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec1125a0 con 0x7fe9ec1008c0 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f4532700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec1128a0 con 0x7fe9ec101280 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f2acf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec107340 0x7fe9ec198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f2acf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec107340 0x7fe9ec198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59386/0 (socket says 172.21.15.77:59386) 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f2acf700 1 -- 172.21.15.77:0/3388163743 learned_addr learned my addr 172.21.15.77:0/3388163743 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f22ce700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec19e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f1acd700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ec101280 0x7fe9ec19ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f2acf700 1 -- 172.21.15.77:0/3388163743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ec101280 msgr2=0x7fe9ec19ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f2acf700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ec101280 0x7fe9ec19ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f2acf700 1 -- 172.21.15.77:0/3388163743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ec1008c0 msgr2=0x7fe9ec19e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:01.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f2acf700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec19e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:01.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.139+0000 7fe9f2acf700 1 -- 172.21.15.77:0/3388163743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ec1990c0 con 0x7fe9ec107340 2024-03-16T10:16:01.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.140+0000 7fe9f2acf700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec107340 0x7fe9ec198960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe9e800b780 tx=0x7fe9e8004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:01.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.140+0000 7fe9e37fe700 1 -- 172.21.15.77:0/3388163743 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8021070 con 0x7fe9ec107340 2024-03-16T10:16:01.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.140+0000 7fe9f4532700 1 -- 172.21.15.77:0/3388163743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9ec199350 con 0x7fe9ec107340 2024-03-16T10:16:01.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.140+0000 7fe9f4532700 1 -- 172.21.15.77:0/3388163743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9ec1a5670 con 0x7fe9ec107340 2024-03-16T10:16:01.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.140+0000 7fe9e37fe700 1 -- 172.21.15.77:0/3388163743 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9e8007b80 con 0x7fe9ec107340 2024-03-16T10:16:01.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.140+0000 7fe9e37fe700 1 -- 172.21.15.77:0/3388163743 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8005040 con 0x7fe9ec107340 2024-03-16T10:16:01.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.141+0000 7fe9f4532700 1 -- 172.21.15.77:0/3388163743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9ec10a8a0 con 0x7fe9ec107340 2024-03-16T10:16:01.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.142+0000 7fe9e37fe700 1 -- 172.21.15.77:0/3388163743 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe9e8024070 con 0x7fe9ec107340 2024-03-16T10:16:01.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.143+0000 7fe9e37fe700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe9d80a29a0 0x7fe9d80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:01.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.143+0000 7fe9f22ce700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe9d80a29a0 0x7fe9d80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:01.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.143+0000 7fe9e37fe700 1 -- 172.21.15.77:0/3388163743 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe9e80cf7f0 con 0x7fe9ec107340 2024-03-16T10:16:01.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.143+0000 7fe9f22ce700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe9d80a29a0 0x7fe9d80a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec199aa0 tx=0x7fe9e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:01.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.147+0000 7fe9e37fe700 1 -- 172.21.15.77:0/3388163743 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9e802b030 con 0x7fe9ec107340 2024-03-16T10:16:01.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:00 smithi192 ceph-mon[36354]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T10:16:01.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:00 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:16:01.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:00 smithi193 ceph-mon[36311]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T10:16:01.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:00 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:16:01.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.362+0000 7fe9f4532700 1 -- 172.21.15.77:0/3388163743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fe9ec199e90 con 0x7fe9ec107340 2024-03-16T10:16:01.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.362+0000 7fe9e37fe700 1 -- 172.21.15.77:0/3388163743 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fe9e80253b0 con 0x7fe9ec107340 2024-03-16T10:16:01.363 INFO:teuthology.orchestra.run.smithi077.stdout:309237645342 2024-03-16T10:16:01.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.365+0000 7fe9f4532700 1 -- 172.21.15.77:0/3388163743 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe9d80a29a0 msgr2=0x7fe9d80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:01.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.365+0000 7fe9f4532700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe9d80a29a0 0x7fe9d80a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec199aa0 tx=0x7fe9e4009260 comp rx=0 tx=0).stop 2024-03-16T10:16:01.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.365+0000 7fe9f4532700 1 -- 172.21.15.77:0/3388163743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec107340 msgr2=0x7fe9ec198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:01.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.365+0000 7fe9f4532700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec107340 0x7fe9ec198960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe9e800b780 tx=0x7fe9e8004d00 comp rx=0 tx=0).stop 2024-03-16T10:16:01.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.365+0000 7fe9f4532700 1 -- 172.21.15.77:0/3388163743 shutdown_connections 2024-03-16T10:16:01.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.365+0000 7fe9f4532700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe9d80a29a0 0x7fe9d80a4e60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:01.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.365+0000 7fe9f4532700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ec107340 0x7fe9ec198960 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:01.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.365+0000 7fe9f4532700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec19e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:01.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.365+0000 7fe9f4532700 1 --2- 172.21.15.77:0/3388163743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ec101280 0x7fe9ec19ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:01.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.365+0000 7fe9f4532700 1 -- 172.21.15.77:0/3388163743 >> 172.21.15.77:0/3388163743 conn(0x7fe9ec0777d0 msgr2=0x7fe9ec1052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:01.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.366+0000 7fe9f4532700 1 -- 172.21.15.77:0/3388163743 shutdown_connections 2024-03-16T10:16:01.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:01.366+0000 7fe9f4532700 1 -- 172.21.15.77:0/3388163743 wait complete. 2024-03-16T10:16:01.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:01 smithi077 ceph-mon[32400]: pgmap v318: 97 pgs: 3 peering, 94 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-16T10:16:01.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:01 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3388163743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T10:16:02.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:01 smithi192 ceph-mon[36354]: pgmap v318: 97 pgs: 3 peering, 94 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-16T10:16:02.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:01 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3388163743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T10:16:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:01 smithi193 ceph-mon[36311]: pgmap v318: 97 pgs: 3 peering, 94 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-16T10:16:02.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:01 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3388163743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T10:16:02.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.298+0000 7f44c79de700 1 -- 172.21.15.77:0/1618408888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c01011d0 msgr2=0x7f44c0101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:02.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.298+0000 7f44c79de700 1 --2- 172.21.15.77:0/1618408888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c01011d0 0x7f44c0101630 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009990 tx=0x7f44b0009ca0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 -- 172.21.15.77:0/1618408888 shutdown_connections 2024-03-16T10:16:02.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 --2- 172.21.15.77:0/1618408888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44c0101b70 0x7f44c010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 --2- 172.21.15.77:0/1618408888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44c0107210 0x7f44c01075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 --2- 172.21.15.77:0/1618408888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c01011d0 0x7f44c0101630 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 -- 172.21.15.77:0/1618408888 >> 172.21.15.77:0/1618408888 conn(0x7f44c0078100 msgr2=0x7f44c0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:02.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 -- 172.21.15.77:0/1618408888 shutdown_connections 2024-03-16T10:16:02.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 -- 172.21.15.77:0/1618408888 wait complete. 2024-03-16T10:16:02.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 Processor -- start 2024-03-16T10:16:02.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 -- start start 2024-03-16T10:16:02.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44c01011d0 0x7f44c019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:02.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44c0101b70 0x7f44c019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:02.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0107210 0x7f44c019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:02.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f44c0111510 con 0x7f44c0107210 2024-03-16T10:16:02.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f44c0111390 con 0x7f44c0101b70 2024-03-16T10:16:02.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.299+0000 7f44c79de700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f44c0111690 con 0x7f44c01011d0 2024-03-16T10:16:02.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44c5f7b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0107210 0x7f44c019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:02.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44c5f7b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0107210 0x7f44c019f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33870/0 (socket says 172.21.15.77:33870) 2024-03-16T10:16:02.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44c4f79700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44c0101b70 0x7f44c019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:02.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44c577a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44c01011d0 0x7f44c019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:02.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44c5f7b700 1 -- 172.21.15.77:0/2134017805 learned_addr learned my addr 172.21.15.77:0/2134017805 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:02.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44c5f7b700 1 -- 172.21.15.77:0/2134017805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44c01011d0 msgr2=0x7f44c019adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:02.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44c5f7b700 1 --2- 172.21.15.77:0/2134017805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44c01011d0 0x7f44c019adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44c5f7b700 1 -- 172.21.15.77:0/2134017805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44c0101b70 msgr2=0x7f44c019b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:02.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44c5f7b700 1 --2- 172.21.15.77:0/2134017805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44c0101b70 0x7f44c019b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44c5f7b700 1 -- 172.21.15.77:0/2134017805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44c019fdd0 con 0x7f44c0107210 2024-03-16T10:16:02.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44c5f7b700 1 --2- 172.21.15.77:0/2134017805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0107210 0x7f44c019f690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f44b800b970 tx=0x7f44b800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:02.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44b67fc700 1 -- 172.21.15.77:0/2134017805 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b800d860 con 0x7f44c0107210 2024-03-16T10:16:02.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44c79de700 1 -- 172.21.15.77:0/2134017805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44c01a00c0 con 0x7f44c0107210 2024-03-16T10:16:02.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44b67fc700 1 -- 172.21.15.77:0/2134017805 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44b800e430 con 0x7f44c0107210 2024-03-16T10:16:02.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.300+0000 7f44b67fc700 1 -- 172.21.15.77:0/2134017805 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b8017660 con 0x7f44c0107210 2024-03-16T10:16:02.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.301+0000 7f44c79de700 1 -- 172.21.15.77:0/2134017805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44c01a7940 con 0x7f44c0107210 2024-03-16T10:16:02.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.302+0000 7f44c79de700 1 -- 172.21.15.77:0/2134017805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44c0068f90 con 0x7f44c0107210 2024-03-16T10:16:02.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.302+0000 7f44b67fc700 1 -- 172.21.15.77:0/2134017805 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f44b800e5e0 con 0x7f44c0107210 2024-03-16T10:16:02.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.303+0000 7f44b67fc700 1 --2- 172.21.15.77:0/2134017805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f44ac0a2920 0x7f44ac0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:02.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.303+0000 7f44c577a700 1 --2- 172.21.15.77:0/2134017805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f44ac0a2920 0x7f44ac0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:02.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.303+0000 7f44b67fc700 1 -- 172.21.15.77:0/2134017805 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f44b80ce750 con 0x7f44c0107210 2024-03-16T10:16:02.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.304+0000 7f44c577a700 1 --2- 172.21.15.77:0/2134017805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f44ac0a2920 0x7f44ac0a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009c60 tx=0x7f44bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:02.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.307+0000 7f44b67fc700 1 -- 172.21.15.77:0/2134017805 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f44b8013030 con 0x7f44c0107210 2024-03-16T10:16:02.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.560+0000 7f44c79de700 1 -- 172.21.15.77:0/2134017805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f44c01a7b10 con 0x7f44c0107210 2024-03-16T10:16:02.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.561+0000 7f44b67fc700 1 -- 172.21.15.77:0/2134017805 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f44b809b780 con 0x7f44c0107210 2024-03-16T10:16:02.562 INFO:teuthology.orchestra.run.smithi077.stdout:163208757303 2024-03-16T10:16:02.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.563+0000 7f44c79de700 1 -- 172.21.15.77:0/2134017805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f44ac0a2920 msgr2=0x7f44ac0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:02.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.563+0000 7f44c79de700 1 --2- 172.21.15.77:0/2134017805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f44ac0a2920 0x7f44ac0a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009c60 tx=0x7f44bc009260 comp rx=0 tx=0).stop 2024-03-16T10:16:02.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.563+0000 7f44c79de700 1 -- 172.21.15.77:0/2134017805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0107210 msgr2=0x7f44c019f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:02.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.563+0000 7f44c79de700 1 --2- 172.21.15.77:0/2134017805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0107210 0x7f44c019f690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f44b800b970 tx=0x7f44b800bc80 comp rx=0 tx=0).stop 2024-03-16T10:16:02.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.563+0000 7f44c79de700 1 -- 172.21.15.77:0/2134017805 shutdown_connections 2024-03-16T10:16:02.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.563+0000 7f44c79de700 1 --2- 172.21.15.77:0/2134017805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f44ac0a2920 0x7f44ac0a4de0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.563+0000 7f44c79de700 1 --2- 172.21.15.77:0/2134017805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0107210 0x7f44c019f690 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.563+0000 7f44c79de700 1 --2- 172.21.15.77:0/2134017805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44c01011d0 0x7f44c019adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.563+0000 7f44c79de700 1 --2- 172.21.15.77:0/2134017805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44c0101b70 0x7f44c019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.563+0000 7f44c79de700 1 -- 172.21.15.77:0/2134017805 >> 172.21.15.77:0/2134017805 conn(0x7f44c0078100 msgr2=0x7f44c006a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:02.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.563+0000 7f44c79de700 1 -- 172.21.15.77:0/2134017805 shutdown_connections 2024-03-16T10:16:02.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.563+0000 7f44c79de700 1 -- 172.21.15.77:0/2134017805 wait complete. 2024-03-16T10:16:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.753+0000 7f6219bee700 1 -- 172.21.15.77:0/609504508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62141064e0 msgr2=0x7f621410e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.753+0000 7f6219bee700 1 --2- 172.21.15.77:0/609504508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62141064e0 0x7f621410e0b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f620c00b0a0 tx=0x7f620c00b3b0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.753+0000 7f6219bee700 1 -- 172.21.15.77:0/609504508 shutdown_connections 2024-03-16T10:16:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.753+0000 7f6219bee700 1 --2- 172.21.15.77:0/609504508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62141064e0 0x7f621410e0b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.753+0000 7f6219bee700 1 --2- 172.21.15.77:0/609504508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6214101420 0x7f6214101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.753+0000 7f6219bee700 1 --2- 172.21.15.77:0/609504508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6214101dd0 0x7f6214105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.753+0000 7f6219bee700 1 -- 172.21.15.77:0/609504508 >> 172.21.15.77:0/609504508 conn(0x7f6214078140 msgr2=0x7f6214078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.753+0000 7f6219bee700 1 -- 172.21.15.77:0/609504508 shutdown_connections 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.753+0000 7f6219bee700 1 -- 172.21.15.77:0/609504508 wait complete. 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f6219bee700 1 Processor -- start 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f6219bee700 1 -- start start 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f6219bee700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214101420 0x7f621419d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f6219bee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6214101dd0 0x7f621419db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f6219bee700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62141064e0 0x7f6214197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f6219bee700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f62141114c0 con 0x7f6214101420 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f6219bee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6214111340 con 0x7f6214101dd0 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f6219bee700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6214111640 con 0x7f62141064e0 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f62137fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214101420 0x7f621419d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f62137fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214101420 0x7f621419d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33894/0 (socket says 172.21.15.77:33894) 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f62137fe700 1 -- 172.21.15.77:0/2410589611 learned_addr learned my addr 172.21.15.77:0/2410589611 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:02.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f620bfff700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6214101dd0 0x7f621419db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.754+0000 7f6213fff700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62141064e0 0x7f6214197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.755+0000 7f62137fe700 1 -- 172.21.15.77:0/2410589611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62141064e0 msgr2=0x7f6214197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.755+0000 7f62137fe700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62141064e0 0x7f6214197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.755+0000 7f62137fe700 1 -- 172.21.15.77:0/2410589611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6214101dd0 msgr2=0x7f621419db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.755+0000 7f62137fe700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6214101dd0 0x7f621419db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.755+0000 7f62137fe700 1 -- 172.21.15.77:0/2410589611 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6214197fc0 con 0x7f6214101420 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.755+0000 7f62137fe700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214101420 0x7f621419d640 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f620400e9a0 tx=0x7f620400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.755+0000 7f62117fa700 1 -- 172.21.15.77:0/2410589611 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620400ca10 con 0x7f6214101420 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.755+0000 7f62117fa700 1 -- 172.21.15.77:0/2410589611 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f620400cbb0 con 0x7f6214101420 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.755+0000 7f6219bee700 1 -- 172.21.15.77:0/2410589611 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62141982b0 con 0x7f6214101420 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.755+0000 7f62117fa700 1 -- 172.21.15.77:0/2410589611 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6204018680 con 0x7f6214101420 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.755+0000 7f6219bee700 1 -- 172.21.15.77:0/2410589611 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f621406ab40 con 0x7f6214101420 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.756+0000 7f6219bee700 1 -- 172.21.15.77:0/2410589611 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6214198620 con 0x7f6214101420 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.757+0000 7f62117fa700 1 -- 172.21.15.77:0/2410589611 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6204010430 con 0x7f6214101420 2024-03-16T10:16:02.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.758+0000 7f62117fa700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f62000a2920 0x7f62000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:02.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.758+0000 7f620bfff700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f62000a2920 0x7f62000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:02.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.758+0000 7f62117fa700 1 -- 172.21.15.77:0/2410589611 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f62040cf1d0 con 0x7f6214101420 2024-03-16T10:16:02.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.758+0000 7f620bfff700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f62000a2920 0x7f62000a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6214198ed0 tx=0x7f61fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:02.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.761+0000 7f62117fa700 1 -- 172.21.15.77:0/2410589611 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f620409c200 con 0x7f6214101420 2024-03-16T10:16:02.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.997+0000 7f6219bee700 1 -- 172.21.15.77:0/2410589611 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f6214198c40 con 0x7f6214101420 2024-03-16T10:16:02.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.997+0000 7f62117fa700 1 -- 172.21.15.77:0/2410589611 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f620409bb00 con 0x7f6214101420 2024-03-16T10:16:02.998 INFO:teuthology.orchestra.run.smithi077.stdout:219043332141 2024-03-16T10:16:03.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.999+0000 7f6219bee700 1 -- 172.21.15.77:0/2410589611 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f62000a2920 msgr2=0x7f62000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:03.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.999+0000 7f6219bee700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f62000a2920 0x7f62000a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6214198ed0 tx=0x7f61fc009260 comp rx=0 tx=0).stop 2024-03-16T10:16:03.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.999+0000 7f6219bee700 1 -- 172.21.15.77:0/2410589611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214101420 msgr2=0x7f621419d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:03.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.999+0000 7f6219bee700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214101420 0x7f621419d640 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f620400e9a0 tx=0x7f620400ecb0 comp rx=0 tx=0).stop 2024-03-16T10:16:03.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.999+0000 7f6219bee700 1 -- 172.21.15.77:0/2410589611 shutdown_connections 2024-03-16T10:16:03.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.999+0000 7f6219bee700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f62000a2920 0x7f62000a4de0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:03.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.999+0000 7f6219bee700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6214101420 0x7f621419d640 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:03.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.999+0000 7f6219bee700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62141064e0 0x7f6214197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:03.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.999+0000 7f6219bee700 1 --2- 172.21.15.77:0/2410589611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6214101dd0 0x7f621419db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:03.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.999+0000 7f6219bee700 1 -- 172.21.15.77:0/2410589611 >> 172.21.15.77:0/2410589611 conn(0x7f6214078140 msgr2=0x7f62140ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:03.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.999+0000 7f6219bee700 1 -- 172.21.15.77:0/2410589611 shutdown_connections 2024-03-16T10:16:03.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:02.999+0000 7f6219bee700 1 -- 172.21.15.77:0/2410589611 wait complete. 2024-03-16T10:16:03.599 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:03.599 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:03.599 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:03.599 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:03 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:03.599 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:03 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2134017805' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T10:16:03.599 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:03 smithi077 ceph-mon[32400]: pgmap v319: 97 pgs: 3 peering, 94 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-03-16T10:16:03.599 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:03 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2410589611' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T10:16:03.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:03.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:03.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:03.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:03 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:03.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:03 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2134017805' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T10:16:03.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:03 smithi192 ceph-mon[36354]: pgmap v319: 97 pgs: 3 peering, 94 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-03-16T10:16:03.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:03 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2410589611' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T10:16:03.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:03.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:03 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:03 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2134017805' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T10:16:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:03 smithi193 ceph-mon[36311]: pgmap v319: 97 pgs: 3 peering, 94 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-03-16T10:16:03.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:03 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2410589611' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T10:16:03.921 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757297 got 163208757303 for osd.5 2024-03-16T10:16:03.921 DEBUG:teuthology.parallel:result is None 2024-03-16T10:16:04.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.583+0000 7f7e1a884700 1 -- 172.21.15.77:0/487318503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e14101460 msgr2=0x7f7e14101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:04.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.583+0000 7f7e1a884700 1 --2- 172.21.15.77:0/487318503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e14101460 0x7f7e14101840 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7e040099c0 tx=0x7f7e04009cd0 comp rx=0 tx=0).stop 2024-03-16T10:16:04.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.583+0000 7f7e1a884700 1 -- 172.21.15.77:0/487318503 shutdown_connections 2024-03-16T10:16:04.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.583+0000 7f7e1a884700 1 --2- 172.21.15.77:0/487318503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e14101460 0x7f7e14101840 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:04.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.583+0000 7f7e1a884700 1 --2- 172.21.15.77:0/487318503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e141064b0 0x7f7e1410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:04.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.583+0000 7f7e1a884700 1 --2- 172.21.15.77:0/487318503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e14101e10 0x7f7e14105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:04.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.583+0000 7f7e1a884700 1 -- 172.21.15.77:0/487318503 >> 172.21.15.77:0/487318503 conn(0x7f7e14077220 msgr2=0x7f7e14077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:04.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.583+0000 7f7e1a884700 1 -- 172.21.15.77:0/487318503 shutdown_connections 2024-03-16T10:16:04.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.583+0000 7f7e1a884700 1 -- 172.21.15.77:0/487318503 wait complete. 2024-03-16T10:16:04.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.583+0000 7f7e1a884700 1 Processor -- start 2024-03-16T10:16:04.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.584+0000 7f7e1a884700 1 -- start start 2024-03-16T10:16:04.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.584+0000 7f7e1a884700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e14101460 0x7f7e1419e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:04.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.584+0000 7f7e1a884700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e14101e10 0x7f7e1419ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:04.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.584+0000 7f7e1a884700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e141064b0 0x7f7e141988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:04.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.584+0000 7f7e1a884700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e141114b0 con 0x7f7e141064b0 2024-03-16T10:16:04.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.584+0000 7f7e1a884700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7e14111330 con 0x7f7e14101e10 2024-03-16T10:16:04.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.584+0000 7f7e1a884700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7e14111630 con 0x7f7e14101460 2024-03-16T10:16:04.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.584+0000 7f7e13fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e14101460 0x7f7e1419e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:04.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.584+0000 7f7e137fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e14101e10 0x7f7e1419ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:04.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.584+0000 7f7e13fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e14101460 0x7f7e1419e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:49046/0 (socket says 172.21.15.77:49046) 2024-03-16T10:16:04.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.584+0000 7f7e13fff700 1 -- 172.21.15.77:0/3635452867 learned_addr learned my addr 172.21.15.77:0/3635452867 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:04.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.584+0000 7f7e18e21700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e141064b0 0x7f7e141988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:04.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.585+0000 7f7e13fff700 1 -- 172.21.15.77:0/3635452867 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e14101e10 msgr2=0x7f7e1419ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:04.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.585+0000 7f7e13fff700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e14101e10 0x7f7e1419ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:04.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.585+0000 7f7e13fff700 1 -- 172.21.15.77:0/3635452867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e141064b0 msgr2=0x7f7e141988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:04.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.585+0000 7f7e13fff700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e141064b0 0x7f7e141988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:04.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.585+0000 7f7e13fff700 1 -- 172.21.15.77:0/3635452867 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e14199050 con 0x7f7e14101460 2024-03-16T10:16:04.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.585+0000 7f7e137fe700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e14101e10 0x7f7e1419ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:04.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.585+0000 7f7e13fff700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e14101460 0x7f7e1419e730 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7e04006010 tx=0x7f7e0400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:04.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.585+0000 7f7e117fa700 1 -- 172.21.15.77:0/3635452867 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04003680 con 0x7f7e14101460 2024-03-16T10:16:04.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.585+0000 7f7e1a884700 1 -- 172.21.15.77:0/3635452867 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e141992e0 con 0x7f7e14101460 2024-03-16T10:16:04.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.585+0000 7f7e117fa700 1 -- 172.21.15.77:0/3635452867 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e040050d0 con 0x7f7e14101460 2024-03-16T10:16:04.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.585+0000 7f7e117fa700 1 -- 172.21.15.77:0/3635452867 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04003db0 con 0x7f7e14101460 2024-03-16T10:16:04.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.585+0000 7f7e1a884700 1 -- 172.21.15.77:0/3635452867 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e141a5600 con 0x7f7e14101460 2024-03-16T10:16:04.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.586+0000 7f7e1a884700 1 -- 172.21.15.77:0/3635452867 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e141028b0 con 0x7f7e14101460 2024-03-16T10:16:04.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.588+0000 7f7e117fa700 1 -- 172.21.15.77:0/3635452867 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7e04005280 con 0x7f7e14101460 2024-03-16T10:16:04.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.588+0000 7f7e117fa700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7dfc0a2ac0 0x7f7dfc0a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:04.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.588+0000 7f7e117fa700 1 -- 172.21.15.77:0/3635452867 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7e040cf830 con 0x7f7e14101460 2024-03-16T10:16:04.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.588+0000 7f7e137fe700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7dfc0a2ac0 0x7f7dfc0a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:04.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.589+0000 7f7e137fe700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7dfc0a2ac0 0x7f7dfc0a4f80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7e1406afe0 tx=0x7f7e08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:04.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.592+0000 7f7e117fa700 1 -- 172.21.15.77:0/3635452867 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7e04013070 con 0x7f7e14101460 2024-03-16T10:16:04.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.809+0000 7f7e1a884700 1 -- 172.21.15.77:0/3635452867 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f7e14068f90 con 0x7f7e14101460 2024-03-16T10:16:04.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.810+0000 7f7e117fa700 1 -- 172.21.15.77:0/3635452867 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7e0409c870 con 0x7f7e14101460 2024-03-16T10:16:04.811 INFO:teuthology.orchestra.run.smithi077.stdout:287762808869 2024-03-16T10:16:04.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.812+0000 7f7e1a884700 1 -- 172.21.15.77:0/3635452867 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7dfc0a2ac0 msgr2=0x7f7dfc0a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:04.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.812+0000 7f7e1a884700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7dfc0a2ac0 0x7f7dfc0a4f80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7e1406afe0 tx=0x7f7e08009260 comp rx=0 tx=0).stop 2024-03-16T10:16:04.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.812+0000 7f7e1a884700 1 -- 172.21.15.77:0/3635452867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e14101460 msgr2=0x7f7e1419e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:04.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.812+0000 7f7e1a884700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e14101460 0x7f7e1419e730 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7e04006010 tx=0x7f7e0400f690 comp rx=0 tx=0).stop 2024-03-16T10:16:04.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.812+0000 7f7e1a884700 1 -- 172.21.15.77:0/3635452867 shutdown_connections 2024-03-16T10:16:04.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.812+0000 7f7e1a884700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7dfc0a2ac0 0x7f7dfc0a4f80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:04.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.812+0000 7f7e1a884700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e141064b0 0x7f7e141988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:04.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.812+0000 7f7e1a884700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e14101460 0x7f7e1419e730 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:04.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.812+0000 7f7e1a884700 1 --2- 172.21.15.77:0/3635452867 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e14101e10 0x7f7e1419ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:04.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.812+0000 7f7e1a884700 1 -- 172.21.15.77:0/3635452867 >> 172.21.15.77:0/3635452867 conn(0x7f7e14077220 msgr2=0x7f7e14104db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:04.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.812+0000 7f7e1a884700 1 -- 172.21.15.77:0/3635452867 shutdown_connections 2024-03-16T10:16:04.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:04.812+0000 7f7e1a884700 1 -- 172.21.15.77:0/3635452867 wait complete. 2024-03-16T10:16:05.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.302+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1560577919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb344102550 msgr2=0x7fb344102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.302+0000 7fb34b2ab700 1 --2- 172.21.15.77:0/1560577919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb344102550 0x7fb344102930 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb3400099c0 tx=0x7fb340009cd0 comp rx=0 tx=0).stop 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.303+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1560577919 shutdown_connections 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.303+0000 7fb34b2ab700 1 --2- 172.21.15.77:0/1560577919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb344102550 0x7fb344102930 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.303+0000 7fb34b2ab700 1 --2- 172.21.15.77:0/1560577919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb344107630 0x7fb34410f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.303+0000 7fb34b2ab700 1 --2- 172.21.15.77:0/1560577919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb344102f00 0x7fb344106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.303+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1560577919 >> 172.21.15.77:0/1560577919 conn(0x7fb3440fdda0 msgr2=0x7fb3441001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.303+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1560577919 shutdown_connections 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.303+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1560577919 wait complete. 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.303+0000 7fb34b2ab700 1 Processor -- start 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.303+0000 7fb34b2ab700 1 -- start start 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.303+0000 7fb34b2ab700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb344102550 0x7fb3440745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.303+0000 7fb34b2ab700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb344102f00 0x7fb344074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:05.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.303+0000 7fb34b2ab700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb344107630 0x7fb344075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:05.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb34b2ab700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb344112610 con 0x7fb344102f00 2024-03-16T10:16:05.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb34b2ab700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb344112490 con 0x7fb344107630 2024-03-16T10:16:05.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb34b2ab700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb344112790 con 0x7fb344102550 2024-03-16T10:16:05.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb348846700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb344102f00 0x7fb344074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:05.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb348846700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb344102f00 0x7fb344074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33928/0 (socket says 172.21.15.77:33928) 2024-03-16T10:16:05.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb348846700 1 -- 172.21.15.77:0/1113304614 learned_addr learned my addr 172.21.15.77:0/1113304614 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:05.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb349047700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb344102550 0x7fb3440745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:05.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb349848700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb344107630 0x7fb344075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:05.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb348846700 1 -- 172.21.15.77:0/1113304614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb344102550 msgr2=0x7fb3440745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:05.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb348846700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb344102550 0x7fb3440745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:05.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb348846700 1 -- 172.21.15.77:0/1113304614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb344107630 msgr2=0x7fb344075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:05.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb348846700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb344107630 0x7fb344075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:05.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb348846700 1 -- 172.21.15.77:0/1113304614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3441a6f90 con 0x7fb344102f00 2024-03-16T10:16:05.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb349047700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb344102550 0x7fb3440745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:05.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb349848700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb344107630 0x7fb344075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:05.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb348846700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb344102f00 0x7fb344074b10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb33400ea30 tx=0x7fb33400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:05.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb33a7fc700 1 -- 172.21.15.77:0/1113304614 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33400cbd0 con 0x7fb344102f00 2024-03-16T10:16:05.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb33a7fc700 1 -- 172.21.15.77:0/1113304614 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb334004500 con 0x7fb344102f00 2024-03-16T10:16:05.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1113304614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3441a7280 con 0x7fb344102f00 2024-03-16T10:16:05.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.304+0000 7fb33a7fc700 1 -- 172.21.15.77:0/1113304614 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334003e30 con 0x7fb344102f00 2024-03-16T10:16:05.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.305+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1113304614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3441a77c0 con 0x7fb344102f00 2024-03-16T10:16:05.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.305+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1113304614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb344068f90 con 0x7fb344102f00 2024-03-16T10:16:05.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.307+0000 7fb33a7fc700 1 -- 172.21.15.77:0/1113304614 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb334004000 con 0x7fb344102f00 2024-03-16T10:16:05.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.307+0000 7fb33a7fc700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb3300a2920 0x7fb3300a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:05.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.307+0000 7fb349047700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb3300a2920 0x7fb3300a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:05.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.307+0000 7fb33a7fc700 1 -- 172.21.15.77:0/1113304614 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb3340ced60 con 0x7fb344102f00 2024-03-16T10:16:05.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.308+0000 7fb349047700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb3300a2920 0x7fb3300a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb340000c00 tx=0x7fb340005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:05.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.311+0000 7fb33a7fc700 1 -- 172.21.15.77:0/1113304614 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb33409bda0 con 0x7fb344102f00 2024-03-16T10:16:05.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.560+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1113304614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb34402db70 con 0x7fb344102f00 2024-03-16T10:16:05.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.560+0000 7fb33a7fc700 1 -- 172.21.15.77:0/1113304614 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb33409b6a0 con 0x7fb344102f00 2024-03-16T10:16:05.561 INFO:teuthology.orchestra.run.smithi077.stdout:107374182466 2024-03-16T10:16:05.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.562+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1113304614 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb3300a2920 msgr2=0x7fb3300a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:05.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.562+0000 7fb34b2ab700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb3300a2920 0x7fb3300a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb340000c00 tx=0x7fb340005fb0 comp rx=0 tx=0).stop 2024-03-16T10:16:05.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.562+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1113304614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb344102f00 msgr2=0x7fb344074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:05.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.562+0000 7fb34b2ab700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb344102f00 0x7fb344074b10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb33400ea30 tx=0x7fb33400ed40 comp rx=0 tx=0).stop 2024-03-16T10:16:05.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.562+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1113304614 shutdown_connections 2024-03-16T10:16:05.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.562+0000 7fb34b2ab700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb3300a2920 0x7fb3300a4de0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:05.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.562+0000 7fb34b2ab700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb344107630 0x7fb344075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:05.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.562+0000 7fb34b2ab700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb344102550 0x7fb3440745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.562+0000 7fb34b2ab700 1 --2- 172.21.15.77:0/1113304614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb344102f00 0x7fb344074b10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.562+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1113304614 >> 172.21.15.77:0/1113304614 conn(0x7fb3440fdda0 msgr2=0x7fb3441000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.562+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1113304614 shutdown_connections 2024-03-16T10:16:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:05.562+0000 7fb34b2ab700 1 -- 172.21.15.77:0/1113304614 wait complete. 2024-03-16T10:16:06.074 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:05 smithi077 ceph-mon[32400]: pgmap v320: 97 pgs: 2 peering, 95 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:06.075 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:05 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3635452867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T10:16:06.075 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:05 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1113304614' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T10:16:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:05 smithi192 ceph-mon[36354]: pgmap v320: 97 pgs: 2 peering, 95 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:05 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3635452867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T10:16:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:05 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1113304614' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T10:16:06.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.180+0000 7f2807428700 1 -- 172.21.15.77:0/4294097368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2800107340 msgr2=0x7f2800107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:06.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.180+0000 7f2807428700 1 --2- 172.21.15.77:0/4294097368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2800107340 0x7f2800107720 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009990 tx=0x7f27fc009ca0 comp rx=0 tx=0).stop 2024-03-16T10:16:06.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.180+0000 7f2807428700 1 -- 172.21.15.77:0/4294097368 shutdown_connections 2024-03-16T10:16:06.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.180+0000 7f2807428700 1 --2- 172.21.15.77:0/4294097368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2800107340 0x7f2800107720 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:06.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.180+0000 7f2807428700 1 --2- 172.21.15.77:0/4294097368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2800101280 0x7f280010f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:06.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.180+0000 7f2807428700 1 --2- 172.21.15.77:0/4294097368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28001008c0 0x7f2800100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:06.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.180+0000 7f2807428700 1 -- 172.21.15.77:0/4294097368 >> 172.21.15.77:0/4294097368 conn(0x7f28000777d0 msgr2=0x7f28000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:06.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.180+0000 7f2807428700 1 -- 172.21.15.77:0/4294097368 shutdown_connections 2024-03-16T10:16:06.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.180+0000 7f2807428700 1 -- 172.21.15.77:0/4294097368 wait complete. 2024-03-16T10:16:06.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f2807428700 1 Processor -- start 2024-03-16T10:16:06.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f2807428700 1 -- start start 2024-03-16T10:16:06.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f2807428700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28001008c0 0x7f2800101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:06.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f2807428700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2800101280 0x7f2800101fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:06.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f2807428700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2800107340 0x7f2800102520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:06.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f2807428700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2800112720 con 0x7f28001008c0 2024-03-16T10:16:06.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f2807428700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f28001125a0 con 0x7f2800107340 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f2807428700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f28001128a0 con 0x7f2800101280 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f28051c4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28001008c0 0x7f2800101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f28051c4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28001008c0 0x7f2800101aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33954/0 (socket says 172.21.15.77:33954) 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f28051c4700 1 -- 172.21.15.77:0/2185425825 learned_addr learned my addr 172.21.15.77:0/2185425825 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f28049c3700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2800101280 0x7f2800101fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f28059c5700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2800107340 0x7f2800102520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f28051c4700 1 -- 172.21.15.77:0/2185425825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2800101280 msgr2=0x7f2800101fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f28051c4700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2800101280 0x7f2800101fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f28051c4700 1 -- 172.21.15.77:0/2185425825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2800107340 msgr2=0x7f2800102520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f28051c4700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2800107340 0x7f2800102520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f28051c4700 1 -- 172.21.15.77:0/2185425825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28001a70c0 con 0x7f28001008c0 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.181+0000 7f28049c3700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2800101280 0x7f2800101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.182+0000 7f28051c4700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28001008c0 0x7f2800101aa0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f27fc000c00 tx=0x7f27fc00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.182+0000 7f27f67fc700 1 -- 172.21.15.77:0/2185425825 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc027070 con 0x7f28001008c0 2024-03-16T10:16:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.182+0000 7f2807428700 1 -- 172.21.15.77:0/2185425825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28001a7350 con 0x7f28001008c0 2024-03-16T10:16:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.182+0000 7f27f67fc700 1 -- 172.21.15.77:0/2185425825 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27fc003d70 con 0x7f28001008c0 2024-03-16T10:16:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.182+0000 7f27f67fc700 1 -- 172.21.15.77:0/2185425825 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc00f460 con 0x7f28001008c0 2024-03-16T10:16:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.182+0000 7f2807428700 1 -- 172.21.15.77:0/2185425825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28001a7800 con 0x7f28001008c0 2024-03-16T10:16:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.183+0000 7f2807428700 1 -- 172.21.15.77:0/2185425825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f280010a910 con 0x7f28001008c0 2024-03-16T10:16:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.184+0000 7f27f67fc700 1 -- 172.21.15.77:0/2185425825 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f27fc02a070 con 0x7f28001008c0 2024-03-16T10:16:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.184+0000 7f27f67fc700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f27ec0a29a0 0x7f27ec0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.185+0000 7f28049c3700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f27ec0a29a0 0x7f27ec0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.185+0000 7f27f67fc700 1 -- 172.21.15.77:0/2185425825 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f27fc0cf850 con 0x7f28001008c0 2024-03-16T10:16:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.185+0000 7f28049c3700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f27ec0a29a0 0x7f27ec0a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f28000ffd70 tx=0x7f27f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:06.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.189+0000 7f27f67fc700 1 -- 172.21.15.77:0/2185425825 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f27fc013070 con 0x7f28001008c0 2024-03-16T10:16:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:05 smithi193 ceph-mon[36311]: pgmap v320: 97 pgs: 2 peering, 95 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:05 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3635452867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T10:16:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:05 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1113304614' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T10:16:06.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.393+0000 7f2807428700 1 -- 172.21.15.77:0/2185425825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f2800102fd0 con 0x7f28001008c0 2024-03-16T10:16:06.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.393+0000 7f27f67fc700 1 -- 172.21.15.77:0/2185425825 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f27fc09c910 con 0x7f28001008c0 2024-03-16T10:16:06.394 INFO:teuthology.orchestra.run.smithi077.stdout:343597383704 2024-03-16T10:16:06.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.395+0000 7f2807428700 1 -- 172.21.15.77:0/2185425825 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f27ec0a29a0 msgr2=0x7f27ec0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.395+0000 7f2807428700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f27ec0a29a0 0x7f27ec0a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f28000ffd70 tx=0x7f27f0009260 comp rx=0 tx=0).stop 2024-03-16T10:16:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.395+0000 7f2807428700 1 -- 172.21.15.77:0/2185425825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28001008c0 msgr2=0x7f2800101aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.395+0000 7f2807428700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28001008c0 0x7f2800101aa0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f27fc000c00 tx=0x7f27fc00bab0 comp rx=0 tx=0).stop 2024-03-16T10:16:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.396+0000 7f2807428700 1 -- 172.21.15.77:0/2185425825 shutdown_connections 2024-03-16T10:16:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.396+0000 7f2807428700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f27ec0a29a0 0x7f27ec0a4e60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.396+0000 7f2807428700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28001008c0 0x7f2800101aa0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.396+0000 7f2807428700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2800107340 0x7f2800102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.396+0000 7f2807428700 1 --2- 172.21.15.77:0/2185425825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2800101280 0x7f2800101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.396+0000 7f2807428700 1 -- 172.21.15.77:0/2185425825 >> 172.21.15.77:0/2185425825 conn(0x7f28000777d0 msgr2=0x7f2800105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.396+0000 7f2807428700 1 -- 172.21.15.77:0/2185425825 shutdown_connections 2024-03-16T10:16:06.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:06.396+0000 7f2807428700 1 -- 172.21.15.77:0/2185425825 wait complete. 2024-03-16T10:16:06.772 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808865 got 287762808869 for osd.9 2024-03-16T10:16:06.772 DEBUG:teuthology.parallel:result is None 2024-03-16T10:16:07.019 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:06 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2185425825' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T10:16:07.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:06 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2185425825' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T10:16:07.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:06 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2185425825' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T10:16:07.450 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738444 got 34359738448 for osd.0 2024-03-16T10:16:07.450 DEBUG:teuthology.parallel:result is None 2024-03-16T10:16:07.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:07 smithi077 ceph-mon[32400]: pgmap v321: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:08.029 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383699 got 343597383704 for osd.11 2024-03-16T10:16:08.030 DEBUG:teuthology.parallel:result is None 2024-03-16T10:16:08.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:07 smithi192 ceph-mon[36354]: pgmap v321: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:07 smithi193 ceph-mon[36311]: pgmap v321: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:08.331 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332135 got 219043332141 for osd.7 2024-03-16T10:16:08.331 DEBUG:teuthology.parallel:result is None 2024-03-16T10:16:08.700 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182461 got 107374182466 for osd.3 2024-03-16T10:16:08.700 DEBUG:teuthology.parallel:result is None 2024-03-16T10:16:10.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:09 smithi192 ceph-mon[36354]: pgmap v322: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:10.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:09 smithi077 ceph-mon[32400]: pgmap v322: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:10.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:09 smithi193 ceph-mon[36311]: pgmap v322: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:10.519 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:10.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.526+0000 7fd6ea519700 1 -- 172.21.15.77:0/2572840172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e4101e70 msgr2=0x7fd6e410e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:10.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.526+0000 7fd6ea519700 1 --2- 172.21.15.77:0/2572840172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e4101e70 0x7fd6e410e090 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009f90 tx=0x7fd6dc00b330 comp rx=0 tx=0).stop 2024-03-16T10:16:10.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.526+0000 7fd6ea519700 1 -- 172.21.15.77:0/2572840172 shutdown_connections 2024-03-16T10:16:10.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.526+0000 7fd6ea519700 1 --2- 172.21.15.77:0/2572840172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e4101e70 0x7fd6e410e090 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:10.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.526+0000 7fd6ea519700 1 --2- 172.21.15.77:0/2572840172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6e4107300 0x7fd6e41076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:10.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.526+0000 7fd6ea519700 1 --2- 172.21.15.77:0/2572840172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6e41014b0 0x7fd6e4101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:10.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.526+0000 7fd6ea519700 1 -- 172.21.15.77:0/2572840172 >> 172.21.15.77:0/2572840172 conn(0x7fd6e4078090 msgr2=0x7fd6e40784a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:10.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.526+0000 7fd6ea519700 1 -- 172.21.15.77:0/2572840172 shutdown_connections 2024-03-16T10:16:10.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.526+0000 7fd6ea519700 1 -- 172.21.15.77:0/2572840172 wait complete. 2024-03-16T10:16:10.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.527+0000 7fd6ea519700 1 Processor -- start 2024-03-16T10:16:10.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.527+0000 7fd6ea519700 1 -- start start 2024-03-16T10:16:10.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.528+0000 7fd6ea519700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6e41014b0 0x7fd6e419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:10.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.528+0000 7fd6ea519700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6e4101e70 0x7fd6e419ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:10.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.528+0000 7fd6ea519700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e4107300 0x7fd6e4198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:10.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.528+0000 7fd6e3fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6e41014b0 0x7fd6e419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:10.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.528+0000 7fd6e3fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6e41014b0 0x7fd6e419e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:55670/0 (socket says 172.21.15.77:55670) 2024-03-16T10:16:10.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.528+0000 7fd6e37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6e4101e70 0x7fd6e419ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:10.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.528+0000 7fd6e37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6e4101e70 0x7fd6e419ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33966/0 (socket says 172.21.15.77:33966) 2024-03-16T10:16:10.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.528+0000 7fd6e3fff700 1 -- 172.21.15.77:0/3980508564 learned_addr learned my addr 172.21.15.77:0/3980508564 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:10.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.528+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e41114d0 con 0x7fd6e4101e70 2024-03-16T10:16:10.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.528+0000 7fd6e8ab6700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e4107300 0x7fd6e4198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:10.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.529+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e4111350 con 0x7fd6e41014b0 2024-03-16T10:16:10.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.529+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e4111650 con 0x7fd6e4107300 2024-03-16T10:16:10.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.529+0000 7fd6e3fff700 1 -- 172.21.15.77:0/3980508564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e4107300 msgr2=0x7fd6e4198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:10.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.529+0000 7fd6e3fff700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e4107300 0x7fd6e4198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:10.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.529+0000 7fd6e3fff700 1 -- 172.21.15.77:0/3980508564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6e4101e70 msgr2=0x7fd6e419ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:10.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.530+0000 7fd6e3fff700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6e4101e70 0x7fd6e419ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:10.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.530+0000 7fd6e3fff700 1 -- 172.21.15.77:0/3980508564 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e4199010 con 0x7fd6e41014b0 2024-03-16T10:16:10.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.530+0000 7fd6e37fe700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6e4101e70 0x7fd6e419ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:10.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.531+0000 7fd6e3fff700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6e41014b0 0x7fd6e419e6b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400e9a0 tx=0x7fd6d400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:10.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.531+0000 7fd6e17fa700 1 -- 172.21.15.77:0/3980508564 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d400c9c0 con 0x7fd6e41014b0 2024-03-16T10:16:10.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.531+0000 7fd6e17fa700 1 -- 172.21.15.77:0/3980508564 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6d400cb60 con 0x7fd6e41014b0 2024-03-16T10:16:10.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.532+0000 7fd6e17fa700 1 -- 172.21.15.77:0/3980508564 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4018660 con 0x7fd6e41014b0 2024-03-16T10:16:10.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.532+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e4199300 con 0x7fd6e41014b0 2024-03-16T10:16:10.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.532+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e41a5510 con 0x7fd6e41014b0 2024-03-16T10:16:10.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.533+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e4102900 con 0x7fd6e41014b0 2024-03-16T10:16:10.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.534+0000 7fd6e17fa700 1 -- 172.21.15.77:0/3980508564 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd6d400cd10 con 0x7fd6e41014b0 2024-03-16T10:16:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.535+0000 7fd6e17fa700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd6cc0a29a0 0x7fd6cc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.535+0000 7fd6e37fe700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd6cc0a29a0 0x7fd6cc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.535+0000 7fd6e17fa700 1 -- 172.21.15.77:0/3980508564 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd6d40ceeb0 con 0x7fd6e41014b0 2024-03-16T10:16:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.536+0000 7fd6e37fe700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd6cc0a29a0 0x7fd6cc0a4e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd6e41013e0 tx=0x7fd6d8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:10.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.543+0000 7fd6e17fa700 1 -- 172.21.15.77:0/3980508564 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd6d409bde0 con 0x7fd6e41014b0 2024-03-16T10:16:10.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.767+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd6e41a56e0 con 0x7fd6e41014b0 2024-03-16T10:16:10.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.768+0000 7fd6e17fa700 1 -- 172.21.15.77:0/3980508564 <== mon.2 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd6d409b6e0 con 0x7fd6e41014b0 2024-03-16T10:16:10.768 INFO:teuthology.orchestra.run.smithi077.stdout:77309411401 2024-03-16T10:16:10.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.770+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd6cc0a29a0 msgr2=0x7fd6cc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:10.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.770+0000 7fd6ea519700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd6cc0a29a0 0x7fd6cc0a4e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd6e41013e0 tx=0x7fd6d8009310 comp rx=0 tx=0).stop 2024-03-16T10:16:10.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.770+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6e41014b0 msgr2=0x7fd6e419e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:10.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.770+0000 7fd6ea519700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6e41014b0 0x7fd6e419e6b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400e9a0 tx=0x7fd6d400ecb0 comp rx=0 tx=0).stop 2024-03-16T10:16:10.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.770+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 shutdown_connections 2024-03-16T10:16:10.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.770+0000 7fd6ea519700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd6cc0a29a0 0x7fd6cc0a4e60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:10.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.770+0000 7fd6ea519700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e4107300 0x7fd6e4198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:10.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.770+0000 7fd6ea519700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6e41014b0 0x7fd6e419e6b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:10.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.770+0000 7fd6ea519700 1 --2- 172.21.15.77:0/3980508564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6e4101e70 0x7fd6e419ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:10.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.770+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 >> 172.21.15.77:0/3980508564 conn(0x7fd6e4078090 msgr2=0x7fd6e40ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:10.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.770+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 shutdown_connections 2024-03-16T10:16:10.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:10.770+0000 7fd6ea519700 1 -- 172.21.15.77:0/3980508564 wait complete. 2024-03-16T10:16:11.017 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645338 got 309237645342 for osd.10 2024-03-16T10:16:11.017 DEBUG:teuthology.parallel:result is None 2024-03-16T10:16:11.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:10 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3980508564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T10:16:11.188 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:11.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:10 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3980508564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T10:16:11.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:10 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3980508564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T10:16:11.864 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:11 smithi077 ceph-mon[32400]: pgmap v323: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:11.894 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411396 got 77309411401 for osd.2 2024-03-16T10:16:11.894 DEBUG:teuthology.parallel:result is None 2024-03-16T10:16:12.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:11 smithi192 ceph-mon[36354]: pgmap v323: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:12.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:11 smithi193 ceph-mon[36311]: pgmap v323: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:13.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.666+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/2152407281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc88101b70 msgr2=0x7fbc8810e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:13.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.666+0000 7fbc8ca7d700 1 --2- 172.21.15.77:0/2152407281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc88101b70 0x7fbc8810e050 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000b0a0 tx=0x7fbc8000b3b0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.671 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.666+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/2152407281 shutdown_connections 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.666+0000 7fbc8ca7d700 1 --2- 172.21.15.77:0/2152407281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc88101b70 0x7fbc8810e050 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.666+0000 7fbc8ca7d700 1 --2- 172.21.15.77:0/2152407281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc88107210 0x7fbc881075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.666+0000 7fbc8ca7d700 1 --2- 172.21.15.77:0/2152407281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc881011d0 0x7fbc88101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.666+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/2152407281 >> 172.21.15.77:0/2152407281 conn(0x7fbc88078100 msgr2=0x7fbc88078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.666+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/2152407281 shutdown_connections 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.666+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/2152407281 wait complete. 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.667+0000 7fbc8ca7d700 1 Processor -- start 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.667+0000 7fbc8ca7d700 1 -- start start 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.667+0000 7fbc8ca7d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc881011d0 0x7fbc8819d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.667+0000 7fbc8ca7d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc88101b70 0x7fbc8819db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.667+0000 7fbc8ca7d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc88107210 0x7fbc881976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.667+0000 7fbc8ca7d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbc88111490 con 0x7fbc881011d0 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.667+0000 7fbc8ca7d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbc88111310 con 0x7fbc88101b70 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.667+0000 7fbc8ca7d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbc88111610 con 0x7fbc88107210 2024-03-16T10:16:13.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc8659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc881011d0 0x7fbc8819d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc8659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc881011d0 0x7fbc8819d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46766/0 (socket says 172.21.15.77:46766) 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc85d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc88101b70 0x7fbc8819db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc8659c700 1 -- 172.21.15.77:0/1167650815 learned_addr learned my addr 172.21.15.77:0/1167650815 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc86d9d700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc88107210 0x7fbc881976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc8659c700 1 -- 172.21.15.77:0/1167650815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc88107210 msgr2=0x7fbc881976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc8659c700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc88107210 0x7fbc881976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc8659c700 1 -- 172.21.15.77:0/1167650815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc88101b70 msgr2=0x7fbc8819db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc8659c700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc88101b70 0x7fbc8819db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc8659c700 1 -- 172.21.15.77:0/1167650815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc88197fd0 con 0x7fbc881011d0 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc8659c700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc881011d0 0x7fbc8819d5d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800e9c0 tx=0x7fbc7800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc777fe700 1 -- 172.21.15.77:0/1167650815 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7800cbd0 con 0x7fbc881011d0 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/1167650815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc881982c0 con 0x7fbc881011d0 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc777fe700 1 -- 172.21.15.77:0/1167650815 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc7800ee60 con 0x7fbc881011d0 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc777fe700 1 -- 172.21.15.77:0/1167650815 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc78005630 con 0x7fbc881011d0 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.668+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/1167650815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc881a4550 con 0x7fbc881011d0 2024-03-16T10:16:13.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.669+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/1167650815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc881028c0 con 0x7fbc881011d0 2024-03-16T10:16:13.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.671+0000 7fbc777fe700 1 -- 172.21.15.77:0/1167650815 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbc78004750 con 0x7fbc881011d0 2024-03-16T10:16:13.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.672+0000 7fbc777fe700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbc700a2920 0x7fbc700a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:13.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.672+0000 7fbc85d9b700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbc700a2920 0x7fbc700a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:13.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.672+0000 7fbc777fe700 1 -- 172.21.15.77:0/1167650815 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fbc780cef90 con 0x7fbc881011d0 2024-03-16T10:16:13.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.672+0000 7fbc85d9b700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbc700a2920 0x7fbc700a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbc88198e60 tx=0x7fbc7c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:13.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.675+0000 7fbc777fe700 1 -- 172.21.15.77:0/1167650815 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbc78053740 con 0x7fbc881011d0 2024-03-16T10:16:13.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.863+0000 7f118eb8b700 1 -- 172.21.15.77:0/82821012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11881011d0 msgr2=0x7f1188101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:13.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.863+0000 7f118eb8b700 1 --2- 172.21.15.77:0/82821012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11881011d0 0x7f1188101630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1174009990 tx=0x7f1174009ca0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.863+0000 7f118eb8b700 1 -- 172.21.15.77:0/82821012 shutdown_connections 2024-03-16T10:16:13.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.863+0000 7f118eb8b700 1 --2- 172.21.15.77:0/82821012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1188101b70 0x7f118810e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.863+0000 7f118eb8b700 1 --2- 172.21.15.77:0/82821012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1188107210 0x7f11881075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.863+0000 7f118eb8b700 1 --2- 172.21.15.77:0/82821012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11881011d0 0x7f1188101630 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.863+0000 7f118eb8b700 1 -- 172.21.15.77:0/82821012 >> 172.21.15.77:0/82821012 conn(0x7f1188078100 msgr2=0x7f1188078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:13.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.863+0000 7f118eb8b700 1 -- 172.21.15.77:0/82821012 shutdown_connections 2024-03-16T10:16:13.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.863+0000 7f118eb8b700 1 -- 172.21.15.77:0/82821012 wait complete. 2024-03-16T10:16:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118eb8b700 1 Processor -- start 2024-03-16T10:16:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118eb8b700 1 -- start start 2024-03-16T10:16:13.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118eb8b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11881011d0 0x7f11881a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118eb8b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1188101b70 0x7f11881a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118eb8b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1188107210 0x7f118819ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118eb8b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f11881114c0 con 0x7f1188107210 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118eb8b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1188111340 con 0x7f11881011d0 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118eb8b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1188111640 con 0x7f1188101b70 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118c927700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11881011d0 0x7f11881a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118c927700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11881011d0 0x7f11881a0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:47262/0 (socket says 172.21.15.77:47262) 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118c927700 1 -- 172.21.15.77:0/4178075616 learned_addr learned my addr 172.21.15.77:0/4178075616 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f1187fff700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1188101b70 0x7f11881a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118d128700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1188107210 0x7f118819ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118c927700 1 -- 172.21.15.77:0/4178075616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1188101b70 msgr2=0x7f11881a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118c927700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1188101b70 0x7f11881a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118c927700 1 -- 172.21.15.77:0/4178075616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1188107210 msgr2=0x7f118819ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118c927700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1188107210 0x7f118819ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.864+0000 7f118c927700 1 -- 172.21.15.77:0/4178075616 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f118819b410 con 0x7f11881011d0 2024-03-16T10:16:13.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.865+0000 7f118c927700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11881011d0 0x7f11881a0a60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f117c00eb30 tx=0x7f117c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:13.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.865+0000 7f1185ffb700 1 -- 172.21.15.77:0/4178075616 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c00cbd0 con 0x7f11881011d0 2024-03-16T10:16:13.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.865+0000 7f1185ffb700 1 -- 172.21.15.77:0/4178075616 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f117c004500 con 0x7f11881011d0 2024-03-16T10:16:13.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.865+0000 7f1185ffb700 1 -- 172.21.15.77:0/4178075616 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c010430 con 0x7f11881011d0 2024-03-16T10:16:13.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.865+0000 7f118eb8b700 1 -- 172.21.15.77:0/4178075616 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f118819b700 con 0x7f11881011d0 2024-03-16T10:16:13.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.865+0000 7f118eb8b700 1 -- 172.21.15.77:0/4178075616 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11881a79e0 con 0x7f11881011d0 2024-03-16T10:16:13.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.866+0000 7f118eb8b700 1 -- 172.21.15.77:0/4178075616 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11881028c0 con 0x7f11881011d0 2024-03-16T10:16:13.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.867+0000 7f1185ffb700 1 -- 172.21.15.77:0/4178075616 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f117c004020 con 0x7f11881011d0 2024-03-16T10:16:13.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.868+0000 7f1185ffb700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f11780a2920 0x7f11780a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:13.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.868+0000 7f1187fff700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f11780a2920 0x7f11780a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:13.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.868+0000 7f1185ffb700 1 -- 172.21.15.77:0/4178075616 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f117c0cec80 con 0x7f11881011d0 2024-03-16T10:16:13.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.868+0000 7f1187fff700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f11780a2920 0x7f11780a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f118819c280 tx=0x7f117400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:13.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.873+0000 7f1185ffb700 1 -- 172.21.15.77:0/4178075616 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f117c09bcb0 con 0x7f11881011d0 2024-03-16T10:16:13.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.926+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/1167650815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fbc88198bd0 con 0x7fbc881011d0 2024-03-16T10:16:13.927 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:13 smithi077 ceph-mon[32400]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:13.928 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.927+0000 7fbc777fe700 1 -- 172.21.15.77:0/1167650815 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbc7809bf40 con 0x7fbc881011d0 2024-03-16T10:16:13.928 INFO:teuthology.orchestra.run.smithi077.stdout:197568495670 2024-03-16T10:16:13.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.929+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/1167650815 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbc700a2920 msgr2=0x7fbc700a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:13.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.929+0000 7fbc8ca7d700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbc700a2920 0x7fbc700a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbc88198e60 tx=0x7fbc7c00a380 comp rx=0 tx=0).stop 2024-03-16T10:16:13.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.929+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/1167650815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc881011d0 msgr2=0x7fbc8819d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:13.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.929+0000 7fbc8ca7d700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc881011d0 0x7fbc8819d5d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800e9c0 tx=0x7fbc7800ecd0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.929+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/1167650815 shutdown_connections 2024-03-16T10:16:13.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.929+0000 7fbc8ca7d700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbc700a2920 0x7fbc700a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.929+0000 7fbc8ca7d700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc881011d0 0x7fbc8819d5d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.929+0000 7fbc8ca7d700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc88107210 0x7fbc881976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.929+0000 7fbc8ca7d700 1 --2- 172.21.15.77:0/1167650815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc88101b70 0x7fbc8819db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:13.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.929+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/1167650815 >> 172.21.15.77:0/1167650815 conn(0x7fbc88078100 msgr2=0x7fbc880ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:13.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.929+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/1167650815 shutdown_connections 2024-03-16T10:16:13.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:13.929+0000 7fbc8ca7d700 1 -- 172.21.15.77:0/1167650815 wait complete. 2024-03-16T10:16:14.054 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:13 smithi193 ceph-mon[36311]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:14.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.088+0000 7f01408b1700 1 -- 172.21.15.77:0/598850209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f013c106990 msgr2=0x7f013c106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.088+0000 7f01408b1700 1 --2- 172.21.15.77:0/598850209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f013c106990 0x7f013c106d70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f012c009990 tx=0x7f012c009ca0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.089+0000 7f01408b1700 1 -- 172.21.15.77:0/598850209 shutdown_connections 2024-03-16T10:16:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.089+0000 7f01408b1700 1 --2- 172.21.15.77:0/598850209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f013c106990 0x7f013c106d70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.089+0000 7f01408b1700 1 --2- 172.21.15.77:0/598850209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f013c101350 0x7f013c10d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.089+0000 7f01408b1700 1 --2- 172.21.15.77:0/598850209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f013c100990 0x7f013c100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.089+0000 7f01408b1700 1 -- 172.21.15.77:0/598850209 >> 172.21.15.77:0/598850209 conn(0x7f013c0777d0 msgr2=0x7f013c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.089+0000 7f01408b1700 1 -- 172.21.15.77:0/598850209 shutdown_connections 2024-03-16T10:16:14.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.089+0000 7f01408b1700 1 -- 172.21.15.77:0/598850209 wait complete. 2024-03-16T10:16:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.089+0000 7f01408b1700 1 Processor -- start 2024-03-16T10:16:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.089+0000 7f01408b1700 1 -- start start 2024-03-16T10:16:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f01408b1700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f013c100990 0x7f013c19e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f01408b1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f013c101350 0x7f013c19eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f01408b1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f013c106990 0x7f013c198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f01408b1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f013c1126a0 con 0x7f013c106990 2024-03-16T10:16:14.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f01408b1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f013c112520 con 0x7f013c101350 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f01408b1700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f013c112820 con 0x7f013c100990 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f013ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f013c106990 0x7f013c198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f0139d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f013c101350 0x7f013c19eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f013a59c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f013c100990 0x7f013c19e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f0139d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f013c101350 0x7f013c19eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:47284/0 (socket says 172.21.15.77:47284) 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f0139d9b700 1 -- 172.21.15.77:0/1303087385 learned_addr learned my addr 172.21.15.77:0/1303087385 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f013ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f013c106990 0x7f013c198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46814/0 (socket says 172.21.15.77:46814) 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f0139d9b700 1 -- 172.21.15.77:0/1303087385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f013c100990 msgr2=0x7f013c19e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f0139d9b700 1 --2- 172.21.15.77:0/1303087385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f013c100990 0x7f013c19e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f0139d9b700 1 -- 172.21.15.77:0/1303087385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f013c106990 msgr2=0x7f013c198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f0139d9b700 1 --2- 172.21.15.77:0/1303087385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f013c106990 0x7f013c198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.090+0000 7f0139d9b700 1 -- 172.21.15.77:0/1303087385 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013c199080 con 0x7f013c101350 2024-03-16T10:16:14.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.091+0000 7f0139d9b700 1 --2- 172.21.15.77:0/1303087385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f013c101350 0x7f013c19eca0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f013000e9a0 tx=0x7f013000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:14.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.091+0000 7f012b7fe700 1 -- 172.21.15.77:0/1303087385 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013000ca10 con 0x7f013c101350 2024-03-16T10:16:14.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.091+0000 7f01408b1700 1 -- 172.21.15.77:0/1303087385 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013c199370 con 0x7f013c101350 2024-03-16T10:16:14.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.091+0000 7f01408b1700 1 -- 172.21.15.77:0/1303087385 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f013c10f030 con 0x7f013c101350 2024-03-16T10:16:14.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.091+0000 7f012b7fe700 1 -- 172.21.15.77:0/1303087385 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f013000cbb0 con 0x7f013c101350 2024-03-16T10:16:14.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.091+0000 7f012b7fe700 1 -- 172.21.15.77:0/1303087385 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130018680 con 0x7f013c101350 2024-03-16T10:16:14.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.092+0000 7f01408b1700 1 -- 172.21.15.77:0/1303087385 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f013c1020d0 con 0x7f013c101350 2024-03-16T10:16:14.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.093+0000 7f012b7fe700 1 -- 172.21.15.77:0/1303087385 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f0130010430 con 0x7f013c101350 2024-03-16T10:16:14.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.094+0000 7f012b7fe700 1 --2- 172.21.15.77:0/1303087385 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f01240a29a0 0x7f01240a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:14.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.094+0000 7f012b7fe700 1 -- 172.21.15.77:0/1303087385 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f01300cecc0 con 0x7f013c101350 2024-03-16T10:16:14.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.094+0000 7f013a59c700 1 --2- 172.21.15.77:0/1303087385 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f01240a29a0 0x7f01240a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:14.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.094+0000 7f013a59c700 1 --2- 172.21.15.77:0/1303087385 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f01240a29a0 0x7f01240a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f013c199a60 tx=0x7f012c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:14.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.098+0000 7f012b7fe700 1 -- 172.21.15.77:0/1303087385 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f013009bcf0 con 0x7f013c101350 2024-03-16T10:16:14.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.104+0000 7f118eb8b700 1 -- 172.21.15.77:0/4178075616 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f118819c020 con 0x7f11881011d0 2024-03-16T10:16:14.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.104+0000 7f1185ffb700 1 -- 172.21.15.77:0/4178075616 <== mon.2 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f118819c020 con 0x7f11881011d0 2024-03-16T10:16:14.105 INFO:teuthology.orchestra.run.smithi077.stdout:253403070508 2024-03-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.107+0000 7f118eb8b700 1 -- 172.21.15.77:0/4178075616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f11780a2920 msgr2=0x7f11780a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.107+0000 7f118eb8b700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f11780a2920 0x7f11780a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f118819c280 tx=0x7f117400b260 comp rx=0 tx=0).stop 2024-03-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.107+0000 7f118eb8b700 1 -- 172.21.15.77:0/4178075616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11881011d0 msgr2=0x7f11881a0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.107+0000 7f118eb8b700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11881011d0 0x7f11881a0a60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f117c00eb30 tx=0x7f117c00ee40 comp rx=0 tx=0).stop 2024-03-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.107+0000 7f118eb8b700 1 -- 172.21.15.77:0/4178075616 shutdown_connections 2024-03-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.107+0000 7f118eb8b700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f11780a2920 0x7f11780a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.107+0000 7f118eb8b700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1188107210 0x7f118819ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.107+0000 7f118eb8b700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11881011d0 0x7f11881a0a60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.107+0000 7f118eb8b700 1 --2- 172.21.15.77:0/4178075616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1188101b70 0x7f11881a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.107+0000 7f118eb8b700 1 -- 172.21.15.77:0/4178075616 >> 172.21.15.77:0/4178075616 conn(0x7f1188078100 msgr2=0x7f1188069fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:14.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.107+0000 7f118eb8b700 1 -- 172.21.15.77:0/4178075616 shutdown_connections 2024-03-16T10:16:14.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.107+0000 7f118eb8b700 1 -- 172.21.15.77:0/4178075616 wait complete. 2024-03-16T10:16:14.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:13 smithi192 ceph-mon[36354]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:14.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.323+0000 7f01408b1700 1 -- 172.21.15.77:0/1303087385 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f013c199e50 con 0x7f013c101350 2024-03-16T10:16:14.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.324+0000 7f012b7fe700 1 -- 172.21.15.77:0/1303087385 <== mon.2 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f013009b5f0 con 0x7f013c101350 2024-03-16T10:16:14.324 INFO:teuthology.orchestra.run.smithi077.stdout:141733920831 2024-03-16T10:16:14.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.326+0000 7f01408b1700 1 -- 172.21.15.77:0/1303087385 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f01240a29a0 msgr2=0x7f01240a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:14.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.326+0000 7f01408b1700 1 --2- 172.21.15.77:0/1303087385 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f01240a29a0 0x7f01240a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f013c199a60 tx=0x7f012c01a040 comp rx=0 tx=0).stop 2024-03-16T10:16:14.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.326+0000 7f01408b1700 1 -- 172.21.15.77:0/1303087385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f013c101350 msgr2=0x7f013c19eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:14.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.326+0000 7f01408b1700 1 --2- 172.21.15.77:0/1303087385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f013c101350 0x7f013c19eca0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f013000e9a0 tx=0x7f013000ecb0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.326+0000 7f01408b1700 1 -- 172.21.15.77:0/1303087385 shutdown_connections 2024-03-16T10:16:14.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.326+0000 7f01408b1700 1 --2- 172.21.15.77:0/1303087385 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f01240a29a0 0x7f01240a4e60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.326+0000 7f01408b1700 1 --2- 172.21.15.77:0/1303087385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f013c106990 0x7f013c198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.326+0000 7f01408b1700 1 --2- 172.21.15.77:0/1303087385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f013c100990 0x7f013c19e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.326+0000 7f01408b1700 1 --2- 172.21.15.77:0/1303087385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f013c101350 0x7f013c19eca0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:14.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.326+0000 7f01408b1700 1 -- 172.21.15.77:0/1303087385 >> 172.21.15.77:0/1303087385 conn(0x7f013c0777d0 msgr2=0x7f013c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:14.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.326+0000 7f01408b1700 1 -- 172.21.15.77:0/1303087385 shutdown_connections 2024-03-16T10:16:14.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:14.326+0000 7f01408b1700 1 -- 172.21.15.77:0/1303087385 wait complete. 2024-03-16T10:16:15.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:14 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1167650815' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T10:16:15.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:14 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/4178075616' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T10:16:15.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:14 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1303087385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T10:16:15.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:14 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1167650815' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T10:16:15.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:14 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/4178075616' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T10:16:15.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:14 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1303087385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T10:16:15.187 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:14 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1167650815' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T10:16:15.187 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:14 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/4178075616' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T10:16:15.187 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:14 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1303087385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T10:16:15.214 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495664 got 197568495670 for osd.6 2024-03-16T10:16:15.214 DEBUG:teuthology.parallel:result is None 2024-03-16T10:16:15.656 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070502 got 253403070508 for osd.8 2024-03-16T10:16:15.657 DEBUG:teuthology.parallel:result is None 2024-03-16T10:16:16.084 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:15 smithi077 ceph-mon[32400]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:16.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:15 smithi192 ceph-mon[36354]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:16.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:15 smithi193 ceph-mon[36311]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:16.736 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920828 got 141733920831 for osd.4 2024-03-16T10:16:16.736 DEBUG:teuthology.parallel:result is None 2024-03-16T10:16:18.077 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:17 smithi077 ceph-mon[32400]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:18.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:17 smithi192 ceph-mon[36354]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:17 smithi193 ceph-mon[36311]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T10:16:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.350+0000 7ffa59d0e700 1 -- 172.21.15.77:0/3730121576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa54102510 msgr2=0x7ffa541028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.350+0000 7ffa59d0e700 1 --2- 172.21.15.77:0/3730121576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa54102510 0x7ffa541028f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ffa440099c0 tx=0x7ffa44009cd0 comp rx=0 tx=0).stop 2024-03-16T10:16:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.350+0000 7ffa59d0e700 1 -- 172.21.15.77:0/3730121576 shutdown_connections 2024-03-16T10:16:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.350+0000 7ffa59d0e700 1 --2- 172.21.15.77:0/3730121576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa54102510 0x7ffa541028f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.350+0000 7ffa59d0e700 1 --2- 172.21.15.77:0/3730121576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa541075f0 0x7ffa5410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:18.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.350+0000 7ffa59d0e700 1 --2- 172.21.15.77:0/3730121576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa54102ec0 0x7ffa54106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.350+0000 7ffa59d0e700 1 -- 172.21.15.77:0/3730121576 >> 172.21.15.77:0/3730121576 conn(0x7ffa540fdd80 msgr2=0x7ffa541001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.350+0000 7ffa59d0e700 1 -- 172.21.15.77:0/3730121576 shutdown_connections 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.350+0000 7ffa59d0e700 1 -- 172.21.15.77:0/3730121576 wait complete. 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.351+0000 7ffa59d0e700 1 Processor -- start 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.351+0000 7ffa59d0e700 1 -- start start 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.351+0000 7ffa59d0e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa54102510 0x7ffa540745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.351+0000 7ffa59d0e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa54102ec0 0x7ffa54074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.351+0000 7ffa59d0e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa541075f0 0x7ffa54075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.351+0000 7ffa59d0e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa541125f0 con 0x7ffa54102510 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.351+0000 7ffa59d0e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ffa54112470 con 0x7ffa541075f0 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.351+0000 7ffa59d0e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffa54112770 con 0x7ffa54102ec0 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa537fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa54102510 0x7ffa540745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:18.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa537fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa54102510 0x7ffa540745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46820/0 (socket says 172.21.15.77:46820) 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa537fe700 1 -- 172.21.15.77:0/2084965820 learned_addr learned my addr 172.21.15.77:0/2084965820 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa53fff700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa541075f0 0x7ffa54075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa52ffd700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa54102ec0 0x7ffa54074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa537fe700 1 -- 172.21.15.77:0/2084965820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa54102ec0 msgr2=0x7ffa54074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa537fe700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa54102ec0 0x7ffa54074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa537fe700 1 -- 172.21.15.77:0/2084965820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa541075f0 msgr2=0x7ffa54075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa537fe700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa541075f0 0x7ffa54075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa537fe700 1 -- 172.21.15.77:0/2084965820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa541a6fa0 con 0x7ffa54102510 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa53fff700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa541075f0 0x7ffa54075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa52ffd700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa54102ec0 0x7ffa54074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa537fe700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa54102510 0x7ffa540745d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffa44006010 tx=0x7ffa4400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa50ff9700 1 -- 172.21.15.77:0/2084965820 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa44003680 con 0x7ffa54102510 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa50ff9700 1 -- 172.21.15.77:0/2084965820 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa440050d0 con 0x7ffa54102510 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa50ff9700 1 -- 172.21.15.77:0/2084965820 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa44003db0 con 0x7ffa54102510 2024-03-16T10:16:18.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa59d0e700 1 -- 172.21.15.77:0/2084965820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa541a7230 con 0x7ffa54102510 2024-03-16T10:16:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.352+0000 7ffa59d0e700 1 -- 172.21.15.77:0/2084965820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa541a7630 con 0x7ffa54102510 2024-03-16T10:16:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.354+0000 7ffa59d0e700 1 -- 172.21.15.77:0/2084965820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa54068f90 con 0x7ffa54102510 2024-03-16T10:16:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.354+0000 7ffa50ff9700 1 -- 172.21.15.77:0/2084965820 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ffa44005280 con 0x7ffa54102510 2024-03-16T10:16:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.355+0000 7ffa50ff9700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ffa3c0a2ac0 0x7ffa3c0a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.355+0000 7ffa52ffd700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ffa3c0a2ac0 0x7ffa3c0a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.355+0000 7ffa50ff9700 1 -- 172.21.15.77:0/2084965820 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ffa440cf7c0 con 0x7ffa54102510 2024-03-16T10:16:18.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.355+0000 7ffa52ffd700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ffa3c0a2ac0 0x7ffa3c0a4f80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa54108bb0 tx=0x7ffa48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:18.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.360+0000 7ffa50ff9700 1 -- 172.21.15.77:0/2084965820 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffa44013070 con 0x7ffa54102510 2024-03-16T10:16:18.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.594+0000 7ffa59d0e700 1 -- 172.21.15.77:0/2084965820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ffa5402db70 con 0x7ffa54102510 2024-03-16T10:16:18.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.594+0000 7ffa50ff9700 1 -- 172.21.15.77:0/2084965820 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ffa4409c880 con 0x7ffa54102510 2024-03-16T10:16:18.595 INFO:teuthology.orchestra.run.smithi077.stdout:55834574927 2024-03-16T10:16:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.597+0000 7ffa59d0e700 1 -- 172.21.15.77:0/2084965820 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ffa3c0a2ac0 msgr2=0x7ffa3c0a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.597+0000 7ffa59d0e700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ffa3c0a2ac0 0x7ffa3c0a4f80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa54108bb0 tx=0x7ffa48009260 comp rx=0 tx=0).stop 2024-03-16T10:16:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.597+0000 7ffa59d0e700 1 -- 172.21.15.77:0/2084965820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa54102510 msgr2=0x7ffa540745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.597+0000 7ffa59d0e700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa54102510 0x7ffa540745d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffa44006010 tx=0x7ffa4400f690 comp rx=0 tx=0).stop 2024-03-16T10:16:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.597+0000 7ffa59d0e700 1 -- 172.21.15.77:0/2084965820 shutdown_connections 2024-03-16T10:16:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.597+0000 7ffa59d0e700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ffa3c0a2ac0 0x7ffa3c0a4f80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.598+0000 7ffa59d0e700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa54102510 0x7ffa540745d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.598+0000 7ffa59d0e700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa541075f0 0x7ffa54075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.598+0000 7ffa59d0e700 1 --2- 172.21.15.77:0/2084965820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa54102ec0 0x7ffa54074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.598+0000 7ffa59d0e700 1 -- 172.21.15.77:0/2084965820 >> 172.21.15.77:0/2084965820 conn(0x7ffa540fdd80 msgr2=0x7ffa54100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.598+0000 7ffa59d0e700 1 -- 172.21.15.77:0/2084965820 shutdown_connections 2024-03-16T10:16:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:18.598+0000 7ffa59d0e700 1 -- 172.21.15.77:0/2084965820 wait complete. 2024-03-16T10:16:18.973 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:18 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2084965820' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T10:16:18.974 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:18 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:18.974 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:18 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:19.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:18 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2084965820' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T10:16:19.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:18 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:19.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:18 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:19.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:18 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2084965820' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T10:16:19.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:18 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:18 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:19.272 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574921 got 55834574927 for osd.1 2024-03-16T10:16:19.272 DEBUG:teuthology.parallel:result is None 2024-03-16T10:16:19.272 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-16T10:16:19.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T10:16:19.535 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:19.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:19 smithi077 ceph-mon[32400]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T10:16:19.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:16:19.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:16:19.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:19 smithi192 ceph-mon[36354]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T10:16:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:16:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:16:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:19 smithi193 ceph-mon[36311]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T10:16:20.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:16:20.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:16:20.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:16:21.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.144+0000 7fd77573b700 1 -- 172.21.15.77:0/1172860246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd770102510 msgr2=0x7fd7701028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:21.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.144+0000 7fd77573b700 1 --2- 172.21.15.77:0/1172860246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd770102510 0x7fd7701028f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd7600099c0 tx=0x7fd760009cd0 comp rx=0 tx=0).stop 2024-03-16T10:16:21.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.144+0000 7fd77573b700 1 -- 172.21.15.77:0/1172860246 shutdown_connections 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.144+0000 7fd77573b700 1 --2- 172.21.15.77:0/1172860246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd770102510 0x7fd7701028f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.144+0000 7fd77573b700 1 --2- 172.21.15.77:0/1172860246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7701075f0 0x7fd77010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.144+0000 7fd77573b700 1 --2- 172.21.15.77:0/1172860246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd770102ec0 0x7fd770106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.144+0000 7fd77573b700 1 -- 172.21.15.77:0/1172860246 >> 172.21.15.77:0/1172860246 conn(0x7fd7700fdd80 msgr2=0x7fd7701001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.144+0000 7fd77573b700 1 -- 172.21.15.77:0/1172860246 shutdown_connections 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.144+0000 7fd77573b700 1 -- 172.21.15.77:0/1172860246 wait complete. 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd77573b700 1 Processor -- start 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd77573b700 1 -- start start 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd77573b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd770102510 0x7fd7701a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd77573b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd770102ec0 0x7fd7701a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd77573b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7701075f0 0x7fd77019aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd77573b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd7701125f0 con 0x7fd7701075f0 2024-03-16T10:16:21.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd77573b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd770112470 con 0x7fd770102510 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd77573b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd770112770 con 0x7fd770102ec0 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd76f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7701075f0 0x7fd77019aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd767fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd770102ec0 0x7fd7701a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd76f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7701075f0 0x7fd77019aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46826/0 (socket says 172.21.15.77:46826) 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd76effd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd770102510 0x7fd7701a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd76f7fe700 1 -- 172.21.15.77:0/1196932847 learned_addr learned my addr 172.21.15.77:0/1196932847 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd767fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd770102ec0 0x7fd7701a0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44298/0 (socket says 172.21.15.77:44298) 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd76f7fe700 1 -- 172.21.15.77:0/1196932847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd770102ec0 msgr2=0x7fd7701a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd76f7fe700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd770102ec0 0x7fd7701a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd76f7fe700 1 -- 172.21.15.77:0/1196932847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd770102510 msgr2=0x7fd7701a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd76f7fe700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd770102510 0x7fd7701a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd76f7fe700 1 -- 172.21.15.77:0/1196932847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77019b1d0 con 0x7fd7701075f0 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.145+0000 7fd76effd700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd770102510 0x7fd7701a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.146+0000 7fd76f7fe700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7701075f0 0x7fd77019aa70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd76800ef30 tx=0x7fd76800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:21.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.146+0000 7fd76cff9700 1 -- 172.21.15.77:0/1196932847 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7680092e0 con 0x7fd7701075f0 2024-03-16T10:16:21.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.146+0000 7fd76cff9700 1 -- 172.21.15.77:0/1196932847 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd768004bb0 con 0x7fd7701075f0 2024-03-16T10:16:21.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.146+0000 7fd76cff9700 1 -- 172.21.15.77:0/1196932847 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd768004280 con 0x7fd7701075f0 2024-03-16T10:16:21.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.146+0000 7fd77573b700 1 -- 172.21.15.77:0/1196932847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77019b460 con 0x7fd7701075f0 2024-03-16T10:16:21.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.146+0000 7fd77573b700 1 -- 172.21.15.77:0/1196932847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7701a7780 con 0x7fd7701075f0 2024-03-16T10:16:21.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.147+0000 7fd77573b700 1 -- 172.21.15.77:0/1196932847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd770068f90 con 0x7fd7701075f0 2024-03-16T10:16:21.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.148+0000 7fd76cff9700 1 -- 172.21.15.77:0/1196932847 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd76801b030 con 0x7fd7701075f0 2024-03-16T10:16:21.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.148+0000 7fd76cff9700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd75c0a29a0 0x7fd75c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:21.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.149+0000 7fd76effd700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd75c0a29a0 0x7fd75c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:21.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.149+0000 7fd76cff9700 1 -- 172.21.15.77:0/1196932847 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd7680ce470 con 0x7fd7701075f0 2024-03-16T10:16:21.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.149+0000 7fd76effd700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd75c0a29a0 0x7fd75c0a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd760000c00 tx=0x7fd760005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:21.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.154+0000 7fd76cff9700 1 -- 172.21.15.77:0/1196932847 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd768022020 con 0x7fd7701075f0 2024-03-16T10:16:21.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.353+0000 7fd77573b700 1 -- 172.21.15.77:0/1196932847 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd770075510 con 0x7fd75c0a29a0 2024-03-16T10:16:21.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.360+0000 7fd76cff9700 1 -- 172.21.15.77:0/1196932847 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258978 (secure 0 0 0) 0x7fd770075510 con 0x7fd75c0a29a0 2024-03-16T10:16:21.365 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:21.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.363+0000 7fd77573b700 1 -- 172.21.15.77:0/1196932847 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd75c0a29a0 msgr2=0x7fd75c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:21.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.363+0000 7fd77573b700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd75c0a29a0 0x7fd75c0a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd760000c00 tx=0x7fd760005fb0 comp rx=0 tx=0).stop 2024-03-16T10:16:21.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.363+0000 7fd77573b700 1 -- 172.21.15.77:0/1196932847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7701075f0 msgr2=0x7fd77019aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:21.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.363+0000 7fd77573b700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7701075f0 0x7fd77019aa70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd76800ef30 tx=0x7fd76800c3a0 comp rx=0 tx=0).stop 2024-03-16T10:16:21.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.364+0000 7fd77573b700 1 -- 172.21.15.77:0/1196932847 shutdown_connections 2024-03-16T10:16:21.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.364+0000 7fd77573b700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd75c0a29a0 0x7fd75c0a4e60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:21.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.364+0000 7fd77573b700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7701075f0 0x7fd77019aa70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:21.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.364+0000 7fd77573b700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd770102510 0x7fd7701a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:21.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.364+0000 7fd77573b700 1 --2- 172.21.15.77:0/1196932847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd770102ec0 0x7fd7701a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:21.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.364+0000 7fd77573b700 1 -- 172.21.15.77:0/1196932847 >> 172.21.15.77:0/1196932847 conn(0x7fd7700fdd80 msgr2=0x7fd7700feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:21.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.364+0000 7fd77573b700 1 -- 172.21.15.77:0/1196932847 shutdown_connections 2024-03-16T10:16:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:21.364+0000 7fd77573b700 1 -- 172.21.15.77:0/1196932847 wait complete. 2024-03-16T10:16:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T10:16:21.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:21 smithi077 ceph-mon[32400]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:22.022 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-03-16T10:16:20.733021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657438,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":864,"num_read_kb":555,"num_write":423,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":347,"ondisk_log_size":347,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360928,"kb_used_data":11700,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580896,"statfs":{"total":1151940427776,"available":1151570837504,"internally_reserved":0,"allocated":11980800,"data_stored":7437894,"data_compressed":112533,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":100,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002664"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.836343+0000","last_change":"2024-03-16T10:15:03.016880+0000","last_active":"2024-03-16T10:15:59.836343+0000","last_peered":"2024-03-16T10:15:59.836343+0000","last_clean":"2024-03-16T10:15:59.836343+0000","last_became_active":"2024-03-16T10:15:03.016513+0000","last_became_peered":"2024-03-16T10:15:03.016513+0000","last_unstale":"2024-03-16T10:15:59.836343+0000","last_undegraded":"2024-03-16T10:15:59.836343+0000","last_fullsized":"2024-03-16T10:15:59.836343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:39.834015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834492+0000","last_change":"2024-03-16T10:15:03.015467+0000","last_active":"2024-03-16T10:15:59.834492+0000","last_peered":"2024-03-16T10:15:59.834492+0000","last_clean":"2024-03-16T10:15:59.834492+0000","last_became_active":"2024-03-16T10:15:03.015152+0000","last_became_peered":"2024-03-16T10:15:03.015152+0000","last_unstale":"2024-03-16T10:15:59.834492+0000","last_undegraded":"2024-03-16T10:15:59.834492+0000","last_fullsized":"2024-03-16T10:15:59.834492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:17:29.988569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834256+0000","last_change":"2024-03-16T10:15:03.023836+0000","last_active":"2024-03-16T10:15:59.834256+0000","last_peered":"2024-03-16T10:15:59.834256+0000","last_clean":"2024-03-16T10:15:59.834256+0000","last_became_active":"2024-03-16T10:15:03.022774+0000","last_became_peered":"2024-03-16T10:15:03.022774+0000","last_unstale":"2024-03-16T10:15:59.834256+0000","last_undegraded":"2024-03-16T10:15:59.834256+0000","last_fullsized":"2024-03-16T10:15:59.834256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:49.837157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134667+0000","last_change":"2024-03-16T10:15:03.030891+0000","last_active":"2024-03-16T10:16:00.134667+0000","last_peered":"2024-03-16T10:16:00.134667+0000","last_clean":"2024-03-16T10:16:00.134667+0000","last_became_active":"2024-03-16T10:15:03.030278+0000","last_became_peered":"2024-03-16T10:15:03.030278+0000","last_unstale":"2024-03-16T10:16:00.134667+0000","last_undegraded":"2024-03-16T10:16:00.134667+0000","last_fullsized":"2024-03-16T10:16:00.134667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:16.166885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834382+0000","last_change":"2024-03-16T10:15:03.017610+0000","last_active":"2024-03-16T10:15:59.834382+0000","last_peered":"2024-03-16T10:15:59.834382+0000","last_clean":"2024-03-16T10:15:59.834382+0000","last_became_active":"2024-03-16T10:15:03.017250+0000","last_became_peered":"2024-03-16T10:15:03.017250+0000","last_unstale":"2024-03-16T10:15:59.834382+0000","last_undegraded":"2024-03-16T10:15:59.834382+0000","last_fullsized":"2024-03-16T10:15:59.834382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:07.899425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.856216+0000","last_change":"2024-03-16T10:15:59.856216+0000","last_active":"2024-03-16T10:15:59.856216+0000","last_peered":"2024-03-16T10:15:59.856216+0000","last_clean":"2024-03-16T10:15:59.856216+0000","last_became_active":"2024-03-16T10:15:59.855882+0000","last_became_peered":"2024-03-16T10:15:59.855882+0000","last_unstale":"2024-03-16T10:15:59.856216+0000","last_undegraded":"2024-03-16T10:15:59.856216+0000","last_fullsized":"2024-03-16T10:15:59.856216+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:27:42.682580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838614+0000","last_change":"2024-03-16T10:15:03.017686+0000","last_active":"2024-03-16T10:15:59.838614+0000","last_peered":"2024-03-16T10:15:59.838614+0000","last_clean":"2024-03-16T10:15:59.838614+0000","last_became_active":"2024-03-16T10:15:03.017262+0000","last_became_peered":"2024-03-16T10:15:03.017262+0000","last_unstale":"2024-03-16T10:15:59.838614+0000","last_undegraded":"2024-03-16T10:15:59.838614+0000","last_fullsized":"2024-03-16T10:15:59.838614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:36:22.270651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837763+0000","last_change":"2024-03-16T10:15:03.020130+0000","last_active":"2024-03-16T10:15:59.837763+0000","last_peered":"2024-03-16T10:15:59.837763+0000","last_clean":"2024-03-16T10:15:59.837763+0000","last_became_active":"2024-03-16T10:15:03.019742+0000","last_became_peered":"2024-03-16T10:15:03.019742+0000","last_unstale":"2024-03-16T10:15:59.837763+0000","last_undegraded":"2024-03-16T10:15:59.837763+0000","last_fullsized":"2024-03-16T10:15:59.837763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:52.450792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835751+0000","last_change":"2024-03-16T10:15:03.027589+0000","last_active":"2024-03-16T10:15:59.835751+0000","last_peered":"2024-03-16T10:15:59.835751+0000","last_clean":"2024-03-16T10:15:59.835751+0000","last_became_active":"2024-03-16T10:15:03.027226+0000","last_became_peered":"2024-03-16T10:15:03.027226+0000","last_unstale":"2024-03-16T10:15:59.835751+0000","last_undegraded":"2024-03-16T10:15:59.835751+0000","last_fullsized":"2024-03-16T10:15:59.835751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:14:35.354629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.842118+0000","last_change":"2024-03-16T10:15:03.021163+0000","last_active":"2024-03-16T10:15:59.842118+0000","last_peered":"2024-03-16T10:15:59.842118+0000","last_clean":"2024-03-16T10:15:59.842118+0000","last_became_active":"2024-03-16T10:15:03.020483+0000","last_became_peered":"2024-03-16T10:15:03.020483+0000","last_unstale":"2024-03-16T10:15:59.842118+0000","last_undegraded":"2024-03-16T10:15:59.842118+0000","last_fullsized":"2024-03-16T10:15:59.842118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:01:00.873471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839232+0000","last_change":"2024-03-16T10:15:03.026111+0000","last_active":"2024-03-16T10:15:59.839232+0000","last_peered":"2024-03-16T10:15:59.839232+0000","last_clean":"2024-03-16T10:15:59.839232+0000","last_became_active":"2024-03-16T10:15:03.025296+0000","last_became_peered":"2024-03-16T10:15:03.025296+0000","last_unstale":"2024-03-16T10:15:59.839232+0000","last_undegraded":"2024-03-16T10:15:59.839232+0000","last_fullsized":"2024-03-16T10:15:59.839232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:33:29.606876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134634+0000","last_change":"2024-03-16T10:15:03.023013+0000","last_active":"2024-03-16T10:16:00.134634+0000","last_peered":"2024-03-16T10:16:00.134634+0000","last_clean":"2024-03-16T10:16:00.134634+0000","last_became_active":"2024-03-16T10:15:03.022251+0000","last_became_peered":"2024-03-16T10:15:03.022251+0000","last_unstale":"2024-03-16T10:16:00.134634+0000","last_undegraded":"2024-03-16T10:16:00.134634+0000","last_fullsized":"2024-03-16T10:16:00.134634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:33.544709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835554+0000","last_change":"2024-03-16T10:15:03.009548+0000","last_active":"2024-03-16T10:15:59.835554+0000","last_peered":"2024-03-16T10:15:59.835554+0000","last_clean":"2024-03-16T10:15:59.835554+0000","last_became_active":"2024-03-16T10:15:03.007675+0000","last_became_peered":"2024-03-16T10:15:03.007675+0000","last_unstale":"2024-03-16T10:15:59.835554+0000","last_undegraded":"2024-03-16T10:15:59.835554+0000","last_fullsized":"2024-03-16T10:15:59.835554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:08:02.561602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.833958+0000","last_change":"2024-03-16T10:15:03.020146+0000","last_active":"2024-03-16T10:15:59.833958+0000","last_peered":"2024-03-16T10:15:59.833958+0000","last_clean":"2024-03-16T10:15:59.833958+0000","last_became_active":"2024-03-16T10:15:03.019823+0000","last_became_peered":"2024-03-16T10:15:03.019823+0000","last_unstale":"2024-03-16T10:15:59.833958+0000","last_undegraded":"2024-03-16T10:15:59.833958+0000","last_fullsized":"2024-03-16T10:15:59.833958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:52.276506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.833939+0000","last_change":"2024-03-16T10:15:03.019150+0000","last_active":"2024-03-16T10:15:59.833939+0000","last_peered":"2024-03-16T10:15:59.833939+0000","last_clean":"2024-03-16T10:15:59.833939+0000","last_became_active":"2024-03-16T10:15:03.018830+0000","last_became_peered":"2024-03-16T10:15:03.018830+0000","last_unstale":"2024-03-16T10:15:59.833939+0000","last_undegraded":"2024-03-16T10:15:59.833939+0000","last_fullsized":"2024-03-16T10:15:59.833939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:46.207187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835981+0000","last_change":"2024-03-16T10:15:03.033321+0000","last_active":"2024-03-16T10:15:59.835981+0000","last_peered":"2024-03-16T10:15:59.835981+0000","last_clean":"2024-03-16T10:15:59.835981+0000","last_became_active":"2024-03-16T10:15:03.032944+0000","last_became_peered":"2024-03-16T10:15:03.032944+0000","last_unstale":"2024-03-16T10:15:59.835981+0000","last_undegraded":"2024-03-16T10:15:59.835981+0000","last_fullsized":"2024-03-16T10:15:59.835981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:14.345488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.836582+0000","last_change":"2024-03-16T10:15:03.019897+0000","last_active":"2024-03-16T10:15:59.836582+0000","last_peered":"2024-03-16T10:15:59.836582+0000","last_clean":"2024-03-16T10:15:59.836582+0000","last_became_active":"2024-03-16T10:15:03.019555+0000","last_became_peered":"2024-03-16T10:15:03.019555+0000","last_unstale":"2024-03-16T10:15:59.836582+0000","last_undegraded":"2024-03-16T10:15:59.836582+0000","last_fullsized":"2024-03-16T10:15:59.836582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:58.307285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.855420+0000","last_change":"2024-03-16T10:15:03.017482+0000","last_active":"2024-03-16T10:15:59.855420+0000","last_peered":"2024-03-16T10:15:59.855420+0000","last_clean":"2024-03-16T10:15:59.855420+0000","last_became_active":"2024-03-16T10:15:03.016521+0000","last_became_peered":"2024-03-16T10:15:03.016521+0000","last_unstale":"2024-03-16T10:15:59.855420+0000","last_undegraded":"2024-03-16T10:15:59.855420+0000","last_fullsized":"2024-03-16T10:15:59.855420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:11.889111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.836040+0000","last_change":"2024-03-16T10:15:03.018625+0000","last_active":"2024-03-16T10:15:59.836040+0000","last_peered":"2024-03-16T10:15:59.836040+0000","last_clean":"2024-03-16T10:15:59.836040+0000","last_became_active":"2024-03-16T10:15:03.018313+0000","last_became_peered":"2024-03-16T10:15:03.018313+0000","last_unstale":"2024-03-16T10:15:59.836040+0000","last_undegraded":"2024-03-16T10:15:59.836040+0000","last_fullsized":"2024-03-16T10:15:59.836040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:35:58.533559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":103,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:17.311576+0000","last_change":"2024-03-16T10:15:02.016545+0000","last_active":"2024-03-16T10:16:17.311576+0000","last_peered":"2024-03-16T10:16:17.311576+0000","last_clean":"2024-03-16T10:16:17.311576+0000","last_became_active":"2024-03-16T10:15:02.012088+0000","last_became_peered":"2024-03-16T10:15:02.012088+0000","last_unstale":"2024-03-16T10:16:17.311576+0000","last_undegraded":"2024-03-16T10:16:17.311576+0000","last_fullsized":"2024-03-16T10:16:17.311576+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:50.633539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838214+0000","last_change":"2024-03-16T10:15:03.026565+0000","last_active":"2024-03-16T10:15:59.838214+0000","last_peered":"2024-03-16T10:15:59.838214+0000","last_clean":"2024-03-16T10:15:59.838214+0000","last_became_active":"2024-03-16T10:15:03.025386+0000","last_became_peered":"2024-03-16T10:15:03.025386+0000","last_unstale":"2024-03-16T10:15:59.838214+0000","last_undegraded":"2024-03-16T10:15:59.838214+0000","last_fullsized":"2024-03-16T10:15:59.838214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:28.831764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838304+0000","last_change":"2024-03-16T10:15:02.018097+0000","last_active":"2024-03-16T10:15:59.838304+0000","last_peered":"2024-03-16T10:15:59.838304+0000","last_clean":"2024-03-16T10:15:59.838304+0000","last_became_active":"2024-03-16T10:15:02.017777+0000","last_became_peered":"2024-03-16T10:15:02.017777+0000","last_unstale":"2024-03-16T10:15:59.838304+0000","last_undegraded":"2024-03-16T10:15:59.838304+0000","last_fullsized":"2024-03-16T10:15:59.838304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:04:58.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834600+0000","last_change":"2024-03-16T10:15:03.023384+0000","last_active":"2024-03-16T10:15:59.834600+0000","last_peered":"2024-03-16T10:15:59.834600+0000","last_clean":"2024-03-16T10:15:59.834600+0000","last_became_active":"2024-03-16T10:15:03.022996+0000","last_became_peered":"2024-03-16T10:15:03.022996+0000","last_unstale":"2024-03-16T10:15:59.834600+0000","last_undegraded":"2024-03-16T10:15:59.834600+0000","last_fullsized":"2024-03-16T10:15:59.834600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:29:20.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834807+0000","last_change":"2024-03-16T10:15:02.014409+0000","last_active":"2024-03-16T10:15:59.834807+0000","last_peered":"2024-03-16T10:15:59.834807+0000","last_clean":"2024-03-16T10:15:59.834807+0000","last_became_active":"2024-03-16T10:15:02.014015+0000","last_became_peered":"2024-03-16T10:15:02.014015+0000","last_unstale":"2024-03-16T10:15:59.834807+0000","last_undegraded":"2024-03-16T10:15:59.834807+0000","last_fullsized":"2024-03-16T10:15:59.834807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:25:53.133466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835418+0000","last_change":"2024-03-16T10:15:03.031328+0000","last_active":"2024-03-16T10:15:59.835418+0000","last_peered":"2024-03-16T10:15:59.835418+0000","last_clean":"2024-03-16T10:15:59.835418+0000","last_became_active":"2024-03-16T10:15:03.029787+0000","last_became_peered":"2024-03-16T10:15:03.029787+0000","last_unstale":"2024-03-16T10:15:59.835418+0000","last_undegraded":"2024-03-16T10:15:59.835418+0000","last_fullsized":"2024-03-16T10:15:59.835418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:34.321883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834594+0000","last_change":"2024-03-16T10:15:02.018032+0000","last_active":"2024-03-16T10:15:59.834594+0000","last_peered":"2024-03-16T10:15:59.834594+0000","last_clean":"2024-03-16T10:15:59.834594+0000","last_became_active":"2024-03-16T10:15:02.017677+0000","last_became_peered":"2024-03-16T10:15:02.017677+0000","last_unstale":"2024-03-16T10:15:59.834594+0000","last_undegraded":"2024-03-16T10:15:59.834594+0000","last_fullsized":"2024-03-16T10:15:59.834594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:58:07.179129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.899027+0000","last_change":"2024-03-16T10:15:03.024958+0000","last_active":"2024-03-16T10:15:59.899027+0000","last_peered":"2024-03-16T10:15:59.899027+0000","last_clean":"2024-03-16T10:15:59.899027+0000","last_became_active":"2024-03-16T10:15:03.023964+0000","last_became_peered":"2024-03-16T10:15:03.023964+0000","last_unstale":"2024-03-16T10:15:59.899027+0000","last_undegraded":"2024-03-16T10:15:59.899027+0000","last_fullsized":"2024-03-16T10:15:59.899027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:55:00.953258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":94,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:14.295573+0000","last_change":"2024-03-16T10:15:02.015062+0000","last_active":"2024-03-16T10:16:14.295573+0000","last_peered":"2024-03-16T10:16:14.295573+0000","last_clean":"2024-03-16T10:16:14.295573+0000","last_became_active":"2024-03-16T10:15:02.014714+0000","last_became_peered":"2024-03-16T10:15:02.014714+0000","last_unstale":"2024-03-16T10:16:14.295573+0000","last_undegraded":"2024-03-16T10:16:14.295573+0000","last_fullsized":"2024-03-16T10:16:14.295573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:23:55.313009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834337+0000","last_change":"2024-03-16T10:15:03.013956+0000","last_active":"2024-03-16T10:15:59.834337+0000","last_peered":"2024-03-16T10:15:59.834337+0000","last_clean":"2024-03-16T10:15:59.834337+0000","last_became_active":"2024-03-16T10:15:03.013607+0000","last_became_peered":"2024-03-16T10:15:03.013607+0000","last_unstale":"2024-03-16T10:15:59.834337+0000","last_undegraded":"2024-03-16T10:15:59.834337+0000","last_fullsized":"2024-03-16T10:15:59.834337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:40:55.445395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.841646+0000","last_change":"2024-03-16T10:15:02.015651+0000","last_active":"2024-03-16T10:15:59.841646+0000","last_peered":"2024-03-16T10:15:59.841646+0000","last_clean":"2024-03-16T10:15:59.841646+0000","last_became_active":"2024-03-16T10:15:02.015254+0000","last_became_peered":"2024-03-16T10:15:02.015254+0000","last_unstale":"2024-03-16T10:15:59.841646+0000","last_undegraded":"2024-03-16T10:15:59.841646+0000","last_fullsized":"2024-03-16T10:15:59.841646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:41:59.013606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898887+0000","last_change":"2024-03-16T10:15:03.023074+0000","last_active":"2024-03-16T10:15:59.898887+0000","last_peered":"2024-03-16T10:15:59.898887+0000","last_clean":"2024-03-16T10:15:59.898887+0000","last_became_active":"2024-03-16T10:15:03.022700+0000","last_became_peered":"2024-03-16T10:15:03.022700+0000","last_unstale":"2024-03-16T10:15:59.898887+0000","last_undegraded":"2024-03-16T10:15:59.898887+0000","last_fullsized":"2024-03-16T10:15:59.898887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:22.047929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835321+0000","last_change":"2024-03-16T10:15:02.014793+0000","last_active":"2024-03-16T10:15:59.835321+0000","last_peered":"2024-03-16T10:15:59.835321+0000","last_clean":"2024-03-16T10:15:59.835321+0000","last_became_active":"2024-03-16T10:15:02.014459+0000","last_became_peered":"2024-03-16T10:15:02.014459+0000","last_unstale":"2024-03-16T10:15:59.835321+0000","last_undegraded":"2024-03-16T10:15:59.835321+0000","last_fullsized":"2024-03-16T10:15:59.835321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:59:45.035045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837733+0000","last_change":"2024-03-16T10:15:03.012713+0000","last_active":"2024-03-16T10:15:59.837733+0000","last_peered":"2024-03-16T10:15:59.837733+0000","last_clean":"2024-03-16T10:15:59.837733+0000","last_became_active":"2024-03-16T10:15:03.012407+0000","last_became_peered":"2024-03-16T10:15:03.012407+0000","last_unstale":"2024-03-16T10:15:59.837733+0000","last_undegraded":"2024-03-16T10:15:59.837733+0000","last_fullsized":"2024-03-16T10:15:59.837733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:23:14.592810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898544+0000","last_change":"2024-03-16T10:15:02.015582+0000","last_active":"2024-03-16T10:15:59.898544+0000","last_peered":"2024-03-16T10:15:59.898544+0000","last_clean":"2024-03-16T10:15:59.898544+0000","last_became_active":"2024-03-16T10:15:02.015073+0000","last_became_peered":"2024-03-16T10:15:02.015073+0000","last_unstale":"2024-03-16T10:15:59.898544+0000","last_undegraded":"2024-03-16T10:15:59.898544+0000","last_fullsized":"2024-03-16T10:15:59.898544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:22:48.212448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.855302+0000","last_change":"2024-03-16T10:15:03.016839+0000","last_active":"2024-03-16T10:15:59.855302+0000","last_peered":"2024-03-16T10:15:59.855302+0000","last_clean":"2024-03-16T10:15:59.855302+0000","last_became_active":"2024-03-16T10:15:03.016463+0000","last_became_peered":"2024-03-16T10:15:03.016463+0000","last_unstale":"2024-03-16T10:15:59.855302+0000","last_undegraded":"2024-03-16T10:15:59.855302+0000","last_fullsized":"2024-03-16T10:15:59.855302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:43:34.259049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":165,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:18.299353+0000","last_change":"2024-03-16T10:15:02.017884+0000","last_active":"2024-03-16T10:16:18.299353+0000","last_peered":"2024-03-16T10:16:18.299353+0000","last_clean":"2024-03-16T10:16:18.299353+0000","last_became_active":"2024-03-16T10:15:02.017511+0000","last_became_peered":"2024-03-16T10:15:02.017511+0000","last_unstale":"2024-03-16T10:16:18.299353+0000","last_undegraded":"2024-03-16T10:16:18.299353+0000","last_fullsized":"2024-03-16T10:16:18.299353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:31.281631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":142,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835597+0000","last_change":"2024-03-16T10:15:03.032259+0000","last_active":"2024-03-16T10:15:59.835597+0000","last_peered":"2024-03-16T10:15:59.835597+0000","last_clean":"2024-03-16T10:15:59.835597+0000","last_became_active":"2024-03-16T10:15:03.031884+0000","last_became_peered":"2024-03-16T10:15:03.031884+0000","last_unstale":"2024-03-16T10:15:59.835597+0000","last_undegraded":"2024-03-16T10:15:59.835597+0000","last_fullsized":"2024-03-16T10:15:59.835597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:37:48.464051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839326+0000","last_change":"2024-03-16T10:15:02.015372+0000","last_active":"2024-03-16T10:15:59.839326+0000","last_peered":"2024-03-16T10:15:59.839326+0000","last_clean":"2024-03-16T10:15:59.839326+0000","last_became_active":"2024-03-16T10:15:02.015022+0000","last_became_peered":"2024-03-16T10:15:02.015022+0000","last_unstale":"2024-03-16T10:15:59.839326+0000","last_undegraded":"2024-03-16T10:15:59.839326+0000","last_fullsized":"2024-03-16T10:15:59.839326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:42.651903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834604+0000","last_change":"2024-03-16T10:15:03.018661+0000","last_active":"2024-03-16T10:15:59.834604+0000","last_peered":"2024-03-16T10:15:59.834604+0000","last_clean":"2024-03-16T10:15:59.834604+0000","last_became_active":"2024-03-16T10:15:03.018351+0000","last_became_peered":"2024-03-16T10:15:03.018351+0000","last_unstale":"2024-03-16T10:15:59.834604+0000","last_undegraded":"2024-03-16T10:15:59.834604+0000","last_fullsized":"2024-03-16T10:15:59.834604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:21:33.939811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.899049+0000","last_change":"2024-03-16T10:15:02.014343+0000","last_active":"2024-03-16T10:15:59.899049+0000","last_peered":"2024-03-16T10:15:59.899049+0000","last_clean":"2024-03-16T10:15:59.899049+0000","last_became_active":"2024-03-16T10:15:02.013983+0000","last_became_peered":"2024-03-16T10:15:02.013983+0000","last_unstale":"2024-03-16T10:15:59.899049+0000","last_undegraded":"2024-03-16T10:15:59.899049+0000","last_fullsized":"2024-03-16T10:15:59.899049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:35:04.768079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835549+0000","last_change":"2024-03-16T10:15:03.031055+0000","last_active":"2024-03-16T10:15:59.835549+0000","last_peered":"2024-03-16T10:15:59.835549+0000","last_clean":"2024-03-16T10:15:59.835549+0000","last_became_active":"2024-03-16T10:15:03.029656+0000","last_became_peered":"2024-03-16T10:15:03.029656+0000","last_unstale":"2024-03-16T10:15:59.835549+0000","last_undegraded":"2024-03-16T10:15:59.835549+0000","last_fullsized":"2024-03-16T10:15:59.835549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:27:35.684774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'32","reported_seq":202,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:18.301781+0000","last_change":"2024-03-16T10:15:02.014752+0000","last_active":"2024-03-16T10:16:18.301781+0000","last_peered":"2024-03-16T10:16:18.301781+0000","last_clean":"2024-03-16T10:16:18.301781+0000","last_became_active":"2024-03-16T10:15:02.013763+0000","last_became_peered":"2024-03-16T10:15:02.013763+0000","last_unstale":"2024-03-16T10:16:18.301781+0000","last_undegraded":"2024-03-16T10:16:18.301781+0000","last_fullsized":"2024-03-16T10:16:18.301781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:12:12.759643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60023,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":151,"num_read_kb":135,"num_write":34,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834673+0000","last_change":"2024-03-16T10:15:03.018838+0000","last_active":"2024-03-16T10:15:59.834673+0000","last_peered":"2024-03-16T10:15:59.834673+0000","last_clean":"2024-03-16T10:15:59.834673+0000","last_became_active":"2024-03-16T10:15:03.017713+0000","last_became_peered":"2024-03-16T10:15:03.017713+0000","last_unstale":"2024-03-16T10:15:59.834673+0000","last_undegraded":"2024-03-16T10:15:59.834673+0000","last_fullsized":"2024-03-16T10:15:59.834673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:03:56.936729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835759+0000","last_change":"2024-03-16T10:15:02.016996+0000","last_active":"2024-03-16T10:15:59.835759+0000","last_peered":"2024-03-16T10:15:59.835759+0000","last_clean":"2024-03-16T10:15:59.835759+0000","last_became_active":"2024-03-16T10:15:02.016421+0000","last_became_peered":"2024-03-16T10:15:02.016421+0000","last_unstale":"2024-03-16T10:15:59.835759+0000","last_undegraded":"2024-03-16T10:15:59.835759+0000","last_fullsized":"2024-03-16T10:15:59.835759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:25:09.866730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835486+0000","last_change":"2024-03-16T10:15:03.017842+0000","last_active":"2024-03-16T10:15:59.835486+0000","last_peered":"2024-03-16T10:15:59.835486+0000","last_clean":"2024-03-16T10:15:59.835486+0000","last_became_active":"2024-03-16T10:15:03.015119+0000","last_became_peered":"2024-03-16T10:15:03.015119+0000","last_unstale":"2024-03-16T10:15:59.835486+0000","last_undegraded":"2024-03-16T10:15:59.835486+0000","last_fullsized":"2024-03-16T10:15:59.835486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:14:34.474519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898795+0000","last_change":"2024-03-16T10:15:02.021407+0000","last_active":"2024-03-16T10:15:59.898795+0000","last_peered":"2024-03-16T10:15:59.898795+0000","last_clean":"2024-03-16T10:15:59.898795+0000","last_became_active":"2024-03-16T10:15:02.020884+0000","last_became_peered":"2024-03-16T10:15:02.020884+0000","last_unstale":"2024-03-16T10:15:59.898795+0000","last_undegraded":"2024-03-16T10:15:59.898795+0000","last_fullsized":"2024-03-16T10:15:59.898795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:24.858281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838848+0000","last_change":"2024-03-16T10:15:03.028607+0000","last_active":"2024-03-16T10:15:59.838848+0000","last_peered":"2024-03-16T10:15:59.838848+0000","last_clean":"2024-03-16T10:15:59.838848+0000","last_became_active":"2024-03-16T10:15:03.028280+0000","last_became_peered":"2024-03-16T10:15:03.028280+0000","last_unstale":"2024-03-16T10:15:59.838848+0000","last_undegraded":"2024-03-16T10:15:59.838848+0000","last_fullsized":"2024-03-16T10:15:59.838848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:39.905921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'5","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:16.307079+0000","last_change":"2024-03-16T10:15:02.008558+0000","last_active":"2024-03-16T10:16:16.307079+0000","last_peered":"2024-03-16T10:16:16.307079+0000","last_clean":"2024-03-16T10:16:16.307079+0000","last_became_active":"2024-03-16T10:15:02.008014+0000","last_became_peered":"2024-03-16T10:15:02.008014+0000","last_unstale":"2024-03-16T10:16:16.307079+0000","last_undegraded":"2024-03-16T10:16:16.307079+0000","last_fullsized":"2024-03-16T10:16:16.307079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:40:09.217991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838543+0000","last_change":"2024-03-16T10:15:03.020847+0000","last_active":"2024-03-16T10:15:59.838543+0000","last_peered":"2024-03-16T10:15:59.838543+0000","last_clean":"2024-03-16T10:15:59.838543+0000","last_became_active":"2024-03-16T10:15:03.020512+0000","last_became_peered":"2024-03-16T10:15:03.020512+0000","last_unstale":"2024-03-16T10:15:59.838543+0000","last_undegraded":"2024-03-16T10:15:59.838543+0000","last_fullsized":"2024-03-16T10:15:59.838543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:04.436195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834281+0000","last_change":"2024-03-16T10:15:02.010690+0000","last_active":"2024-03-16T10:15:59.834281+0000","last_peered":"2024-03-16T10:15:59.834281+0000","last_clean":"2024-03-16T10:15:59.834281+0000","last_became_active":"2024-03-16T10:15:02.010228+0000","last_became_peered":"2024-03-16T10:15:02.010228+0000","last_unstale":"2024-03-16T10:15:59.834281+0000","last_undegraded":"2024-03-16T10:15:59.834281+0000","last_fullsized":"2024-03-16T10:15:59.834281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:30:07.513225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898533+0000","last_change":"2024-03-16T10:15:03.025257+0000","last_active":"2024-03-16T10:15:59.898533+0000","last_peered":"2024-03-16T10:15:59.898533+0000","last_clean":"2024-03-16T10:15:59.898533+0000","last_became_active":"2024-03-16T10:15:03.024498+0000","last_became_peered":"2024-03-16T10:15:03.024498+0000","last_unstale":"2024-03-16T10:15:59.898533+0000","last_undegraded":"2024-03-16T10:15:59.898533+0000","last_fullsized":"2024-03-16T10:15:59.898533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:27:55.894728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834712+0000","last_change":"2024-03-16T10:15:02.014136+0000","last_active":"2024-03-16T10:15:59.834712+0000","last_peered":"2024-03-16T10:15:59.834712+0000","last_clean":"2024-03-16T10:15:59.834712+0000","last_became_active":"2024-03-16T10:15:02.013721+0000","last_became_peered":"2024-03-16T10:15:02.013721+0000","last_unstale":"2024-03-16T10:15:59.834712+0000","last_undegraded":"2024-03-16T10:15:59.834712+0000","last_fullsized":"2024-03-16T10:15:59.834712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:28:12.301450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.856566+0000","last_change":"2024-03-16T10:15:59.856566+0000","last_active":"2024-03-16T10:15:59.856566+0000","last_peered":"2024-03-16T10:15:59.856566+0000","last_clean":"2024-03-16T10:15:59.856566+0000","last_became_active":"2024-03-16T10:15:59.856152+0000","last_became_peered":"2024-03-16T10:15:59.856152+0000","last_unstale":"2024-03-16T10:15:59.856566+0000","last_undegraded":"2024-03-16T10:15:59.856566+0000","last_fullsized":"2024-03-16T10:15:59.856566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:42:39.104928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.855367+0000","last_change":"2024-03-16T10:15:02.005754+0000","last_active":"2024-03-16T10:15:59.855367+0000","last_peered":"2024-03-16T10:15:59.855367+0000","last_clean":"2024-03-16T10:15:59.855367+0000","last_became_active":"2024-03-16T10:15:02.005344+0000","last_became_peered":"2024-03-16T10:15:02.005344+0000","last_unstale":"2024-03-16T10:15:59.855367+0000","last_undegraded":"2024-03-16T10:15:59.855367+0000","last_fullsized":"2024-03-16T10:15:59.855367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:37:37.918861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.842019+0000","last_change":"2024-03-16T10:15:03.012887+0000","last_active":"2024-03-16T10:15:59.842019+0000","last_peered":"2024-03-16T10:15:59.842019+0000","last_clean":"2024-03-16T10:15:59.842019+0000","last_became_active":"2024-03-16T10:15:03.012496+0000","last_became_peered":"2024-03-16T10:15:03.012496+0000","last_unstale":"2024-03-16T10:15:59.842019+0000","last_undegraded":"2024-03-16T10:15:59.842019+0000","last_fullsized":"2024-03-16T10:15:59.842019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:11.111049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":93,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:15.309976+0000","last_change":"2024-03-16T10:15:02.014334+0000","last_active":"2024-03-16T10:16:15.309976+0000","last_peered":"2024-03-16T10:16:15.309976+0000","last_clean":"2024-03-16T10:16:15.309976+0000","last_became_active":"2024-03-16T10:15:02.013832+0000","last_became_peered":"2024-03-16T10:15:02.013832+0000","last_unstale":"2024-03-16T10:16:15.309976+0000","last_undegraded":"2024-03-16T10:16:15.309976+0000","last_fullsized":"2024-03-16T10:16:15.309976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:20.391594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.841895+0000","last_change":"2024-03-16T10:15:03.021276+0000","last_active":"2024-03-16T10:15:59.841895+0000","last_peered":"2024-03-16T10:15:59.841895+0000","last_clean":"2024-03-16T10:15:59.841895+0000","last_became_active":"2024-03-16T10:15:03.020927+0000","last_became_peered":"2024-03-16T10:15:03.020927+0000","last_unstale":"2024-03-16T10:15:59.841895+0000","last_undegraded":"2024-03-16T10:15:59.841895+0000","last_fullsized":"2024-03-16T10:15:59.841895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:53:54.127506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898661+0000","last_change":"2024-03-16T10:15:03.022083+0000","last_active":"2024-03-16T10:15:59.898661+0000","last_peered":"2024-03-16T10:15:59.898661+0000","last_clean":"2024-03-16T10:15:59.898661+0000","last_became_active":"2024-03-16T10:15:03.021609+0000","last_became_peered":"2024-03-16T10:15:03.021609+0000","last_unstale":"2024-03-16T10:15:59.898661+0000","last_undegraded":"2024-03-16T10:15:59.898661+0000","last_fullsized":"2024-03-16T10:15:59.898661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:53.864402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":94,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:15.305620+0000","last_change":"2024-03-16T10:15:02.015497+0000","last_active":"2024-03-16T10:16:15.305620+0000","last_peered":"2024-03-16T10:16:15.305620+0000","last_clean":"2024-03-16T10:16:15.305620+0000","last_became_active":"2024-03-16T10:15:02.015183+0000","last_became_peered":"2024-03-16T10:15:02.015183+0000","last_unstale":"2024-03-16T10:16:15.305620+0000","last_undegraded":"2024-03-16T10:16:15.305620+0000","last_fullsized":"2024-03-16T10:16:15.305620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:44:05.913030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838082+0000","last_change":"2024-03-16T10:15:03.020046+0000","last_active":"2024-03-16T10:15:59.838082+0000","last_peered":"2024-03-16T10:15:59.838082+0000","last_clean":"2024-03-16T10:15:59.838082+0000","last_became_active":"2024-03-16T10:15:03.019647+0000","last_became_peered":"2024-03-16T10:15:03.019647+0000","last_unstale":"2024-03-16T10:15:59.838082+0000","last_undegraded":"2024-03-16T10:15:59.838082+0000","last_fullsized":"2024-03-16T10:15:59.838082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:18.343495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134445+0000","last_change":"2024-03-16T10:15:02.008560+0000","last_active":"2024-03-16T10:16:00.134445+0000","last_peered":"2024-03-16T10:16:00.134445+0000","last_clean":"2024-03-16T10:16:00.134445+0000","last_became_active":"2024-03-16T10:15:02.008094+0000","last_became_peered":"2024-03-16T10:15:02.008094+0000","last_unstale":"2024-03-16T10:16:00.134445+0000","last_undegraded":"2024-03-16T10:16:00.134445+0000","last_fullsized":"2024-03-16T10:16:00.134445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:21:43.555120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":341,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838644+0000","last_change":"2024-03-16T10:13:47.211815+0000","last_active":"2024-03-16T10:15:59.838644+0000","last_peered":"2024-03-16T10:15:59.838644+0000","last_clean":"2024-03-16T10:15:59.838644+0000","last_became_active":"2024-03-16T10:13:47.211353+0000","last_became_peered":"2024-03-16T10:13:47.211353+0000","last_unstale":"2024-03-16T10:15:59.838644+0000","last_undegraded":"2024-03-16T10:15:59.838644+0000","last_fullsized":"2024-03-16T10:15:59.838644+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_clean_scrub_stamp":"2024-03-16T10:10:22.965421+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:11:20.607686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835188+0000","last_change":"2024-03-16T10:15:03.020181+0000","last_active":"2024-03-16T10:15:59.835188+0000","last_peered":"2024-03-16T10:15:59.835188+0000","last_clean":"2024-03-16T10:15:59.835188+0000","last_became_active":"2024-03-16T10:15:03.019781+0000","last_became_peered":"2024-03-16T10:15:03.019781+0000","last_unstale":"2024-03-16T10:15:59.835188+0000","last_undegraded":"2024-03-16T10:15:59.835188+0000","last_fullsized":"2024-03-16T10:15:59.835188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:04:49.369484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838788+0000","last_change":"2024-03-16T10:15:02.014461+0000","last_active":"2024-03-16T10:15:59.838788+0000","last_peered":"2024-03-16T10:15:59.838788+0000","last_clean":"2024-03-16T10:15:59.838788+0000","last_became_active":"2024-03-16T10:15:02.014137+0000","last_became_peered":"2024-03-16T10:15:02.014137+0000","last_unstale":"2024-03-16T10:15:59.838788+0000","last_undegraded":"2024-03-16T10:15:59.838788+0000","last_fullsized":"2024-03-16T10:15:59.838788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:36:41.830893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834705+0000","last_change":"2024-03-16T10:15:02.019441+0000","last_active":"2024-03-16T10:15:59.834705+0000","last_peered":"2024-03-16T10:15:59.834705+0000","last_clean":"2024-03-16T10:15:59.834705+0000","last_became_active":"2024-03-16T10:15:02.019091+0000","last_became_peered":"2024-03-16T10:15:02.019091+0000","last_unstale":"2024-03-16T10:15:59.834705+0000","last_undegraded":"2024-03-16T10:15:59.834705+0000","last_fullsized":"2024-03-16T10:15:59.834705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:04:51.758722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835787+0000","last_change":"2024-03-16T10:15:03.012141+0000","last_active":"2024-03-16T10:15:59.835787+0000","last_peered":"2024-03-16T10:15:59.835787+0000","last_clean":"2024-03-16T10:15:59.835787+0000","last_became_active":"2024-03-16T10:15:03.011771+0000","last_became_peered":"2024-03-16T10:15:03.011771+0000","last_unstale":"2024-03-16T10:15:59.835787+0000","last_undegraded":"2024-03-16T10:15:59.835787+0000","last_fullsized":"2024-03-16T10:15:59.835787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:02:28.522530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'13","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.855528+0000","last_change":"2024-03-16T10:15:02.011898+0000","last_active":"2024-03-16T10:15:59.855528+0000","last_peered":"2024-03-16T10:15:59.855528+0000","last_clean":"2024-03-16T10:15:59.855528+0000","last_became_active":"2024-03-16T10:15:02.011578+0000","last_became_peered":"2024-03-16T10:15:02.011578+0000","last_unstale":"2024-03-16T10:15:59.855528+0000","last_undegraded":"2024-03-16T10:15:59.855528+0000","last_fullsized":"2024-03-16T10:15:59.855528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:27:35.756031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838261+0000","last_change":"2024-03-16T10:15:03.024923+0000","last_active":"2024-03-16T10:15:59.838261+0000","last_peered":"2024-03-16T10:15:59.838261+0000","last_clean":"2024-03-16T10:15:59.838261+0000","last_became_active":"2024-03-16T10:15:03.024577+0000","last_became_peered":"2024-03-16T10:15:03.024577+0000","last_unstale":"2024-03-16T10:15:59.838261+0000","last_undegraded":"2024-03-16T10:15:59.838261+0000","last_fullsized":"2024-03-16T10:15:59.838261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:04.650499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134719+0000","last_change":"2024-03-16T10:15:02.006842+0000","last_active":"2024-03-16T10:16:00.134719+0000","last_peered":"2024-03-16T10:16:00.134719+0000","last_clean":"2024-03-16T10:16:00.134719+0000","last_became_active":"2024-03-16T10:15:02.006483+0000","last_became_peered":"2024-03-16T10:15:02.006483+0000","last_unstale":"2024-03-16T10:16:00.134719+0000","last_undegraded":"2024-03-16T10:16:00.134719+0000","last_fullsized":"2024-03-16T10:16:00.134719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:30:55.856007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834392+0000","last_change":"2024-03-16T10:15:03.018691+0000","last_active":"2024-03-16T10:15:59.834392+0000","last_peered":"2024-03-16T10:15:59.834392+0000","last_clean":"2024-03-16T10:15:59.834392+0000","last_became_active":"2024-03-16T10:15:03.018304+0000","last_became_peered":"2024-03-16T10:15:03.018304+0000","last_unstale":"2024-03-16T10:15:59.834392+0000","last_undegraded":"2024-03-16T10:15:59.834392+0000","last_fullsized":"2024-03-16T10:15:59.834392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:15.446759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.841306+0000","last_change":"2024-03-16T10:15:59.841306+0000","last_active":"2024-03-16T10:15:59.841306+0000","last_peered":"2024-03-16T10:15:59.841306+0000","last_clean":"2024-03-16T10:15:59.841306+0000","last_became_active":"2024-03-16T10:15:59.838434+0000","last_became_peered":"2024-03-16T10:15:59.838434+0000","last_unstale":"2024-03-16T10:15:59.841306+0000","last_undegraded":"2024-03-16T10:15:59.841306+0000","last_fullsized":"2024-03-16T10:15:59.841306+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:14:21.702809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834199+0000","last_change":"2024-03-16T10:15:03.017262+0000","last_active":"2024-03-16T10:15:59.834199+0000","last_peered":"2024-03-16T10:15:59.834199+0000","last_clean":"2024-03-16T10:15:59.834199+0000","last_became_active":"2024-03-16T10:15:03.016909+0000","last_became_peered":"2024-03-16T10:15:03.016909+0000","last_unstale":"2024-03-16T10:15:59.834199+0000","last_undegraded":"2024-03-16T10:15:59.834199+0000","last_fullsized":"2024-03-16T10:15:59.834199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:53:11.385236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837856+0000","last_change":"2024-03-16T10:15:02.011325+0000","last_active":"2024-03-16T10:15:59.837856+0000","last_peered":"2024-03-16T10:15:59.837856+0000","last_clean":"2024-03-16T10:15:59.837856+0000","last_became_active":"2024-03-16T10:15:02.010999+0000","last_became_peered":"2024-03-16T10:15:02.010999+0000","last_unstale":"2024-03-16T10:15:59.837856+0000","last_undegraded":"2024-03-16T10:15:59.837856+0000","last_fullsized":"2024-03-16T10:15:59.837856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:30:08.137693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835889+0000","last_change":"2024-03-16T10:15:03.012146+0000","last_active":"2024-03-16T10:15:59.835889+0000","last_peered":"2024-03-16T10:15:59.835889+0000","last_clean":"2024-03-16T10:15:59.835889+0000","last_became_active":"2024-03-16T10:15:03.011790+0000","last_became_peered":"2024-03-16T10:15:03.011790+0000","last_unstale":"2024-03-16T10:15:59.835889+0000","last_undegraded":"2024-03-16T10:15:59.835889+0000","last_fullsized":"2024-03-16T10:15:59.835889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:45:22.642925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.842829+0000","last_change":"2024-03-16T10:15:59.842829+0000","last_active":"2024-03-16T10:15:59.842829+0000","last_peered":"2024-03-16T10:15:59.842829+0000","last_clean":"2024-03-16T10:15:59.842829+0000","last_became_active":"2024-03-16T10:15:59.838787+0000","last_became_peered":"2024-03-16T10:15:59.838787+0000","last_unstale":"2024-03-16T10:15:59.842829+0000","last_undegraded":"2024-03-16T10:15:59.842829+0000","last_fullsized":"2024-03-16T10:15:59.842829+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:47:27.573657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838065+0000","last_change":"2024-03-16T10:15:03.021094+0000","last_active":"2024-03-16T10:15:59.838065+0000","last_peered":"2024-03-16T10:15:59.838065+0000","last_clean":"2024-03-16T10:15:59.838065+0000","last_became_active":"2024-03-16T10:15:03.017640+0000","last_became_peered":"2024-03-16T10:15:03.017640+0000","last_unstale":"2024-03-16T10:15:59.838065+0000","last_undegraded":"2024-03-16T10:15:59.838065+0000","last_fullsized":"2024-03-16T10:15:59.838065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:02.030112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834498+0000","last_change":"2024-03-16T10:15:02.015652+0000","last_active":"2024-03-16T10:15:59.834498+0000","last_peered":"2024-03-16T10:15:59.834498+0000","last_clean":"2024-03-16T10:15:59.834498+0000","last_became_active":"2024-03-16T10:15:02.014046+0000","last_became_peered":"2024-03-16T10:15:02.014046+0000","last_unstale":"2024-03-16T10:15:59.834498+0000","last_undegraded":"2024-03-16T10:15:59.834498+0000","last_fullsized":"2024-03-16T10:15:59.834498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839375+0000","last_change":"2024-03-16T10:15:03.023233+0000","last_active":"2024-03-16T10:15:59.839375+0000","last_peered":"2024-03-16T10:15:59.839375+0000","last_clean":"2024-03-16T10:15:59.839375+0000","last_became_active":"2024-03-16T10:15:03.022906+0000","last_became_peered":"2024-03-16T10:15:03.022906+0000","last_unstale":"2024-03-16T10:15:59.839375+0000","last_undegraded":"2024-03-16T10:15:59.839375+0000","last_fullsized":"2024-03-16T10:15:59.839375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:22.943087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":97,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:16.298015+0000","last_change":"2024-03-16T10:15:02.010755+0000","last_active":"2024-03-16T10:16:16.298015+0000","last_peered":"2024-03-16T10:16:16.298015+0000","last_clean":"2024-03-16T10:16:16.298015+0000","last_became_active":"2024-03-16T10:15:02.010422+0000","last_became_peered":"2024-03-16T10:15:02.010422+0000","last_unstale":"2024-03-16T10:16:16.298015+0000","last_undegraded":"2024-03-16T10:16:16.298015+0000","last_fullsized":"2024-03-16T10:16:16.298015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:17.606346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839284+0000","last_change":"2024-03-16T10:15:03.025789+0000","last_active":"2024-03-16T10:15:59.839284+0000","last_peered":"2024-03-16T10:15:59.839284+0000","last_clean":"2024-03-16T10:15:59.839284+0000","last_became_active":"2024-03-16T10:15:03.024812+0000","last_became_peered":"2024-03-16T10:15:03.024812+0000","last_unstale":"2024-03-16T10:15:59.839284+0000","last_undegraded":"2024-03-16T10:15:59.839284+0000","last_fullsized":"2024-03-16T10:15:59.839284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:53.041234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898933+0000","last_change":"2024-03-16T10:15:02.020599+0000","last_active":"2024-03-16T10:15:59.898933+0000","last_peered":"2024-03-16T10:15:59.898933+0000","last_clean":"2024-03-16T10:15:59.898933+0000","last_became_active":"2024-03-16T10:15:02.020203+0000","last_became_peered":"2024-03-16T10:15:02.020203+0000","last_unstale":"2024-03-16T10:15:59.898933+0000","last_undegraded":"2024-03-16T10:15:59.898933+0000","last_fullsized":"2024-03-16T10:15:59.898933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:46:33.032310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838966+0000","last_change":"2024-03-16T10:15:03.016223+0000","last_active":"2024-03-16T10:15:59.838966+0000","last_peered":"2024-03-16T10:15:59.838966+0000","last_clean":"2024-03-16T10:15:59.838966+0000","last_became_active":"2024-03-16T10:15:03.015857+0000","last_became_peered":"2024-03-16T10:15:03.015857+0000","last_unstale":"2024-03-16T10:15:59.838966+0000","last_undegraded":"2024-03-16T10:15:59.838966+0000","last_fullsized":"2024-03-16T10:15:59.838966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:40:10.329722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835797+0000","last_change":"2024-03-16T10:15:02.015690+0000","last_active":"2024-03-16T10:15:59.835797+0000","last_peered":"2024-03-16T10:15:59.835797+0000","last_clean":"2024-03-16T10:15:59.835797+0000","last_became_active":"2024-03-16T10:15:02.015375+0000","last_became_peered":"2024-03-16T10:15:02.015375+0000","last_unstale":"2024-03-16T10:15:59.835797+0000","last_undegraded":"2024-03-16T10:15:59.835797+0000","last_fullsized":"2024-03-16T10:15:59.835797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:39.118343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839461+0000","last_change":"2024-03-16T10:15:03.012702+0000","last_active":"2024-03-16T10:15:59.839461+0000","last_peered":"2024-03-16T10:15:59.839461+0000","last_clean":"2024-03-16T10:15:59.839461+0000","last_became_active":"2024-03-16T10:15:03.012387+0000","last_became_peered":"2024-03-16T10:15:03.012387+0000","last_unstale":"2024-03-16T10:15:59.839461+0000","last_undegraded":"2024-03-16T10:15:59.839461+0000","last_fullsized":"2024-03-16T10:15:59.839461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:16.938372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835944+0000","last_change":"2024-03-16T10:15:03.015290+0000","last_active":"2024-03-16T10:15:59.835944+0000","last_peered":"2024-03-16T10:15:59.835944+0000","last_clean":"2024-03-16T10:15:59.835944+0000","last_became_active":"2024-03-16T10:15:03.013846+0000","last_became_peered":"2024-03-16T10:15:03.013846+0000","last_unstale":"2024-03-16T10:15:59.835944+0000","last_undegraded":"2024-03-16T10:15:59.835944+0000","last_fullsized":"2024-03-16T10:15:59.835944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:33.377793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835614+0000","last_change":"2024-03-16T10:15:03.023733+0000","last_active":"2024-03-16T10:15:59.835614+0000","last_peered":"2024-03-16T10:15:59.835614+0000","last_clean":"2024-03-16T10:15:59.835614+0000","last_became_active":"2024-03-16T10:15:03.023409+0000","last_became_peered":"2024-03-16T10:15:03.023409+0000","last_unstale":"2024-03-16T10:15:59.835614+0000","last_undegraded":"2024-03-16T10:15:59.835614+0000","last_fullsized":"2024-03-16T10:15:59.835614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:44.748152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.856644+0000","last_change":"2024-03-16T10:15:59.856644+0000","last_active":"2024-03-16T10:15:59.856644+0000","last_peered":"2024-03-16T10:15:59.856644+0000","last_clean":"2024-03-16T10:15:59.856644+0000","last_became_active":"2024-03-16T10:15:59.856243+0000","last_became_peered":"2024-03-16T10:15:59.856243+0000","last_unstale":"2024-03-16T10:15:59.856644+0000","last_undegraded":"2024-03-16T10:15:59.856644+0000","last_fullsized":"2024-03-16T10:15:59.856644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:22.831073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834103+0000","last_change":"2024-03-16T10:15:03.029277+0000","last_active":"2024-03-16T10:15:59.834103+0000","last_peered":"2024-03-16T10:15:59.834103+0000","last_clean":"2024-03-16T10:15:59.834103+0000","last_became_active":"2024-03-16T10:15:03.028960+0000","last_became_peered":"2024-03-16T10:15:03.028960+0000","last_unstale":"2024-03-16T10:15:59.834103+0000","last_undegraded":"2024-03-16T10:15:59.834103+0000","last_fullsized":"2024-03-16T10:15:59.834103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:50:48.362914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.856671+0000","last_change":"2024-03-16T10:15:59.856671+0000","last_active":"2024-03-16T10:15:59.856671+0000","last_peered":"2024-03-16T10:15:59.856671+0000","last_clean":"2024-03-16T10:15:59.856671+0000","last_became_active":"2024-03-16T10:15:59.856287+0000","last_became_peered":"2024-03-16T10:15:59.856287+0000","last_unstale":"2024-03-16T10:15:59.856671+0000","last_undegraded":"2024-03-16T10:15:59.856671+0000","last_fullsized":"2024-03-16T10:15:59.856671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:15:58.409554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837196+0000","last_change":"2024-03-16T10:15:03.017499+0000","last_active":"2024-03-16T10:15:59.837196+0000","last_peered":"2024-03-16T10:15:59.837196+0000","last_clean":"2024-03-16T10:15:59.837196+0000","last_became_active":"2024-03-16T10:15:03.017085+0000","last_became_peered":"2024-03-16T10:15:03.017085+0000","last_unstale":"2024-03-16T10:15:59.837196+0000","last_undegraded":"2024-03-16T10:15:59.837196+0000","last_fullsized":"2024-03-16T10:15:59.837196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:57:09.592420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.836371+0000","last_change":"2024-03-16T10:15:03.018943+0000","last_active":"2024-03-16T10:15:59.836371+0000","last_peered":"2024-03-16T10:15:59.836371+0000","last_clean":"2024-03-16T10:15:59.836371+0000","last_became_active":"2024-03-16T10:15:03.017519+0000","last_became_peered":"2024-03-16T10:15:03.017519+0000","last_unstale":"2024-03-16T10:15:59.836371+0000","last_undegraded":"2024-03-16T10:15:59.836371+0000","last_fullsized":"2024-03-16T10:15:59.836371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:29.026988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838223+0000","last_change":"2024-03-16T10:15:03.024697+0000","last_active":"2024-03-16T10:15:59.838223+0000","last_peered":"2024-03-16T10:15:59.838223+0000","last_clean":"2024-03-16T10:15:59.838223+0000","last_became_active":"2024-03-16T10:15:03.024305+0000","last_became_peered":"2024-03-16T10:15:03.024305+0000","last_unstale":"2024-03-16T10:15:59.838223+0000","last_undegraded":"2024-03-16T10:15:59.838223+0000","last_fullsized":"2024-03-16T10:15:59.838223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:07:35.160746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898566+0000","last_change":"2024-03-16T10:15:03.017504+0000","last_active":"2024-03-16T10:15:59.898566+0000","last_peered":"2024-03-16T10:15:59.898566+0000","last_clean":"2024-03-16T10:15:59.898566+0000","last_became_active":"2024-03-16T10:15:03.017100+0000","last_became_peered":"2024-03-16T10:15:03.017100+0000","last_unstale":"2024-03-16T10:15:59.898566+0000","last_undegraded":"2024-03-16T10:15:59.898566+0000","last_fullsized":"2024-03-16T10:15:59.898566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:23:17.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837770+0000","last_change":"2024-03-16T10:15:03.024684+0000","last_active":"2024-03-16T10:15:59.837770+0000","last_peered":"2024-03-16T10:15:59.837770+0000","last_clean":"2024-03-16T10:15:59.837770+0000","last_became_active":"2024-03-16T10:15:03.024282+0000","last_became_peered":"2024-03-16T10:15:03.024282+0000","last_unstale":"2024-03-16T10:15:59.837770+0000","last_undegraded":"2024-03-16T10:15:59.837770+0000","last_fullsized":"2024-03-16T10:15:59.837770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:29.595162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835443+0000","last_change":"2024-03-16T10:15:03.025337+0000","last_active":"2024-03-16T10:15:59.835443+0000","last_peered":"2024-03-16T10:15:59.835443+0000","last_clean":"2024-03-16T10:15:59.835443+0000","last_became_active":"2024-03-16T10:15:03.024938+0000","last_became_peered":"2024-03-16T10:15:03.024938+0000","last_unstale":"2024-03-16T10:15:59.835443+0000","last_undegraded":"2024-03-16T10:15:59.835443+0000","last_fullsized":"2024-03-16T10:15:59.835443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:13:15.892354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835652+0000","last_change":"2024-03-16T10:15:03.015468+0000","last_active":"2024-03-16T10:15:59.835652+0000","last_peered":"2024-03-16T10:15:59.835652+0000","last_clean":"2024-03-16T10:15:59.835652+0000","last_became_active":"2024-03-16T10:15:03.015097+0000","last_became_peered":"2024-03-16T10:15:03.015097+0000","last_unstale":"2024-03-16T10:15:59.835652+0000","last_undegraded":"2024-03-16T10:15:59.835652+0000","last_fullsized":"2024-03-16T10:15:59.835652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:46.114346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.836102+0000","last_change":"2024-03-16T10:15:03.020690+0000","last_active":"2024-03-16T10:15:59.836102+0000","last_peered":"2024-03-16T10:15:59.836102+0000","last_clean":"2024-03-16T10:15:59.836102+0000","last_became_active":"2024-03-16T10:15:03.020365+0000","last_became_peered":"2024-03-16T10:15:03.020365+0000","last_unstale":"2024-03-16T10:15:59.836102+0000","last_undegraded":"2024-03-16T10:15:59.836102+0000","last_fullsized":"2024-03-16T10:15:59.836102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:25.671533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280280,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66836,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":738,"num_read_kb":446,"num_write":240,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":200508,"data_compressed":5583,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383708,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":950272,"data_stored":457048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645346,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1101824,"data_stored":1107709,"data_compressed":37511,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":456587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070509,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":456965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332145,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1049092,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495671,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":456304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":517240,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":458034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":517851,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182469,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":456129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920832,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":456165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757307,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":1048770,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61775,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":62386,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":61759,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T10:16:22.026 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T10:16:22.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:21 smithi192 ceph-mon[36354]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:22.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:21 smithi193 ceph-mon[36311]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:22.284 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:22 smithi192 ceph-mon[36354]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:16:23.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:22 smithi077 ceph-mon[32400]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:16:23.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:22 smithi193 ceph-mon[36311]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:16:23.862 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:23 smithi077 ceph-mon[32400]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:16:23.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.880+0000 7fa76711d700 1 -- 172.21.15.77:0/1282491122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7601073a0 msgr2=0x7fa760107780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:23.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.880+0000 7fa76711d700 1 --2- 172.21.15.77:0/1282491122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7601073a0 0x7fa760107780 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa7540099c0 tx=0x7fa754009cd0 comp rx=0 tx=0).stop 2024-03-16T10:16:23.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.880+0000 7fa76711d700 1 -- 172.21.15.77:0/1282491122 shutdown_connections 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.880+0000 7fa76711d700 1 --2- 172.21.15.77:0/1282491122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7601073a0 0x7fa760107780 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.880+0000 7fa76711d700 1 --2- 172.21.15.77:0/1282491122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa760101690 0x7fa76010f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.880+0000 7fa76711d700 1 --2- 172.21.15.77:0/1282491122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa760100cd0 0x7fa760101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.880+0000 7fa76711d700 1 -- 172.21.15.77:0/1282491122 >> 172.21.15.77:0/1282491122 conn(0x7fa760076520 msgr2=0x7fa760076930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.881+0000 7fa76711d700 1 -- 172.21.15.77:0/1282491122 shutdown_connections 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.881+0000 7fa76711d700 1 -- 172.21.15.77:0/1282491122 wait complete. 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.881+0000 7fa76711d700 1 Processor -- start 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.881+0000 7fa76711d700 1 -- start start 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.881+0000 7fa76711d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa760100cd0 0x7fa760198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa76711d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa760101690 0x7fa760199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa76711d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7601073a0 0x7fa76019d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa76711d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa760112650 con 0x7fa7601073a0 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa76711d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601124d0 con 0x7fa760100cd0 2024-03-16T10:16:23.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa76711d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601127d0 con 0x7fa760101690 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa7656ba700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7601073a0 0x7fa76019d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa764eb9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa760100cd0 0x7fa760198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa7656ba700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7601073a0 0x7fa76019d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51492/0 (socket says 172.21.15.77:51492) 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa7656ba700 1 -- 172.21.15.77:0/1601810001 learned_addr learned my addr 172.21.15.77:0/1601810001 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa75ffff700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa760101690 0x7fa760199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa764eb9700 1 -- 172.21.15.77:0/1601810001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa760101690 msgr2=0x7fa760199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa764eb9700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa760101690 0x7fa760199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa764eb9700 1 -- 172.21.15.77:0/1601810001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7601073a0 msgr2=0x7fa76019d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa764eb9700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7601073a0 0x7fa76019d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa764eb9700 1 -- 172.21.15.77:0/1601810001 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76019dae0 con 0x7fa760100cd0 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa7656ba700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7601073a0 0x7fa76019d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa764eb9700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa760100cd0 0x7fa760198ae0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa7540098e0 tx=0x7fa754005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.882+0000 7fa75dffb700 1 -- 172.21.15.77:0/1601810001 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754027070 con 0x7fa760100cd0 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.883+0000 7fa75dffb700 1 -- 172.21.15.77:0/1601810001 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa754003680 con 0x7fa760100cd0 2024-03-16T10:16:23.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.883+0000 7fa76711d700 1 -- 172.21.15.77:0/1601810001 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa76019dd70 con 0x7fa760100cd0 2024-03-16T10:16:23.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.883+0000 7fa75dffb700 1 -- 172.21.15.77:0/1601810001 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754022e10 con 0x7fa760100cd0 2024-03-16T10:16:23.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.883+0000 7fa76711d700 1 -- 172.21.15.77:0/1601810001 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa760101de0 con 0x7fa760100cd0 2024-03-16T10:16:23.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.884+0000 7fa76711d700 1 -- 172.21.15.77:0/1601810001 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa76006a450 con 0x7fa760100cd0 2024-03-16T10:16:23.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.885+0000 7fa75dffb700 1 -- 172.21.15.77:0/1601810001 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa75402a070 con 0x7fa760100cd0 2024-03-16T10:16:23.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.886+0000 7fa75dffb700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa7500a29a0 0x7fa7500a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:23.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.886+0000 7fa75ffff700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa7500a29a0 0x7fa7500a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:23.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.886+0000 7fa75dffb700 1 -- 172.21.15.77:0/1601810001 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa7540cf590 con 0x7fa760100cd0 2024-03-16T10:16:23.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.886+0000 7fa75ffff700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa7500a29a0 0x7fa7500a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa7601089e0 tx=0x7fa74c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:23.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:23.890+0000 7fa75dffb700 1 -- 172.21.15.77:0/1601810001 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa754013070 con 0x7fa760100cd0 2024-03-16T10:16:24.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.098+0000 7fa76711d700 1 -- 172.21.15.77:0/1601810001 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa760199e40 con 0x7fa7500a29a0 2024-03-16T10:16:24.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.105+0000 7fa75dffb700 1 -- 172.21.15.77:0/1601810001 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258979 (secure 0 0 0) 0x7fa760199e40 con 0x7fa7500a29a0 2024-03-16T10:16:24.109 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:24.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.108+0000 7fa76711d700 1 -- 172.21.15.77:0/1601810001 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa7500a29a0 msgr2=0x7fa7500a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:24.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.108+0000 7fa76711d700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa7500a29a0 0x7fa7500a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa7601089e0 tx=0x7fa74c009260 comp rx=0 tx=0).stop 2024-03-16T10:16:24.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.108+0000 7fa76711d700 1 -- 172.21.15.77:0/1601810001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa760100cd0 msgr2=0x7fa760198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:24.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.108+0000 7fa76711d700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa760100cd0 0x7fa760198ae0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa7540098e0 tx=0x7fa754005f10 comp rx=0 tx=0).stop 2024-03-16T10:16:24.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.108+0000 7fa76711d700 1 -- 172.21.15.77:0/1601810001 shutdown_connections 2024-03-16T10:16:24.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.108+0000 7fa76711d700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa7500a29a0 0x7fa7500a4e60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:24.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.108+0000 7fa76711d700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7601073a0 0x7fa76019d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:24.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.108+0000 7fa76711d700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa760100cd0 0x7fa760198ae0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:24.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.108+0000 7fa76711d700 1 --2- 172.21.15.77:0/1601810001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa760101690 0x7fa760199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:24.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.108+0000 7fa76711d700 1 -- 172.21.15.77:0/1601810001 >> 172.21.15.77:0/1601810001 conn(0x7fa760076520 msgr2=0x7fa7600fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:24.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.109+0000 7fa76711d700 1 -- 172.21.15.77:0/1601810001 shutdown_connections 2024-03-16T10:16:24.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:24.109+0000 7fa76711d700 1 -- 172.21.15.77:0/1601810001 wait complete. 2024-03-16T10:16:24.112 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T10:16:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:23 smithi192 ceph-mon[36354]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:16:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:23 smithi193 ceph-mon[36311]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:16:24.862 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-03-16T10:16:22.733375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657438,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":884,"num_read_kb":560,"num_write":423,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":347,"ondisk_log_size":347,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360928,"kb_used_data":11700,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580896,"statfs":{"total":1151940427776,"available":1151570837504,"internally_reserved":0,"allocated":11980800,"data_stored":7437894,"data_compressed":112533,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002553"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.836343+0000","last_change":"2024-03-16T10:15:03.016880+0000","last_active":"2024-03-16T10:15:59.836343+0000","last_peered":"2024-03-16T10:15:59.836343+0000","last_clean":"2024-03-16T10:15:59.836343+0000","last_became_active":"2024-03-16T10:15:03.016513+0000","last_became_peered":"2024-03-16T10:15:03.016513+0000","last_unstale":"2024-03-16T10:15:59.836343+0000","last_undegraded":"2024-03-16T10:15:59.836343+0000","last_fullsized":"2024-03-16T10:15:59.836343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:39.834015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834492+0000","last_change":"2024-03-16T10:15:03.015467+0000","last_active":"2024-03-16T10:15:59.834492+0000","last_peered":"2024-03-16T10:15:59.834492+0000","last_clean":"2024-03-16T10:15:59.834492+0000","last_became_active":"2024-03-16T10:15:03.015152+0000","last_became_peered":"2024-03-16T10:15:03.015152+0000","last_unstale":"2024-03-16T10:15:59.834492+0000","last_undegraded":"2024-03-16T10:15:59.834492+0000","last_fullsized":"2024-03-16T10:15:59.834492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:17:29.988569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834256+0000","last_change":"2024-03-16T10:15:03.023836+0000","last_active":"2024-03-16T10:15:59.834256+0000","last_peered":"2024-03-16T10:15:59.834256+0000","last_clean":"2024-03-16T10:15:59.834256+0000","last_became_active":"2024-03-16T10:15:03.022774+0000","last_became_peered":"2024-03-16T10:15:03.022774+0000","last_unstale":"2024-03-16T10:15:59.834256+0000","last_undegraded":"2024-03-16T10:15:59.834256+0000","last_fullsized":"2024-03-16T10:15:59.834256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:49.837157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134667+0000","last_change":"2024-03-16T10:15:03.030891+0000","last_active":"2024-03-16T10:16:00.134667+0000","last_peered":"2024-03-16T10:16:00.134667+0000","last_clean":"2024-03-16T10:16:00.134667+0000","last_became_active":"2024-03-16T10:15:03.030278+0000","last_became_peered":"2024-03-16T10:15:03.030278+0000","last_unstale":"2024-03-16T10:16:00.134667+0000","last_undegraded":"2024-03-16T10:16:00.134667+0000","last_fullsized":"2024-03-16T10:16:00.134667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:16.166885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834382+0000","last_change":"2024-03-16T10:15:03.017610+0000","last_active":"2024-03-16T10:15:59.834382+0000","last_peered":"2024-03-16T10:15:59.834382+0000","last_clean":"2024-03-16T10:15:59.834382+0000","last_became_active":"2024-03-16T10:15:03.017250+0000","last_became_peered":"2024-03-16T10:15:03.017250+0000","last_unstale":"2024-03-16T10:15:59.834382+0000","last_undegraded":"2024-03-16T10:15:59.834382+0000","last_fullsized":"2024-03-16T10:15:59.834382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:07.899425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.856216+0000","last_change":"2024-03-16T10:15:59.856216+0000","last_active":"2024-03-16T10:15:59.856216+0000","last_peered":"2024-03-16T10:15:59.856216+0000","last_clean":"2024-03-16T10:15:59.856216+0000","last_became_active":"2024-03-16T10:15:59.855882+0000","last_became_peered":"2024-03-16T10:15:59.855882+0000","last_unstale":"2024-03-16T10:15:59.856216+0000","last_undegraded":"2024-03-16T10:15:59.856216+0000","last_fullsized":"2024-03-16T10:15:59.856216+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:27:42.682580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838614+0000","last_change":"2024-03-16T10:15:03.017686+0000","last_active":"2024-03-16T10:15:59.838614+0000","last_peered":"2024-03-16T10:15:59.838614+0000","last_clean":"2024-03-16T10:15:59.838614+0000","last_became_active":"2024-03-16T10:15:03.017262+0000","last_became_peered":"2024-03-16T10:15:03.017262+0000","last_unstale":"2024-03-16T10:15:59.838614+0000","last_undegraded":"2024-03-16T10:15:59.838614+0000","last_fullsized":"2024-03-16T10:15:59.838614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:36:22.270651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837763+0000","last_change":"2024-03-16T10:15:03.020130+0000","last_active":"2024-03-16T10:15:59.837763+0000","last_peered":"2024-03-16T10:15:59.837763+0000","last_clean":"2024-03-16T10:15:59.837763+0000","last_became_active":"2024-03-16T10:15:03.019742+0000","last_became_peered":"2024-03-16T10:15:03.019742+0000","last_unstale":"2024-03-16T10:15:59.837763+0000","last_undegraded":"2024-03-16T10:15:59.837763+0000","last_fullsized":"2024-03-16T10:15:59.837763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:52.450792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835751+0000","last_change":"2024-03-16T10:15:03.027589+0000","last_active":"2024-03-16T10:15:59.835751+0000","last_peered":"2024-03-16T10:15:59.835751+0000","last_clean":"2024-03-16T10:15:59.835751+0000","last_became_active":"2024-03-16T10:15:03.027226+0000","last_became_peered":"2024-03-16T10:15:03.027226+0000","last_unstale":"2024-03-16T10:15:59.835751+0000","last_undegraded":"2024-03-16T10:15:59.835751+0000","last_fullsized":"2024-03-16T10:15:59.835751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:14:35.354629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.842118+0000","last_change":"2024-03-16T10:15:03.021163+0000","last_active":"2024-03-16T10:15:59.842118+0000","last_peered":"2024-03-16T10:15:59.842118+0000","last_clean":"2024-03-16T10:15:59.842118+0000","last_became_active":"2024-03-16T10:15:03.020483+0000","last_became_peered":"2024-03-16T10:15:03.020483+0000","last_unstale":"2024-03-16T10:15:59.842118+0000","last_undegraded":"2024-03-16T10:15:59.842118+0000","last_fullsized":"2024-03-16T10:15:59.842118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:01:00.873471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839232+0000","last_change":"2024-03-16T10:15:03.026111+0000","last_active":"2024-03-16T10:15:59.839232+0000","last_peered":"2024-03-16T10:15:59.839232+0000","last_clean":"2024-03-16T10:15:59.839232+0000","last_became_active":"2024-03-16T10:15:03.025296+0000","last_became_peered":"2024-03-16T10:15:03.025296+0000","last_unstale":"2024-03-16T10:15:59.839232+0000","last_undegraded":"2024-03-16T10:15:59.839232+0000","last_fullsized":"2024-03-16T10:15:59.839232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:33:29.606876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134634+0000","last_change":"2024-03-16T10:15:03.023013+0000","last_active":"2024-03-16T10:16:00.134634+0000","last_peered":"2024-03-16T10:16:00.134634+0000","last_clean":"2024-03-16T10:16:00.134634+0000","last_became_active":"2024-03-16T10:15:03.022251+0000","last_became_peered":"2024-03-16T10:15:03.022251+0000","last_unstale":"2024-03-16T10:16:00.134634+0000","last_undegraded":"2024-03-16T10:16:00.134634+0000","last_fullsized":"2024-03-16T10:16:00.134634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:33.544709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835554+0000","last_change":"2024-03-16T10:15:03.009548+0000","last_active":"2024-03-16T10:15:59.835554+0000","last_peered":"2024-03-16T10:15:59.835554+0000","last_clean":"2024-03-16T10:15:59.835554+0000","last_became_active":"2024-03-16T10:15:03.007675+0000","last_became_peered":"2024-03-16T10:15:03.007675+0000","last_unstale":"2024-03-16T10:15:59.835554+0000","last_undegraded":"2024-03-16T10:15:59.835554+0000","last_fullsized":"2024-03-16T10:15:59.835554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:08:02.561602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.833958+0000","last_change":"2024-03-16T10:15:03.020146+0000","last_active":"2024-03-16T10:15:59.833958+0000","last_peered":"2024-03-16T10:15:59.833958+0000","last_clean":"2024-03-16T10:15:59.833958+0000","last_became_active":"2024-03-16T10:15:03.019823+0000","last_became_peered":"2024-03-16T10:15:03.019823+0000","last_unstale":"2024-03-16T10:15:59.833958+0000","last_undegraded":"2024-03-16T10:15:59.833958+0000","last_fullsized":"2024-03-16T10:15:59.833958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:52.276506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.833939+0000","last_change":"2024-03-16T10:15:03.019150+0000","last_active":"2024-03-16T10:15:59.833939+0000","last_peered":"2024-03-16T10:15:59.833939+0000","last_clean":"2024-03-16T10:15:59.833939+0000","last_became_active":"2024-03-16T10:15:03.018830+0000","last_became_peered":"2024-03-16T10:15:03.018830+0000","last_unstale":"2024-03-16T10:15:59.833939+0000","last_undegraded":"2024-03-16T10:15:59.833939+0000","last_fullsized":"2024-03-16T10:15:59.833939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:46.207187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835981+0000","last_change":"2024-03-16T10:15:03.033321+0000","last_active":"2024-03-16T10:15:59.835981+0000","last_peered":"2024-03-16T10:15:59.835981+0000","last_clean":"2024-03-16T10:15:59.835981+0000","last_became_active":"2024-03-16T10:15:03.032944+0000","last_became_peered":"2024-03-16T10:15:03.032944+0000","last_unstale":"2024-03-16T10:15:59.835981+0000","last_undegraded":"2024-03-16T10:15:59.835981+0000","last_fullsized":"2024-03-16T10:15:59.835981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:14.345488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.836582+0000","last_change":"2024-03-16T10:15:03.019897+0000","last_active":"2024-03-16T10:15:59.836582+0000","last_peered":"2024-03-16T10:15:59.836582+0000","last_clean":"2024-03-16T10:15:59.836582+0000","last_became_active":"2024-03-16T10:15:03.019555+0000","last_became_peered":"2024-03-16T10:15:03.019555+0000","last_unstale":"2024-03-16T10:15:59.836582+0000","last_undegraded":"2024-03-16T10:15:59.836582+0000","last_fullsized":"2024-03-16T10:15:59.836582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:58.307285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.855420+0000","last_change":"2024-03-16T10:15:03.017482+0000","last_active":"2024-03-16T10:15:59.855420+0000","last_peered":"2024-03-16T10:15:59.855420+0000","last_clean":"2024-03-16T10:15:59.855420+0000","last_became_active":"2024-03-16T10:15:03.016521+0000","last_became_peered":"2024-03-16T10:15:03.016521+0000","last_unstale":"2024-03-16T10:15:59.855420+0000","last_undegraded":"2024-03-16T10:15:59.855420+0000","last_fullsized":"2024-03-16T10:15:59.855420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:11.889111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.836040+0000","last_change":"2024-03-16T10:15:03.018625+0000","last_active":"2024-03-16T10:15:59.836040+0000","last_peered":"2024-03-16T10:15:59.836040+0000","last_clean":"2024-03-16T10:15:59.836040+0000","last_became_active":"2024-03-16T10:15:03.018313+0000","last_became_peered":"2024-03-16T10:15:03.018313+0000","last_unstale":"2024-03-16T10:15:59.836040+0000","last_undegraded":"2024-03-16T10:15:59.836040+0000","last_fullsized":"2024-03-16T10:15:59.836040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:35:58.533559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":103,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:17.311576+0000","last_change":"2024-03-16T10:15:02.016545+0000","last_active":"2024-03-16T10:16:17.311576+0000","last_peered":"2024-03-16T10:16:17.311576+0000","last_clean":"2024-03-16T10:16:17.311576+0000","last_became_active":"2024-03-16T10:15:02.012088+0000","last_became_peered":"2024-03-16T10:15:02.012088+0000","last_unstale":"2024-03-16T10:16:17.311576+0000","last_undegraded":"2024-03-16T10:16:17.311576+0000","last_fullsized":"2024-03-16T10:16:17.311576+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:50.633539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838214+0000","last_change":"2024-03-16T10:15:03.026565+0000","last_active":"2024-03-16T10:15:59.838214+0000","last_peered":"2024-03-16T10:15:59.838214+0000","last_clean":"2024-03-16T10:15:59.838214+0000","last_became_active":"2024-03-16T10:15:03.025386+0000","last_became_peered":"2024-03-16T10:15:03.025386+0000","last_unstale":"2024-03-16T10:15:59.838214+0000","last_undegraded":"2024-03-16T10:15:59.838214+0000","last_fullsized":"2024-03-16T10:15:59.838214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:28.831764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838304+0000","last_change":"2024-03-16T10:15:02.018097+0000","last_active":"2024-03-16T10:15:59.838304+0000","last_peered":"2024-03-16T10:15:59.838304+0000","last_clean":"2024-03-16T10:15:59.838304+0000","last_became_active":"2024-03-16T10:15:02.017777+0000","last_became_peered":"2024-03-16T10:15:02.017777+0000","last_unstale":"2024-03-16T10:15:59.838304+0000","last_undegraded":"2024-03-16T10:15:59.838304+0000","last_fullsized":"2024-03-16T10:15:59.838304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:04:58.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834600+0000","last_change":"2024-03-16T10:15:03.023384+0000","last_active":"2024-03-16T10:15:59.834600+0000","last_peered":"2024-03-16T10:15:59.834600+0000","last_clean":"2024-03-16T10:15:59.834600+0000","last_became_active":"2024-03-16T10:15:03.022996+0000","last_became_peered":"2024-03-16T10:15:03.022996+0000","last_unstale":"2024-03-16T10:15:59.834600+0000","last_undegraded":"2024-03-16T10:15:59.834600+0000","last_fullsized":"2024-03-16T10:15:59.834600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:29:20.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834807+0000","last_change":"2024-03-16T10:15:02.014409+0000","last_active":"2024-03-16T10:15:59.834807+0000","last_peered":"2024-03-16T10:15:59.834807+0000","last_clean":"2024-03-16T10:15:59.834807+0000","last_became_active":"2024-03-16T10:15:02.014015+0000","last_became_peered":"2024-03-16T10:15:02.014015+0000","last_unstale":"2024-03-16T10:15:59.834807+0000","last_undegraded":"2024-03-16T10:15:59.834807+0000","last_fullsized":"2024-03-16T10:15:59.834807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:25:53.133466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835418+0000","last_change":"2024-03-16T10:15:03.031328+0000","last_active":"2024-03-16T10:15:59.835418+0000","last_peered":"2024-03-16T10:15:59.835418+0000","last_clean":"2024-03-16T10:15:59.835418+0000","last_became_active":"2024-03-16T10:15:03.029787+0000","last_became_peered":"2024-03-16T10:15:03.029787+0000","last_unstale":"2024-03-16T10:15:59.835418+0000","last_undegraded":"2024-03-16T10:15:59.835418+0000","last_fullsized":"2024-03-16T10:15:59.835418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:34.321883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834594+0000","last_change":"2024-03-16T10:15:02.018032+0000","last_active":"2024-03-16T10:15:59.834594+0000","last_peered":"2024-03-16T10:15:59.834594+0000","last_clean":"2024-03-16T10:15:59.834594+0000","last_became_active":"2024-03-16T10:15:02.017677+0000","last_became_peered":"2024-03-16T10:15:02.017677+0000","last_unstale":"2024-03-16T10:15:59.834594+0000","last_undegraded":"2024-03-16T10:15:59.834594+0000","last_fullsized":"2024-03-16T10:15:59.834594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:58:07.179129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.899027+0000","last_change":"2024-03-16T10:15:03.024958+0000","last_active":"2024-03-16T10:15:59.899027+0000","last_peered":"2024-03-16T10:15:59.899027+0000","last_clean":"2024-03-16T10:15:59.899027+0000","last_became_active":"2024-03-16T10:15:03.023964+0000","last_became_peered":"2024-03-16T10:15:03.023964+0000","last_unstale":"2024-03-16T10:15:59.899027+0000","last_undegraded":"2024-03-16T10:15:59.899027+0000","last_fullsized":"2024-03-16T10:15:59.899027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:55:00.953258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":99,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:19.300796+0000","last_change":"2024-03-16T10:15:02.015062+0000","last_active":"2024-03-16T10:16:19.300796+0000","last_peered":"2024-03-16T10:16:19.300796+0000","last_clean":"2024-03-16T10:16:19.300796+0000","last_became_active":"2024-03-16T10:15:02.014714+0000","last_became_peered":"2024-03-16T10:15:02.014714+0000","last_unstale":"2024-03-16T10:16:19.300796+0000","last_undegraded":"2024-03-16T10:16:19.300796+0000","last_fullsized":"2024-03-16T10:16:19.300796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:23:55.313009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834337+0000","last_change":"2024-03-16T10:15:03.013956+0000","last_active":"2024-03-16T10:15:59.834337+0000","last_peered":"2024-03-16T10:15:59.834337+0000","last_clean":"2024-03-16T10:15:59.834337+0000","last_became_active":"2024-03-16T10:15:03.013607+0000","last_became_peered":"2024-03-16T10:15:03.013607+0000","last_unstale":"2024-03-16T10:15:59.834337+0000","last_undegraded":"2024-03-16T10:15:59.834337+0000","last_fullsized":"2024-03-16T10:15:59.834337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:40:55.445395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.841646+0000","last_change":"2024-03-16T10:15:02.015651+0000","last_active":"2024-03-16T10:15:59.841646+0000","last_peered":"2024-03-16T10:15:59.841646+0000","last_clean":"2024-03-16T10:15:59.841646+0000","last_became_active":"2024-03-16T10:15:02.015254+0000","last_became_peered":"2024-03-16T10:15:02.015254+0000","last_unstale":"2024-03-16T10:15:59.841646+0000","last_undegraded":"2024-03-16T10:15:59.841646+0000","last_fullsized":"2024-03-16T10:15:59.841646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:41:59.013606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898887+0000","last_change":"2024-03-16T10:15:03.023074+0000","last_active":"2024-03-16T10:15:59.898887+0000","last_peered":"2024-03-16T10:15:59.898887+0000","last_clean":"2024-03-16T10:15:59.898887+0000","last_became_active":"2024-03-16T10:15:03.022700+0000","last_became_peered":"2024-03-16T10:15:03.022700+0000","last_unstale":"2024-03-16T10:15:59.898887+0000","last_undegraded":"2024-03-16T10:15:59.898887+0000","last_fullsized":"2024-03-16T10:15:59.898887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:22.047929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835321+0000","last_change":"2024-03-16T10:15:02.014793+0000","last_active":"2024-03-16T10:15:59.835321+0000","last_peered":"2024-03-16T10:15:59.835321+0000","last_clean":"2024-03-16T10:15:59.835321+0000","last_became_active":"2024-03-16T10:15:02.014459+0000","last_became_peered":"2024-03-16T10:15:02.014459+0000","last_unstale":"2024-03-16T10:15:59.835321+0000","last_undegraded":"2024-03-16T10:15:59.835321+0000","last_fullsized":"2024-03-16T10:15:59.835321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:59:45.035045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837733+0000","last_change":"2024-03-16T10:15:03.012713+0000","last_active":"2024-03-16T10:15:59.837733+0000","last_peered":"2024-03-16T10:15:59.837733+0000","last_clean":"2024-03-16T10:15:59.837733+0000","last_became_active":"2024-03-16T10:15:03.012407+0000","last_became_peered":"2024-03-16T10:15:03.012407+0000","last_unstale":"2024-03-16T10:15:59.837733+0000","last_undegraded":"2024-03-16T10:15:59.837733+0000","last_fullsized":"2024-03-16T10:15:59.837733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:23:14.592810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898544+0000","last_change":"2024-03-16T10:15:02.015582+0000","last_active":"2024-03-16T10:15:59.898544+0000","last_peered":"2024-03-16T10:15:59.898544+0000","last_clean":"2024-03-16T10:15:59.898544+0000","last_became_active":"2024-03-16T10:15:02.015073+0000","last_became_peered":"2024-03-16T10:15:02.015073+0000","last_unstale":"2024-03-16T10:15:59.898544+0000","last_undegraded":"2024-03-16T10:15:59.898544+0000","last_fullsized":"2024-03-16T10:15:59.898544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:22:48.212448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.855302+0000","last_change":"2024-03-16T10:15:03.016839+0000","last_active":"2024-03-16T10:15:59.855302+0000","last_peered":"2024-03-16T10:15:59.855302+0000","last_clean":"2024-03-16T10:15:59.855302+0000","last_became_active":"2024-03-16T10:15:03.016463+0000","last_became_peered":"2024-03-16T10:15:03.016463+0000","last_unstale":"2024-03-16T10:15:59.855302+0000","last_undegraded":"2024-03-16T10:15:59.855302+0000","last_fullsized":"2024-03-16T10:15:59.855302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:43:34.259049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":165,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:18.299353+0000","last_change":"2024-03-16T10:15:02.017884+0000","last_active":"2024-03-16T10:16:18.299353+0000","last_peered":"2024-03-16T10:16:18.299353+0000","last_clean":"2024-03-16T10:16:18.299353+0000","last_became_active":"2024-03-16T10:15:02.017511+0000","last_became_peered":"2024-03-16T10:15:02.017511+0000","last_unstale":"2024-03-16T10:16:18.299353+0000","last_undegraded":"2024-03-16T10:16:18.299353+0000","last_fullsized":"2024-03-16T10:16:18.299353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:31.281631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":142,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835597+0000","last_change":"2024-03-16T10:15:03.032259+0000","last_active":"2024-03-16T10:15:59.835597+0000","last_peered":"2024-03-16T10:15:59.835597+0000","last_clean":"2024-03-16T10:15:59.835597+0000","last_became_active":"2024-03-16T10:15:03.031884+0000","last_became_peered":"2024-03-16T10:15:03.031884+0000","last_unstale":"2024-03-16T10:15:59.835597+0000","last_undegraded":"2024-03-16T10:15:59.835597+0000","last_fullsized":"2024-03-16T10:15:59.835597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:37:48.464051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839326+0000","last_change":"2024-03-16T10:15:02.015372+0000","last_active":"2024-03-16T10:15:59.839326+0000","last_peered":"2024-03-16T10:15:59.839326+0000","last_clean":"2024-03-16T10:15:59.839326+0000","last_became_active":"2024-03-16T10:15:02.015022+0000","last_became_peered":"2024-03-16T10:15:02.015022+0000","last_unstale":"2024-03-16T10:15:59.839326+0000","last_undegraded":"2024-03-16T10:15:59.839326+0000","last_fullsized":"2024-03-16T10:15:59.839326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:42.651903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834604+0000","last_change":"2024-03-16T10:15:03.018661+0000","last_active":"2024-03-16T10:15:59.834604+0000","last_peered":"2024-03-16T10:15:59.834604+0000","last_clean":"2024-03-16T10:15:59.834604+0000","last_became_active":"2024-03-16T10:15:03.018351+0000","last_became_peered":"2024-03-16T10:15:03.018351+0000","last_unstale":"2024-03-16T10:15:59.834604+0000","last_undegraded":"2024-03-16T10:15:59.834604+0000","last_fullsized":"2024-03-16T10:15:59.834604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:21:33.939811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.899049+0000","last_change":"2024-03-16T10:15:02.014343+0000","last_active":"2024-03-16T10:15:59.899049+0000","last_peered":"2024-03-16T10:15:59.899049+0000","last_clean":"2024-03-16T10:15:59.899049+0000","last_became_active":"2024-03-16T10:15:02.013983+0000","last_became_peered":"2024-03-16T10:15:02.013983+0000","last_unstale":"2024-03-16T10:15:59.899049+0000","last_undegraded":"2024-03-16T10:15:59.899049+0000","last_fullsized":"2024-03-16T10:15:59.899049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:35:04.768079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835549+0000","last_change":"2024-03-16T10:15:03.031055+0000","last_active":"2024-03-16T10:15:59.835549+0000","last_peered":"2024-03-16T10:15:59.835549+0000","last_clean":"2024-03-16T10:15:59.835549+0000","last_became_active":"2024-03-16T10:15:03.029656+0000","last_became_peered":"2024-03-16T10:15:03.029656+0000","last_unstale":"2024-03-16T10:15:59.835549+0000","last_undegraded":"2024-03-16T10:15:59.835549+0000","last_fullsized":"2024-03-16T10:15:59.835549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:27:35.684774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'32","reported_seq":202,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:18.301781+0000","last_change":"2024-03-16T10:15:02.014752+0000","last_active":"2024-03-16T10:16:18.301781+0000","last_peered":"2024-03-16T10:16:18.301781+0000","last_clean":"2024-03-16T10:16:18.301781+0000","last_became_active":"2024-03-16T10:15:02.013763+0000","last_became_peered":"2024-03-16T10:15:02.013763+0000","last_unstale":"2024-03-16T10:16:18.301781+0000","last_undegraded":"2024-03-16T10:16:18.301781+0000","last_fullsized":"2024-03-16T10:16:18.301781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:12:12.759643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60023,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":151,"num_read_kb":135,"num_write":34,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834673+0000","last_change":"2024-03-16T10:15:03.018838+0000","last_active":"2024-03-16T10:15:59.834673+0000","last_peered":"2024-03-16T10:15:59.834673+0000","last_clean":"2024-03-16T10:15:59.834673+0000","last_became_active":"2024-03-16T10:15:03.017713+0000","last_became_peered":"2024-03-16T10:15:03.017713+0000","last_unstale":"2024-03-16T10:15:59.834673+0000","last_undegraded":"2024-03-16T10:15:59.834673+0000","last_fullsized":"2024-03-16T10:15:59.834673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:03:56.936729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835759+0000","last_change":"2024-03-16T10:15:02.016996+0000","last_active":"2024-03-16T10:15:59.835759+0000","last_peered":"2024-03-16T10:15:59.835759+0000","last_clean":"2024-03-16T10:15:59.835759+0000","last_became_active":"2024-03-16T10:15:02.016421+0000","last_became_peered":"2024-03-16T10:15:02.016421+0000","last_unstale":"2024-03-16T10:15:59.835759+0000","last_undegraded":"2024-03-16T10:15:59.835759+0000","last_fullsized":"2024-03-16T10:15:59.835759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:25:09.866730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835486+0000","last_change":"2024-03-16T10:15:03.017842+0000","last_active":"2024-03-16T10:15:59.835486+0000","last_peered":"2024-03-16T10:15:59.835486+0000","last_clean":"2024-03-16T10:15:59.835486+0000","last_became_active":"2024-03-16T10:15:03.015119+0000","last_became_peered":"2024-03-16T10:15:03.015119+0000","last_unstale":"2024-03-16T10:15:59.835486+0000","last_undegraded":"2024-03-16T10:15:59.835486+0000","last_fullsized":"2024-03-16T10:15:59.835486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:14:34.474519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898795+0000","last_change":"2024-03-16T10:15:02.021407+0000","last_active":"2024-03-16T10:15:59.898795+0000","last_peered":"2024-03-16T10:15:59.898795+0000","last_clean":"2024-03-16T10:15:59.898795+0000","last_became_active":"2024-03-16T10:15:02.020884+0000","last_became_peered":"2024-03-16T10:15:02.020884+0000","last_unstale":"2024-03-16T10:15:59.898795+0000","last_undegraded":"2024-03-16T10:15:59.898795+0000","last_fullsized":"2024-03-16T10:15:59.898795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:24.858281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838848+0000","last_change":"2024-03-16T10:15:03.028607+0000","last_active":"2024-03-16T10:15:59.838848+0000","last_peered":"2024-03-16T10:15:59.838848+0000","last_clean":"2024-03-16T10:15:59.838848+0000","last_became_active":"2024-03-16T10:15:03.028280+0000","last_became_peered":"2024-03-16T10:15:03.028280+0000","last_unstale":"2024-03-16T10:15:59.838848+0000","last_undegraded":"2024-03-16T10:15:59.838848+0000","last_fullsized":"2024-03-16T10:15:59.838848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:39.905921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'5","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:21.312689+0000","last_change":"2024-03-16T10:15:02.008558+0000","last_active":"2024-03-16T10:16:21.312689+0000","last_peered":"2024-03-16T10:16:21.312689+0000","last_clean":"2024-03-16T10:16:21.312689+0000","last_became_active":"2024-03-16T10:15:02.008014+0000","last_became_peered":"2024-03-16T10:15:02.008014+0000","last_unstale":"2024-03-16T10:16:21.312689+0000","last_undegraded":"2024-03-16T10:16:21.312689+0000","last_fullsized":"2024-03-16T10:16:21.312689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:40:09.217991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838543+0000","last_change":"2024-03-16T10:15:03.020847+0000","last_active":"2024-03-16T10:15:59.838543+0000","last_peered":"2024-03-16T10:15:59.838543+0000","last_clean":"2024-03-16T10:15:59.838543+0000","last_became_active":"2024-03-16T10:15:03.020512+0000","last_became_peered":"2024-03-16T10:15:03.020512+0000","last_unstale":"2024-03-16T10:15:59.838543+0000","last_undegraded":"2024-03-16T10:15:59.838543+0000","last_fullsized":"2024-03-16T10:15:59.838543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:04.436195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834281+0000","last_change":"2024-03-16T10:15:02.010690+0000","last_active":"2024-03-16T10:15:59.834281+0000","last_peered":"2024-03-16T10:15:59.834281+0000","last_clean":"2024-03-16T10:15:59.834281+0000","last_became_active":"2024-03-16T10:15:02.010228+0000","last_became_peered":"2024-03-16T10:15:02.010228+0000","last_unstale":"2024-03-16T10:15:59.834281+0000","last_undegraded":"2024-03-16T10:15:59.834281+0000","last_fullsized":"2024-03-16T10:15:59.834281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:30:07.513225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898533+0000","last_change":"2024-03-16T10:15:03.025257+0000","last_active":"2024-03-16T10:15:59.898533+0000","last_peered":"2024-03-16T10:15:59.898533+0000","last_clean":"2024-03-16T10:15:59.898533+0000","last_became_active":"2024-03-16T10:15:03.024498+0000","last_became_peered":"2024-03-16T10:15:03.024498+0000","last_unstale":"2024-03-16T10:15:59.898533+0000","last_undegraded":"2024-03-16T10:15:59.898533+0000","last_fullsized":"2024-03-16T10:15:59.898533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:27:55.894728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834712+0000","last_change":"2024-03-16T10:15:02.014136+0000","last_active":"2024-03-16T10:15:59.834712+0000","last_peered":"2024-03-16T10:15:59.834712+0000","last_clean":"2024-03-16T10:15:59.834712+0000","last_became_active":"2024-03-16T10:15:02.013721+0000","last_became_peered":"2024-03-16T10:15:02.013721+0000","last_unstale":"2024-03-16T10:15:59.834712+0000","last_undegraded":"2024-03-16T10:15:59.834712+0000","last_fullsized":"2024-03-16T10:15:59.834712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:28:12.301450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.856566+0000","last_change":"2024-03-16T10:15:59.856566+0000","last_active":"2024-03-16T10:15:59.856566+0000","last_peered":"2024-03-16T10:15:59.856566+0000","last_clean":"2024-03-16T10:15:59.856566+0000","last_became_active":"2024-03-16T10:15:59.856152+0000","last_became_peered":"2024-03-16T10:15:59.856152+0000","last_unstale":"2024-03-16T10:15:59.856566+0000","last_undegraded":"2024-03-16T10:15:59.856566+0000","last_fullsized":"2024-03-16T10:15:59.856566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:42:39.104928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.855367+0000","last_change":"2024-03-16T10:15:02.005754+0000","last_active":"2024-03-16T10:15:59.855367+0000","last_peered":"2024-03-16T10:15:59.855367+0000","last_clean":"2024-03-16T10:15:59.855367+0000","last_became_active":"2024-03-16T10:15:02.005344+0000","last_became_peered":"2024-03-16T10:15:02.005344+0000","last_unstale":"2024-03-16T10:15:59.855367+0000","last_undegraded":"2024-03-16T10:15:59.855367+0000","last_fullsized":"2024-03-16T10:15:59.855367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:37:37.918861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.842019+0000","last_change":"2024-03-16T10:15:03.012887+0000","last_active":"2024-03-16T10:15:59.842019+0000","last_peered":"2024-03-16T10:15:59.842019+0000","last_clean":"2024-03-16T10:15:59.842019+0000","last_became_active":"2024-03-16T10:15:03.012496+0000","last_became_peered":"2024-03-16T10:15:03.012496+0000","last_unstale":"2024-03-16T10:15:59.842019+0000","last_undegraded":"2024-03-16T10:15:59.842019+0000","last_fullsized":"2024-03-16T10:15:59.842019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:11.111049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":98,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:20.314933+0000","last_change":"2024-03-16T10:15:02.014334+0000","last_active":"2024-03-16T10:16:20.314933+0000","last_peered":"2024-03-16T10:16:20.314933+0000","last_clean":"2024-03-16T10:16:20.314933+0000","last_became_active":"2024-03-16T10:15:02.013832+0000","last_became_peered":"2024-03-16T10:15:02.013832+0000","last_unstale":"2024-03-16T10:16:20.314933+0000","last_undegraded":"2024-03-16T10:16:20.314933+0000","last_fullsized":"2024-03-16T10:16:20.314933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:20.391594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.841895+0000","last_change":"2024-03-16T10:15:03.021276+0000","last_active":"2024-03-16T10:15:59.841895+0000","last_peered":"2024-03-16T10:15:59.841895+0000","last_clean":"2024-03-16T10:15:59.841895+0000","last_became_active":"2024-03-16T10:15:03.020927+0000","last_became_peered":"2024-03-16T10:15:03.020927+0000","last_unstale":"2024-03-16T10:15:59.841895+0000","last_undegraded":"2024-03-16T10:15:59.841895+0000","last_fullsized":"2024-03-16T10:15:59.841895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:53:54.127506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898661+0000","last_change":"2024-03-16T10:15:03.022083+0000","last_active":"2024-03-16T10:15:59.898661+0000","last_peered":"2024-03-16T10:15:59.898661+0000","last_clean":"2024-03-16T10:15:59.898661+0000","last_became_active":"2024-03-16T10:15:03.021609+0000","last_became_peered":"2024-03-16T10:15:03.021609+0000","last_unstale":"2024-03-16T10:15:59.898661+0000","last_undegraded":"2024-03-16T10:15:59.898661+0000","last_fullsized":"2024-03-16T10:15:59.898661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:53.864402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":99,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:20.311123+0000","last_change":"2024-03-16T10:15:02.015497+0000","last_active":"2024-03-16T10:16:20.311123+0000","last_peered":"2024-03-16T10:16:20.311123+0000","last_clean":"2024-03-16T10:16:20.311123+0000","last_became_active":"2024-03-16T10:15:02.015183+0000","last_became_peered":"2024-03-16T10:15:02.015183+0000","last_unstale":"2024-03-16T10:16:20.311123+0000","last_undegraded":"2024-03-16T10:16:20.311123+0000","last_fullsized":"2024-03-16T10:16:20.311123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:44:05.913030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838082+0000","last_change":"2024-03-16T10:15:03.020046+0000","last_active":"2024-03-16T10:15:59.838082+0000","last_peered":"2024-03-16T10:15:59.838082+0000","last_clean":"2024-03-16T10:15:59.838082+0000","last_became_active":"2024-03-16T10:15:03.019647+0000","last_became_peered":"2024-03-16T10:15:03.019647+0000","last_unstale":"2024-03-16T10:15:59.838082+0000","last_undegraded":"2024-03-16T10:15:59.838082+0000","last_fullsized":"2024-03-16T10:15:59.838082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:18.343495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134445+0000","last_change":"2024-03-16T10:15:02.008560+0000","last_active":"2024-03-16T10:16:00.134445+0000","last_peered":"2024-03-16T10:16:00.134445+0000","last_clean":"2024-03-16T10:16:00.134445+0000","last_became_active":"2024-03-16T10:15:02.008094+0000","last_became_peered":"2024-03-16T10:15:02.008094+0000","last_unstale":"2024-03-16T10:16:00.134445+0000","last_undegraded":"2024-03-16T10:16:00.134445+0000","last_fullsized":"2024-03-16T10:16:00.134445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:21:43.555120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":341,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838644+0000","last_change":"2024-03-16T10:13:47.211815+0000","last_active":"2024-03-16T10:15:59.838644+0000","last_peered":"2024-03-16T10:15:59.838644+0000","last_clean":"2024-03-16T10:15:59.838644+0000","last_became_active":"2024-03-16T10:13:47.211353+0000","last_became_peered":"2024-03-16T10:13:47.211353+0000","last_unstale":"2024-03-16T10:15:59.838644+0000","last_undegraded":"2024-03-16T10:15:59.838644+0000","last_fullsized":"2024-03-16T10:15:59.838644+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_clean_scrub_stamp":"2024-03-16T10:10:22.965421+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:11:20.607686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835188+0000","last_change":"2024-03-16T10:15:03.020181+0000","last_active":"2024-03-16T10:15:59.835188+0000","last_peered":"2024-03-16T10:15:59.835188+0000","last_clean":"2024-03-16T10:15:59.835188+0000","last_became_active":"2024-03-16T10:15:03.019781+0000","last_became_peered":"2024-03-16T10:15:03.019781+0000","last_unstale":"2024-03-16T10:15:59.835188+0000","last_undegraded":"2024-03-16T10:15:59.835188+0000","last_fullsized":"2024-03-16T10:15:59.835188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:04:49.369484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838788+0000","last_change":"2024-03-16T10:15:02.014461+0000","last_active":"2024-03-16T10:15:59.838788+0000","last_peered":"2024-03-16T10:15:59.838788+0000","last_clean":"2024-03-16T10:15:59.838788+0000","last_became_active":"2024-03-16T10:15:02.014137+0000","last_became_peered":"2024-03-16T10:15:02.014137+0000","last_unstale":"2024-03-16T10:15:59.838788+0000","last_undegraded":"2024-03-16T10:15:59.838788+0000","last_fullsized":"2024-03-16T10:15:59.838788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:36:41.830893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834705+0000","last_change":"2024-03-16T10:15:02.019441+0000","last_active":"2024-03-16T10:15:59.834705+0000","last_peered":"2024-03-16T10:15:59.834705+0000","last_clean":"2024-03-16T10:15:59.834705+0000","last_became_active":"2024-03-16T10:15:02.019091+0000","last_became_peered":"2024-03-16T10:15:02.019091+0000","last_unstale":"2024-03-16T10:15:59.834705+0000","last_undegraded":"2024-03-16T10:15:59.834705+0000","last_fullsized":"2024-03-16T10:15:59.834705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:04:51.758722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835787+0000","last_change":"2024-03-16T10:15:03.012141+0000","last_active":"2024-03-16T10:15:59.835787+0000","last_peered":"2024-03-16T10:15:59.835787+0000","last_clean":"2024-03-16T10:15:59.835787+0000","last_became_active":"2024-03-16T10:15:03.011771+0000","last_became_peered":"2024-03-16T10:15:03.011771+0000","last_unstale":"2024-03-16T10:15:59.835787+0000","last_undegraded":"2024-03-16T10:15:59.835787+0000","last_fullsized":"2024-03-16T10:15:59.835787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:02:28.522530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'13","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.855528+0000","last_change":"2024-03-16T10:15:02.011898+0000","last_active":"2024-03-16T10:15:59.855528+0000","last_peered":"2024-03-16T10:15:59.855528+0000","last_clean":"2024-03-16T10:15:59.855528+0000","last_became_active":"2024-03-16T10:15:02.011578+0000","last_became_peered":"2024-03-16T10:15:02.011578+0000","last_unstale":"2024-03-16T10:15:59.855528+0000","last_undegraded":"2024-03-16T10:15:59.855528+0000","last_fullsized":"2024-03-16T10:15:59.855528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:27:35.756031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838261+0000","last_change":"2024-03-16T10:15:03.024923+0000","last_active":"2024-03-16T10:15:59.838261+0000","last_peered":"2024-03-16T10:15:59.838261+0000","last_clean":"2024-03-16T10:15:59.838261+0000","last_became_active":"2024-03-16T10:15:03.024577+0000","last_became_peered":"2024-03-16T10:15:03.024577+0000","last_unstale":"2024-03-16T10:15:59.838261+0000","last_undegraded":"2024-03-16T10:15:59.838261+0000","last_fullsized":"2024-03-16T10:15:59.838261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:04.650499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134719+0000","last_change":"2024-03-16T10:15:02.006842+0000","last_active":"2024-03-16T10:16:00.134719+0000","last_peered":"2024-03-16T10:16:00.134719+0000","last_clean":"2024-03-16T10:16:00.134719+0000","last_became_active":"2024-03-16T10:15:02.006483+0000","last_became_peered":"2024-03-16T10:15:02.006483+0000","last_unstale":"2024-03-16T10:16:00.134719+0000","last_undegraded":"2024-03-16T10:16:00.134719+0000","last_fullsized":"2024-03-16T10:16:00.134719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:30:55.856007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834392+0000","last_change":"2024-03-16T10:15:03.018691+0000","last_active":"2024-03-16T10:15:59.834392+0000","last_peered":"2024-03-16T10:15:59.834392+0000","last_clean":"2024-03-16T10:15:59.834392+0000","last_became_active":"2024-03-16T10:15:03.018304+0000","last_became_peered":"2024-03-16T10:15:03.018304+0000","last_unstale":"2024-03-16T10:15:59.834392+0000","last_undegraded":"2024-03-16T10:15:59.834392+0000","last_fullsized":"2024-03-16T10:15:59.834392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:15.446759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.841306+0000","last_change":"2024-03-16T10:15:59.841306+0000","last_active":"2024-03-16T10:15:59.841306+0000","last_peered":"2024-03-16T10:15:59.841306+0000","last_clean":"2024-03-16T10:15:59.841306+0000","last_became_active":"2024-03-16T10:15:59.838434+0000","last_became_peered":"2024-03-16T10:15:59.838434+0000","last_unstale":"2024-03-16T10:15:59.841306+0000","last_undegraded":"2024-03-16T10:15:59.841306+0000","last_fullsized":"2024-03-16T10:15:59.841306+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:14:21.702809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834199+0000","last_change":"2024-03-16T10:15:03.017262+0000","last_active":"2024-03-16T10:15:59.834199+0000","last_peered":"2024-03-16T10:15:59.834199+0000","last_clean":"2024-03-16T10:15:59.834199+0000","last_became_active":"2024-03-16T10:15:03.016909+0000","last_became_peered":"2024-03-16T10:15:03.016909+0000","last_unstale":"2024-03-16T10:15:59.834199+0000","last_undegraded":"2024-03-16T10:15:59.834199+0000","last_fullsized":"2024-03-16T10:15:59.834199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:53:11.385236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837856+0000","last_change":"2024-03-16T10:15:02.011325+0000","last_active":"2024-03-16T10:15:59.837856+0000","last_peered":"2024-03-16T10:15:59.837856+0000","last_clean":"2024-03-16T10:15:59.837856+0000","last_became_active":"2024-03-16T10:15:02.010999+0000","last_became_peered":"2024-03-16T10:15:02.010999+0000","last_unstale":"2024-03-16T10:15:59.837856+0000","last_undegraded":"2024-03-16T10:15:59.837856+0000","last_fullsized":"2024-03-16T10:15:59.837856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:30:08.137693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835889+0000","last_change":"2024-03-16T10:15:03.012146+0000","last_active":"2024-03-16T10:15:59.835889+0000","last_peered":"2024-03-16T10:15:59.835889+0000","last_clean":"2024-03-16T10:15:59.835889+0000","last_became_active":"2024-03-16T10:15:03.011790+0000","last_became_peered":"2024-03-16T10:15:03.011790+0000","last_unstale":"2024-03-16T10:15:59.835889+0000","last_undegraded":"2024-03-16T10:15:59.835889+0000","last_fullsized":"2024-03-16T10:15:59.835889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:45:22.642925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.842829+0000","last_change":"2024-03-16T10:15:59.842829+0000","last_active":"2024-03-16T10:15:59.842829+0000","last_peered":"2024-03-16T10:15:59.842829+0000","last_clean":"2024-03-16T10:15:59.842829+0000","last_became_active":"2024-03-16T10:15:59.838787+0000","last_became_peered":"2024-03-16T10:15:59.838787+0000","last_unstale":"2024-03-16T10:15:59.842829+0000","last_undegraded":"2024-03-16T10:15:59.842829+0000","last_fullsized":"2024-03-16T10:15:59.842829+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:47:27.573657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838065+0000","last_change":"2024-03-16T10:15:03.021094+0000","last_active":"2024-03-16T10:15:59.838065+0000","last_peered":"2024-03-16T10:15:59.838065+0000","last_clean":"2024-03-16T10:15:59.838065+0000","last_became_active":"2024-03-16T10:15:03.017640+0000","last_became_peered":"2024-03-16T10:15:03.017640+0000","last_unstale":"2024-03-16T10:15:59.838065+0000","last_undegraded":"2024-03-16T10:15:59.838065+0000","last_fullsized":"2024-03-16T10:15:59.838065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:02.030112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834498+0000","last_change":"2024-03-16T10:15:02.015652+0000","last_active":"2024-03-16T10:15:59.834498+0000","last_peered":"2024-03-16T10:15:59.834498+0000","last_clean":"2024-03-16T10:15:59.834498+0000","last_became_active":"2024-03-16T10:15:02.014046+0000","last_became_peered":"2024-03-16T10:15:02.014046+0000","last_unstale":"2024-03-16T10:15:59.834498+0000","last_undegraded":"2024-03-16T10:15:59.834498+0000","last_fullsized":"2024-03-16T10:15:59.834498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839375+0000","last_change":"2024-03-16T10:15:03.023233+0000","last_active":"2024-03-16T10:15:59.839375+0000","last_peered":"2024-03-16T10:15:59.839375+0000","last_clean":"2024-03-16T10:15:59.839375+0000","last_became_active":"2024-03-16T10:15:03.022906+0000","last_became_peered":"2024-03-16T10:15:03.022906+0000","last_unstale":"2024-03-16T10:15:59.839375+0000","last_undegraded":"2024-03-16T10:15:59.839375+0000","last_fullsized":"2024-03-16T10:15:59.839375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:22.943087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":97,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:16.298015+0000","last_change":"2024-03-16T10:15:02.010755+0000","last_active":"2024-03-16T10:16:16.298015+0000","last_peered":"2024-03-16T10:16:16.298015+0000","last_clean":"2024-03-16T10:16:16.298015+0000","last_became_active":"2024-03-16T10:15:02.010422+0000","last_became_peered":"2024-03-16T10:15:02.010422+0000","last_unstale":"2024-03-16T10:16:16.298015+0000","last_undegraded":"2024-03-16T10:16:16.298015+0000","last_fullsized":"2024-03-16T10:16:16.298015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:17.606346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839284+0000","last_change":"2024-03-16T10:15:03.025789+0000","last_active":"2024-03-16T10:15:59.839284+0000","last_peered":"2024-03-16T10:15:59.839284+0000","last_clean":"2024-03-16T10:15:59.839284+0000","last_became_active":"2024-03-16T10:15:03.024812+0000","last_became_peered":"2024-03-16T10:15:03.024812+0000","last_unstale":"2024-03-16T10:15:59.839284+0000","last_undegraded":"2024-03-16T10:15:59.839284+0000","last_fullsized":"2024-03-16T10:15:59.839284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:53.041234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898933+0000","last_change":"2024-03-16T10:15:02.020599+0000","last_active":"2024-03-16T10:15:59.898933+0000","last_peered":"2024-03-16T10:15:59.898933+0000","last_clean":"2024-03-16T10:15:59.898933+0000","last_became_active":"2024-03-16T10:15:02.020203+0000","last_became_peered":"2024-03-16T10:15:02.020203+0000","last_unstale":"2024-03-16T10:15:59.898933+0000","last_undegraded":"2024-03-16T10:15:59.898933+0000","last_fullsized":"2024-03-16T10:15:59.898933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:46:33.032310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838966+0000","last_change":"2024-03-16T10:15:03.016223+0000","last_active":"2024-03-16T10:15:59.838966+0000","last_peered":"2024-03-16T10:15:59.838966+0000","last_clean":"2024-03-16T10:15:59.838966+0000","last_became_active":"2024-03-16T10:15:03.015857+0000","last_became_peered":"2024-03-16T10:15:03.015857+0000","last_unstale":"2024-03-16T10:15:59.838966+0000","last_undegraded":"2024-03-16T10:15:59.838966+0000","last_fullsized":"2024-03-16T10:15:59.838966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:40:10.329722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835797+0000","last_change":"2024-03-16T10:15:02.015690+0000","last_active":"2024-03-16T10:15:59.835797+0000","last_peered":"2024-03-16T10:15:59.835797+0000","last_clean":"2024-03-16T10:15:59.835797+0000","last_became_active":"2024-03-16T10:15:02.015375+0000","last_became_peered":"2024-03-16T10:15:02.015375+0000","last_unstale":"2024-03-16T10:15:59.835797+0000","last_undegraded":"2024-03-16T10:15:59.835797+0000","last_fullsized":"2024-03-16T10:15:59.835797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:39.118343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839461+0000","last_change":"2024-03-16T10:15:03.012702+0000","last_active":"2024-03-16T10:15:59.839461+0000","last_peered":"2024-03-16T10:15:59.839461+0000","last_clean":"2024-03-16T10:15:59.839461+0000","last_became_active":"2024-03-16T10:15:03.012387+0000","last_became_peered":"2024-03-16T10:15:03.012387+0000","last_unstale":"2024-03-16T10:15:59.839461+0000","last_undegraded":"2024-03-16T10:15:59.839461+0000","last_fullsized":"2024-03-16T10:15:59.839461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:16.938372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835944+0000","last_change":"2024-03-16T10:15:03.015290+0000","last_active":"2024-03-16T10:15:59.835944+0000","last_peered":"2024-03-16T10:15:59.835944+0000","last_clean":"2024-03-16T10:15:59.835944+0000","last_became_active":"2024-03-16T10:15:03.013846+0000","last_became_peered":"2024-03-16T10:15:03.013846+0000","last_unstale":"2024-03-16T10:15:59.835944+0000","last_undegraded":"2024-03-16T10:15:59.835944+0000","last_fullsized":"2024-03-16T10:15:59.835944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:33.377793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835614+0000","last_change":"2024-03-16T10:15:03.023733+0000","last_active":"2024-03-16T10:15:59.835614+0000","last_peered":"2024-03-16T10:15:59.835614+0000","last_clean":"2024-03-16T10:15:59.835614+0000","last_became_active":"2024-03-16T10:15:03.023409+0000","last_became_peered":"2024-03-16T10:15:03.023409+0000","last_unstale":"2024-03-16T10:15:59.835614+0000","last_undegraded":"2024-03-16T10:15:59.835614+0000","last_fullsized":"2024-03-16T10:15:59.835614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:44.748152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.856644+0000","last_change":"2024-03-16T10:15:59.856644+0000","last_active":"2024-03-16T10:15:59.856644+0000","last_peered":"2024-03-16T10:15:59.856644+0000","last_clean":"2024-03-16T10:15:59.856644+0000","last_became_active":"2024-03-16T10:15:59.856243+0000","last_became_peered":"2024-03-16T10:15:59.856243+0000","last_unstale":"2024-03-16T10:15:59.856644+0000","last_undegraded":"2024-03-16T10:15:59.856644+0000","last_fullsized":"2024-03-16T10:15:59.856644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:22.831073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834103+0000","last_change":"2024-03-16T10:15:03.029277+0000","last_active":"2024-03-16T10:15:59.834103+0000","last_peered":"2024-03-16T10:15:59.834103+0000","last_clean":"2024-03-16T10:15:59.834103+0000","last_became_active":"2024-03-16T10:15:03.028960+0000","last_became_peered":"2024-03-16T10:15:03.028960+0000","last_unstale":"2024-03-16T10:15:59.834103+0000","last_undegraded":"2024-03-16T10:15:59.834103+0000","last_fullsized":"2024-03-16T10:15:59.834103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:50:48.362914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.856671+0000","last_change":"2024-03-16T10:15:59.856671+0000","last_active":"2024-03-16T10:15:59.856671+0000","last_peered":"2024-03-16T10:15:59.856671+0000","last_clean":"2024-03-16T10:15:59.856671+0000","last_became_active":"2024-03-16T10:15:59.856287+0000","last_became_peered":"2024-03-16T10:15:59.856287+0000","last_unstale":"2024-03-16T10:15:59.856671+0000","last_undegraded":"2024-03-16T10:15:59.856671+0000","last_fullsized":"2024-03-16T10:15:59.856671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:15:58.409554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837196+0000","last_change":"2024-03-16T10:15:03.017499+0000","last_active":"2024-03-16T10:15:59.837196+0000","last_peered":"2024-03-16T10:15:59.837196+0000","last_clean":"2024-03-16T10:15:59.837196+0000","last_became_active":"2024-03-16T10:15:03.017085+0000","last_became_peered":"2024-03-16T10:15:03.017085+0000","last_unstale":"2024-03-16T10:15:59.837196+0000","last_undegraded":"2024-03-16T10:15:59.837196+0000","last_fullsized":"2024-03-16T10:15:59.837196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:57:09.592420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.836371+0000","last_change":"2024-03-16T10:15:03.018943+0000","last_active":"2024-03-16T10:15:59.836371+0000","last_peered":"2024-03-16T10:15:59.836371+0000","last_clean":"2024-03-16T10:15:59.836371+0000","last_became_active":"2024-03-16T10:15:03.017519+0000","last_became_peered":"2024-03-16T10:15:03.017519+0000","last_unstale":"2024-03-16T10:15:59.836371+0000","last_undegraded":"2024-03-16T10:15:59.836371+0000","last_fullsized":"2024-03-16T10:15:59.836371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:29.026988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838223+0000","last_change":"2024-03-16T10:15:03.024697+0000","last_active":"2024-03-16T10:15:59.838223+0000","last_peered":"2024-03-16T10:15:59.838223+0000","last_clean":"2024-03-16T10:15:59.838223+0000","last_became_active":"2024-03-16T10:15:03.024305+0000","last_became_peered":"2024-03-16T10:15:03.024305+0000","last_unstale":"2024-03-16T10:15:59.838223+0000","last_undegraded":"2024-03-16T10:15:59.838223+0000","last_fullsized":"2024-03-16T10:15:59.838223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:07:35.160746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898566+0000","last_change":"2024-03-16T10:15:03.017504+0000","last_active":"2024-03-16T10:15:59.898566+0000","last_peered":"2024-03-16T10:15:59.898566+0000","last_clean":"2024-03-16T10:15:59.898566+0000","last_became_active":"2024-03-16T10:15:03.017100+0000","last_became_peered":"2024-03-16T10:15:03.017100+0000","last_unstale":"2024-03-16T10:15:59.898566+0000","last_undegraded":"2024-03-16T10:15:59.898566+0000","last_fullsized":"2024-03-16T10:15:59.898566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:23:17.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837770+0000","last_change":"2024-03-16T10:15:03.024684+0000","last_active":"2024-03-16T10:15:59.837770+0000","last_peered":"2024-03-16T10:15:59.837770+0000","last_clean":"2024-03-16T10:15:59.837770+0000","last_became_active":"2024-03-16T10:15:03.024282+0000","last_became_peered":"2024-03-16T10:15:03.024282+0000","last_unstale":"2024-03-16T10:15:59.837770+0000","last_undegraded":"2024-03-16T10:15:59.837770+0000","last_fullsized":"2024-03-16T10:15:59.837770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:29.595162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835443+0000","last_change":"2024-03-16T10:15:03.025337+0000","last_active":"2024-03-16T10:15:59.835443+0000","last_peered":"2024-03-16T10:15:59.835443+0000","last_clean":"2024-03-16T10:15:59.835443+0000","last_became_active":"2024-03-16T10:15:03.024938+0000","last_became_peered":"2024-03-16T10:15:03.024938+0000","last_unstale":"2024-03-16T10:15:59.835443+0000","last_undegraded":"2024-03-16T10:15:59.835443+0000","last_fullsized":"2024-03-16T10:15:59.835443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:13:15.892354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835652+0000","last_change":"2024-03-16T10:15:03.015468+0000","last_active":"2024-03-16T10:15:59.835652+0000","last_peered":"2024-03-16T10:15:59.835652+0000","last_clean":"2024-03-16T10:15:59.835652+0000","last_became_active":"2024-03-16T10:15:03.015097+0000","last_became_peered":"2024-03-16T10:15:03.015097+0000","last_unstale":"2024-03-16T10:15:59.835652+0000","last_undegraded":"2024-03-16T10:15:59.835652+0000","last_fullsized":"2024-03-16T10:15:59.835652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:46.114346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.836102+0000","last_change":"2024-03-16T10:15:03.020690+0000","last_active":"2024-03-16T10:15:59.836102+0000","last_peered":"2024-03-16T10:15:59.836102+0000","last_clean":"2024-03-16T10:15:59.836102+0000","last_became_active":"2024-03-16T10:15:03.020365+0000","last_became_peered":"2024-03-16T10:15:03.020365+0000","last_unstale":"2024-03-16T10:15:59.836102+0000","last_undegraded":"2024-03-16T10:15:59.836102+0000","last_fullsized":"2024-03-16T10:15:59.836102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:25.671533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280280,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66836,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":758,"num_read_kb":451,"num_write":240,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":200508,"data_compressed":5583,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383708,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":950272,"data_stored":457048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645346,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1101824,"data_stored":1107709,"data_compressed":37511,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":456587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":456965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332146,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1049092,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495672,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":456304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":517240,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":458034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":517851,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":456129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920833,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":456165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757308,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":1048770,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61775,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":62386,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":61759,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T10:16:24.866 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-16T10:16:24.866 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-16T10:16:24.866 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-16T10:16:24.866 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-16T10:16:25.117 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:25 smithi192 ceph-mon[36354]: from='client.24797 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:16:26.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:25 smithi192 ceph-mon[36354]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T10:16:26.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:25 smithi077 ceph-mon[32400]: from='client.24797 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:16:26.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:25 smithi077 ceph-mon[32400]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T10:16:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:25 smithi193 ceph-mon[36311]: from='client.24797 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:16:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:25 smithi193 ceph-mon[36311]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T10:16:26.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.771+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1378451437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea0108320 msgr2=0x7f8ea0108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:26.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.771+0000 7f8ea6e19700 1 --2- 172.21.15.77:0/1378451437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea0108320 0x7f8ea0108700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8e940099c0 tx=0x7f8e94009cd0 comp rx=0 tx=0).stop 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.771+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1378451437 shutdown_connections 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.771+0000 7f8ea6e19700 1 --2- 172.21.15.77:0/1378451437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea0108320 0x7f8ea0108700 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.771+0000 7f8ea6e19700 1 --2- 172.21.15.77:0/1378451437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ea006a920 0x7f8ea010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.771+0000 7f8ea6e19700 1 --2- 172.21.15.77:0/1378451437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea0069f80 0x7f8ea006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.771+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1378451437 >> 172.21.15.77:0/1378451437 conn(0x7f8ea0078100 msgr2=0x7f8ea0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.771+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1378451437 shutdown_connections 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.771+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1378451437 wait complete. 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea6e19700 1 Processor -- start 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea6e19700 1 -- start start 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea6e19700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea0069f80 0x7f8ea019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea6e19700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ea006a920 0x7f8ea019ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea6e19700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea0108320 0x7f8ea0198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:26.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea6e19700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea01125b0 con 0x7f8ea0069f80 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea6e19700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea0112430 con 0x7f8ea006a920 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea6e19700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea0112730 con 0x7f8ea0108320 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea4bb5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea0069f80 0x7f8ea019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8e9ffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ea006a920 0x7f8ea019ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea4bb5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea0069f80 0x7f8ea019e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51516/0 (socket says 172.21.15.77:51516) 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea4bb5700 1 -- 172.21.15.77:0/1349775774 learned_addr learned my addr 172.21.15.77:0/1349775774 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea53b6700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea0108320 0x7f8ea0198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea4bb5700 1 -- 172.21.15.77:0/1349775774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea0108320 msgr2=0x7f8ea0198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea4bb5700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea0108320 0x7f8ea0198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea4bb5700 1 -- 172.21.15.77:0/1349775774 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ea006a920 msgr2=0x7f8ea019ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea4bb5700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ea006a920 0x7f8ea019ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8ea4bb5700 1 -- 172.21.15.77:0/1349775774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea0199030 con 0x7f8ea0069f80 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.772+0000 7f8e9ffff700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ea006a920 0x7f8ea019ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.773+0000 7f8ea4bb5700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea0069f80 0x7f8ea019e6d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8e94005910 tx=0x7f8e9400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:26.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.773+0000 7f8e9dffb700 1 -- 172.21.15.77:0/1349775774 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e94027070 con 0x7f8ea0069f80 2024-03-16T10:16:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.773+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1349775774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea01992c0 con 0x7f8ea0069f80 2024-03-16T10:16:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.773+0000 7f8e9dffb700 1 -- 172.21.15.77:0/1349775774 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e94003680 con 0x7f8ea0069f80 2024-03-16T10:16:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.773+0000 7f8e9dffb700 1 -- 172.21.15.77:0/1349775774 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e940173f0 con 0x7f8ea0069f80 2024-03-16T10:16:26.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.773+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1349775774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea01a5530 con 0x7f8ea0069f80 2024-03-16T10:16:26.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.774+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1349775774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ea01039d0 con 0x7f8ea0069f80 2024-03-16T10:16:26.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.775+0000 7f8e9dffb700 1 -- 172.21.15.77:0/1349775774 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8e94022590 con 0x7f8ea0069f80 2024-03-16T10:16:26.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.776+0000 7f8e9dffb700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8e900a2920 0x7f8e900a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:26.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.776+0000 7f8e9ffff700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8e900a2920 0x7f8e900a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:26.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.776+0000 7f8e9dffb700 1 -- 172.21.15.77:0/1349775774 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8e940cf8b0 con 0x7f8ea0069f80 2024-03-16T10:16:26.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.776+0000 7f8e9ffff700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8e900a2920 0x7f8e900a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0078670 tx=0x7f8e8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:26.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:26.779+0000 7f8e9dffb700 1 -- 172.21.15.77:0/1349775774 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8e94013070 con 0x7f8ea0069f80 2024-03-16T10:16:27.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.027+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1349775774 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8ea0199d80 con 0x7f8ea0069f80 2024-03-16T10:16:27.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.028+0000 7f8e9dffb700 1 -- 172.21.15.77:0/1349775774 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f8e9409c970 con 0x7f8ea0069f80 2024-03-16T10:16:27.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:27.029 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-16T10:16:27.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.030+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1349775774 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8e900a2920 msgr2=0x7f8e900a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:27.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.030+0000 7f8ea6e19700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8e900a2920 0x7f8e900a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0078670 tx=0x7f8e8c009260 comp rx=0 tx=0).stop 2024-03-16T10:16:27.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.030+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1349775774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea0069f80 msgr2=0x7f8ea019e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:27.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.030+0000 7f8ea6e19700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea0069f80 0x7f8ea019e6d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8e94005910 tx=0x7f8e9400f690 comp rx=0 tx=0).stop 2024-03-16T10:16:27.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.030+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1349775774 shutdown_connections 2024-03-16T10:16:27.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.030+0000 7f8ea6e19700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8e900a2920 0x7f8e900a4de0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.030+0000 7f8ea6e19700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ea0069f80 0x7f8ea019e6d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.030+0000 7f8ea6e19700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea0108320 0x7f8ea0198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.030+0000 7f8ea6e19700 1 --2- 172.21.15.77:0/1349775774 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ea006a920 0x7f8ea019ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.030+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1349775774 >> 172.21.15.77:0/1349775774 conn(0x7f8ea0078100 msgr2=0x7f8ea00ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:27.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.030+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1349775774 shutdown_connections 2024-03-16T10:16:27.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.030+0000 7f8ea6e19700 1 -- 172.21.15.77:0/1349775774 wait complete. 2024-03-16T10:16:27.675 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-16T10:16:27.676 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:16:27.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.781+0000 7f65246d1640 1 -- 172.21.15.77:0/4099345438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f651c1013b0 msgr2=0x7f651c101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.781+0000 7f65246d1640 1 --2- 172.21.15.77:0/4099345438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f651c1013b0 0x7f651c101830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6514009ca0 tx=0x7f651402c140 comp rx=0 tx=0).stop 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.781+0000 7f65246d1640 1 -- 172.21.15.77:0/4099345438 shutdown_connections 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.781+0000 7f65246d1640 1 --2- 172.21.15.77:0/4099345438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f651c1013b0 0x7f651c101830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.781+0000 7f65246d1640 1 --2- 172.21.15.77:0/4099345438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f651c100a10 0x7f651c100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.781+0000 7f65246d1640 1 --2- 172.21.15.77:0/4099345438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f651c1000b0 0x7f651c1004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.781+0000 7f65246d1640 1 -- 172.21.15.77:0/4099345438 >> 172.21.15.77:0/4099345438 conn(0x7f651c0fbc70 msgr2=0x7f651c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.781+0000 7f65246d1640 1 -- 172.21.15.77:0/4099345438 shutdown_connections 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.781+0000 7f65246d1640 1 -- 172.21.15.77:0/4099345438 wait complete. 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f65246d1640 1 Processor -- start 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f65246d1640 1 -- start start 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f65246d1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f651c1000b0 0x7f651c199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f65246d1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f651c100a10 0x7f651c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f65246d1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f651c1013b0 0x7f651c1a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:27.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f65246d1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f651c10f680 con 0x7f651c1000b0 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f65246d1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f651c10f500 con 0x7f651c100a10 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f65246d1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f651c10f800 con 0x7f651c1013b0 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f652246d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f651c1000b0 0x7f651c199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f652246d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f651c1000b0 0x7f651c199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51534/0 (socket says 172.21.15.77:51534) 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f652246d640 1 -- 172.21.15.77:0/337307551 learned_addr learned my addr 172.21.15.77:0/337307551 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f6521c6c640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f651c100a10 0x7f651c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f6522c6e640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f651c1013b0 0x7f651c1a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f652246d640 1 -- 172.21.15.77:0/337307551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f651c1013b0 msgr2=0x7f651c1a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f652246d640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f651c1013b0 0x7f651c1a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f652246d640 1 -- 172.21.15.77:0/337307551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f651c100a10 msgr2=0x7f651c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f652246d640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f651c100a10 0x7f651c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f652246d640 1 -- 172.21.15.77:0/337307551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651c1a1920 con 0x7f651c1000b0 2024-03-16T10:16:27.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.782+0000 7f652246d640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f651c1000b0 0x7f651c199d10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f651800e9f0 tx=0x7f651800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:27.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.783+0000 7f65137fe640 1 -- 172.21.15.77:0/337307551 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651800cd80 con 0x7f651c1000b0 2024-03-16T10:16:27.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.783+0000 7f65246d1640 1 -- 172.21.15.77:0/337307551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651c1a1c10 con 0x7f651c1000b0 2024-03-16T10:16:27.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.783+0000 7f65137fe640 1 -- 172.21.15.77:0/337307551 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6518010910 con 0x7f651c1000b0 2024-03-16T10:16:27.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.783+0000 7f65137fe640 1 -- 172.21.15.77:0/337307551 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65180188f0 con 0x7f651c1000b0 2024-03-16T10:16:27.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.783+0000 7f65246d1640 1 -- 172.21.15.77:0/337307551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f651c1a2160 con 0x7f651c1000b0 2024-03-16T10:16:27.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.784+0000 7f65246d1640 1 -- 172.21.15.77:0/337307551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f651c10af90 con 0x7f651c1000b0 2024-03-16T10:16:27.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.785+0000 7f65137fe640 1 -- 172.21.15.77:0/337307551 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6518010ac0 con 0x7f651c1000b0 2024-03-16T10:16:27.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.785+0000 7f65137fe640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65080a2890 0x7f65080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:27.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.785+0000 7f6521c6c640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65080a2890 0x7f65080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:27.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.786+0000 7f65137fe640 1 -- 172.21.15.77:0/337307551 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f65180cef30 con 0x7f651c1000b0 2024-03-16T10:16:27.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.786+0000 7f6521c6c640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65080a2890 0x7f65080a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f650c009bd0 tx=0x7f650c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:27.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.790+0000 7f65137fe640 1 -- 172.21.15.77:0/337307551 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f651809b910 con 0x7f651c1000b0 2024-03-16T10:16:27.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.975+0000 7f65246d1640 1 -- 172.21.15.77:0/337307551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f651c02d460 con 0x7f651c1000b0 2024-03-16T10:16:27.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.976+0000 7f65137fe640 1 -- 172.21.15.77:0/337307551 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22496 (secure 0 0 0) 0x7f651809f9b0 con 0x7f651c1000b0 2024-03-16T10:16:27.977 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:27.977 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":87,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:59.826259+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:16:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.978+0000 7f65246d1640 1 -- 172.21.15.77:0/337307551 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65080a2890 msgr2=0x7f65080a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.978+0000 7f65246d1640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65080a2890 0x7f65080a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f650c009bd0 tx=0x7f650c005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.978+0000 7f65246d1640 1 -- 172.21.15.77:0/337307551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f651c1000b0 msgr2=0x7f651c199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.978+0000 7f65246d1640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f651c1000b0 0x7f651c199d10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f651800e9f0 tx=0x7f651800eeb0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.978+0000 7f65246d1640 1 -- 172.21.15.77:0/337307551 shutdown_connections 2024-03-16T10:16:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.978+0000 7f65246d1640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65080a2890 0x7f65080a4d50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.978+0000 7f65246d1640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f651c1013b0 0x7f651c1a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.978+0000 7f65246d1640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f651c100a10 0x7f651c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.978+0000 7f65246d1640 1 --2- 172.21.15.77:0/337307551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f651c1000b0 0x7f651c199d10 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.978+0000 7f65246d1640 1 -- 172.21.15.77:0/337307551 >> 172.21.15.77:0/337307551 conn(0x7f651c0fbc70 msgr2=0x7f651c0fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.978+0000 7f65246d1640 1 -- 172.21.15.77:0/337307551 shutdown_connections 2024-03-16T10:16:27.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:27.978+0000 7f65246d1640 1 -- 172.21.15.77:0/337307551 wait complete. 2024-03-16T10:16:27.989 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:27 smithi077 ceph-mon[32400]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:27.989 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:27 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1349775774' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T10:16:27.990 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:10:22.965004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T10:15:00.264039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T10:15:01.536067+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T10:16:27.991 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T10:16:28.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 -- 172.21.15.77:0/2252301432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6101006e0 msgr2=0x7fe610100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 --2- 172.21.15.77:0/2252301432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6101006e0 0x7fe610100b00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe604009860 tx=0x7fe60402bb70 comp rx=0 tx=0).stop 2024-03-16T10:16:28.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 -- 172.21.15.77:0/2252301432 shutdown_connections 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 --2- 172.21.15.77:0/2252301432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe610102290 0x7fe610108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 --2- 172.21.15.77:0/2252301432 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6101018d0 0x7fe610101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 --2- 172.21.15.77:0/2252301432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6101006e0 0x7fe610100b00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 -- 172.21.15.77:0/2252301432 >> 172.21.15.77:0/2252301432 conn(0x7fe6100fbc70 msgr2=0x7fe6100fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 -- 172.21.15.77:0/2252301432 shutdown_connections 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 -- 172.21.15.77:0/2252301432 wait complete. 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 Processor -- start 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 -- start start 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6101006e0 0x7fe610196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6101018d0 0x7fe610196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe610102290 0x7fe61019e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe61010c330 con 0x7fe6101006e0 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe61010c1b0 con 0x7fe6101018d0 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.098+0000 7fe616c97640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe61010c4b0 con 0x7fe610102290 2024-03-16T10:16:28.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe614a33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6101006e0 0x7fe610196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe614a33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6101006e0 0x7fe610196a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51544/0 (socket says 172.21.15.77:51544) 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe614a33640 1 -- 172.21.15.77:0/1397241986 learned_addr learned my addr 172.21.15.77:0/1397241986 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe60ffff640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6101018d0 0x7fe610196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe615234640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe610102290 0x7fe61019e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe614a33640 1 -- 172.21.15.77:0/1397241986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe610102290 msgr2=0x7fe61019e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe614a33640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe610102290 0x7fe61019e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe614a33640 1 -- 172.21.15.77:0/1397241986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6101018d0 msgr2=0x7fe610196fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe614a33640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6101018d0 0x7fe610196fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe614a33640 1 -- 172.21.15.77:0/1397241986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe61019e6a0 con 0x7fe6101006e0 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe614a33640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6101006e0 0x7fe610196a80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe60402c0b0 tx=0x7fe604032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe60dffb640 1 -- 172.21.15.77:0/1397241986 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604041020 con 0x7fe6101006e0 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe60dffb640 1 -- 172.21.15.77:0/1397241986 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe60402c9f0 con 0x7fe6101006e0 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe616c97640 1 -- 172.21.15.77:0/1397241986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe61019e930 con 0x7fe6101006e0 2024-03-16T10:16:28.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe60dffb640 1 -- 172.21.15.77:0/1397241986 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604039040 con 0x7fe6101006e0 2024-03-16T10:16:28.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.099+0000 7fe616c97640 1 -- 172.21.15.77:0/1397241986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe61019ee70 con 0x7fe6101006e0 2024-03-16T10:16:28.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.100+0000 7fe616c97640 1 -- 172.21.15.77:0/1397241986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe610107ac0 con 0x7fe6101006e0 2024-03-16T10:16:28.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.101+0000 7fe60dffb640 1 -- 172.21.15.77:0/1397241986 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe604044020 con 0x7fe6101006e0 2024-03-16T10:16:28.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.102+0000 7fe60dffb640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe6000a2910 0x7fe6000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:28.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.102+0000 7fe60ffff640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe6000a2910 0x7fe6000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:28.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.102+0000 7fe60dffb640 1 -- 172.21.15.77:0/1397241986 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe6040f05f0 con 0x7fe6101006e0 2024-03-16T10:16:28.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.102+0000 7fe60ffff640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe6000a2910 0x7fe6000a4dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe610197f90 tx=0x7fe5fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:28.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.106+0000 7fe60dffb640 1 -- 172.21.15.77:0/1397241986 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe60404b020 con 0x7fe6101006e0 2024-03-16T10:16:28.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:27 smithi192 ceph-mon[36354]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:28.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:27 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1349775774' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T10:16:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:27 smithi193 ceph-mon[36311]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:28.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:27 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1349775774' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T10:16:28.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.286+0000 7fe616c97640 1 -- 172.21.15.77:0/1397241986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe6100623e0 con 0x7fe6101006e0 2024-03-16T10:16:28.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.286+0000 7fe60dffb640 1 -- 172.21.15.77:0/1397241986 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fe6040bd7b0 con 0x7fe6101006e0 2024-03-16T10:16:28.287 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T10:16:28.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.288+0000 7fe616c97640 1 -- 172.21.15.77:0/1397241986 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe6000a2910 msgr2=0x7fe6000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.288+0000 7fe616c97640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe6000a2910 0x7fe6000a4dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe610197f90 tx=0x7fe5fc005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:28.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.288+0000 7fe616c97640 1 -- 172.21.15.77:0/1397241986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6101006e0 msgr2=0x7fe610196a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.288+0000 7fe616c97640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6101006e0 0x7fe610196a80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe60402c0b0 tx=0x7fe604032430 comp rx=0 tx=0).stop 2024-03-16T10:16:28.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.288+0000 7fe616c97640 1 -- 172.21.15.77:0/1397241986 shutdown_connections 2024-03-16T10:16:28.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.289+0000 7fe616c97640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe6000a2910 0x7fe6000a4dd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.289+0000 7fe616c97640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe610102290 0x7fe61019e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.289+0000 7fe616c97640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6101018d0 0x7fe610196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.289+0000 7fe616c97640 1 --2- 172.21.15.77:0/1397241986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6101006e0 0x7fe610196a80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.289+0000 7fe616c97640 1 -- 172.21.15.77:0/1397241986 >> 172.21.15.77:0/1397241986 conn(0x7fe6100fbc70 msgr2=0x7fe610104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:28.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.289+0000 7fe616c97640 1 -- 172.21.15.77:0/1397241986 shutdown_connections 2024-03-16T10:16:28.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.289+0000 7fe616c97640 1 -- 172.21.15.77:0/1397241986 wait complete. 2024-03-16T10:16:28.303 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.409+0000 7fe9b0beb640 1 -- 172.21.15.77:0/4010694708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ac100700 msgr2=0x7fe9ac100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.409+0000 7fe9b0beb640 1 --2- 172.21.15.77:0/4010694708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ac100700 0x7fe9ac100b20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009900 tx=0x7fe99c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.409+0000 7fe9b0beb640 1 -- 172.21.15.77:0/4010694708 shutdown_connections 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.409+0000 7fe9b0beb640 1 --2- 172.21.15.77:0/4010694708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ac1022b0 0x7fe9ac108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.409+0000 7fe9b0beb640 1 --2- 172.21.15.77:0/4010694708 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ac1018f0 0x7fe9ac101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.409+0000 7fe9b0beb640 1 --2- 172.21.15.77:0/4010694708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ac100700 0x7fe9ac100b20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.409+0000 7fe9b0beb640 1 -- 172.21.15.77:0/4010694708 >> 172.21.15.77:0/4010694708 conn(0x7fe9ac0fbc90 msgr2=0x7fe9ac0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.409+0000 7fe9b0beb640 1 -- 172.21.15.77:0/4010694708 shutdown_connections 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.409+0000 7fe9b0beb640 1 -- 172.21.15.77:0/4010694708 wait complete. 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9b0beb640 1 Processor -- start 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9b0beb640 1 -- start start 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9b0beb640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ac100700 0x7fe9ac197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9b0beb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ac1018f0 0x7fe9ac1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9b0beb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ac1022b0 0x7fe9ac19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9b0beb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac10c350 con 0x7fe9ac1018f0 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9b0beb640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac10c1d0 con 0x7fe9ac100700 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9b0beb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac10c4d0 con 0x7fe9ac1022b0 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9a1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ac1018f0 0x7fe9ac1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9a1d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ac1018f0 0x7fe9ac1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51568/0 (socket says 172.21.15.77:51568) 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9a1d9b640 1 -- 172.21.15.77:0/3286987381 learned_addr learned my addr 172.21.15.77:0/3286987381 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9aad9d640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ac1022b0 0x7fe9ac19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.410+0000 7fe9aa59c640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ac100700 0x7fe9ac197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.411+0000 7fe9a1d9b640 1 -- 172.21.15.77:0/3286987381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ac1022b0 msgr2=0x7fe9ac19f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.411+0000 7fe9a1d9b640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ac1022b0 0x7fe9ac19f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.411+0000 7fe9a1d9b640 1 -- 172.21.15.77:0/3286987381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ac100700 msgr2=0x7fe9ac197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.411+0000 7fe9a1d9b640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ac100700 0x7fe9ac197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.411+0000 7fe9a1d9b640 1 -- 172.21.15.77:0/3286987381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ac19f770 con 0x7fe9ac1018f0 2024-03-16T10:16:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.411+0000 7fe9aad9d640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ac1022b0 0x7fe9ac19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.411+0000 7fe9a1d9b640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ac1018f0 0x7fe9ac1980e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe99400e9f0 tx=0x7fe99400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.411+0000 7fe9a3fff640 1 -- 172.21.15.77:0/3286987381 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99400cd80 con 0x7fe9ac1018f0 2024-03-16T10:16:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.411+0000 7fe9b0beb640 1 -- 172.21.15.77:0/3286987381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9ac19fa60 con 0x7fe9ac1018f0 2024-03-16T10:16:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.411+0000 7fe9a3fff640 1 -- 172.21.15.77:0/3286987381 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe994010910 con 0x7fe9ac1018f0 2024-03-16T10:16:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.411+0000 7fe9a3fff640 1 -- 172.21.15.77:0/3286987381 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9940188f0 con 0x7fe9ac1018f0 2024-03-16T10:16:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.411+0000 7fe9b0beb640 1 -- 172.21.15.77:0/3286987381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9ac19ffb0 con 0x7fe9ac1018f0 2024-03-16T10:16:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.412+0000 7fe9b0beb640 1 -- 172.21.15.77:0/3286987381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9ac068a20 con 0x7fe9ac1018f0 2024-03-16T10:16:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.413+0000 7fe9a3fff640 1 -- 172.21.15.77:0/3286987381 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe994010ac0 con 0x7fe9ac1018f0 2024-03-16T10:16:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.414+0000 7fe9a3fff640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe9980a2890 0x7fe9980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.414+0000 7fe9aa59c640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe9980a2890 0x7fe9980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.414+0000 7fe9a3fff640 1 -- 172.21.15.77:0/3286987381 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe9940cf270 con 0x7fe9ac1018f0 2024-03-16T10:16:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.414+0000 7fe9aa59c640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe9980a2890 0x7fe9980a4d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe99c000c00 tx=0x7fe99c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:28.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.421+0000 7fe9a3fff640 1 -- 172.21.15.77:0/3286987381 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe99409bc50 con 0x7fe9ac1018f0 2024-03-16T10:16:28.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.599+0000 7fe9b0beb640 1 -- 172.21.15.77:0/3286987381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe9ac1a03f0 con 0x7fe9ac1018f0 2024-03-16T10:16:28.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.599+0000 7fe9a3fff640 1 -- 172.21.15.77:0/3286987381 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fe99409fcf0 con 0x7fe9ac1018f0 2024-03-16T10:16:28.600 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T10:16:28.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.601+0000 7fe9b0beb640 1 -- 172.21.15.77:0/3286987381 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe9980a2890 msgr2=0x7fe9980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.601+0000 7fe9b0beb640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe9980a2890 0x7fe9980a4d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe99c000c00 tx=0x7fe99c005fd0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.601+0000 7fe9b0beb640 1 -- 172.21.15.77:0/3286987381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ac1018f0 msgr2=0x7fe9ac1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.601+0000 7fe9b0beb640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ac1018f0 0x7fe9ac1980e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe99400e9f0 tx=0x7fe99400eeb0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.601+0000 7fe9b0beb640 1 -- 172.21.15.77:0/3286987381 shutdown_connections 2024-03-16T10:16:28.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.601+0000 7fe9b0beb640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe9980a2890 0x7fe9980a4d50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.601+0000 7fe9b0beb640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9ac1022b0 0x7fe9ac19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.601+0000 7fe9b0beb640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9ac1018f0 0x7fe9ac1980e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.601+0000 7fe9b0beb640 1 --2- 172.21.15.77:0/3286987381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ac100700 0x7fe9ac197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.601+0000 7fe9b0beb640 1 -- 172.21.15.77:0/3286987381 >> 172.21.15.77:0/3286987381 conn(0x7fe9ac0fbc90 msgr2=0x7fe9ac0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:28.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.601+0000 7fe9b0beb640 1 -- 172.21.15.77:0/3286987381 shutdown_connections 2024-03-16T10:16:28.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.601+0000 7fe9b0beb640 1 -- 172.21.15.77:0/3286987381 wait complete. 2024-03-16T10:16:28.613 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T10:16:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.732+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/133880809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5adc1051c0 msgr2=0x7f5adc105640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.732+0000 7f5ae0bb0640 1 --2- 172.21.15.77:0/133880809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5adc1051c0 0x7f5adc105640 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5ad401acb0 tx=0x7f5ad403d130 comp rx=0 tx=0).stop 2024-03-16T10:16:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.732+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/133880809 shutdown_connections 2024-03-16T10:16:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.732+0000 7f5ae0bb0640 1 --2- 172.21.15.77:0/133880809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5adc1051c0 0x7f5adc105640 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.732+0000 7f5ae0bb0640 1 --2- 172.21.15.77:0/133880809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5adc102810 0x7f5adc104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.732+0000 7f5ae0bb0640 1 --2- 172.21.15.77:0/133880809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5adc0ffea0 0x7f5adc1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.732+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/133880809 >> 172.21.15.77:0/133880809 conn(0x7f5adc0fbaa0 msgr2=0x7f5adc0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.732+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/133880809 shutdown_connections 2024-03-16T10:16:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.732+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/133880809 wait complete. 2024-03-16T10:16:28.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.733+0000 7f5ae0bb0640 1 Processor -- start 2024-03-16T10:16:28.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.733+0000 7f5ae0bb0640 1 -- start start 2024-03-16T10:16:28.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.734+0000 7f5ae0bb0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5adc0ffea0 0x7f5adc1956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:28.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.734+0000 7f5ae0bb0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5adc102810 0x7f5adc195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:28.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.734+0000 7f5ae0bb0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5adc1051c0 0x7f5adc19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:28.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.734+0000 7f5ae0bb0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1090b0 con 0x7f5adc102810 2024-03-16T10:16:28.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.734+0000 7f5ae0bb0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc108f30 con 0x7f5adc1051c0 2024-03-16T10:16:28.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.734+0000 7f5ae0bb0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc109230 con 0x7f5adc0ffea0 2024-03-16T10:16:28.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.735+0000 7f5adb7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5adc0ffea0 0x7f5adc1956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.735+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5adc1051c0 0x7f5adc19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.735+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5adc1051c0 0x7f5adc19cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:37050/0 (socket says 172.21.15.77:37050) 2024-03-16T10:16:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.735+0000 7f5adbfff640 1 -- 172.21.15.77:0/2238000297 learned_addr learned my addr 172.21.15.77:0/2238000297 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.735+0000 7f5adaffd640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5adc102810 0x7f5adc195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.735+0000 7f5adbfff640 1 -- 172.21.15.77:0/2238000297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5adc0ffea0 msgr2=0x7f5adc1956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.735+0000 7f5adbfff640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5adc0ffea0 0x7f5adc1956c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.735+0000 7f5adbfff640 1 -- 172.21.15.77:0/2238000297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5adc102810 msgr2=0x7f5adc195c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.735+0000 7f5adbfff640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5adc102810 0x7f5adc195c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.735+0000 7f5adbfff640 1 -- 172.21.15.77:0/2238000297 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5adc19d2e0 con 0x7f5adc1051c0 2024-03-16T10:16:28.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.736+0000 7f5adbfff640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5adc1051c0 0x7f5adc19cc70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4043420 tx=0x7f5ad4004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:28.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.741+0000 7f5ad8ff9640 1 -- 172.21.15.77:0/2238000297 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad404a040 con 0x7f5adc1051c0 2024-03-16T10:16:28.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.741+0000 7f5ad8ff9640 1 -- 172.21.15.77:0/2238000297 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ad4055070 con 0x7f5adc1051c0 2024-03-16T10:16:28.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.741+0000 7f5ad8ff9640 1 -- 172.21.15.77:0/2238000297 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4043720 con 0x7f5adc1051c0 2024-03-16T10:16:28.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.741+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/2238000297 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5adc19d570 con 0x7f5adc1051c0 2024-03-16T10:16:28.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.741+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/2238000297 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5adc19d950 con 0x7f5adc1051c0 2024-03-16T10:16:28.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.743+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/2238000297 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5adc19dc70 con 0x7f5adc1051c0 2024-03-16T10:16:28.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.745+0000 7f5ad8ff9640 1 -- 172.21.15.77:0/2238000297 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5ad4082b70 con 0x7f5adc1051c0 2024-03-16T10:16:28.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.746+0000 7f5ad8ff9640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ac40a2a90 0x7f5ac40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:28.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.746+0000 7f5adb7fe640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ac40a2a90 0x7f5ac40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:28.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.746+0000 7f5ad8ff9640 1 -- 172.21.15.77:0/2238000297 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f5ad4101ea0 con 0x7f5adc1051c0 2024-03-16T10:16:28.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.747+0000 7f5adb7fe640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ac40a2a90 0x7f5ac40a4f50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009bd0 tx=0x7f5acc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:28.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.756+0000 7f5ad8ff9640 1 -- 172.21.15.77:0/2238000297 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ad40cef60 con 0x7f5adc1051c0 2024-03-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.947+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/2238000297 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5adc19e000 con 0x7f5adc1051c0 2024-03-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.947+0000 7f5ad8ff9640 1 -- 172.21.15.77:0/2238000297 <== mon.2 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f5ad4004620 con 0x7f5adc1051c0 2024-03-16T10:16:28.948 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.949+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/2238000297 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ac40a2a90 msgr2=0x7f5ac40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.949+0000 7f5ae0bb0640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ac40a2a90 0x7f5ac40a4f50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009bd0 tx=0x7f5acc005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.949+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/2238000297 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5adc1051c0 msgr2=0x7f5adc19cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:28.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.949+0000 7f5ae0bb0640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5adc1051c0 0x7f5adc19cc70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4043420 tx=0x7f5ad4004530 comp rx=0 tx=0).stop 2024-03-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.949+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/2238000297 shutdown_connections 2024-03-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.949+0000 7f5ae0bb0640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5adc1051c0 0x7f5adc19cc70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.949+0000 7f5ae0bb0640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5adc102810 0x7f5adc195c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.949+0000 7f5ae0bb0640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5ac40a2a90 0x7f5ac40a4f50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.949+0000 7f5ae0bb0640 1 --2- 172.21.15.77:0/2238000297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5adc0ffea0 0x7f5adc1956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.949+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/2238000297 >> 172.21.15.77:0/2238000297 conn(0x7f5adc0fbaa0 msgr2=0x7f5adc0fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.949+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/2238000297 shutdown_connections 2024-03-16T10:16:28.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:28.950+0000 7f5ae0bb0640 1 -- 172.21.15.77:0/2238000297 wait complete. 2024-03-16T10:16:28.960 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:28 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/337307551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:28.960 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:28 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1397241986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:16:28.960 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:28 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3286987381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:16:28.962 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:16:29.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.064+0000 7f5e2a33b640 1 -- 172.21.15.77:0/2034790950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e24101810 msgr2=0x7f5e24101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.064+0000 7f5e2a33b640 1 --2- 172.21.15.77:0/2034790950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e24101810 0x7f5e24101c30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009900 tx=0x7f5e1402bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.064+0000 7f5e2a33b640 1 -- 172.21.15.77:0/2034790950 shutdown_connections 2024-03-16T10:16:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.064+0000 7f5e2a33b640 1 --2- 172.21.15.77:0/2034790950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e241033c0 0x7f5e24109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.064+0000 7f5e2a33b640 1 --2- 172.21.15.77:0/2034790950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e24102a00 0x7f5e24102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.064+0000 7f5e2a33b640 1 --2- 172.21.15.77:0/2034790950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e24101810 0x7f5e24101c30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.064+0000 7f5e2a33b640 1 -- 172.21.15.77:0/2034790950 >> 172.21.15.77:0/2034790950 conn(0x7f5e240fcd80 msgr2=0x7f5e240ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.064+0000 7f5e2a33b640 1 -- 172.21.15.77:0/2034790950 shutdown_connections 2024-03-16T10:16:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.065+0000 7f5e2a33b640 1 -- 172.21.15.77:0/2034790950 wait complete. 2024-03-16T10:16:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.065+0000 7f5e2a33b640 1 Processor -- start 2024-03-16T10:16:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.065+0000 7f5e2a33b640 1 -- start start 2024-03-16T10:16:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.065+0000 7f5e2a33b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e24101810 0x7f5e24197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.065+0000 7f5e2a33b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e24102a00 0x7f5e24198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.065+0000 7f5e2a33b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e241033c0 0x7f5e2419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.065+0000 7f5e2a33b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2410d490 con 0x7f5e24101810 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.065+0000 7f5e2a33b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2410d310 con 0x7f5e24102a00 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.065+0000 7f5e2a33b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2410d610 con 0x7f5e241033c0 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.065+0000 7f5e23fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e24101810 0x7f5e24197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e23fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e24101810 0x7f5e24197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51610/0 (socket says 172.21.15.77:51610) 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e23fff640 1 -- 172.21.15.77:0/86345845 learned_addr learned my addr 172.21.15.77:0/86345845 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e237fe640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e24102a00 0x7f5e24198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e288d8640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e241033c0 0x7f5e2419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e23fff640 1 -- 172.21.15.77:0/86345845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e241033c0 msgr2=0x7f5e2419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e23fff640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e241033c0 0x7f5e2419f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e23fff640 1 -- 172.21.15.77:0/86345845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e24102a00 msgr2=0x7f5e24198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e23fff640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e24102a00 0x7f5e24198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e23fff640 1 -- 172.21.15.77:0/86345845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2419f720 con 0x7f5e24101810 2024-03-16T10:16:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e23fff640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e24101810 0x7f5e24197b00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5e14000c00 tx=0x7f5e1402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e217fa640 1 -- 172.21.15.77:0/86345845 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14041070 con 0x7f5e24101810 2024-03-16T10:16:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e2a33b640 1 -- 172.21.15.77:0/86345845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2419f9b0 con 0x7f5e24101810 2024-03-16T10:16:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e217fa640 1 -- 172.21.15.77:0/86345845 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e14004450 con 0x7f5e24101810 2024-03-16T10:16:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e217fa640 1 -- 172.21.15.77:0/86345845 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14034440 con 0x7f5e24101810 2024-03-16T10:16:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.066+0000 7f5e2a33b640 1 -- 172.21.15.77:0/86345845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e2419fef0 con 0x7f5e24101810 2024-03-16T10:16:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.067+0000 7f5e2a33b640 1 -- 172.21.15.77:0/86345845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e24108b30 con 0x7f5e24101810 2024-03-16T10:16:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.069+0000 7f5e217fa640 1 -- 172.21.15.77:0/86345845 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5e14044070 con 0x7f5e24101810 2024-03-16T10:16:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.070+0000 7f5e217fa640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5e0c0a2890 0x7f5e0c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.070+0000 7f5e237fe640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5e0c0a2890 0x7f5e0c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.070+0000 7f5e217fa640 1 -- 172.21.15.77:0/86345845 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f5e140f06e0 con 0x7f5e24101810 2024-03-16T10:16:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.070+0000 7f5e237fe640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5e0c0a2890 0x7f5e0c0a4d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5e24199010 tx=0x7f5e18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:29.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.072+0000 7f5e217fa640 1 -- 172.21.15.77:0/86345845 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5e140bd260 con 0x7f5e24101810 2024-03-16T10:16:29.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:28 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/337307551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:29.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:28 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1397241986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:16:29.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:28 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3286987381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:16:29.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:28 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/337307551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:29.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:28 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1397241986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:16:29.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:28 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3286987381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:16:29.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.289+0000 7f5e2a33b640 1 -- 172.21.15.77:0/86345845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e2402d1b0 con 0x7f5e24101810 2024-03-16T10:16:29.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.290+0000 7f5e217fa640 1 -- 172.21.15.77:0/86345845 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11938 (secure 0 0 0) 0x7f5e140c1300 con 0x7f5e24101810 2024-03-16T10:16:29.290 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:29.291 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:15:09.227516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[14658,24455,24482,24454,24475]},"id":1}]} 2024-03-16T10:16:29.291 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-16T10:16:29.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.292+0000 7f5e2a33b640 1 -- 172.21.15.77:0/86345845 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5e0c0a2890 msgr2=0x7f5e0c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.292+0000 7f5e2a33b640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5e0c0a2890 0x7f5e0c0a4d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5e24199010 tx=0x7f5e18005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:29.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.292+0000 7f5e2a33b640 1 -- 172.21.15.77:0/86345845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e24101810 msgr2=0x7f5e24197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.292+0000 7f5e2a33b640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e24101810 0x7f5e24197b00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5e14000c00 tx=0x7f5e1402c640 comp rx=0 tx=0).stop 2024-03-16T10:16:29.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.292+0000 7f5e2a33b640 1 -- 172.21.15.77:0/86345845 shutdown_connections 2024-03-16T10:16:29.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.292+0000 7f5e2a33b640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5e0c0a2890 0x7f5e0c0a4d50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.292+0000 7f5e2a33b640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e241033c0 0x7f5e2419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.292+0000 7f5e2a33b640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e24102a00 0x7f5e24198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.292+0000 7f5e2a33b640 1 --2- 172.21.15.77:0/86345845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e24101810 0x7f5e24197b00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.292+0000 7f5e2a33b640 1 -- 172.21.15.77:0/86345845 >> 172.21.15.77:0/86345845 conn(0x7f5e240fcd80 msgr2=0x7f5e24105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:29.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.292+0000 7f5e2a33b640 1 -- 172.21.15.77:0/86345845 shutdown_connections 2024-03-16T10:16:29.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.292+0000 7f5e2a33b640 1 -- 172.21.15.77:0/86345845 wait complete. 2024-03-16T10:16:29.307 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:16:29.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.409+0000 7f08bffd3640 1 -- 172.21.15.77:0/311478574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b81018f0 msgr2=0x7f08b8101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.409+0000 7f08bffd3640 1 --2- 172.21.15.77:0/311478574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b81018f0 0x7f08b8101d70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f08a8009900 tx=0x7f08a802bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:29.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.409+0000 7f08bffd3640 1 -- 172.21.15.77:0/311478574 shutdown_connections 2024-03-16T10:16:29.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.409+0000 7f08bffd3640 1 --2- 172.21.15.77:0/311478574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b81022b0 0x7f08b8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.409+0000 7f08bffd3640 1 --2- 172.21.15.77:0/311478574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b81018f0 0x7f08b8101d70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.409+0000 7f08bffd3640 1 --2- 172.21.15.77:0/311478574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08b8100700 0x7f08b8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.409+0000 7f08bffd3640 1 -- 172.21.15.77:0/311478574 >> 172.21.15.77:0/311478574 conn(0x7f08b80fbc70 msgr2=0x7f08b80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:29.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.409+0000 7f08bffd3640 1 -- 172.21.15.77:0/311478574 shutdown_connections 2024-03-16T10:16:29.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.409+0000 7f08bffd3640 1 -- 172.21.15.77:0/311478574 wait complete. 2024-03-16T10:16:29.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08bffd3640 1 Processor -- start 2024-03-16T10:16:29.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08bffd3640 1 -- start start 2024-03-16T10:16:29.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08bffd3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b8100700 0x7f08b8196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:29.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08bffd3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08b81018f0 0x7f08b8196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:29.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08bffd3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b81022b0 0x7f08b819dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:29.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08bffd3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f08b810c380 con 0x7f08b81022b0 2024-03-16T10:16:29.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08bffd3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f08b810c200 con 0x7f08b81018f0 2024-03-16T10:16:29.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08bffd3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f08b810c500 con 0x7f08b8100700 2024-03-16T10:16:29.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08be570640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b81022b0 0x7f08b819dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:29.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08be570640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b81022b0 0x7f08b819dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51624/0 (socket says 172.21.15.77:51624) 2024-03-16T10:16:29.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08be570640 1 -- 172.21.15.77:0/3619895055 learned_addr learned my addr 172.21.15.77:0/3619895055 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:29.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08bd56e640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08b81018f0 0x7f08b8196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:29.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08bdd6f640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b8100700 0x7f08b8196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:29.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08be570640 1 -- 172.21.15.77:0/3619895055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b8100700 msgr2=0x7f08b8196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08be570640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b8100700 0x7f08b8196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08be570640 1 -- 172.21.15.77:0/3619895055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08b81018f0 msgr2=0x7f08b8196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08be570640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08b81018f0 0x7f08b8196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08be570640 1 -- 172.21.15.77:0/3619895055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b819e640 con 0x7f08b81022b0 2024-03-16T10:16:29.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.410+0000 7f08be570640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b81022b0 0x7f08b819dfd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f08b000eb10 tx=0x7f08b000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:29.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.411+0000 7f08aeffd640 1 -- 172.21.15.77:0/3619895055 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b0004b20 con 0x7f08b81022b0 2024-03-16T10:16:29.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.411+0000 7f08aeffd640 1 -- 172.21.15.77:0/3619895055 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08b0004cc0 con 0x7f08b81022b0 2024-03-16T10:16:29.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.411+0000 7f08bffd3640 1 -- 172.21.15.77:0/3619895055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08b819e930 con 0x7f08b81022b0 2024-03-16T10:16:29.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.411+0000 7f08aeffd640 1 -- 172.21.15.77:0/3619895055 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b0010760 con 0x7f08b81022b0 2024-03-16T10:16:29.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.411+0000 7f08bffd3640 1 -- 172.21.15.77:0/3619895055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08b819eea0 con 0x7f08b81022b0 2024-03-16T10:16:29.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.412+0000 7f08bffd3640 1 -- 172.21.15.77:0/3619895055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08b8107a20 con 0x7f08b81022b0 2024-03-16T10:16:29.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.413+0000 7f08aeffd640 1 -- 172.21.15.77:0/3619895055 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f08b0005020 con 0x7f08b81022b0 2024-03-16T10:16:29.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.413+0000 7f08aeffd640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f08a40a2910 0x7f08a40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:29.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.414+0000 7f08bdd6f640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f08a40a2910 0x7f08a40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:29.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.414+0000 7f08aeffd640 1 -- 172.21.15.77:0/3619895055 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f08b00cf150 con 0x7f08b81022b0 2024-03-16T10:16:29.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.414+0000 7f08bdd6f640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f08a40a2910 0x7f08a40a4dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f08b8101750 tx=0x7f08b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:29.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.418+0000 7f08aeffd640 1 -- 172.21.15.77:0/3619895055 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f08b009c180 con 0x7f08b81022b0 2024-03-16T10:16:29.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.597+0000 7f08bffd3640 1 -- 172.21.15.77:0/3619895055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08b802d1b0 con 0x7f08b81022b0 2024-03-16T10:16:29.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.599+0000 7f08aeffd640 1 -- 172.21.15.77:0/3619895055 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22496 (secure 0 0 0) 0x7f08b009ba80 con 0x7f08b81022b0 2024-03-16T10:16:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:29.600 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":87,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:59.826259+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:16:29.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.601+0000 7f08bffd3640 1 -- 172.21.15.77:0/3619895055 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f08a40a2910 msgr2=0x7f08a40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.601+0000 7f08bffd3640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f08a40a2910 0x7f08a40a4dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f08b8101750 tx=0x7f08b4005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:29.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.601+0000 7f08bffd3640 1 -- 172.21.15.77:0/3619895055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b81022b0 msgr2=0x7f08b819dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.601+0000 7f08bffd3640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b81022b0 0x7f08b819dfd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f08b000eb10 tx=0x7f08b000c2d0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.601+0000 7f08bffd3640 1 -- 172.21.15.77:0/3619895055 shutdown_connections 2024-03-16T10:16:29.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.601+0000 7f08bffd3640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b81022b0 0x7f08b819dfd0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.601+0000 7f08bffd3640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08b81018f0 0x7f08b8196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.601+0000 7f08bffd3640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f08a40a2910 0x7f08a40a4dd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.601+0000 7f08bffd3640 1 --2- 172.21.15.77:0/3619895055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b8100700 0x7f08b8196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.601+0000 7f08bffd3640 1 -- 172.21.15.77:0/3619895055 >> 172.21.15.77:0/3619895055 conn(0x7f08b80fbc70 msgr2=0x7f08b8104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:29.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.601+0000 7f08bffd3640 1 -- 172.21.15.77:0/3619895055 shutdown_connections 2024-03-16T10:16:29.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.601+0000 7f08bffd3640 1 -- 172.21.15.77:0/3619895055 wait complete. 2024-03-16T10:16:29.613 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:10:22.965004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T10:15:00.264039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T10:15:01.536067+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T10:16:29.613 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T10:16:29.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.719+0000 7f046b1b3640 1 -- 172.21.15.77:0/1071505938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04641018d0 msgr2=0x7f0464101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.719+0000 7f046b1b3640 1 --2- 172.21.15.77:0/1071505938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04641018d0 0x7f0464101d50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0458009860 tx=0x7f045802bb70 comp rx=0 tx=0).stop 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.719+0000 7f046b1b3640 1 -- 172.21.15.77:0/1071505938 shutdown_connections 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.719+0000 7f046b1b3640 1 --2- 172.21.15.77:0/1071505938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0464102290 0x7f0464108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.719+0000 7f046b1b3640 1 --2- 172.21.15.77:0/1071505938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04641018d0 0x7f0464101d50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.719+0000 7f046b1b3640 1 --2- 172.21.15.77:0/1071505938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04641006e0 0x7f0464100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.719+0000 7f046b1b3640 1 -- 172.21.15.77:0/1071505938 >> 172.21.15.77:0/1071505938 conn(0x7f04640fbc70 msgr2=0x7f04640fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.719+0000 7f046b1b3640 1 -- 172.21.15.77:0/1071505938 shutdown_connections 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.719+0000 7f046b1b3640 1 -- 172.21.15.77:0/1071505938 wait complete. 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f046b1b3640 1 Processor -- start 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f046b1b3640 1 -- start start 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f046b1b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04641006e0 0x7f0464197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f046b1b3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04641018d0 0x7f0464198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f046b1b3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0464102290 0x7f046419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:29.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f046b1b3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f046410d480 con 0x7f04641006e0 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f046b1b3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f046410d300 con 0x7f0464102290 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f046b1b3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f046410d600 con 0x7f04641018d0 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f0468f4f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04641006e0 0x7f0464197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f0468f4f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04641006e0 0x7f0464197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51634/0 (socket says 172.21.15.77:51634) 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f0468f4f640 1 -- 172.21.15.77:0/3781237627 learned_addr learned my addr 172.21.15.77:0/3781237627 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f0463fff640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04641018d0 0x7f0464198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.720+0000 7f0469750640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0464102290 0x7f046419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.721+0000 7f0468f4f640 1 -- 172.21.15.77:0/3781237627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04641018d0 msgr2=0x7f0464198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.721+0000 7f0468f4f640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04641018d0 0x7f0464198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.721+0000 7f0468f4f640 1 -- 172.21.15.77:0/3781237627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0464102290 msgr2=0x7f046419f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.721+0000 7f0468f4f640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0464102290 0x7f046419f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.721+0000 7f0468f4f640 1 -- 172.21.15.77:0/3781237627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046419f730 con 0x7f04641006e0 2024-03-16T10:16:29.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.721+0000 7f0468f4f640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04641006e0 0x7f0464197b10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f045c00c3f0 tx=0x7f045c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.721+0000 7f0461ffb640 1 -- 172.21.15.77:0/3781237627 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c01b070 con 0x7f04641006e0 2024-03-16T10:16:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.721+0000 7f046b1b3640 1 -- 172.21.15.77:0/3781237627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f046419fa20 con 0x7f04641006e0 2024-03-16T10:16:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.721+0000 7f0461ffb640 1 -- 172.21.15.77:0/3781237627 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f045c00bb10 con 0x7f04641006e0 2024-03-16T10:16:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.721+0000 7f0461ffb640 1 -- 172.21.15.77:0/3781237627 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c016d80 con 0x7f04641006e0 2024-03-16T10:16:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.721+0000 7f046b1b3640 1 -- 172.21.15.77:0/3781237627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f046419ff90 con 0x7f04641006e0 2024-03-16T10:16:29.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.722+0000 7f046b1b3640 1 -- 172.21.15.77:0/3781237627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0464107aa0 con 0x7f04641006e0 2024-03-16T10:16:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.724+0000 7f0461ffb640 1 -- 172.21.15.77:0/3781237627 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f045c01e070 con 0x7f04641006e0 2024-03-16T10:16:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.724+0000 7f0461ffb640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f044c0a2890 0x7f044c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.724+0000 7f0463fff640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f044c0a2890 0x7f044c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.724+0000 7f0461ffb640 1 -- 172.21.15.77:0/3781237627 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f045c0ce9a0 con 0x7f04641006e0 2024-03-16T10:16:29.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.725+0000 7f0463fff640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f044c0a2890 0x7f044c0a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f045802c180 tx=0x7f0458034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:29.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.729+0000 7f0461ffb640 1 -- 172.21.15.77:0/3781237627 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f045c09b380 con 0x7f04641006e0 2024-03-16T10:16:29.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.840+0000 7f0461ffb640 1 -- 172.21.15.77:0/3781237627 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f045c0979f0 con 0x7f04641006e0 2024-03-16T10:16:29.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.959+0000 7f046b1b3640 1 -- 172.21.15.77:0/3781237627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f046402d3d0 con 0x7f04641006e0 2024-03-16T10:16:29.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.959+0000 7f0461ffb640 1 -- 172.21.15.77:0/3781237627 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f045c08cad0 con 0x7f04641006e0 2024-03-16T10:16:29.960 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T10:16:29.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.961+0000 7f046b1b3640 1 -- 172.21.15.77:0/3781237627 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f044c0a2890 msgr2=0x7f044c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.961+0000 7f046b1b3640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f044c0a2890 0x7f044c0a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f045802c180 tx=0x7f0458034ad0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.961+0000 7f046b1b3640 1 -- 172.21.15.77:0/3781237627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04641006e0 msgr2=0x7f0464197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:29.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.961+0000 7f046b1b3640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04641006e0 0x7f0464197b10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f045c00c3f0 tx=0x7f045c009f90 comp rx=0 tx=0).stop 2024-03-16T10:16:29.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.961+0000 7f046b1b3640 1 -- 172.21.15.77:0/3781237627 shutdown_connections 2024-03-16T10:16:29.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.961+0000 7f046b1b3640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0464102290 0x7f046419f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.961+0000 7f046b1b3640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f044c0a2890 0x7f044c0a4d50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.961+0000 7f046b1b3640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04641018d0 0x7f0464198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.961+0000 7f046b1b3640 1 --2- 172.21.15.77:0/3781237627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04641006e0 0x7f0464197b10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:29.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.961+0000 7f046b1b3640 1 -- 172.21.15.77:0/3781237627 >> 172.21.15.77:0/3781237627 conn(0x7f04640fbc70 msgr2=0x7f0464104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:29.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.961+0000 7f046b1b3640 1 -- 172.21.15.77:0/3781237627 shutdown_connections 2024-03-16T10:16:29.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:29.962+0000 7f046b1b3640 1 -- 172.21.15.77:0/3781237627 wait complete. 2024-03-16T10:16:29.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:29 smithi077 ceph-mon[32400]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T10:16:29.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:29 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2238000297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:16:29.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:29 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/86345845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:29.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:29 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3619895055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:29.976 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T10:16:30.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.090+0000 7f19f8c85640 1 -- 172.21.15.77:0/1792672328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19f4102a00 msgr2=0x7f19f4102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.090+0000 7f19f8c85640 1 --2- 172.21.15.77:0/1792672328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19f4102a00 0x7f19f4102e80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009900 tx=0x7f19dc02bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:30.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.090+0000 7f19f8c85640 1 -- 172.21.15.77:0/1792672328 shutdown_connections 2024-03-16T10:16:30.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.090+0000 7f19f8c85640 1 --2- 172.21.15.77:0/1792672328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f41033c0 0x7f19f4109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.090+0000 7f19f8c85640 1 --2- 172.21.15.77:0/1792672328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19f4102a00 0x7f19f4102e80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.090+0000 7f19f8c85640 1 --2- 172.21.15.77:0/1792672328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f4101810 0x7f19f4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.090+0000 7f19f8c85640 1 -- 172.21.15.77:0/1792672328 >> 172.21.15.77:0/1792672328 conn(0x7f19f40fcd80 msgr2=0x7f19f40ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.090+0000 7f19f8c85640 1 -- 172.21.15.77:0/1792672328 shutdown_connections 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.090+0000 7f19f8c85640 1 -- 172.21.15.77:0/1792672328 wait complete. 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19f8c85640 1 Processor -- start 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19f8c85640 1 -- start start 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19f8c85640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f4101810 0x7f19f4197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19f8c85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19f4102a00 0x7f19f4198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19f8c85640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f41033c0 0x7f19f419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19f8c85640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f19f410d490 con 0x7f19f4102a00 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19f8c85640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f19f410d310 con 0x7f19f4101810 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19f8c85640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f19f410d610 con 0x7f19f41033c0 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19ebfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19f4102a00 0x7f19f4198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19ebfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19f4102a00 0x7f19f4198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51648/0 (socket says 172.21.15.77:51648) 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19f259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f4101810 0x7f19f4197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:30.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19ebfff640 1 -- 172.21.15.77:0/3668476787 learned_addr learned my addr 172.21.15.77:0/3668476787 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.091+0000 7f19f2d9d640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f41033c0 0x7f19f419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.092+0000 7f19ebfff640 1 -- 172.21.15.77:0/3668476787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f41033c0 msgr2=0x7f19f419f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.092+0000 7f19ebfff640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f41033c0 0x7f19f419f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.092+0000 7f19ebfff640 1 -- 172.21.15.77:0/3668476787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f4101810 msgr2=0x7f19f4197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.092+0000 7f19ebfff640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f4101810 0x7f19f4197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.092+0000 7f19ebfff640 1 -- 172.21.15.77:0/3668476787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f419f750 con 0x7f19f4102a00 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.092+0000 7f19f259c640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f4101810 0x7f19f4197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.092+0000 7f19ebfff640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19f4102a00 0x7f19f4198070 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009230 tx=0x7f19dc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.092+0000 7f19eb7fe640 1 -- 172.21.15.77:0/3668476787 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc041070 con 0x7f19f4102a00 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.092+0000 7f19eb7fe640 1 -- 172.21.15.77:0/3668476787 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19dc004450 con 0x7f19f4102a00 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.092+0000 7f19f8c85640 1 -- 172.21.15.77:0/3668476787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19f419f9e0 con 0x7f19f4102a00 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.092+0000 7f19eb7fe640 1 -- 172.21.15.77:0/3668476787 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc003e20 con 0x7f19f4102a00 2024-03-16T10:16:30.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.092+0000 7f19f8c85640 1 -- 172.21.15.77:0/3668476787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19f419fed0 con 0x7f19f4102a00 2024-03-16T10:16:30.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.093+0000 7f19f8c85640 1 -- 172.21.15.77:0/3668476787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19f4108ba0 con 0x7f19f4102a00 2024-03-16T10:16:30.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.094+0000 7f19eb7fe640 1 -- 172.21.15.77:0/3668476787 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f19dc044070 con 0x7f19f4102a00 2024-03-16T10:16:30.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.095+0000 7f19eb7fe640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f19e00a2800 0x7f19e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:30.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.095+0000 7f19f259c640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f19e00a2800 0x7f19e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:30.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.095+0000 7f19eb7fe640 1 -- 172.21.15.77:0/3668476787 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f19dc0f0560 con 0x7f19f4102a00 2024-03-16T10:16:30.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.095+0000 7f19f259c640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f19e00a2800 0x7f19e00a4cc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f19f4102860 tx=0x7f19e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:30.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.104+0000 7f19eb7fe640 1 -- 172.21.15.77:0/3668476787 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f19dc0bd040 con 0x7f19f4102a00 2024-03-16T10:16:30.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:29 smithi192 ceph-mon[36354]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T10:16:30.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:29 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2238000297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:16:30.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:29 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/86345845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:30.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:29 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3619895055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:30.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:29 smithi193 ceph-mon[36311]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T10:16:30.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:29 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2238000297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:16:30.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:29 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/86345845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:30.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:29 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3619895055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:30.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.292+0000 7f19f8c85640 1 -- 172.21.15.77:0/3668476787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f19f402d400 con 0x7f19f4102a00 2024-03-16T10:16:30.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.292+0000 7f19eb7fe640 1 -- 172.21.15.77:0/3668476787 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f19dc0c10e0 con 0x7f19f4102a00 2024-03-16T10:16:30.293 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T10:16:30.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.294+0000 7f19f8c85640 1 -- 172.21.15.77:0/3668476787 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f19e00a2800 msgr2=0x7f19e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.294+0000 7f19f8c85640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f19e00a2800 0x7f19e00a4cc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f19f4102860 tx=0x7f19e4005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:30.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.294+0000 7f19f8c85640 1 -- 172.21.15.77:0/3668476787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19f4102a00 msgr2=0x7f19f4198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.294+0000 7f19f8c85640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19f4102a00 0x7f19f4198070 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009230 tx=0x7f19dc02c650 comp rx=0 tx=0).stop 2024-03-16T10:16:30.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.294+0000 7f19f8c85640 1 -- 172.21.15.77:0/3668476787 shutdown_connections 2024-03-16T10:16:30.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.294+0000 7f19f8c85640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f19e00a2800 0x7f19e00a4cc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.294+0000 7f19f8c85640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19f41033c0 0x7f19f419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.294+0000 7f19f8c85640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19f4102a00 0x7f19f4198070 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.294+0000 7f19f8c85640 1 --2- 172.21.15.77:0/3668476787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19f4101810 0x7f19f4197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.294+0000 7f19f8c85640 1 -- 172.21.15.77:0/3668476787 >> 172.21.15.77:0/3668476787 conn(0x7f19f40fcd80 msgr2=0x7f19f4105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:30.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.294+0000 7f19f8c85640 1 -- 172.21.15.77:0/3668476787 shutdown_connections 2024-03-16T10:16:30.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.294+0000 7f19f8c85640 1 -- 172.21.15.77:0/3668476787 wait complete. 2024-03-16T10:16:30.306 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T10:16:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.411+0000 7fafbe568640 1 -- 172.21.15.77:0/2475843624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb81006e0 msgr2=0x7fafb8100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.411+0000 7fafbe568640 1 --2- 172.21.15.77:0/2475843624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb81006e0 0x7fafb8100b00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009860 tx=0x7fafa802bb70 comp rx=0 tx=0).stop 2024-03-16T10:16:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.411+0000 7fafbe568640 1 -- 172.21.15.77:0/2475843624 shutdown_connections 2024-03-16T10:16:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.411+0000 7fafbe568640 1 --2- 172.21.15.77:0/2475843624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb8102290 0x7fafb8108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.411+0000 7fafbe568640 1 --2- 172.21.15.77:0/2475843624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb81018d0 0x7fafb8101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.411+0000 7fafbe568640 1 --2- 172.21.15.77:0/2475843624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb81006e0 0x7fafb8100b00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.411+0000 7fafbe568640 1 -- 172.21.15.77:0/2475843624 >> 172.21.15.77:0/2475843624 conn(0x7fafb80fbc70 msgr2=0x7fafb80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.411+0000 7fafbe568640 1 -- 172.21.15.77:0/2475843624 shutdown_connections 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.411+0000 7fafbe568640 1 -- 172.21.15.77:0/2475843624 wait complete. 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.411+0000 7fafbe568640 1 Processor -- start 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.411+0000 7fafbe568640 1 -- start start 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbe568640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb81006e0 0x7fafb8196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbe568640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb81018d0 0x7fafb8196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbe568640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb8102290 0x7fafb819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbe568640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fafb810c330 con 0x7fafb8102290 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbe568640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fafb810c1b0 con 0x7fafb81018d0 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbe568640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fafb810c4b0 con 0x7fafb81006e0 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbcb05640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb8102290 0x7fafb819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbcb05640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb8102290 0x7fafb819e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51660/0 (socket says 172.21.15.77:51660) 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbcb05640 1 -- 172.21.15.77:0/1419455519 learned_addr learned my addr 172.21.15.77:0/1419455519 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafb77fe640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb81018d0 0x7fafb8196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafb7fff640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb81006e0 0x7fafb8196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbcb05640 1 -- 172.21.15.77:0/1419455519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb81006e0 msgr2=0x7fafb8196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbcb05640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb81006e0 0x7fafb8196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbcb05640 1 -- 172.21.15.77:0/1419455519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb81018d0 msgr2=0x7fafb8196fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbcb05640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb81018d0 0x7fafb8196fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbcb05640 1 -- 172.21.15.77:0/1419455519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb819e680 con 0x7fafb8102290 2024-03-16T10:16:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbcb05640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb8102290 0x7fafb819e010 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fafb000efc0 tx=0x7fafb000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafb57fa640 1 -- 172.21.15.77:0/1419455519 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb0003970 con 0x7fafb8102290 2024-03-16T10:16:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.412+0000 7fafbe568640 1 -- 172.21.15.77:0/1419455519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb819e970 con 0x7fafb8102290 2024-03-16T10:16:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.413+0000 7fafb57fa640 1 -- 172.21.15.77:0/1419455519 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb0004020 con 0x7fafb8102290 2024-03-16T10:16:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.413+0000 7fafbe568640 1 -- 172.21.15.77:0/1419455519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb819eec0 con 0x7fafb8102290 2024-03-16T10:16:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.413+0000 7fafb57fa640 1 -- 172.21.15.77:0/1419455519 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb000b9f0 con 0x7fafb8102290 2024-03-16T10:16:30.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.413+0000 7fafbe568640 1 -- 172.21.15.77:0/1419455519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafb81074e0 con 0x7fafb8102290 2024-03-16T10:16:30.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.415+0000 7fafb57fa640 1 -- 172.21.15.77:0/1419455519 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fafb001d020 con 0x7fafb8102290 2024-03-16T10:16:30.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.415+0000 7fafb57fa640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fafa00a4a10 0x7fafa00a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:30.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.416+0000 7fafb7fff640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fafa00a4a10 0x7fafa00a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:30.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.416+0000 7fafb57fa640 1 -- 172.21.15.77:0/1419455519 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fafb00ceea0 con 0x7fafb8102290 2024-03-16T10:16:30.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.416+0000 7fafb7fff640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fafa00a4a10 0x7fafa00a6ed0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009830 tx=0x7fafa8034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:30.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.419+0000 7fafb57fa640 1 -- 172.21.15.77:0/1419455519 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fafb009bfd0 con 0x7fafb8102290 2024-03-16T10:16:30.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.600+0000 7fafbe568640 1 -- 172.21.15.77:0/1419455519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fafb8069100 con 0x7fafb8102290 2024-03-16T10:16:30.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.600+0000 7fafb57fa640 1 -- 172.21.15.77:0/1419455519 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fafb009b8d0 con 0x7fafb8102290 2024-03-16T10:16:30.601 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T10:16:30.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.602+0000 7fafbe568640 1 -- 172.21.15.77:0/1419455519 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fafa00a4a10 msgr2=0x7fafa00a6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.602+0000 7fafbe568640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fafa00a4a10 0x7fafa00a6ed0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009830 tx=0x7fafa8034ad0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.602+0000 7fafbe568640 1 -- 172.21.15.77:0/1419455519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb8102290 msgr2=0x7fafb819e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.602+0000 7fafbe568640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb8102290 0x7fafb819e010 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fafb000efc0 tx=0x7fafb000c800 comp rx=0 tx=0).stop 2024-03-16T10:16:30.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.602+0000 7fafbe568640 1 -- 172.21.15.77:0/1419455519 shutdown_connections 2024-03-16T10:16:30.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.602+0000 7fafbe568640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb8102290 0x7fafb819e010 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.602+0000 7fafbe568640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb81018d0 0x7fafb8196fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.602+0000 7fafbe568640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fafa00a4a10 0x7fafa00a6ed0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.602+0000 7fafbe568640 1 --2- 172.21.15.77:0/1419455519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb81006e0 0x7fafb8196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.602+0000 7fafbe568640 1 -- 172.21.15.77:0/1419455519 >> 172.21.15.77:0/1419455519 conn(0x7fafb80fbc70 msgr2=0x7fafb8104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:30.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.602+0000 7fafbe568640 1 -- 172.21.15.77:0/1419455519 shutdown_connections 2024-03-16T10:16:30.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.602+0000 7fafbe568640 1 -- 172.21.15.77:0/1419455519 wait complete. 2024-03-16T10:16:30.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:16:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 -- 172.21.15.77:0/411487030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e80100700 msgr2=0x7f3e80100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 --2- 172.21.15.77:0/411487030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e80100700 0x7f3e80100b20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009900 tx=0x7f3e7002bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 -- 172.21.15.77:0/411487030 shutdown_connections 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 --2- 172.21.15.77:0/411487030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e801022b0 0x7f3e80108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 --2- 172.21.15.77:0/411487030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e801018f0 0x7f3e80101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 --2- 172.21.15.77:0/411487030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e80100700 0x7f3e80100b20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 -- 172.21.15.77:0/411487030 >> 172.21.15.77:0/411487030 conn(0x7f3e800fbc70 msgr2=0x7f3e800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 -- 172.21.15.77:0/411487030 shutdown_connections 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 -- 172.21.15.77:0/411487030 wait complete. 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 Processor -- start 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 -- start start 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e80100700 0x7f3e80196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e801018f0 0x7f3e80196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e801022b0 0x7f3e8019e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8010c370 con 0x7f3e801022b0 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8010c1f0 con 0x7f3e801018f0 2024-03-16T10:16:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.721+0000 7f3e85d7e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8010c4f0 con 0x7f3e80100700 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e801022b0 0x7f3e8019e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e801022b0 0x7f3e8019e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51674/0 (socket says 172.21.15.77:51674) 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7ffff640 1 -- 172.21.15.77:0/796752218 learned_addr learned my addr 172.21.15.77:0/796752218 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7f7fe640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e80100700 0x7f3e80196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7effd640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e801018f0 0x7f3e80196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7ffff640 1 -- 172.21.15.77:0/796752218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e80100700 msgr2=0x7f3e80196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7ffff640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e80100700 0x7f3e80196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7ffff640 1 -- 172.21.15.77:0/796752218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e801018f0 msgr2=0x7f3e80196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7ffff640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e801018f0 0x7f3e80196fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7ffff640 1 -- 172.21.15.77:0/796752218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8019e690 con 0x7f3e801022b0 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7f7fe640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e80100700 0x7f3e80196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7ffff640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e801022b0 0x7f3e8019e020 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3e7800eb10 tx=0x7f3e7800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7cff9640 1 -- 172.21.15.77:0/796752218 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e78004b20 con 0x7f3e801022b0 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7cff9640 1 -- 172.21.15.77:0/796752218 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e78004cc0 con 0x7f3e801022b0 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e85d7e640 1 -- 172.21.15.77:0/796752218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8019e980 con 0x7f3e801022b0 2024-03-16T10:16:30.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e7cff9640 1 -- 172.21.15.77:0/796752218 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e78010760 con 0x7f3e801022b0 2024-03-16T10:16:30.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.722+0000 7f3e85d7e640 1 -- 172.21.15.77:0/796752218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e8019ee80 con 0x7f3e801022b0 2024-03-16T10:16:30.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.723+0000 7f3e85d7e640 1 -- 172.21.15.77:0/796752218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e80107a90 con 0x7f3e801022b0 2024-03-16T10:16:30.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.724+0000 7f3e7cff9640 1 -- 172.21.15.77:0/796752218 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3e78005020 con 0x7f3e801022b0 2024-03-16T10:16:30.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.725+0000 7f3e7cff9640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e680a2800 0x7f3e680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:30.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.725+0000 7f3e7f7fe640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e680a2800 0x7f3e680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:30.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.725+0000 7f3e7cff9640 1 -- 172.21.15.77:0/796752218 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f3e780cef40 con 0x7f3e801022b0 2024-03-16T10:16:30.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.725+0000 7f3e7f7fe640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e680a2800 0x7f3e680a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3e70000c00 tx=0x7f3e70005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:30.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.729+0000 7f3e7cff9640 1 -- 172.21.15.77:0/796752218 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e7809c070 con 0x7f3e801022b0 2024-03-16T10:16:30.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.952+0000 7f3e85d7e640 1 -- 172.21.15.77:0/796752218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e8002d1b0 con 0x7f3e801022b0 2024-03-16T10:16:30.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.953+0000 7f3e7cff9640 1 -- 172.21.15.77:0/796752218 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11938 (secure 0 0 0) 0x7f3e7809b970 con 0x7f3e801022b0 2024-03-16T10:16:30.953 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:30.953 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:15:09.227516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[14658,24455,24482,24454,24475]},"id":1}]} 2024-03-16T10:16:30.954 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-16T10:16:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.955+0000 7f3e85d7e640 1 -- 172.21.15.77:0/796752218 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e680a2800 msgr2=0x7f3e680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.955+0000 7f3e85d7e640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e680a2800 0x7f3e680a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3e70000c00 tx=0x7f3e70005fd0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.955+0000 7f3e85d7e640 1 -- 172.21.15.77:0/796752218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e801022b0 msgr2=0x7f3e8019e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.955+0000 7f3e85d7e640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e801022b0 0x7f3e8019e020 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3e7800eb10 tx=0x7f3e7800c2d0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.955+0000 7f3e85d7e640 1 -- 172.21.15.77:0/796752218 shutdown_connections 2024-03-16T10:16:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.955+0000 7f3e85d7e640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e801022b0 0x7f3e8019e020 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.955+0000 7f3e85d7e640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e801018f0 0x7f3e80196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.955+0000 7f3e85d7e640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e680a2800 0x7f3e680a4cc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.955+0000 7f3e85d7e640 1 --2- 172.21.15.77:0/796752218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e80100700 0x7f3e80196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.955+0000 7f3e85d7e640 1 -- 172.21.15.77:0/796752218 >> 172.21.15.77:0/796752218 conn(0x7f3e800fbc70 msgr2=0x7f3e80104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.955+0000 7f3e85d7e640 1 -- 172.21.15.77:0/796752218 shutdown_connections 2024-03-16T10:16:30.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:30.955+0000 7f3e85d7e640 1 -- 172.21.15.77:0/796752218 wait complete. 2024-03-16T10:16:30.965 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:30 smithi077 ceph-mon[32400]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-16T10:16:30.965 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:30 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3781237627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:16:30.965 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:30 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3668476787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:16:30.966 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:30 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1419455519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:16:30.967 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:16:31.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.072+0000 7fbc80d74640 1 -- 172.21.15.77:0/3327773685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc7c1018f0 msgr2=0x7fbc7c101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.072+0000 7fbc80d74640 1 --2- 172.21.15.77:0/3327773685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc7c1018f0 0x7fbc7c101d70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009900 tx=0x7fbc6402bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:31.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.072+0000 7fbc80d74640 1 -- 172.21.15.77:0/3327773685 shutdown_connections 2024-03-16T10:16:31.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.072+0000 7fbc80d74640 1 --2- 172.21.15.77:0/3327773685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c1022b0 0x7fbc7c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.072+0000 7fbc80d74640 1 --2- 172.21.15.77:0/3327773685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc7c1018f0 0x7fbc7c101d70 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.072+0000 7fbc80d74640 1 --2- 172.21.15.77:0/3327773685 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc7c100700 0x7fbc7c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.072+0000 7fbc80d74640 1 -- 172.21.15.77:0/3327773685 >> 172.21.15.77:0/3327773685 conn(0x7fbc7c0fbc70 msgr2=0x7fbc7c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.072+0000 7fbc80d74640 1 -- 172.21.15.77:0/3327773685 shutdown_connections 2024-03-16T10:16:31.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.072+0000 7fbc80d74640 1 -- 172.21.15.77:0/3327773685 wait complete. 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc80d74640 1 Processor -- start 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc80d74640 1 -- start start 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc80d74640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c100700 0x7fbc7c197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc80d74640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc7c1018f0 0x7fbc7c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc80d74640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc7c1022b0 0x7fbc7c19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc80d74640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10d4a0 con 0x7fbc7c1022b0 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc80d74640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10d320 con 0x7fbc7c1018f0 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc80d74640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10d620 con 0x7fbc7c100700 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc7ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc7c1022b0 0x7fbc7c19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc7a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c100700 0x7fbc7c197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc7ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc7c1022b0 0x7fbc7c19f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51692/0 (socket says 172.21.15.77:51692) 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc7ad9d640 1 -- 172.21.15.77:0/862720284 learned_addr learned my addr 172.21.15.77:0/862720284 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:31.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc79d9b640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc7c1018f0 0x7fbc7c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc7ad9d640 1 -- 172.21.15.77:0/862720284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c100700 msgr2=0x7fbc7c197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc7ad9d640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c100700 0x7fbc7c197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc7ad9d640 1 -- 172.21.15.77:0/862720284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc7c1018f0 msgr2=0x7fbc7c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc7ad9d640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc7c1018f0 0x7fbc7c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc7ad9d640 1 -- 172.21.15.77:0/862720284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c19f710 con 0x7fbc7c1022b0 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.073+0000 7fbc7a59c640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c100700 0x7fbc7c197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.074+0000 7fbc7ad9d640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc7c1022b0 0x7fbc7c19f0a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400eb10 tx=0x7fbc7400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.074+0000 7fbc737fe640 1 -- 172.21.15.77:0/862720284 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74004b20 con 0x7fbc7c1022b0 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.074+0000 7fbc737fe640 1 -- 172.21.15.77:0/862720284 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc74004cc0 con 0x7fbc7c1022b0 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.074+0000 7fbc80d74640 1 -- 172.21.15.77:0/862720284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7c19fa00 con 0x7fbc7c1022b0 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.074+0000 7fbc737fe640 1 -- 172.21.15.77:0/862720284 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74010760 con 0x7fbc7c1022b0 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.074+0000 7fbc80d74640 1 -- 172.21.15.77:0/862720284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7c19ff70 con 0x7fbc7c1022b0 2024-03-16T10:16:31.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.075+0000 7fbc80d74640 1 -- 172.21.15.77:0/862720284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7c107a20 con 0x7fbc7c1022b0 2024-03-16T10:16:31.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.076+0000 7fbc737fe640 1 -- 172.21.15.77:0/862720284 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbc74005020 con 0x7fbc7c1022b0 2024-03-16T10:16:31.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.076+0000 7fbc737fe640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbc680a2880 0x7fbc680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.076+0000 7fbc7a59c640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbc680a2880 0x7fbc680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.076+0000 7fbc737fe640 1 -- 172.21.15.77:0/862720284 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fbc740cf020 con 0x7fbc7c1022b0 2024-03-16T10:16:31.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.077+0000 7fbc7a59c640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbc680a2880 0x7fbc680a4d40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c101750 tx=0x7fbc6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.083+0000 7fbc737fe640 1 -- 172.21.15.77:0/862720284 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbc7409c150 con 0x7fbc7c1022b0 2024-03-16T10:16:31.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:30 smithi192 ceph-mon[36354]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-16T10:16:31.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:30 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3781237627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:16:31.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:30 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3668476787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:16:31.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:30 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1419455519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:16:31.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:30 smithi193 ceph-mon[36311]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-16T10:16:31.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:30 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3781237627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:16:31.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:30 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3668476787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:16:31.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:30 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1419455519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:16:31.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.265+0000 7fbc80d74640 1 -- 172.21.15.77:0/862720284 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc7c02d3b0 con 0x7fbc7c1022b0 2024-03-16T10:16:31.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.266+0000 7fbc737fe640 1 -- 172.21.15.77:0/862720284 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22496 (secure 0 0 0) 0x7fbc7409ba50 con 0x7fbc7c1022b0 2024-03-16T10:16:31.267 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:31.267 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":87,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:59.826259+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:16:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.268+0000 7fbc80d74640 1 -- 172.21.15.77:0/862720284 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbc680a2880 msgr2=0x7fbc680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.268+0000 7fbc80d74640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbc680a2880 0x7fbc680a4d40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c101750 tx=0x7fbc6c005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.268+0000 7fbc80d74640 1 -- 172.21.15.77:0/862720284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc7c1022b0 msgr2=0x7fbc7c19f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.268+0000 7fbc80d74640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc7c1022b0 0x7fbc7c19f0a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400eb10 tx=0x7fbc7400c2d0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.268+0000 7fbc80d74640 1 -- 172.21.15.77:0/862720284 shutdown_connections 2024-03-16T10:16:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.268+0000 7fbc80d74640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc7c1022b0 0x7fbc7c19f0a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.268+0000 7fbc80d74640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc7c1018f0 0x7fbc7c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.268+0000 7fbc80d74640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbc680a2880 0x7fbc680a4d40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.268+0000 7fbc80d74640 1 --2- 172.21.15.77:0/862720284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c100700 0x7fbc7c197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.268+0000 7fbc80d74640 1 -- 172.21.15.77:0/862720284 >> 172.21.15.77:0/862720284 conn(0x7fbc7c0fbc70 msgr2=0x7fbc7c104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.268+0000 7fbc80d74640 1 -- 172.21.15.77:0/862720284 shutdown_connections 2024-03-16T10:16:31.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.268+0000 7fbc80d74640 1 -- 172.21.15.77:0/862720284 wait complete. 2024-03-16T10:16:31.280 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:16:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.382+0000 7f13830dc640 1 -- 172.21.15.77:0/1034852528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f137c1022d0 msgr2=0x7f137c108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.382+0000 7f13830dc640 1 --2- 172.21.15.77:0/1034852528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f137c1022d0 0x7f137c108b60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f136c009900 tx=0x7f136c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.382+0000 7f13830dc640 1 -- 172.21.15.77:0/1034852528 shutdown_connections 2024-03-16T10:16:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.382+0000 7f13830dc640 1 --2- 172.21.15.77:0/1034852528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f137c1022d0 0x7f137c108b60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.382+0000 7f13830dc640 1 --2- 172.21.15.77:0/1034852528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f137c101910 0x7f137c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.382+0000 7f13830dc640 1 --2- 172.21.15.77:0/1034852528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f137c100720 0x7f137c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.382+0000 7f13830dc640 1 -- 172.21.15.77:0/1034852528 >> 172.21.15.77:0/1034852528 conn(0x7f137c0fbc90 msgr2=0x7f137c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.382+0000 7f13830dc640 1 -- 172.21.15.77:0/1034852528 shutdown_connections 2024-03-16T10:16:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.382+0000 7f13830dc640 1 -- 172.21.15.77:0/1034852528 wait complete. 2024-03-16T10:16:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f13830dc640 1 Processor -- start 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f13830dc640 1 -- start start 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f13830dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f137c100720 0x7f137c196ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f13830dc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f137c101910 0x7f137c197010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f13830dc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f137c1022d0 0x7f137c19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f13830dc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f137c10c3a0 con 0x7f137c100720 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f13830dc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f137c10c220 con 0x7f137c1022d0 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f13830dc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f137c10c520 con 0x7f137c101910 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f1380e78640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f137c100720 0x7f137c196ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f1380e78640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f137c100720 0x7f137c196ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51706/0 (socket says 172.21.15.77:51706) 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f1380e78640 1 -- 172.21.15.77:0/93064952 learned_addr learned my addr 172.21.15.77:0/93064952 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f137bfff640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f137c101910 0x7f137c197010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f1381679640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f137c1022d0 0x7f137c19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f1380e78640 1 -- 172.21.15.77:0/93064952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f137c101910 msgr2=0x7f137c197010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f1380e78640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f137c101910 0x7f137c197010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f1380e78640 1 -- 172.21.15.77:0/93064952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f137c1022d0 msgr2=0x7f137c19dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f1380e78640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f137c1022d0 0x7f137c19dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.383+0000 7f1380e78640 1 -- 172.21.15.77:0/93064952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137c19e650 con 0x7f137c100720 2024-03-16T10:16:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.384+0000 7f1380e78640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f137c100720 0x7f137c196ad0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1374009d20 tx=0x7f137400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.384+0000 7f1379ffb640 1 -- 172.21.15.77:0/93064952 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374010630 con 0x7f137c100720 2024-03-16T10:16:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.384+0000 7f1379ffb640 1 -- 172.21.15.77:0/93064952 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1374010cb0 con 0x7f137c100720 2024-03-16T10:16:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.384+0000 7f13830dc640 1 -- 172.21.15.77:0/93064952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f137c19e940 con 0x7f137c100720 2024-03-16T10:16:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.384+0000 7f1379ffb640 1 -- 172.21.15.77:0/93064952 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374018de0 con 0x7f137c100720 2024-03-16T10:16:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.384+0000 7f13830dc640 1 -- 172.21.15.77:0/93064952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f137c19ee90 con 0x7f137c100720 2024-03-16T10:16:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.385+0000 7f13830dc640 1 -- 172.21.15.77:0/93064952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f137c107a40 con 0x7f137c100720 2024-03-16T10:16:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.386+0000 7f1379ffb640 1 -- 172.21.15.77:0/93064952 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f137401f020 con 0x7f137c100720 2024-03-16T10:16:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.386+0000 7f1379ffb640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f13640a2800 0x7f13640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.387+0000 7f137bfff640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f13640a2800 0x7f13640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.387+0000 7f1379ffb640 1 -- 172.21.15.77:0/93064952 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f13740cee40 con 0x7f137c100720 2024-03-16T10:16:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.387+0000 7f137bfff640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f13640a2800 0x7f13640a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f137c197f40 tx=0x7f1370005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.393+0000 7f1379ffb640 1 -- 172.21.15.77:0/93064952 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f137409bf70 con 0x7f137c100720 2024-03-16T10:16:31.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.613+0000 7f13830dc640 1 -- 172.21.15.77:0/93064952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f137c02d1b0 con 0x7f137c100720 2024-03-16T10:16:31.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.614+0000 7f1379ffb640 1 -- 172.21.15.77:0/93064952 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11938 (secure 0 0 0) 0x7f137409b870 con 0x7f137c100720 2024-03-16T10:16:31.615 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:31.615 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:15:09.227516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[14658,24455,24482,24454,24475]},"id":1}]} 2024-03-16T10:16:31.615 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-16T10:16:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.616+0000 7f13830dc640 1 -- 172.21.15.77:0/93064952 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f13640a2800 msgr2=0x7f13640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.616+0000 7f13830dc640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f13640a2800 0x7f13640a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f137c197f40 tx=0x7f1370005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.616+0000 7f13830dc640 1 -- 172.21.15.77:0/93064952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f137c100720 msgr2=0x7f137c196ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.616+0000 7f13830dc640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f137c100720 0x7f137c196ad0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1374009d20 tx=0x7f137400c6a0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.616+0000 7f13830dc640 1 -- 172.21.15.77:0/93064952 shutdown_connections 2024-03-16T10:16:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.616+0000 7f13830dc640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f137c1022d0 0x7f137c19dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.616+0000 7f13830dc640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f13640a2800 0x7f13640a4cc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.616+0000 7f13830dc640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f137c101910 0x7f137c197010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.616+0000 7f13830dc640 1 --2- 172.21.15.77:0/93064952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f137c100720 0x7f137c196ad0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.616+0000 7f13830dc640 1 -- 172.21.15.77:0/93064952 >> 172.21.15.77:0/93064952 conn(0x7f137c0fbc90 msgr2=0x7f137c104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.616+0000 7f13830dc640 1 -- 172.21.15.77:0/93064952 shutdown_connections 2024-03-16T10:16:31.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.616+0000 7f13830dc640 1 -- 172.21.15.77:0/93064952 wait complete. 2024-03-16T10:16:31.628 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-16T10:15:02.344673+0000', 'modified': '2024-03-16T10:15:09.227516+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24482, 'mds_1': 24475, 'mds_2': 24455, 'mds_3': 24454, 'mds_4': 14658}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14646': {'gid': 14646, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6839/3943375297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 3943375297}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 3943375297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14658': {'gid': 14658, 'name': 'a', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.77:6841/857863693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6840', 'nonce': 857863693}, {'type': 'v1', 'addr': '172.21.15.77:6841', 'nonce': 857863693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24395': {'gid': 24395, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6833/3214007033', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6832', 'nonce': 3214007033}, {'type': 'v1', 'addr': '172.21.15.192:6833', 'nonce': 3214007033}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24403': {'gid': 24403, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6833/1430295144', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6832', 'nonce': 1430295144}, {'type': 'v1', 'addr': '172.21.15.193:6833', 'nonce': 1430295144}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6835/3288720506', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6834', 'nonce': 3288720506}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 3288720506}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/3768363028', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 3768363028}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 3768363028}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.193:6837/3018320989', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 3018320989}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 3018320989}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.192:6837/440836568', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6836', 'nonce': 440836568}, {'type': 'v1', 'addr': '172.21.15.192:6837', 'nonce': 440836568}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.193:6839/3991082280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 3991082280}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 3991082280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24482': {'gid': 24482, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.192:6839/3223311954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6838', 'nonce': 3223311954}, {'type': 'v1', 'addr': '172.21.15.192:6839', 'nonce': 3223311954}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24482, 'qdb_cluster': [14658, 24455, 24482, 24454, 24475]} 2024-03-16T10:16:31.628 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-16T10:16:31.628 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-16T10:16:31.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.732+0000 7fe899ad1640 1 -- 172.21.15.77:0/1210970296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8941006e0 msgr2=0x7fe894100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.732+0000 7fe899ad1640 1 --2- 172.21.15.77:0/1210970296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8941006e0 0x7fe894100b00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe888009900 tx=0x7fe88802bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.732+0000 7fe899ad1640 1 -- 172.21.15.77:0/1210970296 shutdown_connections 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.732+0000 7fe899ad1640 1 --2- 172.21.15.77:0/1210970296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe894102290 0x7fe894108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.732+0000 7fe899ad1640 1 --2- 172.21.15.77:0/1210970296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8941018d0 0x7fe894101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.732+0000 7fe899ad1640 1 --2- 172.21.15.77:0/1210970296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8941006e0 0x7fe894100b00 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.732+0000 7fe899ad1640 1 -- 172.21.15.77:0/1210970296 >> 172.21.15.77:0/1210970296 conn(0x7fe8940fbc70 msgr2=0x7fe8940fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.732+0000 7fe899ad1640 1 -- 172.21.15.77:0/1210970296 shutdown_connections 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.732+0000 7fe899ad1640 1 -- 172.21.15.77:0/1210970296 wait complete. 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.732+0000 7fe899ad1640 1 Processor -- start 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.732+0000 7fe899ad1640 1 -- start start 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe899ad1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8941006e0 0x7fe894197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe899ad1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8941018d0 0x7fe894198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe899ad1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe894102290 0x7fe89419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe899ad1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe89410c360 con 0x7fe894102290 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe899ad1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe89410c1e0 con 0x7fe8941018d0 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe899ad1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe89410c4e0 con 0x7fe8941006e0 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe893fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe894102290 0x7fe89419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe893fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe894102290 0x7fe89419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49930/0 (socket says 172.21.15.77:49930) 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe8937fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8941006e0 0x7fe894197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe893fff640 1 -- 172.21.15.77:0/2981689794 learned_addr learned my addr 172.21.15.77:0/2981689794 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe892ffd640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8941018d0 0x7fe894198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe893fff640 1 -- 172.21.15.77:0/2981689794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8941006e0 msgr2=0x7fe894197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe893fff640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8941006e0 0x7fe894197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe893fff640 1 -- 172.21.15.77:0/2981689794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8941018d0 msgr2=0x7fe894198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe893fff640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8941018d0 0x7fe894198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe893fff640 1 -- 172.21.15.77:0/2981689794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89419f700 con 0x7fe894102290 2024-03-16T10:16:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe892ffd640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8941018d0 0x7fe894198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:31.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe893fff640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe894102290 0x7fe89419f090 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe88c00eb10 tx=0x7fe88c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe890ff9640 1 -- 172.21.15.77:0/2981689794 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c004b20 con 0x7fe894102290 2024-03-16T10:16:31.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe899ad1640 1 -- 172.21.15.77:0/2981689794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe89419f9f0 con 0x7fe894102290 2024-03-16T10:16:31.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe890ff9640 1 -- 172.21.15.77:0/2981689794 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe88c004cc0 con 0x7fe894102290 2024-03-16T10:16:31.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe890ff9640 1 -- 172.21.15.77:0/2981689794 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c010760 con 0x7fe894102290 2024-03-16T10:16:31.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.733+0000 7fe899ad1640 1 -- 172.21.15.77:0/2981689794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe89419ff90 con 0x7fe894102290 2024-03-16T10:16:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.735+0000 7fe890ff9640 1 -- 172.21.15.77:0/2981689794 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe88c005020 con 0x7fe894102290 2024-03-16T10:16:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe89c798740 1 Processor -- start 2024-03-16T10:16:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe89c798740 1 -- start start 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe89c798740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd633bd0 0x55a8dd633ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe89c798740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd6345c0 0x55a8dd6390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe89c798740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd639730 0x55a8dd63bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe89c798740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a8dd597c80 con 0x55a8dd639730 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe89c798740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a8dd540aa0 con 0x55a8dd6345c0 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe89c798740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a8dd593d20 con 0x55a8dd633bd0 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd639730 0x55a8dd63bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd639730 0x55a8dd63bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49936/0 (socket says 172.21.15.77:49936) 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe892ffd640 1 -- 172.21.15.77:0/2718021971 learned_addr learned my addr 172.21.15.77:0/2718021971 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe8937fe640 1 --2- 172.21.15.77:0/2718021971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd6345c0 0x55a8dd6390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe892ffd640 1 --2- 172.21.15.77:0/2718021971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd633bd0 0x55a8dd633ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe890ff9640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe87c0a2800 0x7fe87c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.736+0000 7fe893fff640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe87c0a2800 0x7fe87c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.737+0000 7fe892ffd640 1 -- 172.21.15.77:0/2718021971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd633bd0 msgr2=0x55a8dd633ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.737+0000 7fe892ffd640 1 --2- 172.21.15.77:0/2718021971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd633bd0 0x55a8dd633ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.737+0000 7fe892ffd640 1 -- 172.21.15.77:0/2718021971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd6345c0 msgr2=0x55a8dd6390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.737+0000 7fe892ffd640 1 --2- 172.21.15.77:0/2718021971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd6345c0 0x55a8dd6390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.737+0000 7fe892ffd640 1 -- 172.21.15.77:0/2718021971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8dd562ff0 con 0x55a8dd639730 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.737+0000 7fe8937fe640 1 --2- 172.21.15.77:0/2718021971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd6345c0 0x55a8dd6390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.737+0000 7fe890ff9640 1 -- 172.21.15.77:0/2981689794 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe88c0cf0b0 con 0x7fe894102290 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.737+0000 7fe892ffd640 1 --2- 172.21.15.77:0/2718021971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd639730 0x55a8dd63bbc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe884009fd0 tx=0x7fe884009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6828c72479b2f123 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.737+0000 7fe879ffb640 1 -- 172.21.15.77:0/2718021971 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88400fc50 con 0x55a8dd639730 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.737+0000 7fe879ffb640 1 -- 172.21.15.77:0/2718021971 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe88400fdf0 con 0x55a8dd639730 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.737+0000 7fe893fff640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe87c0a2800 0x7fe87c0a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0a3d50 tx=0x7fe88c0c6530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.737+0000 7fe879ffb640 1 -- 172.21.15.77:0/2718021971 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88401d5a0 con 0x55a8dd639730 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 -- 172.21.15.77:0/2718021971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd639730 msgr2=0x55a8dd63bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 --2- 172.21.15.77:0/2718021971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd639730 0x55a8dd63bbc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe884009fd0 tx=0x7fe884009ba0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 -- 172.21.15.77:0/2718021971 shutdown_connections 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 --2- 172.21.15.77:0/2718021971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd639730 0x55a8dd63bbc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 --2- 172.21.15.77:0/2718021971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd6345c0 0x55a8dd6390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 --2- 172.21.15.77:0/2718021971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd633bd0 0x55a8dd633ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 -- 172.21.15.77:0/2718021971 >> 172.21.15.77:0/2718021971 conn(0x55a8dd62f140 msgr2=0x55a8dd62f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 -- 172.21.15.77:0/2718021971 shutdown_connections 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 -- 172.21.15.77:0/2718021971 wait complete. 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 -- start start 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 Processor -- start 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd633bd0 0x55a8dd63dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd6345c0 0x55a8dd63e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd639730 0x55a8dd645310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a8dd63c6e0 con 0x55a8dd6345c0 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a8dd553ef0 con 0x55a8dd639730 2024-03-16T10:16:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.738+0000 7fe89c798740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a8dd63c860 con 0x55a8dd633bd0 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe89c798740 10 client.0 _ll_register_callbacks cb 0x55a8dd54edc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe8937fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd6345c0 0x55a8dd63e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd633bd0 0x55a8dd63dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd633bd0 0x55a8dd63dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:56644/0 (socket says 172.21.15.77:56644) 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe892ffd640 1 -- 172.21.15.77:0/1282052733 learned_addr learned my addr 172.21.15.77:0/1282052733 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe892ffd640 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd639730 0x55a8dd645310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe8937fe640 1 -- 172.21.15.77:0/1282052733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd633bd0 msgr2=0x55a8dd63dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe8937fe640 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd633bd0 0x55a8dd63dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe8937fe640 1 -- 172.21.15.77:0/1282052733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd639730 msgr2=0x55a8dd645310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe8937fe640 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd639730 0x55a8dd645310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe8937fe640 1 -- 172.21.15.77:0/1282052733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8dd560a30 con 0x55a8dd6345c0 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe892ffd640 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd633bd0 0x55a8dd63dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe8937fe640 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd6345c0 0x55a8dd63e2d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe88802c170 tx=0x7fe888004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a273c43732e190b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe872ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:31.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe872ffd640 1 -- 172.21.15.77:0/1282052733 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888005d50 con 0x55a8dd6345c0 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe872ffd640 1 -- 172.21.15.77:0/1282052733 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe888005230 con 0x55a8dd6345c0 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe89c798740 1 -- 172.21.15.77:0/1282052733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe88400fdf0 con 0x55a8dd6345c0 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe872ffd640 1 -- 172.21.15.77:0/1282052733 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe888005430 con 0x55a8dd6345c0 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe89c798740 10 client.15081 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.739+0000 7fe89c798740 1 -- 172.21.15.77:0/1282052733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a8dd2ac2d0 con 0x55a8dd6345c0 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.740+0000 7fe872ffd640 1 -- 172.21.15.77:0/1282052733 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe88400fdf0 con 0x55a8dd6345c0 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.740+0000 7fe872ffd640 1 -- 172.21.15.77:0/1282052733 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe888003950 con 0x55a8dd6345c0 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.740+0000 7fe89c798740 10 client.15081 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.740+0000 7fe89c798740 10 client.15081 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.740+0000 7fe89c798740 20 client.15081 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.740+0000 7fe89c798740 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a8dd64a7b0 0x55a8dd64bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.740+0000 7fe89c798740 4 client.15081 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.740+0000 7fe89c798740 1 -- 172.21.15.77:0/1282052733 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8dd5a51c0 con 0x55a8dd64a7b0 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.740+0000 7fe892ffd640 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a8dd64a7b0 0x55a8dd64bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.740+0000 7fe872ffd640 10 client.15081 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.740+0000 7fe892ffd640 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a8dd64a7b0 0x55a8dd64bc20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4ac1669113652c31 server_cookie=4dd434f31670fdac in_seq=0 out_seq=0 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.741+0000 7fe872ffd640 1 -- 172.21.15.77:0/1282052733 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a8dd5a51c0 con 0x55a8dd64a7b0 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.741+0000 7fe872ffd640 10 client.15081 handle_command_reply: tid=0 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.741+0000 7fe89c798740 1 client.15081 shutdown 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.742+0000 7fe89c798740 1 -- 172.21.15.77:0/1282052733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd6345c0 msgr2=0x55a8dd63e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.742+0000 7fe89c798740 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd6345c0 0x55a8dd63e2d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe88802c170 tx=0x7fe888004270 comp rx=0 tx=0).stop 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.742+0000 7fe89c798740 1 -- 172.21.15.77:0/1282052733 shutdown_connections 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.742+0000 7fe89c798740 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd639730 0x55a8dd645310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.742+0000 7fe89c798740 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd6345c0 0x55a8dd63e2d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.742+0000 7fe89c798740 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a8dd64a7b0 0x55a8dd64bc20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.742+0000 7fe89c798740 1 --2- 172.21.15.77:0/1282052733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd633bd0 0x55a8dd63dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.742+0000 7fe872ffd640 0 client.15081 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.742+0000 7fe89c798740 1 -- 172.21.15.77:0/1282052733 >> 172.21.15.77:0/1282052733 conn(0x55a8dd62f140 msgr2=0x55a8dd62e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.742+0000 7fe89c798740 1 -- 172.21.15.77:0/1282052733 shutdown_connections 2024-03-16T10:16:31.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.742+0000 7fe89c798740 1 -- 172.21.15.77:0/1282052733 wait complete. 2024-03-16T10:16:31.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.742+0000 7fe89c798740 20 client.15081 trim_cache size 0 max 16384 2024-03-16T10:16:31.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe89c798740 1 Processor -- start 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe89c798740 1 -- start start 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe89c798740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 0x55a8dd62f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe89c798740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd6345c0 0x55a8dd6452d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe89c798740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd639730 0x55a8dd63bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe89c798740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a8dd649f60 con 0x55a8dd633bd0 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe89c798740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a8dd64a630 con 0x55a8dd6345c0 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe89c798740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a8dd64ad00 con 0x55a8dd639730 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 0x55a8dd62f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 0x55a8dd62f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49952/0 (socket says 172.21.15.77:49952) 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe892ffd640 1 -- 172.21.15.77:0/4033569724 learned_addr learned my addr 172.21.15.77:0/4033569724 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe8937fe640 1 --2- 172.21.15.77:0/4033569724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd6345c0 0x55a8dd6452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe892ffd640 1 --2- 172.21.15.77:0/4033569724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd639730 0x55a8dd63bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe892ffd640 1 -- 172.21.15.77:0/4033569724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd639730 msgr2=0x55a8dd63bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe892ffd640 1 --2- 172.21.15.77:0/4033569724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd639730 0x55a8dd63bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe892ffd640 1 -- 172.21.15.77:0/4033569724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd6345c0 msgr2=0x55a8dd6452d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe892ffd640 1 --2- 172.21.15.77:0/4033569724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd6345c0 0x55a8dd6452d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.760+0000 7fe892ffd640 1 -- 172.21.15.77:0/4033569724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8dd3766b0 con 0x55a8dd633bd0 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe892ffd640 1 --2- 172.21.15.77:0/4033569724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 0x55a8dd62f520 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe884011bb0 tx=0x7fe884009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2441a01e0fe84b29 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe8797fa640 1 -- 172.21.15.77:0/4033569724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88400fe20 con 0x55a8dd633bd0 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe8797fa640 1 -- 172.21.15.77:0/4033569724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe884011040 con 0x55a8dd633bd0 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe8797fa640 1 -- 172.21.15.77:0/4033569724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884004970 con 0x55a8dd633bd0 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe89c798740 1 -- 172.21.15.77:0/4033569724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 msgr2=0x55a8dd62f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe89c798740 1 --2- 172.21.15.77:0/4033569724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 0x55a8dd62f520 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe884011bb0 tx=0x7fe884009ba0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe89c798740 1 -- 172.21.15.77:0/4033569724 shutdown_connections 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe89c798740 1 --2- 172.21.15.77:0/4033569724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd639730 0x55a8dd63bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe89c798740 1 --2- 172.21.15.77:0/4033569724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd6345c0 0x55a8dd6452d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe89c798740 1 --2- 172.21.15.77:0/4033569724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 0x55a8dd62f520 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe89c798740 1 -- 172.21.15.77:0/4033569724 >> 172.21.15.77:0/4033569724 conn(0x55a8dd62f140 msgr2=0x55a8dd62daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe89c798740 1 -- 172.21.15.77:0/4033569724 shutdown_connections 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.761+0000 7fe89c798740 1 -- 172.21.15.77:0/4033569724 wait complete. 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe89c798740 1 -- start start 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe89c798740 1 Processor -- start 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe89c798740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 0x55a8dd640420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe89c798740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd6345c0 0x55a8dd640960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe89c798740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd639730 0x55a8dd643980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe89c798740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a8dd64baa0 con 0x55a8dd633bd0 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe89c798740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a8dd64b3d0 con 0x55a8dd639730 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe89c798740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a8dd64c170 con 0x55a8dd6345c0 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe89c798740 10 client.0 _ll_register_callbacks cb 0x7fe888009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 0x55a8dd640420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe8937fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd6345c0 0x55a8dd640960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 0x55a8dd640420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49958/0 (socket says 172.21.15.77:49958) 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe892ffd640 1 -- 172.21.15.77:0/3885798857 learned_addr learned my addr 172.21.15.77:0/3885798857 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe892ffd640 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd639730 0x55a8dd643980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe892ffd640 1 -- 172.21.15.77:0/3885798857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd6345c0 msgr2=0x55a8dd640960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe892ffd640 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd6345c0 0x55a8dd640960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe892ffd640 1 -- 172.21.15.77:0/3885798857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd639730 msgr2=0x55a8dd643980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe892ffd640 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd639730 0x55a8dd643980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.762+0000 7fe892ffd640 1 -- 172.21.15.77:0/3885798857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8dd643f70 con 0x55a8dd633bd0 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe8937fe640 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd6345c0 0x55a8dd640960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe892ffd640 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 0x55a8dd640420 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe884011320 tx=0x7fe884005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2249ca49abd8bc37 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe872ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe872ffd640 1 -- 172.21.15.77:0/3885798857 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884011400 con 0x55a8dd633bd0 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe872ffd640 1 -- 172.21.15.77:0/3885798857 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe884028040 con 0x55a8dd633bd0 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe89c798740 1 -- 172.21.15.77:0/3885798857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe884011040 con 0x55a8dd633bd0 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe872ffd640 1 -- 172.21.15.77:0/3885798857 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe88402f3d0 con 0x55a8dd633bd0 2024-03-16T10:16:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe89c798740 10 client.15087 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe89c798740 1 -- 172.21.15.77:0/3885798857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a8dd645270 con 0x55a8dd633bd0 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe872ffd640 1 -- 172.21.15.77:0/3885798857 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe884011040 con 0x55a8dd633bd0 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe872ffd640 1 -- 172.21.15.77:0/3885798857 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe88401d020 con 0x55a8dd633bd0 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe89c798740 10 client.15087 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe89c798740 10 client.15087 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe89c798740 20 client.15087 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe89c798740 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a8dd6b45a0 0x55a8dd6b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe89c798740 4 client.15087 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.763+0000 7fe89c798740 1 -- 172.21.15.77:0/3885798857 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "status"}) -- 0x55a8dd59f510 con 0x55a8dd6b45a0 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe8937fe640 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a8dd6b45a0 0x55a8dd6b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe872ffd640 10 client.15087 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe8937fe640 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a8dd6b45a0 0x55a8dd6b32b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f9887464f505264a server_cookie=57f9fe5a5b6d3680 in_seq=0 out_seq=0 2024-03-16T10:16:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe872ffd640 1 -- 172.21.15.77:0/3885798857 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a8dd59f510 con 0x55a8dd6b45a0 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe872ffd640 10 client.15087 handle_command_reply: tid=0 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe89c798740 1 client.15087 shutdown 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe89c798740 1 -- 172.21.15.77:0/3885798857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 msgr2=0x55a8dd640420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe89c798740 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 0x55a8dd640420 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe884011320 tx=0x7fe884005700 comp rx=0 tx=0).stop 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe89c798740 1 -- 172.21.15.77:0/3885798857 shutdown_connections 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe89c798740 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a8dd639730 0x55a8dd643980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe89c798740 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a8dd6b45a0 0x55a8dd6b32b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe89c798740 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8dd6345c0 0x55a8dd640960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe89c798740 1 --2- 172.21.15.77:0/3885798857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8dd633bd0 0x55a8dd640420 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe872ffd640 0 client.15087 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe89c798740 1 -- 172.21.15.77:0/3885798857 >> 172.21.15.77:0/3885798857 conn(0x55a8dd62f140 msgr2=0x55a8dd632c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe89c798740 1 -- 172.21.15.77:0/3885798857 shutdown_connections 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.764+0000 7fe89c798740 1 -- 172.21.15.77:0/3885798857 wait complete. 2024-03-16T10:16:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe89c798740 20 client.15087 trim_cache size 0 max 16384 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 4, 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 14658, 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 83.543181555999993, 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 26, 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 87, 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 98.711915899999994 2024-03-16T10:16:31.772 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:16:31.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe899ad1640 1 -- 172.21.15.77:0/2981689794 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe87c0a2800 msgr2=0x7fe87c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe899ad1640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe87c0a2800 0x7fe87c0a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0a3d50 tx=0x7fe88c0c6530 comp rx=0 tx=0).stop 2024-03-16T10:16:31.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe899ad1640 1 -- 172.21.15.77:0/2981689794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe894102290 msgr2=0x7fe89419f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe899ad1640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe894102290 0x7fe89419f090 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe88c00eb10 tx=0x7fe88c00c2d0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe899ad1640 1 -- 172.21.15.77:0/2981689794 shutdown_connections 2024-03-16T10:16:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe899ad1640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe894102290 0x7fe89419f090 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe899ad1640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8941018d0 0x7fe894198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe899ad1640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe87c0a2800 0x7fe87c0a4cc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe899ad1640 1 --2- 172.21.15.77:0/2981689794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8941006e0 0x7fe894197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe899ad1640 1 -- 172.21.15.77:0/2981689794 >> 172.21.15.77:0/2981689794 conn(0x7fe8940fbc70 msgr2=0x7fe894068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe899ad1640 1 -- 172.21.15.77:0/2981689794 shutdown_connections 2024-03-16T10:16:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.765+0000 7fe899ad1640 1 -- 172.21.15.77:0/2981689794 wait complete. 2024-03-16T10:16:31.779 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-16T10:16:31.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.895+0000 7fc99ec12640 1 -- 172.21.15.77:0/2683965710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9981018d0 msgr2=0x7fc998101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.895+0000 7fc99ec12640 1 --2- 172.21.15.77:0/2683965710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9981018d0 0x7fc998101d50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc980009900 tx=0x7fc98002bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.895+0000 7fc99ec12640 1 -- 172.21.15.77:0/2683965710 shutdown_connections 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.895+0000 7fc99ec12640 1 --2- 172.21.15.77:0/2683965710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc998102290 0x7fc998108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.895+0000 7fc99ec12640 1 --2- 172.21.15.77:0/2683965710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9981018d0 0x7fc998101d50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.895+0000 7fc99ec12640 1 --2- 172.21.15.77:0/2683965710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9981006e0 0x7fc998100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.895+0000 7fc99ec12640 1 -- 172.21.15.77:0/2683965710 >> 172.21.15.77:0/2683965710 conn(0x7fc9980fbc70 msgr2=0x7fc9980fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.895+0000 7fc99ec12640 1 -- 172.21.15.77:0/2683965710 shutdown_connections 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.895+0000 7fc99ec12640 1 -- 172.21.15.77:0/2683965710 wait complete. 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc99ec12640 1 Processor -- start 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc99ec12640 1 -- start start 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc99ec12640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9981006e0 0x7fc998069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc99ec12640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9981018d0 0x7fc99819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc99ec12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc998102290 0x7fc99819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc99ec12640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc99810c340 con 0x7fc998102290 2024-03-16T10:16:31.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc99ec12640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc99810c1c0 con 0x7fc9981006e0 2024-03-16T10:16:31.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc99ec12640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc99810c4c0 con 0x7fc9981018d0 2024-03-16T10:16:31.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc99d1af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc998102290 0x7fc99819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc99c9ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9981006e0 0x7fc998069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc99d1af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc998102290 0x7fc99819ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49974/0 (socket says 172.21.15.77:49974) 2024-03-16T10:16:31.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc99d1af640 1 -- 172.21.15.77:0/4072317625 learned_addr learned my addr 172.21.15.77:0/4072317625 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:31.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.896+0000 7fc997fff640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9981018d0 0x7fc99819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.897+0000 7fc99c9ae640 1 -- 172.21.15.77:0/4072317625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9981018d0 msgr2=0x7fc99819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.897+0000 7fc99c9ae640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9981018d0 0x7fc99819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.897+0000 7fc99c9ae640 1 -- 172.21.15.77:0/4072317625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc998102290 msgr2=0x7fc99819ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.897+0000 7fc99c9ae640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc998102290 0x7fc99819ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.897+0000 7fc99c9ae640 1 -- 172.21.15.77:0/4072317625 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99819f750 con 0x7fc9981006e0 2024-03-16T10:16:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.897+0000 7fc99c9ae640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9981006e0 0x7fc998069510 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc990005fd0 tx=0x7fc99000ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.897+0000 7fc995ffb640 1 -- 172.21.15.77:0/4072317625 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc990003970 con 0x7fc9981006e0 2024-03-16T10:16:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.897+0000 7fc99ec12640 1 -- 172.21.15.77:0/4072317625 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc99819fa40 con 0x7fc9981006e0 2024-03-16T10:16:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.897+0000 7fc995ffb640 1 -- 172.21.15.77:0/4072317625 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc990004030 con 0x7fc9981006e0 2024-03-16T10:16:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.897+0000 7fc995ffb640 1 -- 172.21.15.77:0/4072317625 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9900107f0 con 0x7fc9981006e0 2024-03-16T10:16:31.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.897+0000 7fc99ec12640 1 -- 172.21.15.77:0/4072317625 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc99819ffb0 con 0x7fc9981006e0 2024-03-16T10:16:31.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.899+0000 7fc995ffb640 1 -- 172.21.15.77:0/4072317625 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc9900041e0 con 0x7fc9981006e0 2024-03-16T10:16:31.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc9a1804740 1 Processor -- start 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc995ffb640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc9840a2800 0x7fc9840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc9a1804740 1 -- start start 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc99d1af640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc9840a2800 0x7fc9840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc9a1804740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe118ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc9a1804740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 0x55bbfe11e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc995ffb640 1 -- 172.21.15.77:0/4072317625 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc9900cf1d0 con 0x7fc9981006e0 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc9a1804740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe11e730 0x55bbfe120bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc9a1804740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bbfe07cc80 con 0x55bbfe1195c0 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc9a1804740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55bbfe025aa0 con 0x55bbfe11e730 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc9a1804740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bbfe078d20 con 0x55bbfe118bd0 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc997fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 0x55bbfe11e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc997fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 0x55bbfe11e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49986/0 (socket says 172.21.15.77:49986) 2024-03-16T10:16:31.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc997fff640 1 -- 172.21.15.77:0/2174816505 learned_addr learned my addr 172.21.15.77:0/2174816505 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc99c9ae640 1 --2- 172.21.15.77:0/2174816505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe118ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc99d1af640 1 --2- 172.21.15.77:0/2174816505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe11e730 0x55bbfe120bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc997fff640 1 -- 172.21.15.77:0/2174816505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 msgr2=0x55bbfe118ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc997fff640 1 --2- 172.21.15.77:0/2174816505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe118ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc997fff640 1 -- 172.21.15.77:0/2174816505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe11e730 msgr2=0x55bbfe120bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc997fff640 1 --2- 172.21.15.77:0/2174816505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe11e730 0x55bbfe120bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc997fff640 1 -- 172.21.15.77:0/2174816505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbfe047ff0 con 0x55bbfe1195c0 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc99d1af640 1 --2- 172.21.15.77:0/2174816505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe11e730 0x55bbfe120bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc99d1af640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc9840a2800 0x7fc9840a4cc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc988011550 tx=0x7fc988011950 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.900+0000 7fc997fff640 1 --2- 172.21.15.77:0/2174816505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 0x55bbfe11e0c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc980009900 tx=0x7fc980009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b038d11c05b51d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.901+0000 7fc98effd640 1 -- 172.21.15.77:0/2174816505 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98002cb80 con 0x55bbfe1195c0 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.901+0000 7fc98effd640 1 -- 172.21.15.77:0/2174816505 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc980005040 con 0x55bbfe1195c0 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.901+0000 7fc98effd640 1 -- 172.21.15.77:0/2174816505 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980004900 con 0x55bbfe1195c0 2024-03-16T10:16:31.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.901+0000 7fc9a1804740 1 -- 172.21.15.77:0/2174816505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 msgr2=0x55bbfe11e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.901+0000 7fc9a1804740 1 --2- 172.21.15.77:0/2174816505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 0x55bbfe11e0c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc980009900 tx=0x7fc980009560 comp rx=0 tx=0).stop 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.901+0000 7fc9a1804740 1 -- 172.21.15.77:0/2174816505 shutdown_connections 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.901+0000 7fc9a1804740 1 --2- 172.21.15.77:0/2174816505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe11e730 0x55bbfe120bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.901+0000 7fc9a1804740 1 --2- 172.21.15.77:0/2174816505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 0x55bbfe11e0c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.901+0000 7fc9a1804740 1 --2- 172.21.15.77:0/2174816505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe118ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.901+0000 7fc9a1804740 1 -- 172.21.15.77:0/2174816505 >> 172.21.15.77:0/2174816505 conn(0x55bbfe114140 msgr2=0x55bbfe114540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.901+0000 7fc9a1804740 1 -- 172.21.15.77:0/2174816505 shutdown_connections 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.901+0000 7fc9a1804740 1 -- 172.21.15.77:0/2174816505 wait complete. 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc9a1804740 1 -- start start 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc9a1804740 1 Processor -- start 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc9a1804740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe122d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc9a1804740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe1195c0 0x55bbfe1232c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc9a1804740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 0x55bbfe12a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc9a1804740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bbfe121680 con 0x55bbfe11e730 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc9a1804740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55bbfe038ef0 con 0x55bbfe1195c0 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc9a1804740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bbfe121800 con 0x55bbfe118bd0 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc9a1804740 10 client.0 _ll_register_callbacks cb 0x55bbfe033dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc99d1af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 0x55bbfe12a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc99c9ae640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe122d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc997fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe1195c0 0x55bbfe1232c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc99d1af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 0x55bbfe12a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49990/0 (socket says 172.21.15.77:49990) 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.902+0000 7fc99d1af640 1 -- 172.21.15.77:0/3144754554 learned_addr learned my addr 172.21.15.77:0/3144754554 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc99c9ae640 1 -- 172.21.15.77:0/3144754554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe1195c0 msgr2=0x55bbfe1232c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc99c9ae640 1 --2- 172.21.15.77:0/3144754554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe1195c0 0x55bbfe1232c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc99c9ae640 1 -- 172.21.15.77:0/3144754554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 msgr2=0x55bbfe12a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc99c9ae640 1 --2- 172.21.15.77:0/3144754554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 0x55bbfe12a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc99c9ae640 1 -- 172.21.15.77:0/3144754554 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbfe045a30 con 0x55bbfe118bd0 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc99d1af640 1 --2- 172.21.15.77:0/3144754554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 0x55bbfe12a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc99c9ae640 1 --2- 172.21.15.77:0/3144754554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe122d80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc9900ccdb0 tx=0x7fc99009c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7703a7de45f72424 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc977fff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:16:31.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc977fff640 1 -- 172.21.15.77:0/3144754554 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9900b9060 con 0x55bbfe118bd0 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc977fff640 1 -- 172.21.15.77:0/3144754554 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc9900a3f10 con 0x55bbfe118bd0 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc9a1804740 1 -- 172.21.15.77:0/3144754554 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc980005040 con 0x55bbfe118bd0 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc977fff640 1 -- 172.21.15.77:0/3144754554 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc9900a4110 con 0x55bbfe118bd0 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc9a1804740 10 client.24889 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc9a1804740 1 -- 172.21.15.77:0/3144754554 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bbfdd912d0 con 0x55bbfe118bd0 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc977fff640 1 -- 172.21.15.77:0/3144754554 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9900ce030 con 0x55bbfe118bd0 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc977fff640 1 -- 172.21.15.77:0/3144754554 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fc990004c60 con 0x55bbfe118bd0 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc9a1804740 10 client.24889 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.903+0000 7fc9a1804740 10 client.24889 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.904+0000 7fc9a1804740 20 client.24889 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.904+0000 7fc9a1804740 1 --2- 172.21.15.77:0/3144754554 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55bbfe12f6f0 0x55bbfe130b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.904+0000 7fc9a1804740 4 client.24889 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.904+0000 7fc9a1804740 1 -- 172.21.15.77:0/3144754554 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc980005040 con 0x55bbfe12f6f0 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.904+0000 7fc997fff640 1 --2- 172.21.15.77:0/3144754554 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55bbfe12f6f0 0x55bbfe130b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.904+0000 7fc977fff640 10 client.24889 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:16:31.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.904+0000 7fc997fff640 1 --2- 172.21.15.77:0/3144754554 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55bbfe12f6f0 0x55bbfe130b60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=295dc34e700c4dec server_cookie=8b7d34b9e42ec664 in_seq=0 out_seq=0 2024-03-16T10:16:31.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc977fff640 1 -- 172.21.15.77:0/3144754554 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fc980005040 con 0x55bbfe12f6f0 2024-03-16T10:16:31.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc977fff640 10 client.24889 handle_command_reply: tid=0 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc9a1804740 1 client.24889 shutdown 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc9a1804740 1 -- 172.21.15.77:0/3144754554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 msgr2=0x55bbfe122d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc9a1804740 1 --2- 172.21.15.77:0/3144754554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe122d80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc9900ccdb0 tx=0x7fc99009c440 comp rx=0 tx=0).stop 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc9a1804740 1 -- 172.21.15.77:0/3144754554 shutdown_connections 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc9a1804740 1 --2- 172.21.15.77:0/3144754554 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55bbfe12f6f0 0x55bbfe130b60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc9a1804740 1 --2- 172.21.15.77:0/3144754554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 0x55bbfe12a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc9a1804740 1 --2- 172.21.15.77:0/3144754554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe1195c0 0x55bbfe1232c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc9a1804740 1 --2- 172.21.15.77:0/3144754554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe122d80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc977fff640 0 client.24889 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc9a1804740 1 -- 172.21.15.77:0/3144754554 >> 172.21.15.77:0/3144754554 conn(0x55bbfe114140 msgr2=0x55bbfe113380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc9a1804740 1 -- 172.21.15.77:0/3144754554 shutdown_connections 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc9a1804740 1 -- 172.21.15.77:0/3144754554 wait complete. 2024-03-16T10:16:31.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.905+0000 7fc9a1804740 20 client.24889 trim_cache size 0 max 16384 2024-03-16T10:16:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc9a1804740 1 Processor -- start 2024-03-16T10:16:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc9a1804740 1 -- start start 2024-03-16T10:16:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc9a1804740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe1230a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.929 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc9a1804740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe1195c0 0x55bbfe12a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc9a1804740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 0x55bbfe112af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc9a1804740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bbfe12eea0 con 0x55bbfe11e730 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc9a1804740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55bbfe12f570 con 0x55bbfe1195c0 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc9a1804740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bbfe12fc40 con 0x55bbfe118bd0 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc99d1af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 0x55bbfe112af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc99d1af640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 0x55bbfe112af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49996/0 (socket says 172.21.15.77:49996) 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc99d1af640 1 -- 172.21.15.77:0/4101227477 learned_addr learned my addr 172.21.15.77:0/4101227477 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc99c9ae640 1 --2- 172.21.15.77:0/4101227477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe1230a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc997fff640 1 --2- 172.21.15.77:0/4101227477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe1195c0 0x55bbfe12a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc99d1af640 1 -- 172.21.15.77:0/4101227477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 msgr2=0x55bbfe1230a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc99d1af640 1 --2- 172.21.15.77:0/4101227477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe1230a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc99d1af640 1 -- 172.21.15.77:0/4101227477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe1195c0 msgr2=0x55bbfe12a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc99d1af640 1 --2- 172.21.15.77:0/4101227477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe1195c0 0x55bbfe12a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.923+0000 7fc99d1af640 1 -- 172.21.15.77:0/4101227477 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbfde5b6b0 con 0x55bbfe11e730 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.924+0000 7fc99d1af640 1 --2- 172.21.15.77:0/4101227477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 0x55bbfe112af0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc98800f880 tx=0x7fc98801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c441eaa4f6239c7f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.924+0000 7fc98e7fc640 1 -- 172.21.15.77:0/4101227477 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98801a040 con 0x55bbfe11e730 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.924+0000 7fc98e7fc640 1 -- 172.21.15.77:0/4101227477 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc988018040 con 0x55bbfe11e730 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.924+0000 7fc98e7fc640 1 -- 172.21.15.77:0/4101227477 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc988028e50 con 0x55bbfe11e730 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.924+0000 7fc9a1804740 1 -- 172.21.15.77:0/4101227477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 msgr2=0x55bbfe112af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.924+0000 7fc9a1804740 1 --2- 172.21.15.77:0/4101227477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 0x55bbfe112af0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc98800f880 tx=0x7fc98801f000 comp rx=0 tx=0).stop 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.924+0000 7fc9a1804740 1 -- 172.21.15.77:0/4101227477 shutdown_connections 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.924+0000 7fc9a1804740 1 --2- 172.21.15.77:0/4101227477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe11e730 0x55bbfe112af0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.924+0000 7fc9a1804740 1 --2- 172.21.15.77:0/4101227477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe1195c0 0x55bbfe12a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.924+0000 7fc9a1804740 1 --2- 172.21.15.77:0/4101227477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe118bd0 0x55bbfe1230a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.924+0000 7fc9a1804740 1 -- 172.21.15.77:0/4101227477 >> 172.21.15.77:0/4101227477 conn(0x55bbfe114140 msgr2=0x55bbfe122990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.924+0000 7fc9a1804740 1 -- 172.21.15.77:0/4101227477 shutdown_connections 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.925+0000 7fc9a1804740 1 -- 172.21.15.77:0/4101227477 wait complete. 2024-03-16T10:16:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.925+0000 7fc9a1804740 1 -- start start 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.925+0000 7fc9a1804740 1 Processor -- start 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.925+0000 7fc9a1804740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe118bd0 0x55bbfe125bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.925+0000 7fc9a1804740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 0x55bbfe126110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.925+0000 7fc9a1804740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe11e730 0x55bbfe129130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.925+0000 7fc9a1804740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bbfe1309e0 con 0x55bbfe1195c0 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.925+0000 7fc9a1804740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55bbfe130310 con 0x55bbfe118bd0 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.925+0000 7fc9a1804740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bbfe1310b0 con 0x55bbfe11e730 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.925+0000 7fc9a1804740 10 client.0 _ll_register_callbacks cb 0x7fc99009ba10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.925+0000 7fc997fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 0x55bbfe126110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc997fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 0x55bbfe126110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50012/0 (socket says 172.21.15.77:50012) 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc997fff640 1 -- 172.21.15.77:0/2517272961 learned_addr learned my addr 172.21.15.77:0/2517272961 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc99c9ae640 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe118bd0 0x55bbfe125bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc99d1af640 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe11e730 0x55bbfe129130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc997fff640 1 -- 172.21.15.77:0/2517272961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe11e730 msgr2=0x55bbfe129130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc997fff640 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe11e730 0x55bbfe129130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc997fff640 1 -- 172.21.15.77:0/2517272961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe118bd0 msgr2=0x55bbfe125bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc997fff640 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe118bd0 0x55bbfe125bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc997fff640 1 -- 172.21.15.77:0/2517272961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbfe129720 con 0x55bbfe1195c0 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc997fff640 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 0x55bbfe126110 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc98002c170 tx=0x7fc980005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff5ccc944db20d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc977fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc977fff640 1 -- 172.21.15.77:0/2517272961 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980050d90 con 0x55bbfe1195c0 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc977fff640 1 -- 172.21.15.77:0/2517272961 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc980004040 con 0x55bbfe1195c0 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc9a1804740 1 -- 172.21.15.77:0/2517272961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc988018040 con 0x55bbfe1195c0 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc977fff640 1 -- 172.21.15.77:0/2517272961 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc980004240 con 0x55bbfe1195c0 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc9a1804740 10 client.15105 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc9a1804740 1 -- 172.21.15.77:0/2517272961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bbfe19ddf0 con 0x55bbfe1195c0 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.926+0000 7fc977fff640 1 -- 172.21.15.77:0/2517272961 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc988018040 con 0x55bbfe1195c0 2024-03-16T10:16:31.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.927+0000 7fc977fff640 1 -- 172.21.15.77:0/2517272961 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fc980050330 con 0x55bbfe1195c0 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.927+0000 7fc9a1804740 10 client.15105 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.927+0000 7fc9a1804740 10 client.15105 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.927+0000 7fc9a1804740 20 client.15105 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.927+0000 7fc9a1804740 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55bbfe12f6f0 0x55bbfe1a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.927+0000 7fc9a1804740 4 client.15105 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.927+0000 7fc9a1804740 1 -- 172.21.15.77:0/2517272961 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "status"}) -- 0x55bbfe08a1c0 con 0x55bbfe12f6f0 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.927+0000 7fc99c9ae640 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55bbfe12f6f0 0x55bbfe1a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.927+0000 7fc977fff640 10 client.15105 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.927+0000 7fc99c9ae640 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55bbfe12f6f0 0x55bbfe1a13f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1332d0908d6a909d server_cookie=c96d97884e3bc46a in_seq=0 out_seq=0 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.927+0000 7fc977fff640 1 -- 172.21.15.77:0/2517272961 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55bbfe08a1c0 con 0x55bbfe12f6f0 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc977fff640 10 client.15105 handle_command_reply: tid=0 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc9a1804740 1 client.15105 shutdown 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc9a1804740 1 -- 172.21.15.77:0/2517272961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 msgr2=0x55bbfe126110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc9a1804740 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 0x55bbfe126110 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc98002c170 tx=0x7fc980005be0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc9a1804740 1 -- 172.21.15.77:0/2517272961 shutdown_connections 2024-03-16T10:16:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc9a1804740 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55bbfe12f6f0 0x55bbfe1a13f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc9a1804740 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbfe11e730 0x55bbfe129130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc977fff640 0 client.15105 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:16:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc9a1804740 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bbfe1195c0 0x55bbfe126110 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc9a1804740 1 --2- 172.21.15.77:0/2517272961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bbfe118bd0 0x55bbfe125bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc9a1804740 1 -- 172.21.15.77:0/2517272961 >> 172.21.15.77:0/2517272961 conn(0x55bbfe114140 msgr2=0x55bbfe122970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc9a1804740 1 -- 172.21.15.77:0/2517272961 shutdown_connections 2024-03-16T10:16:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc9a1804740 1 -- 172.21.15.77:0/2517272961 wait complete. 2024-03-16T10:16:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc9a1804740 20 client.15105 trim_cache size 0 max 16384 2024-03-16T10:16:31.935 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:16:31.935 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:16:31.936 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 3, 2024-03-16T10:16:31.936 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24454, 2024-03-16T10:16:31.936 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T10:16:31.936 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T10:16:31.936 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T10:16:31.936 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 84.673140739999994, 2024-03-16T10:16:31.936 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 26, 2024-03-16T10:16:31.936 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 87, 2024-03-16T10:16:31.936 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T10:16:31.936 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 110.226181618 2024-03-16T10:16:31.936 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:16:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc99ec12640 1 -- 172.21.15.77:0/4072317625 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc9840a2800 msgr2=0x7fc9840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc99ec12640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc9840a2800 0x7fc9840a4cc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc988011550 tx=0x7fc988011950 comp rx=0 tx=0).stop 2024-03-16T10:16:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc99ec12640 1 -- 172.21.15.77:0/4072317625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9981006e0 msgr2=0x7fc998069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc99ec12640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9981006e0 0x7fc998069510 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc990005fd0 tx=0x7fc99000ef90 comp rx=0 tx=0).stop 2024-03-16T10:16:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc99ec12640 1 -- 172.21.15.77:0/4072317625 shutdown_connections 2024-03-16T10:16:31.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc99ec12640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc998102290 0x7fc99819ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc99ec12640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc9840a2800 0x7fc9840a4cc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.928+0000 7fc99ec12640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9981018d0 0x7fc99819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.929+0000 7fc99ec12640 1 --2- 172.21.15.77:0/4072317625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9981006e0 0x7fc998069510 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:31.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.929+0000 7fc99ec12640 1 -- 172.21.15.77:0/4072317625 >> 172.21.15.77:0/4072317625 conn(0x7fc9980fbc70 msgr2=0x7fc998104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:31.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.929+0000 7fc99ec12640 1 -- 172.21.15.77:0/4072317625 shutdown_connections 2024-03-16T10:16:31.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:31.929+0000 7fc99ec12640 1 -- 172.21.15.77:0/4072317625 wait complete. 2024-03-16T10:16:31.939 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:31 smithi077 ceph-mon[32400]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:31.939 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:31 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/796752218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:31.939 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:31 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/862720284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:31.939 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:31 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/93064952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:31.940 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-16T10:16:32.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.042+0000 7f2adb57e640 1 -- 172.21.15.77:0/341162924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad4102290 msgr2=0x7f2ad4108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.042+0000 7f2adb57e640 1 --2- 172.21.15.77:0/341162924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad4102290 0x7f2ad4108b20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009ca0 tx=0x7f2acc02c0f0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.042+0000 7f2adb57e640 1 -- 172.21.15.77:0/341162924 shutdown_connections 2024-03-16T10:16:32.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.042+0000 7f2adb57e640 1 --2- 172.21.15.77:0/341162924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad4102290 0x7f2ad4108b20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.042+0000 7f2adb57e640 1 --2- 172.21.15.77:0/341162924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad41018d0 0x7f2ad4101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.042+0000 7f2adb57e640 1 --2- 172.21.15.77:0/341162924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ad41006e0 0x7f2ad4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.042+0000 7f2adb57e640 1 -- 172.21.15.77:0/341162924 >> 172.21.15.77:0/341162924 conn(0x7f2ad40fbc70 msgr2=0x7f2ad40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2adb57e640 1 -- 172.21.15.77:0/341162924 shutdown_connections 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2adb57e640 1 -- 172.21.15.77:0/341162924 wait complete. 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2adb57e640 1 Processor -- start 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2adb57e640 1 -- start start 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2adb57e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad41006e0 0x7f2ad40694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2adb57e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad41018d0 0x7f2ad419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2adb57e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ad4102290 0x7f2ad419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2adb57e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad410c340 con 0x7f2ad41018d0 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2adb57e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad410c1c0 con 0x7f2ad41006e0 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2adb57e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad410c4c0 con 0x7f2ad4102290 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2ad8b19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad41018d0 0x7f2ad419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2ad8b19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad41018d0 0x7f2ad419c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50042/0 (socket says 172.21.15.77:50042) 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2ad8b19640 1 -- 172.21.15.77:0/1714467387 learned_addr learned my addr 172.21.15.77:0/1714467387 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2ad931a640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad41006e0 0x7f2ad40694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.043+0000 7f2ad9b1b640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ad4102290 0x7f2ad419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.044+0000 7f2ad8b19640 1 -- 172.21.15.77:0/1714467387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ad4102290 msgr2=0x7f2ad419ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.044+0000 7f2ad8b19640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ad4102290 0x7f2ad419ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.044+0000 7f2ad8b19640 1 -- 172.21.15.77:0/1714467387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad41006e0 msgr2=0x7f2ad40694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.044+0000 7f2ad8b19640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad41006e0 0x7f2ad40694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.044+0000 7f2ad8b19640 1 -- 172.21.15.77:0/1714467387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad419f6e0 con 0x7f2ad41018d0 2024-03-16T10:16:32.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.044+0000 7f2ad8b19640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad41018d0 0x7f2ad419c7e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000ea60 tx=0x7f2ad000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.044+0000 7f2aca7fc640 1 -- 172.21.15.77:0/1714467387 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad000cde0 con 0x7f2ad41018d0 2024-03-16T10:16:32.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.044+0000 7f2aca7fc640 1 -- 172.21.15.77:0/1714467387 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ad00035f0 con 0x7f2ad41018d0 2024-03-16T10:16:32.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.044+0000 7f2adb57e640 1 -- 172.21.15.77:0/1714467387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad419f9d0 con 0x7f2ad41018d0 2024-03-16T10:16:32.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.044+0000 7f2aca7fc640 1 -- 172.21.15.77:0/1714467387 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0010670 con 0x7f2ad41018d0 2024-03-16T10:16:32.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.044+0000 7f2adb57e640 1 -- 172.21.15.77:0/1714467387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad419ff70 con 0x7f2ad41018d0 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.046+0000 7f2aca7fc640 1 -- 172.21.15.77:0/1714467387 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2ad00037a0 con 0x7f2ad41018d0 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.046+0000 7f2ade16b740 1 Processor -- start 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.046+0000 7f2ade16b740 1 -- start start 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ade16b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5bbd0 0x564ef7a5bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ade16b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5c5c0 0x564ef7a610c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ade16b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 0x564ef7a63bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ade16b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564ef79bfc80 con 0x564ef7a61730 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ade16b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564ef7968aa0 con 0x564ef7a5c5c0 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ade16b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564ef79bbd20 con 0x564ef7a5bbd0 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad8b19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 0x564ef7a63bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad931a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5c5c0 0x564ef7a610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad9b1b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5bbd0 0x564ef7a5bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad8b19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 0x564ef7a63bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50052/0 (socket says 172.21.15.77:50052) 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad931a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5c5c0 0x564ef7a610c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:49352/0 (socket says 172.21.15.77:49352) 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad8b19640 1 -- 172.21.15.77:0/1364079919 learned_addr learned my addr 172.21.15.77:0/1364079919 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2aca7fc640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ac00a2880 0x7f2ac00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad9b1b640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ac00a2880 0x7f2ac00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2aca7fc640 1 -- 172.21.15.77:0/1714467387 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2ad00cec40 con 0x7f2ad41018d0 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad931a640 1 -- 172.21.15.77:0/1364079919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5bbd0 msgr2=0x564ef7a5bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad931a640 1 --2- 172.21.15.77:0/1364079919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5bbd0 0x564ef7a5bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad931a640 1 -- 172.21.15.77:0/1364079919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 msgr2=0x564ef7a63bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad931a640 1 --2- 172.21.15.77:0/1364079919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 0x564ef7a63bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad931a640 1 -- 172.21.15.77:0/1364079919 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ef798aff0 con 0x564ef7a5c5c0 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad8b19640 1 --2- 172.21.15.77:0/1364079919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 0x564ef7a63bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad931a640 1 --2- 172.21.15.77:0/1364079919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5c5c0 0x564ef7a610c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009bf0 tx=0x7f2ac4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f69702898df6e752 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.047+0000 7f2ad9b1b640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ac00a2880 0x7f2ac00a4d40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2acc005e30 tx=0x7f2acc02c0f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.048+0000 7f2abf7fe640 1 -- 172.21.15.77:0/1364079919 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4004050 con 0x564ef7a5c5c0 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.048+0000 7f2abf7fe640 1 -- 172.21.15.77:0/1364079919 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac4005500 con 0x564ef7a5c5c0 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.048+0000 7f2abf7fe640 1 -- 172.21.15.77:0/1364079919 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4004dd0 con 0x564ef7a5c5c0 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.048+0000 7f2ade16b740 1 -- 172.21.15.77:0/1364079919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5c5c0 msgr2=0x564ef7a610c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.048+0000 7f2ade16b740 1 --2- 172.21.15.77:0/1364079919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5c5c0 0x564ef7a610c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009bf0 tx=0x7f2ac4011810 comp rx=0 tx=0).stop 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.048+0000 7f2ade16b740 1 -- 172.21.15.77:0/1364079919 shutdown_connections 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.048+0000 7f2ade16b740 1 --2- 172.21.15.77:0/1364079919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 0x564ef7a63bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.048+0000 7f2ade16b740 1 --2- 172.21.15.77:0/1364079919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5c5c0 0x564ef7a610c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.048+0000 7f2ade16b740 1 --2- 172.21.15.77:0/1364079919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5bbd0 0x564ef7a5bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.048+0000 7f2ade16b740 1 -- 172.21.15.77:0/1364079919 >> 172.21.15.77:0/1364079919 conn(0x564ef7a57140 msgr2=0x564ef7a57540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.048+0000 7f2ade16b740 1 -- 172.21.15.77:0/1364079919 shutdown_connections 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.048+0000 7f2ade16b740 1 -- 172.21.15.77:0/1364079919 wait complete. 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ade16b740 1 -- start start 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ade16b740 1 Processor -- start 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ade16b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5bbd0 0x564ef7a65df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ade16b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5c5c0 0x564ef7a66330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ade16b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a61730 0x564ef7a6d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ade16b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564ef7a646e0 con 0x564ef7a5bbd0 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ade16b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564ef797bef0 con 0x564ef7a61730 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ade16b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564ef7a64860 con 0x564ef7a5c5c0 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ade16b740 10 client.0 _ll_register_callbacks cb 0x564ef7976dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ad931a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5bbd0 0x564ef7a65df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ad931a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5bbd0 0x564ef7a65df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50064/0 (socket says 172.21.15.77:50064) 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ad931a640 1 -- 172.21.15.77:0/824872711 learned_addr learned my addr 172.21.15.77:0/824872711 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ad9b1b640 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a61730 0x564ef7a6d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ad8b19640 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5c5c0 0x564ef7a66330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ad931a640 1 -- 172.21.15.77:0/824872711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5c5c0 msgr2=0x564ef7a66330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ad931a640 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5c5c0 0x564ef7a66330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ad931a640 1 -- 172.21.15.77:0/824872711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a61730 msgr2=0x564ef7a6d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ad931a640 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a61730 0x564ef7a6d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.049+0000 7f2ad931a640 1 -- 172.21.15.77:0/824872711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ef7988a30 con 0x564ef7a5bbd0 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2ad931a640 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5bbd0 0x564ef7a65df0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2ac401b040 tx=0x7f2ac4009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78002ceec747c6f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2abcff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2abcff9640 1 -- 172.21.15.77:0/824872711 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4029070 con 0x564ef7a5bbd0 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2abcff9640 1 -- 172.21.15.77:0/824872711 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2ac4004f20 con 0x564ef7a5bbd0 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2ade16b740 1 -- 172.21.15.77:0/824872711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ac4005500 con 0x564ef7a5bbd0 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2abcff9640 1 -- 172.21.15.77:0/824872711 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2ac4035050 con 0x564ef7a5bbd0 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2ade16b740 10 client.15117 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2ade16b740 1 -- 172.21.15.77:0/824872711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564ef76d42d0 con 0x564ef7a5bbd0 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2abcff9640 1 -- 172.21.15.77:0/824872711 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac4005500 con 0x564ef7a5bbd0 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2abcff9640 1 -- 172.21.15.77:0/824872711 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2ac4003c90 con 0x564ef7a5bbd0 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2ade16b740 10 client.15117 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2ade16b740 10 client.15117 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2ade16b740 20 client.15117 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2ade16b740 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564ef7a72570 0x564ef7a739e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2ade16b740 4 client.15117 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.050+0000 7f2ade16b740 1 -- 172.21.15.77:0/824872711 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ef79cd1c0 con 0x564ef7a72570 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.051+0000 7f2ad8b19640 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564ef7a72570 0x564ef7a739e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.051+0000 7f2abcff9640 10 client.15117 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.051+0000 7f2ad8b19640 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564ef7a72570 0x564ef7a739e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f8bdec2196bf3777 server_cookie=fc60ee998e2159e2 in_seq=0 out_seq=0 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2abcff9640 1 -- 172.21.15.77:0/824872711 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564ef79cd1c0 con 0x564ef7a72570 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2abcff9640 10 client.15117 handle_command_reply: tid=0 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2ade16b740 1 client.15117 shutdown 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2ade16b740 1 -- 172.21.15.77:0/824872711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5bbd0 msgr2=0x564ef7a65df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2ade16b740 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5bbd0 0x564ef7a65df0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2ac401b040 tx=0x7f2ac4009fd0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2ade16b740 1 -- 172.21.15.77:0/824872711 shutdown_connections 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2ade16b740 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564ef7a72570 0x564ef7a739e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2ade16b740 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a61730 0x564ef7a6d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2ade16b740 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5c5c0 0x564ef7a66330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2abcff9640 0 client.15117 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2ade16b740 1 --2- 172.21.15.77:0/824872711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5bbd0 0x564ef7a65df0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2ade16b740 1 -- 172.21.15.77:0/824872711 >> 172.21.15.77:0/824872711 conn(0x564ef7a57140 msgr2=0x564ef7a56380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2ade16b740 1 -- 172.21.15.77:0/824872711 shutdown_connections 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2ade16b740 1 -- 172.21.15.77:0/824872711 wait complete. 2024-03-16T10:16:32.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.052+0000 7f2ade16b740 20 client.15117 trim_cache size 0 max 16384 2024-03-16T10:16:32.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ade16b740 1 Processor -- start 2024-03-16T10:16:32.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ade16b740 1 -- start start 2024-03-16T10:16:32.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ade16b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5bbd0 0x564ef7a71520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ade16b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5c5c0 0x564ef7a6d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ade16b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a61730 0x564ef7a60df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ade16b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564ef7a71d20 con 0x564ef7a5c5c0 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ade16b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564ef7a723f0 con 0x564ef7a5bbd0 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ade16b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564ef7a72ac0 con 0x564ef7a61730 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ad8b19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5c5c0 0x564ef7a6d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ad8b19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5c5c0 0x564ef7a6d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50076/0 (socket says 172.21.15.77:50076) 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ad8b19640 1 -- 172.21.15.77:0/3741295209 learned_addr learned my addr 172.21.15.77:0/3741295209 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ad931a640 1 --2- 172.21.15.77:0/3741295209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5bbd0 0x564ef7a71520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ad9b1b640 1 --2- 172.21.15.77:0/3741295209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a61730 0x564ef7a60df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ad8b19640 1 -- 172.21.15.77:0/3741295209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a61730 msgr2=0x564ef7a60df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ad8b19640 1 --2- 172.21.15.77:0/3741295209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a61730 0x564ef7a60df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ad8b19640 1 -- 172.21.15.77:0/3741295209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5bbd0 msgr2=0x564ef7a71520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ad8b19640 1 --2- 172.21.15.77:0/3741295209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5bbd0 0x564ef7a71520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ad8b19640 1 -- 172.21.15.77:0/3741295209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ef779e6b0 con 0x564ef7a5c5c0 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2ad8b19640 1 --2- 172.21.15.77:0/3741295209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5c5c0 0x564ef7a6d2c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2ad001f000 tx=0x7f2ad00d1170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2da92294bfce545 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2abeffd640 1 -- 172.21.15.77:0/3741295209 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad00bb040 con 0x564ef7a5c5c0 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2abeffd640 1 -- 172.21.15.77:0/3741295209 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ad00860b0 con 0x564ef7a5c5c0 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.070+0000 7f2abeffd640 1 -- 172.21.15.77:0/3741295209 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0099970 con 0x564ef7a5c5c0 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.071+0000 7f2ade16b740 1 -- 172.21.15.77:0/3741295209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5c5c0 msgr2=0x564ef7a6d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.071+0000 7f2ade16b740 1 --2- 172.21.15.77:0/3741295209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5c5c0 0x564ef7a6d2c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2ad001f000 tx=0x7f2ad00d1170 comp rx=0 tx=0).stop 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.071+0000 7f2ade16b740 1 -- 172.21.15.77:0/3741295209 shutdown_connections 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.071+0000 7f2ade16b740 1 --2- 172.21.15.77:0/3741295209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a61730 0x564ef7a60df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.071+0000 7f2ade16b740 1 --2- 172.21.15.77:0/3741295209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a5c5c0 0x564ef7a6d2c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.071+0000 7f2ade16b740 1 --2- 172.21.15.77:0/3741295209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5bbd0 0x564ef7a71520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.071+0000 7f2ade16b740 1 -- 172.21.15.77:0/3741295209 >> 172.21.15.77:0/3741295209 conn(0x564ef7a57140 msgr2=0x564ef7a5ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.071+0000 7f2ade16b740 1 -- 172.21.15.77:0/3741295209 shutdown_connections 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.071+0000 7f2ade16b740 1 -- 172.21.15.77:0/3741295209 wait complete. 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.071+0000 7f2ade16b740 1 -- start start 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.071+0000 7f2ade16b740 1 Processor -- start 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ade16b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5bbd0 0x564ef7a6b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ade16b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5c5c0 0x564ef7ad42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ade16b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 0x564ef7ad6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ade16b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564ef7a73860 con 0x564ef7a61730 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ade16b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564ef7a73190 con 0x564ef7a5c5c0 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ade16b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564ef7a73f30 con 0x564ef7a5bbd0 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ade16b740 10 client.0 _ll_register_callbacks cb 0x7f2ad009b4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ad9b1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 0x564ef7ad6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ad9b1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 0x564ef7ad6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50090/0 (socket says 172.21.15.77:50090) 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ad9b1b640 1 -- 172.21.15.77:0/3393402917 learned_addr learned my addr 172.21.15.77:0/3393402917 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ad931a640 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5bbd0 0x564ef7a6b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ad8b19640 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5c5c0 0x564ef7ad42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ad9b1b640 1 -- 172.21.15.77:0/3393402917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5bbd0 msgr2=0x564ef7a6b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ad9b1b640 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5bbd0 0x564ef7a6b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ad9b1b640 1 -- 172.21.15.77:0/3393402917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5c5c0 msgr2=0x564ef7ad42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ad9b1b640 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5c5c0 0x564ef7ad42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ad9b1b640 1 -- 172.21.15.77:0/3393402917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ef7ad6d50 con 0x564ef7a61730 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2ad9b1b640 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 0x564ef7ad6810 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009ca0 tx=0x7f2acc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e310f0df4e43d52 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.072+0000 7f2abcff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2abcff9640 1 -- 172.21.15.77:0/3393402917 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc039040 con 0x564ef7a61730 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2abcff9640 1 -- 172.21.15.77:0/3393402917 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2acc004240 con 0x564ef7a61730 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2ade16b740 1 -- 172.21.15.77:0/3393402917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ad00860b0 con 0x564ef7a61730 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2abcff9640 1 -- 172.21.15.77:0/3393402917 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2acc004440 con 0x564ef7a61730 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2ade16b740 10 client.15123 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2ade16b740 1 -- 172.21.15.77:0/3393402917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564ef7ad7f60 con 0x564ef7a61730 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2abcff9640 1 -- 172.21.15.77:0/3393402917 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ad00860b0 con 0x564ef7a61730 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2abcff9640 1 -- 172.21.15.77:0/3393402917 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2acc03f070 con 0x564ef7a61730 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2ade16b740 10 client.15123 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2ade16b740 10 client.15123 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2ade16b740 20 client.15123 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2ade16b740 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564ef7adcc00 0x564ef7adb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2ade16b740 4 client.15123 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2ade16b740 1 -- 172.21.15.77:0/3393402917 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "status"}) -- 0x564ef79c7510 con 0x564ef7adcc00 2024-03-16T10:16:32.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.073+0000 7f2ad931a640 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564ef7adcc00 0x564ef7adb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2abcff9640 10 client.15123 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ad931a640 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564ef7adcc00 0x564ef7adb910 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=793ab93a17c3fd5c server_cookie=40d8f320c3213202 in_seq=0 out_seq=0 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2abcff9640 1 -- 172.21.15.77:0/3393402917 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564ef79c7510 con 0x564ef7adcc00 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2abcff9640 10 client.15123 handle_command_reply: tid=0 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ade16b740 1 client.15123 shutdown 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ade16b740 1 -- 172.21.15.77:0/3393402917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 msgr2=0x564ef7ad6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ade16b740 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 0x564ef7ad6810 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009ca0 tx=0x7f2acc0095c0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ade16b740 1 -- 172.21.15.77:0/3393402917 shutdown_connections 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ade16b740 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ef7a61730 0x564ef7ad6810 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ade16b740 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564ef7adcc00 0x564ef7adb910 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ade16b740 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef7a5c5c0 0x564ef7ad42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ade16b740 1 --2- 172.21.15.77:0/3393402917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ef7a5bbd0 0x564ef7a6b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ade16b740 1 -- 172.21.15.77:0/3393402917 >> 172.21.15.77:0/3393402917 conn(0x564ef7a57140 msgr2=0x564ef7a5a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2abcff9640 0 client.15123 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ade16b740 1 -- 172.21.15.77:0/3393402917 shutdown_connections 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ade16b740 1 -- 172.21.15.77:0/3393402917 wait complete. 2024-03-16T10:16:32.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.074+0000 7f2ade16b740 20 client.15123 trim_cache size 0 max 16384 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 2, 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24455, 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 85.831373495999998, 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 26, 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 87, 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 107.424143159 2024-03-16T10:16:32.081 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:16:32.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.075+0000 7f2adb57e640 1 -- 172.21.15.77:0/1714467387 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ac00a2880 msgr2=0x7f2ac00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.075+0000 7f2adb57e640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ac00a2880 0x7f2ac00a4d40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2acc005e30 tx=0x7f2acc02c0f0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.075+0000 7f2adb57e640 1 -- 172.21.15.77:0/1714467387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad41018d0 msgr2=0x7f2ad419c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.075+0000 7f2adb57e640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad41018d0 0x7f2ad419c7e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000ea60 tx=0x7f2ad000ef20 comp rx=0 tx=0).stop 2024-03-16T10:16:32.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.075+0000 7f2adb57e640 1 -- 172.21.15.77:0/1714467387 shutdown_connections 2024-03-16T10:16:32.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.075+0000 7f2adb57e640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ac00a2880 0x7f2ac00a4d40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.075+0000 7f2adb57e640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ad4102290 0x7f2ad419ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.075+0000 7f2adb57e640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ad41018d0 0x7f2ad419c7e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.075+0000 7f2adb57e640 1 --2- 172.21.15.77:0/1714467387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ad41006e0 0x7f2ad40694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.075+0000 7f2adb57e640 1 -- 172.21.15.77:0/1714467387 >> 172.21.15.77:0/1714467387 conn(0x7f2ad40fbc70 msgr2=0x7f2ad4104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.075+0000 7f2adb57e640 1 -- 172.21.15.77:0/1714467387 shutdown_connections 2024-03-16T10:16:32.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.075+0000 7f2adb57e640 1 -- 172.21.15.77:0/1714467387 wait complete. 2024-03-16T10:16:32.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T10:16:32.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:31 smithi192 ceph-mon[36354]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:32.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:31 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/796752218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:32.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:31 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/862720284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:32.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:31 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/93064952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:32.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.193+0000 7f605ae0c640 1 -- 172.21.15.77:0/281141800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60541018d0 msgr2=0x7f6054101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.193+0000 7f605ae0c640 1 --2- 172.21.15.77:0/281141800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60541018d0 0x7f6054101d50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6048009900 tx=0x7f604802bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:32.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.193+0000 7f605ae0c640 1 -- 172.21.15.77:0/281141800 shutdown_connections 2024-03-16T10:16:32.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.193+0000 7f605ae0c640 1 --2- 172.21.15.77:0/281141800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6054102290 0x7f6054108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.193+0000 7f605ae0c640 1 --2- 172.21.15.77:0/281141800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60541018d0 0x7f6054101d50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.193+0000 7f605ae0c640 1 --2- 172.21.15.77:0/281141800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60541006e0 0x7f6054100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.193+0000 7f605ae0c640 1 -- 172.21.15.77:0/281141800 >> 172.21.15.77:0/281141800 conn(0x7f60540fbc70 msgr2=0x7f60540fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.193+0000 7f605ae0c640 1 -- 172.21.15.77:0/281141800 shutdown_connections 2024-03-16T10:16:32.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.193+0000 7f605ae0c640 1 -- 172.21.15.77:0/281141800 wait complete. 2024-03-16T10:16:32.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.193+0000 7f605ae0c640 1 Processor -- start 2024-03-16T10:16:32.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.193+0000 7f605ae0c640 1 -- start start 2024-03-16T10:16:32.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f605ae0c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60541006e0 0x7f60540694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f605ae0c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60541018d0 0x7f605419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f605ae0c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6054102290 0x7f605419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f605ae0c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f605410c390 con 0x7f60541006e0 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f605ae0c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f605410c210 con 0x7f60541018d0 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f605ae0c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f605410c510 con 0x7f6054102290 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6058ba8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60541006e0 0x7f60540694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6058ba8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60541006e0 0x7f60540694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50104/0 (socket says 172.21.15.77:50104) 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6058ba8640 1 -- 172.21.15.77:0/1751140576 learned_addr learned my addr 172.21.15.77:0/1751140576 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f60593a9640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6054102290 0x7f605419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6053fff640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60541018d0 0x7f605419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6058ba8640 1 -- 172.21.15.77:0/1751140576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6054102290 msgr2=0x7f605419ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6058ba8640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6054102290 0x7f605419ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6058ba8640 1 -- 172.21.15.77:0/1751140576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60541018d0 msgr2=0x7f605419c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6058ba8640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60541018d0 0x7f605419c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6058ba8640 1 -- 172.21.15.77:0/1751140576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605419f780 con 0x7f60541006e0 2024-03-16T10:16:32.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6053fff640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60541018d0 0x7f605419c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:16:32.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6058ba8640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60541006e0 0x7f60540694b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f604400e9c0 tx=0x7f604400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6051ffb640 1 -- 172.21.15.77:0/1751140576 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604400cd80 con 0x7f60541006e0 2024-03-16T10:16:32.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f605ae0c640 1 -- 172.21.15.77:0/1751140576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605419fa10 con 0x7f60541006e0 2024-03-16T10:16:32.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6051ffb640 1 -- 172.21.15.77:0/1751140576 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60440035f0 con 0x7f60541006e0 2024-03-16T10:16:32.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.194+0000 7f6051ffb640 1 -- 172.21.15.77:0/1751140576 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6044010690 con 0x7f60541006e0 2024-03-16T10:16:32.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.195+0000 7f605ae0c640 1 -- 172.21.15.77:0/1751140576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605419ff30 con 0x7f60541006e0 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f6051ffb640 1 -- 172.21.15.77:0/1751140576 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f60440037a0 con 0x7f60541006e0 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f605d9f0740 1 Processor -- start 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f605d9f0740 1 -- start start 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f605d9f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120adbd0 0x55a0120adff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f605d9f0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 0x55a0120b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f605d9f0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a0120b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f605d9f0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a012011c80 con 0x55a0120ae5c0 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f605d9f0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a011fbaaa0 con 0x55a0120b3730 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f605d9f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a01200dd20 con 0x55a0120adbd0 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f6058ba8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 0x55a0120b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f60593a9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120adbd0 0x55a0120adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f60593a9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120adbd0 0x55a0120adff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:56836/0 (socket says 172.21.15.77:56836) 2024-03-16T10:16:32.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f6058ba8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 0x55a0120b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50110/0 (socket says 172.21.15.77:50110) 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f60593a9640 1 -- 172.21.15.77:0/2858496304 learned_addr learned my addr 172.21.15.77:0/2858496304 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.197+0000 7f6053fff640 1 --2- 172.21.15.77:0/2858496304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a0120b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f6051ffb640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f603c0a2800 0x7f603c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f60593a9640 1 -- 172.21.15.77:0/2858496304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 msgr2=0x55a0120b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f60593a9640 1 --2- 172.21.15.77:0/2858496304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a0120b5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f60593a9640 1 -- 172.21.15.77:0/2858496304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 msgr2=0x55a0120b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f60593a9640 1 --2- 172.21.15.77:0/2858496304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 0x55a0120b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f60593a9640 1 -- 172.21.15.77:0/2858496304 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a011fdcff0 con 0x55a0120adbd0 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f6053fff640 1 --2- 172.21.15.77:0/2858496304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a0120b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f6058ba8640 1 --2- 172.21.15.77:0/2858496304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 0x55a0120b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f6051ffb640 1 -- 172.21.15.77:0/1751140576 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f60440cee30 con 0x7f60541006e0 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f60593a9640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f603c0a2800 0x7f603c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f60593a9640 1 --2- 172.21.15.77:0/2858496304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120adbd0 0x55a0120adff0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f604c011590 tx=0x7f604c011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49173ace03cfbb1e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f603affd640 1 -- 172.21.15.77:0/2858496304 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c017050 con 0x55a0120adbd0 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f60593a9640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f603c0a2800 0x7f603c0a4cc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f604c022000 tx=0x7f604c023000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f603affd640 1 -- 172.21.15.77:0/2858496304 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f604c00fd20 con 0x55a0120adbd0 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.198+0000 7f603affd640 1 -- 172.21.15.77:0/2858496304 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c01bb70 con 0x55a0120adbd0 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.199+0000 7f605d9f0740 1 -- 172.21.15.77:0/2858496304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120adbd0 msgr2=0x55a0120adff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.199+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2858496304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120adbd0 0x55a0120adff0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f604c011590 tx=0x7f604c011a50 comp rx=0 tx=0).stop 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.199+0000 7f605d9f0740 1 -- 172.21.15.77:0/2858496304 shutdown_connections 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.199+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2858496304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a0120b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.199+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2858496304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 0x55a0120b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.199+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2858496304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120adbd0 0x55a0120adff0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.199+0000 7f605d9f0740 1 -- 172.21.15.77:0/2858496304 >> 172.21.15.77:0/2858496304 conn(0x55a0120a9140 msgr2=0x55a0120a9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.199+0000 7f605d9f0740 1 -- 172.21.15.77:0/2858496304 shutdown_connections 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.199+0000 7f605d9f0740 1 -- 172.21.15.77:0/2858496304 wait complete. 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.199+0000 7f605d9f0740 1 -- start start 2024-03-16T10:16:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.199+0000 7f605d9f0740 1 Processor -- start 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f605d9f0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 0x55a0120b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f605d9f0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120ae5c0 0x55a0120b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f605d9f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120b3730 0x55a0120bf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f605d9f0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a0120b66e0 con 0x55a0120adbd0 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f605d9f0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a011fcdef0 con 0x55a0120ae5c0 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f605d9f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a0120b6860 con 0x55a0120b3730 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f605d9f0740 10 client.0 _ll_register_callbacks cb 0x55a011fc8dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6058ba8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 0x55a0120b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6058ba8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 0x55a0120b7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50122/0 (socket says 172.21.15.77:50122) 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6053fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120ae5c0 0x55a0120b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6058ba8640 1 -- 172.21.15.77:0/1096109340 learned_addr learned my addr 172.21.15.77:0/1096109340 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f60593a9640 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120b3730 0x55a0120bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6058ba8640 1 -- 172.21.15.77:0/1096109340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120b3730 msgr2=0x55a0120bf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6058ba8640 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120b3730 0x55a0120bf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6058ba8640 1 -- 172.21.15.77:0/1096109340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120ae5c0 msgr2=0x55a0120b82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6058ba8640 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120ae5c0 0x55a0120b82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6058ba8640 1 -- 172.21.15.77:0/1096109340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a011fdaa30 con 0x55a0120adbd0 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6053fff640 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120ae5c0 0x55a0120b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6058ba8640 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 0x55a0120b7db0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f60440058c0 tx=0x7f60440974a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df93aeeb335c3ac server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6033fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6033fff640 1 -- 172.21.15.77:0/1096109340 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604409b1c0 con 0x55a0120adbd0 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f6033fff640 1 -- 172.21.15.77:0/1096109340 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f6044003b30 con 0x55a0120adbd0 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.200+0000 7f605d9f0740 1 -- 172.21.15.77:0/1096109340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f604c00fd20 con 0x55a0120adbd0 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.201+0000 7f6033fff640 1 -- 172.21.15.77:0/1096109340 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f604409bf60 con 0x55a0120adbd0 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.201+0000 7f605d9f0740 10 client.15135 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.201+0000 7f605d9f0740 1 -- 172.21.15.77:0/1096109340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a011d262d0 con 0x55a0120adbd0 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.201+0000 7f6033fff640 1 -- 172.21.15.77:0/1096109340 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f604c00fd20 con 0x55a0120adbd0 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.201+0000 7f6033fff640 1 -- 172.21.15.77:0/1096109340 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f604409cb90 con 0x55a0120adbd0 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.201+0000 7f605d9f0740 10 client.15135 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.201+0000 7f605d9f0740 10 client.15135 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.201+0000 7f605d9f0740 20 client.15135 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.201+0000 7f605d9f0740 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a0120c4790 0x55a0120c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.201+0000 7f605d9f0740 4 client.15135 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.201+0000 7f605d9f0740 1 -- 172.21.15.77:0/1096109340 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a01201f1c0 con 0x55a0120c4790 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.201+0000 7f6053fff640 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a0120c4790 0x55a0120c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f6033fff640 10 client.15135 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f6053fff640 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a0120c4790 0x55a0120c5c00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e68015062bcbe44e server_cookie=6ec00b44c9494acb in_seq=0 out_seq=0 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f6033fff640 1 -- 172.21.15.77:0/1096109340 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a01201f1c0 con 0x55a0120c4790 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f6033fff640 10 client.15135 handle_command_reply: tid=0 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f605d9f0740 1 client.15135 shutdown 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f605d9f0740 1 -- 172.21.15.77:0/1096109340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 msgr2=0x55a0120b7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f605d9f0740 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 0x55a0120b7db0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f60440058c0 tx=0x7f60440974a0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f605d9f0740 1 -- 172.21.15.77:0/1096109340 shutdown_connections 2024-03-16T10:16:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f605d9f0740 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a0120c4790 0x55a0120c5c00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f605d9f0740 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120b3730 0x55a0120bf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f605d9f0740 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120ae5c0 0x55a0120b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f6033fff640 0 client.15135 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:16:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.202+0000 7f605d9f0740 1 --2- 172.21.15.77:0/1096109340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 0x55a0120b7db0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.203+0000 7f605d9f0740 1 -- 172.21.15.77:0/1096109340 >> 172.21.15.77:0/1096109340 conn(0x55a0120a9140 msgr2=0x55a0120a8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.203+0000 7f605d9f0740 1 -- 172.21.15.77:0/1096109340 shutdown_connections 2024-03-16T10:16:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.203+0000 7f605d9f0740 1 -- 172.21.15.77:0/1096109340 wait complete. 2024-03-16T10:16:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.203+0000 7f605d9f0740 20 client.15135 trim_cache size 0 max 16384 2024-03-16T10:16:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.220+0000 7f605d9f0740 1 Processor -- start 2024-03-16T10:16:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f605d9f0740 1 -- start start 2024-03-16T10:16:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f605d9f0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 0x55a0120a4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f605d9f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120ae5c0 0x55a0120bf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f605d9f0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a0120b79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f605d9f0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a0120c3f40 con 0x55a0120adbd0 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f605d9f0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a0120c4610 con 0x55a0120b3730 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f605d9f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a0120c4ce0 con 0x55a0120ae5c0 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f6058ba8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 0x55a0120a4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f6058ba8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 0x55a0120a4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50132/0 (socket says 172.21.15.77:50132) 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f6058ba8640 1 -- 172.21.15.77:0/2552963001 learned_addr learned my addr 172.21.15.77:0/2552963001 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f6053fff640 1 --2- 172.21.15.77:0/2552963001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120ae5c0 0x55a0120bf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f60593a9640 1 --2- 172.21.15.77:0/2552963001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a0120b79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f6058ba8640 1 -- 172.21.15.77:0/2552963001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120ae5c0 msgr2=0x55a0120bf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f6058ba8640 1 --2- 172.21.15.77:0/2552963001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120ae5c0 0x55a0120bf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f6058ba8640 1 -- 172.21.15.77:0/2552963001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 msgr2=0x55a0120b79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f6058ba8640 1 --2- 172.21.15.77:0/2552963001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a0120b79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f6058ba8640 1 -- 172.21.15.77:0/2552963001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a011df06b0 con 0x55a0120adbd0 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f60593a9640 1 --2- 172.21.15.77:0/2552963001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a0120b79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f6058ba8640 1 --2- 172.21.15.77:0/2552963001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 0x55a0120a4210 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6044097cc0 tx=0x7f6044004720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12e8904079470af server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f603a7fc640 1 -- 172.21.15.77:0/2552963001 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60440d20d0 con 0x55a0120adbd0 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f603a7fc640 1 -- 172.21.15.77:0/2552963001 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60440d2560 con 0x55a0120adbd0 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.221+0000 7f603a7fc640 1 -- 172.21.15.77:0/2552963001 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604409b630 con 0x55a0120adbd0 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.222+0000 7f605d9f0740 1 -- 172.21.15.77:0/2552963001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 msgr2=0x55a0120a4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.222+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2552963001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 0x55a0120a4210 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6044097cc0 tx=0x7f6044004720 comp rx=0 tx=0).stop 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.222+0000 7f605d9f0740 1 -- 172.21.15.77:0/2552963001 shutdown_connections 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.222+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2552963001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a0120b79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.222+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2552963001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120ae5c0 0x55a0120bf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.222+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2552963001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120adbd0 0x55a0120a4210 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.222+0000 7f605d9f0740 1 -- 172.21.15.77:0/2552963001 >> 172.21.15.77:0/2552963001 conn(0x55a0120a9140 msgr2=0x55a0120a9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.222+0000 7f605d9f0740 1 -- 172.21.15.77:0/2552963001 shutdown_connections 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.222+0000 7f605d9f0740 1 -- 172.21.15.77:0/2552963001 wait complete. 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.222+0000 7f605d9f0740 1 -- start start 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.222+0000 7f605d9f0740 1 Processor -- start 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f605d9f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120adbd0 0x55a0120bd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f605d9f0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 0x55a0121264e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f605d9f0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a012128a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f605d9f0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a0120c5a80 con 0x55a0120ae5c0 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f605d9f0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a0120c53b0 con 0x55a0120b3730 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f605d9f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a0120c6150 con 0x55a0120adbd0 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f605d9f0740 10 client.0 _ll_register_callbacks cb 0x7f60440097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f6053fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 0x55a0121264e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f6053fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 0x55a0121264e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50134/0 (socket says 172.21.15.77:50134) 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f6053fff640 1 -- 172.21.15.77:0/2934645324 learned_addr learned my addr 172.21.15.77:0/2934645324 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f6058ba8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120adbd0 0x55a0120bd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f60593a9640 1 --2- 172.21.15.77:0/2934645324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a012128a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f6053fff640 1 -- 172.21.15.77:0/2934645324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120adbd0 msgr2=0x55a0120bd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f6053fff640 1 --2- 172.21.15.77:0/2934645324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120adbd0 0x55a0120bd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f6053fff640 1 -- 172.21.15.77:0/2934645324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 msgr2=0x55a012128a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f6053fff640 1 --2- 172.21.15.77:0/2934645324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a012128a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f6053fff640 1 -- 172.21.15.77:0/2934645324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a012128f70 con 0x55a0120ae5c0 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f6053fff640 1 --2- 172.21.15.77:0/2934645324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 0x55a0121264e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6048009900 tx=0x7f604802c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae118ea54d18019 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.223+0000 7f6033fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f6033fff640 1 -- 172.21.15.77:0/2934645324 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048005040 con 0x55a0120ae5c0 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f6033fff640 1 -- 172.21.15.77:0/2934645324 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f60480051e0 con 0x55a0120ae5c0 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f605d9f0740 1 -- 172.21.15.77:0/2934645324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a012019510 con 0x55a0120ae5c0 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f6033fff640 1 -- 172.21.15.77:0/2934645324 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f60480053e0 con 0x55a0120ae5c0 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f605d9f0740 10 client.15141 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f605d9f0740 1 -- 172.21.15.77:0/2934645324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a01212a240 con 0x55a0120ae5c0 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f6033fff640 1 -- 172.21.15.77:0/2934645324 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a012019510 con 0x55a0120ae5c0 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f6033fff640 1 -- 172.21.15.77:0/2934645324 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f6048003820 con 0x55a0120ae5c0 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f605d9f0740 10 client.15141 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f605d9f0740 10 client.15141 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f605d9f0740 20 client.15141 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2934645324 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a01212eec0 0x55a01212dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f605d9f0740 4 client.15141 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f605d9f0740 1 -- 172.21.15.77:0/2934645324 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "status"}) -- 0x55a011f97a00 con 0x55a01212eec0 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.224+0000 7f6058ba8640 1 --2- 172.21.15.77:0/2934645324 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a01212eec0 0x55a01212dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f6058ba8640 1 --2- 172.21.15.77:0/2934645324 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a01212eec0 0x55a01212dbd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1937a114c182aa97 server_cookie=27466117de07c2e0 in_seq=0 out_seq=0 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f6033fff640 10 client.15141 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f6033fff640 1 -- 172.21.15.77:0/2934645324 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a011f97a00 con 0x55a01212eec0 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f6033fff640 10 client.15141 handle_command_reply: tid=0 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f605d9f0740 1 client.15141 shutdown 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f605d9f0740 1 -- 172.21.15.77:0/2934645324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 msgr2=0x55a0121264e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2934645324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 0x55a0121264e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6048009900 tx=0x7f604802c1f0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f605d9f0740 1 -- 172.21.15.77:0/2934645324 shutdown_connections 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2934645324 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a01212eec0 0x55a01212dbd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2934645324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a0120b3730 0x55a012128a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2934645324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a0120ae5c0 0x55a0121264e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f6033fff640 0 client.15141 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f605d9f0740 1 --2- 172.21.15.77:0/2934645324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a0120adbd0 0x55a0120bd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f605d9f0740 1 -- 172.21.15.77:0/2934645324 >> 172.21.15.77:0/2934645324 conn(0x55a0120a9140 msgr2=0x55a0120b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f605d9f0740 1 -- 172.21.15.77:0/2934645324 shutdown_connections 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f605d9f0740 1 -- 172.21.15.77:0/2934645324 wait complete. 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.225+0000 7f605d9f0740 20 client.15141 trim_cache size 0 max 16384 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 1, 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24475, 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 86.987214648999995, 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 26, 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 87, 2024-03-16T10:16:32.232 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T10:16:32.233 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 104.768582583 2024-03-16T10:16:32.233 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:16:32.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.226+0000 7f605ae0c640 1 -- 172.21.15.77:0/1751140576 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f603c0a2800 msgr2=0x7f603c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.226+0000 7f605ae0c640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f603c0a2800 0x7f603c0a4cc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f604c022000 tx=0x7f604c023000 comp rx=0 tx=0).stop 2024-03-16T10:16:32.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.226+0000 7f605ae0c640 1 -- 172.21.15.77:0/1751140576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60541006e0 msgr2=0x7f60540694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.226+0000 7f605ae0c640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60541006e0 0x7f60540694b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f604400e9c0 tx=0x7f604400ee80 comp rx=0 tx=0).stop 2024-03-16T10:16:32.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.226+0000 7f605ae0c640 1 -- 172.21.15.77:0/1751140576 shutdown_connections 2024-03-16T10:16:32.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.226+0000 7f605ae0c640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f603c0a2800 0x7f603c0a4cc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.226+0000 7f605ae0c640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6054102290 0x7f605419ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.226+0000 7f605ae0c640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60541018d0 0x7f605419c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.226+0000 7f605ae0c640 1 --2- 172.21.15.77:0/1751140576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60541006e0 0x7f60540694b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.226+0000 7f605ae0c640 1 -- 172.21.15.77:0/1751140576 >> 172.21.15.77:0/1751140576 conn(0x7f60540fbc70 msgr2=0x7f6054104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.226+0000 7f605ae0c640 1 -- 172.21.15.77:0/1751140576 shutdown_connections 2024-03-16T10:16:32.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.226+0000 7f605ae0c640 1 -- 172.21.15.77:0/1751140576 wait complete. 2024-03-16T10:16:32.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T10:16:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:31 smithi193 ceph-mon[36311]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:31 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/796752218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:31 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/862720284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:31 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/93064952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:32.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.342+0000 7f4bee8f1640 1 -- 172.21.15.77:0/2167680800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be8107520 msgr2=0x7f4be8109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.342+0000 7f4bee8f1640 1 --2- 172.21.15.77:0/2167680800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be8107520 0x7f4be8109910 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009be0 tx=0x7f4be002bef0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.342+0000 7f4bee8f1640 1 -- 172.21.15.77:0/2167680800 shutdown_connections 2024-03-16T10:16:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.342+0000 7f4bee8f1640 1 --2- 172.21.15.77:0/2167680800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be8107520 0x7f4be8109910 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.342+0000 7f4bee8f1640 1 --2- 172.21.15.77:0/2167680800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4be8104bf0 0x7f4be8106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.342+0000 7f4bee8f1640 1 --2- 172.21.15.77:0/2167680800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be81022c0 0x7f4be81046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.342+0000 7f4bee8f1640 1 -- 172.21.15.77:0/2167680800 >> 172.21.15.77:0/2167680800 conn(0x7f4be80fbc70 msgr2=0x7f4be80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.342+0000 7f4bee8f1640 1 -- 172.21.15.77:0/2167680800 shutdown_connections 2024-03-16T10:16:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.342+0000 7f4bee8f1640 1 -- 172.21.15.77:0/2167680800 wait complete. 2024-03-16T10:16:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.342+0000 7f4bee8f1640 1 Processor -- start 2024-03-16T10:16:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.342+0000 7f4bee8f1640 1 -- start start 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4bee8f1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be81022c0 0x7f4be8197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4bee8f1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4be8104bf0 0x7f4be8198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4bee8f1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8107520 0x7f4be819f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4bee8f1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4be810d460 con 0x7f4be81022c0 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4bee8f1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4be810d2e0 con 0x7f4be8104bf0 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4bee8f1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4be810d5e0 con 0x7f4be8107520 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be81022c0 0x7f4be8197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be81022c0 0x7f4be8197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50160/0 (socket says 172.21.15.77:50160) 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be7fff640 1 -- 172.21.15.77:0/1645735805 learned_addr learned my addr 172.21.15.77:0/1645735805 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be77fe640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4be8104bf0 0x7f4be8198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4bece8e640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8107520 0x7f4be819f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be7fff640 1 -- 172.21.15.77:0/1645735805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8107520 msgr2=0x7f4be819f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be7fff640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8107520 0x7f4be819f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be7fff640 1 -- 172.21.15.77:0/1645735805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4be8104bf0 msgr2=0x7f4be8198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be7fff640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4be8104bf0 0x7f4be8198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be7fff640 1 -- 172.21.15.77:0/1645735805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be819f750 con 0x7f4be81022c0 2024-03-16T10:16:32.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be7fff640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be81022c0 0x7f4be8197b30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800e9c0 tx=0x7f4bd800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be57fa640 1 -- 172.21.15.77:0/1645735805 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd800ccc0 con 0x7f4be81022c0 2024-03-16T10:16:32.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4bee8f1640 1 -- 172.21.15.77:0/1645735805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be819fa40 con 0x7f4be81022c0 2024-03-16T10:16:32.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be57fa640 1 -- 172.21.15.77:0/1645735805 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd8005500 con 0x7f4be81022c0 2024-03-16T10:16:32.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4be57fa640 1 -- 172.21.15.77:0/1645735805 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8010430 con 0x7f4be81022c0 2024-03-16T10:16:32.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.343+0000 7f4bee8f1640 1 -- 172.21.15.77:0/1645735805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be819ff90 con 0x7f4be81022c0 2024-03-16T10:16:32.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.345+0000 7f4be57fa640 1 -- 172.21.15.77:0/1645735805 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4bd80046c0 con 0x7f4be81022c0 2024-03-16T10:16:32.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.346+0000 7f4be57fa640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4bd00a2800 0x7f4bd00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.346+0000 7f4be77fe640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4bd00a2800 0x7f4bd00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.346+0000 7f4be57fa640 1 -- 172.21.15.77:0/1645735805 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f4bd80cecc0 con 0x7f4be81022c0 2024-03-16T10:16:32.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.346+0000 7f4bf15ba740 1 Processor -- start 2024-03-16T10:16:32.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.346+0000 7f4bf15ba740 1 -- start start 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.346+0000 7f4bf15ba740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 0x558addd9eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.346+0000 7f4be77fe640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4bd00a2800 0x7f4bd00a4cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc0095f0 tx=0x7f4bdc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.346+0000 7f4bf15ba740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558addda40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.346+0000 7f4bf15ba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addda4730 0x558addda6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.346+0000 7f4bf15ba740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558addd02c80 con 0x558addd9ebd0 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4bf15ba740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x558addcabaa0 con 0x558addd9f5c0 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4bf15ba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558addcfed20 con 0x558addda4730 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4be7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 0x558addd9eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4be7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 0x558addd9eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50164/0 (socket says 172.21.15.77:50164) 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4bece8e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addda4730 0x558addda6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4be77fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558addda40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4be7fff640 1 -- 172.21.15.77:0/487573895 learned_addr learned my addr 172.21.15.77:0/487573895 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4be7fff640 1 -- 172.21.15.77:0/487573895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addda4730 msgr2=0x558addda6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4be7fff640 1 --2- 172.21.15.77:0/487573895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addda4730 0x558addda6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4be7fff640 1 -- 172.21.15.77:0/487573895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 msgr2=0x558addda40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4be7fff640 1 --2- 172.21.15.77:0/487573895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558addda40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4be7fff640 1 -- 172.21.15.77:0/487573895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558addccdff0 con 0x558addd9ebd0 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4be7fff640 1 --2- 172.21.15.77:0/487573895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 0x558addd9eff0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4bd80a3960 tx=0x7f4bd8097390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c212bd31b044425 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4bce7fc640 1 -- 172.21.15.77:0/487573895 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd80b4070 con 0x558addd9ebd0 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4bce7fc640 1 -- 172.21.15.77:0/487573895 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd809f840 con 0x558addd9ebd0 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.347+0000 7f4bce7fc640 1 -- 172.21.15.77:0/487573895 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd80a3c60 con 0x558addd9ebd0 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.348+0000 7f4bf15ba740 1 -- 172.21.15.77:0/487573895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 msgr2=0x558addd9eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.348+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/487573895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 0x558addd9eff0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4bd80a3960 tx=0x7f4bd8097390 comp rx=0 tx=0).stop 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.348+0000 7f4bf15ba740 1 -- 172.21.15.77:0/487573895 shutdown_connections 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.348+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/487573895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addda4730 0x558addda6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.348+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/487573895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558addda40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.348+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/487573895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 0x558addd9eff0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.348+0000 7f4bf15ba740 1 -- 172.21.15.77:0/487573895 >> 172.21.15.77:0/487573895 conn(0x558addd9a140 msgr2=0x558addd9a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.348+0000 7f4bf15ba740 1 -- 172.21.15.77:0/487573895 shutdown_connections 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.348+0000 7f4bf15ba740 1 -- 172.21.15.77:0/487573895 wait complete. 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.348+0000 7f4bf15ba740 1 -- start start 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bf15ba740 1 Processor -- start 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bf15ba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addd9ebd0 0x558addda8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bf15ba740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558addda9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bf15ba740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 0x558adddb0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bf15ba740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558addda76e0 con 0x558addda4730 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bf15ba740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x558addcbeef0 con 0x558addd9f5c0 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bf15ba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558addda7860 con 0x558addd9ebd0 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bece8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 0x558adddb0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4be7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addd9ebd0 0x558addda8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bece8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 0x558adddb0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50176/0 (socket says 172.21.15.77:50176) 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bf15ba740 10 client.0 _ll_register_callbacks cb 0x558addcb9dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:32.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bece8e640 1 -- 172.21.15.77:0/1104132278 learned_addr learned my addr 172.21.15.77:0/1104132278 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4be77fe640 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558addda9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bece8e640 1 -- 172.21.15.77:0/1104132278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addd9ebd0 msgr2=0x558addda8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bece8e640 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addd9ebd0 0x558addda8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bece8e640 1 -- 172.21.15.77:0/1104132278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 msgr2=0x558addda9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bece8e640 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558addda9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4bece8e640 1 -- 172.21.15.77:0/1104132278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558addccba30 con 0x558addda4730 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4be77fe640 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558addda9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.349+0000 7f4be7fff640 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addd9ebd0 0x558addda8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bece8e640 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 0x558adddb0360 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009be0 tx=0x7f4be0005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e6ac6e74edbde7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bc77fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bc77fe640 1 -- 172.21.15.77:0/1104132278 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be0033680 con 0x558addda4730 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1104132278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bd809f840 con 0x558addda4730 2024-03-16T10:16:32.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bc77fe640 1 -- 172.21.15.77:0/1104132278 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f4be0033820 con 0x558addda4730 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bc77fe640 1 -- 172.21.15.77:0/1104132278 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4be0003970 con 0x558addda4730 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bf15ba740 10 client.15153 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1104132278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558adda172d0 con 0x558addda4730 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bc77fe640 1 -- 172.21.15.77:0/1104132278 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd809f840 con 0x558addda4730 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bc77fe640 1 -- 172.21.15.77:0/1104132278 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f4be0004280 con 0x558addda4730 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bf15ba740 10 client.15153 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bf15ba740 10 client.15153 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bf15ba740 20 client.15153 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558adddb5910 0x558adddb6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bf15ba740 4 client.15153 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.350+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1104132278 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558addd101c0 con 0x558adddb5910 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.351+0000 7f4be7fff640 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558adddb5910 0x558adddb6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.351+0000 7f4be7fff640 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558adddb5910 0x558adddb6d80 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=365d88d794108fdb server_cookie=9f9f1cc48fb98f39 in_seq=0 out_seq=0 2024-03-16T10:16:32.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.351+0000 7f4bc77fe640 10 client.15153 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bc77fe640 1 -- 172.21.15.77:0/1104132278 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558addd101c0 con 0x558adddb5910 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bc77fe640 10 client.15153 handle_command_reply: tid=0 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bf15ba740 1 client.15153 shutdown 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1104132278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 msgr2=0x558adddb0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 0x558adddb0360 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009be0 tx=0x7f4be0005c60 comp rx=0 tx=0).stop 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1104132278 shutdown_connections 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558adddb5910 0x558adddb6d80 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 0x558adddb0360 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558addda9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bc77fe640 0 client.15153 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/1104132278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addd9ebd0 0x558addda8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1104132278 >> 172.21.15.77:0/1104132278 conn(0x558addd9a140 msgr2=0x558addd99380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1104132278 shutdown_connections 2024-03-16T10:16:32.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1104132278 wait complete. 2024-03-16T10:16:32.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.352+0000 7f4bf15ba740 20 client.15153 trim_cache size 0 max 16384 2024-03-16T10:16:32.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.369+0000 7f4bf15ba740 1 Processor -- start 2024-03-16T10:16:32.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4bf15ba740 1 -- start start 2024-03-16T10:16:32.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4bf15ba740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 0x558adddb46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4bf15ba740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558adddb0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4bf15ba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addda4730 0x558addda8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4bf15ba740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558adddb50c0 con 0x558addd9ebd0 2024-03-16T10:16:32.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4bf15ba740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x558adddb5790 con 0x558addd9f5c0 2024-03-16T10:16:32.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4bf15ba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558adddb5e60 con 0x558addda4730 2024-03-16T10:16:32.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4be7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 0x558adddb46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4be7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 0x558adddb46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50186/0 (socket says 172.21.15.77:50186) 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4be7fff640 1 -- 172.21.15.77:0/4050367604 learned_addr learned my addr 172.21.15.77:0/4050367604 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4bece8e640 1 --2- 172.21.15.77:0/4050367604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addda4730 0x558addda8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4be77fe640 1 --2- 172.21.15.77:0/4050367604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558adddb0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4be7fff640 1 -- 172.21.15.77:0/4050367604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addda4730 msgr2=0x558addda8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4be7fff640 1 --2- 172.21.15.77:0/4050367604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addda4730 0x558addda8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4be7fff640 1 -- 172.21.15.77:0/4050367604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 msgr2=0x558adddb0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4be7fff640 1 --2- 172.21.15.77:0/4050367604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558adddb0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4be7fff640 1 -- 172.21.15.77:0/4050367604 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558addae16b0 con 0x558addd9ebd0 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4be7fff640 1 --2- 172.21.15.77:0/4050367604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 0x558adddb46a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4bd80cc8a0 tx=0x7f4bd8004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95a5614ae3034c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4bcdffb640 1 -- 172.21.15.77:0/4050367604 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd80b4040 con 0x558addd9ebd0 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4bcdffb640 1 -- 172.21.15.77:0/4050367604 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd807f070 con 0x558addd9ebd0 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.370+0000 7f4bcdffb640 1 -- 172.21.15.77:0/4050367604 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8099a80 con 0x558addd9ebd0 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.371+0000 7f4bf15ba740 1 -- 172.21.15.77:0/4050367604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 msgr2=0x558adddb46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.371+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/4050367604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 0x558adddb46a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4bd80cc8a0 tx=0x7f4bd8004b00 comp rx=0 tx=0).stop 2024-03-16T10:16:32.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.371+0000 7f4bf15ba740 1 -- 172.21.15.77:0/4050367604 shutdown_connections 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.371+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/4050367604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addda4730 0x558addda8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.371+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/4050367604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558adddb0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.371+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/4050367604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addd9ebd0 0x558adddb46a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.371+0000 7f4bf15ba740 1 -- 172.21.15.77:0/4050367604 >> 172.21.15.77:0/4050367604 conn(0x558addd9a140 msgr2=0x558addda3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.371+0000 7f4bf15ba740 1 -- 172.21.15.77:0/4050367604 shutdown_connections 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.371+0000 7f4bf15ba740 1 -- 172.21.15.77:0/4050367604 wait complete. 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.371+0000 7f4bf15ba740 1 -- start start 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.371+0000 7f4bf15ba740 1 Processor -- start 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bf15ba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addd9ebd0 0x558addda2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bf15ba740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558adddb0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bf15ba740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 0x558addda27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bf15ba740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558adddb6c00 con 0x558addda4730 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bf15ba740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x558adddb6530 con 0x558addd9f5c0 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bf15ba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558adddb72d0 con 0x558addd9ebd0 2024-03-16T10:16:32.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bf15ba740 10 client.0 _ll_register_callbacks cb 0x7f4be0009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bece8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 0x558addda27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4be77fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558adddb0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4be77fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558adddb0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:49476/0 (socket says 172.21.15.77:49476) 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4be77fe640 1 -- 172.21.15.77:0/1922681730 learned_addr learned my addr 172.21.15.77:0/1922681730 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bece8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 0x558addda27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50200/0 (socket says 172.21.15.77:50200) 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4be7fff640 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addd9ebd0 0x558addda2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bece8e640 1 -- 172.21.15.77:0/1922681730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addd9ebd0 msgr2=0x558addda2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bece8e640 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addd9ebd0 0x558addda2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bece8e640 1 -- 172.21.15.77:0/1922681730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 msgr2=0x558adddb0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bece8e640 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558adddb0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bece8e640 1 -- 172.21.15.77:0/1922681730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558adde196a0 con 0x558addda4730 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4be77fe640 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558adddb0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bece8e640 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 0x558addda27d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4be00099a0 tx=0x7f4be00337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82da2117f1fee90e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bc77fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:32.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bc77fe640 1 -- 172.21.15.77:0/1922681730 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be0009800 con 0x558addda4730 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bc77fe640 1 -- 172.21.15.77:0/1922681730 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f4be0034840 con 0x558addda4730 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.372+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1922681730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bd807f070 con 0x558addda4730 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.373+0000 7f4bc77fe640 1 -- 172.21.15.77:0/1922681730 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4be0034a40 con 0x558addda4730 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.373+0000 7f4bf15ba740 10 client.15159 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.373+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1922681730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558adde1aa10 con 0x558addda4730 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.373+0000 7f4bc77fe640 1 -- 172.21.15.77:0/1922681730 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd807f070 con 0x558addda4730 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.373+0000 7f4bc77fe640 1 -- 172.21.15.77:0/1922681730 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f4be0004080 con 0x558addda4730 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.373+0000 7f4bf15ba740 10 client.15159 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.373+0000 7f4bf15ba740 10 client.15159 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.373+0000 7f4bf15ba740 20 client.15159 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.373+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558adde1f5d0 0x558adde1e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.373+0000 7f4bf15ba740 4 client.15159 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.373+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1922681730 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "status"}) -- 0x558addd0a510 con 0x558adde1f5d0 2024-03-16T10:16:32.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.373+0000 7f4be7fff640 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558adde1f5d0 0x558adde1e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bc77fe640 10 client.15159 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4be7fff640 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558adde1f5d0 0x558adde1e2e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f3c2cab7242666b server_cookie=faa463d7129fb461 in_seq=0 out_seq=0 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bc77fe640 1 -- 172.21.15.77:0/1922681730 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558addd0a510 con 0x558adde1f5d0 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bc77fe640 10 client.15159 handle_command_reply: tid=0 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bf15ba740 1 client.15159 shutdown 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1922681730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 msgr2=0x558addda27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 0x558addda27d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4be00099a0 tx=0x7f4be00337b0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1922681730 shutdown_connections 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558adde1f5d0 0x558adde1e2e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558addda4730 0x558addda27d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bc77fe640 0 client.15159 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558addd9f5c0 0x558adddb0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bf15ba740 1 --2- 172.21.15.77:0/1922681730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558addd9ebd0 0x558addda2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1922681730 >> 172.21.15.77:0/1922681730 conn(0x558addd9a140 msgr2=0x558addda8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1922681730 shutdown_connections 2024-03-16T10:16:32.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bf15ba740 1 -- 172.21.15.77:0/1922681730 wait complete. 2024-03-16T10:16:32.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.374+0000 7f4bf15ba740 20 client.15159 trim_cache size 0 max 16384 2024-03-16T10:16:32.381 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:16:32.381 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:16:32.381 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-16T10:16:32.381 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24482, 2024-03-16T10:16:32.381 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T10:16:32.381 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T10:16:32.381 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T10:16:32.381 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 89.365256074000001, 2024-03-16T10:16:32.382 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 26, 2024-03-16T10:16:32.382 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 87, 2024-03-16T10:16:32.382 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T10:16:32.382 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 96.338811231999998 2024-03-16T10:16:32.382 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:16:32.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.375+0000 7f4bee8f1640 1 -- 172.21.15.77:0/1645735805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4bd00a2800 msgr2=0x7f4bd00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.375+0000 7f4bee8f1640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4bd00a2800 0x7f4bd00a4cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc0095f0 tx=0x7f4bdc005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:32.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.375+0000 7f4bee8f1640 1 -- 172.21.15.77:0/1645735805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be81022c0 msgr2=0x7f4be8197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:32.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.375+0000 7f4bee8f1640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be81022c0 0x7f4be8197b30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800e9c0 tx=0x7f4bd800ee80 comp rx=0 tx=0).stop 2024-03-16T10:16:32.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.375+0000 7f4bee8f1640 1 -- 172.21.15.77:0/1645735805 shutdown_connections 2024-03-16T10:16:32.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.375+0000 7f4bee8f1640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4bd00a2800 0x7f4bd00a4cc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.375+0000 7f4bee8f1640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4be8107520 0x7f4be819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.375+0000 7f4bee8f1640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4be8104bf0 0x7f4be8198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.375+0000 7f4bee8f1640 1 --2- 172.21.15.77:0/1645735805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4be81022c0 0x7f4be8197b30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:32.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.375+0000 7f4bee8f1640 1 -- 172.21.15.77:0/1645735805 >> 172.21.15.77:0/1645735805 conn(0x7f4be80fbc70 msgr2=0x7f4be8102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:32.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.375+0000 7f4bee8f1640 1 -- 172.21.15.77:0/1645735805 shutdown_connections 2024-03-16T10:16:32.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:32.375+0000 7f4bee8f1640 1 -- 172.21.15.77:0/1645735805 wait complete. 2024-03-16T10:16:32.388 INFO:tasks.cephadm:Setup complete, yielding 2024-03-16T10:16:32.388 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T10:16:32.399 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T10:16:32.399 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-16T10:16:32.643 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:33.989 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:33 smithi077 ceph-mon[32400]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:16:34.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:33 smithi192 ceph-mon[36354]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:16:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:33 smithi193 ceph-mon[36311]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:16:34.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 -- 172.21.15.77:0/3595836477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6a81035c0 msgr2=0x7fc6a8103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:34.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 --2- 172.21.15.77:0/3595836477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6a81035c0 0x7fc6a8103a20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc698009990 tx=0x7fc698009ca0 comp rx=0 tx=0).stop 2024-03-16T10:16:34.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 -- 172.21.15.77:0/3595836477 shutdown_connections 2024-03-16T10:16:34.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 --2- 172.21.15.77:0/3595836477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6a8103f60 0x7fc6a810a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:34.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 --2- 172.21.15.77:0/3595836477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6a81005e0 0x7fc6a8100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 --2- 172.21.15.77:0/3595836477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6a81035c0 0x7fc6a8103a20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 -- 172.21.15.77:0/3595836477 >> 172.21.15.77:0/3595836477 conn(0x7fc6a80fbf60 msgr2=0x7fc6a80fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 -- 172.21.15.77:0/3595836477 shutdown_connections 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 -- 172.21.15.77:0/3595836477 wait complete. 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 Processor -- start 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 -- start start 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6a81005e0 0x7fc6a81984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6a81035c0 0x7fc6a8198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.266+0000 7fc6af2f9700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6a8103f60 0x7fc6a819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6af2f9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a810df00 con 0x7fc6a81005e0 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6af2f9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a810dd80 con 0x7fc6a81035c0 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6af2f9700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a810e080 con 0x7fc6a8103f60 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6ad095700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6a81005e0 0x7fc6a81984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:34.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6ad095700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6a81005e0 0x7fc6a81984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50218/0 (socket says 172.21.15.77:50218) 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6ac894700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6a81035c0 0x7fc6a8198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6ad095700 1 -- 172.21.15.77:0/661052393 learned_addr learned my addr 172.21.15.77:0/661052393 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6ad896700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6a8103f60 0x7fc6a819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6ad095700 1 -- 172.21.15.77:0/661052393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6a8103f60 msgr2=0x7fc6a819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6ad095700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6a8103f60 0x7fc6a819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6ad095700 1 -- 172.21.15.77:0/661052393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6a81035c0 msgr2=0x7fc6a8198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6ad095700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6a81035c0 0x7fc6a8198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6ad095700 1 -- 172.21.15.77:0/661052393 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a81a0140 con 0x7fc6a81005e0 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6ad095700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6a81005e0 0x7fc6a81984e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4007d50 tx=0x7fc6a400b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc69e7fc700 1 -- 172.21.15.77:0/661052393 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a400bca0 con 0x7fc6a81005e0 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc69e7fc700 1 -- 172.21.15.77:0/661052393 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6a400be40 con 0x7fc6a81005e0 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc69e7fc700 1 -- 172.21.15.77:0/661052393 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a401a950 con 0x7fc6a81005e0 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6af2f9700 1 -- 172.21.15.77:0/661052393 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6a8101410 con 0x7fc6a81005e0 2024-03-16T10:16:34.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.267+0000 7fc6af2f9700 1 -- 172.21.15.77:0/661052393 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6a8101950 con 0x7fc6a81005e0 2024-03-16T10:16:34.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.268+0000 7fc6af2f9700 1 -- 172.21.15.77:0/661052393 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6a8068f90 con 0x7fc6a81005e0 2024-03-16T10:16:34.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.270+0000 7fc69e7fc700 1 -- 172.21.15.77:0/661052393 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc6a4022020 con 0x7fc6a81005e0 2024-03-16T10:16:34.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.270+0000 7fc69e7fc700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc6940a2910 0x7fc6940a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:34.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.270+0000 7fc6ac894700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc6940a2910 0x7fc6940a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:34.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.270+0000 7fc69e7fc700 1 -- 172.21.15.77:0/661052393 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc6a40d2a80 con 0x7fc6a81005e0 2024-03-16T10:16:34.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.271+0000 7fc6ac894700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc6940a2910 0x7fc6940a4dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc69800b2e0 tx=0x7fc698005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:34.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.274+0000 7fc69e7fc700 1 -- 172.21.15.77:0/661052393 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc6a4016070 con 0x7fc6a81005e0 2024-03-16T10:16:34.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.497+0000 7fc6af2f9700 1 -- 172.21.15.77:0/661052393 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fc6a8102da0 con 0x7fc6940a2910 2024-03-16T10:16:34.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.498+0000 7fc69e7fc700 1 -- 172.21.15.77:0/661052393 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fc6a8102da0 con 0x7fc6940a2910 2024-03-16T10:16:34.498 INFO:teuthology.orchestra.run.smithi077.stdout:Backend: cephadm 2024-03-16T10:16:34.499 INFO:teuthology.orchestra.run.smithi077.stdout:Available: Yes 2024-03-16T10:16:34.499 INFO:teuthology.orchestra.run.smithi077.stdout:Paused: No 2024-03-16T10:16:34.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.500+0000 7fc6af2f9700 1 -- 172.21.15.77:0/661052393 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc6940a2910 msgr2=0x7fc6940a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:34.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.500+0000 7fc6af2f9700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc6940a2910 0x7fc6940a4dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc69800b2e0 tx=0x7fc698005fd0 comp rx=0 tx=0).stop 2024-03-16T10:16:34.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.500+0000 7fc6af2f9700 1 -- 172.21.15.77:0/661052393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6a81005e0 msgr2=0x7fc6a81984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:34.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.500+0000 7fc6af2f9700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6a81005e0 0x7fc6a81984e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4007d50 tx=0x7fc6a400b540 comp rx=0 tx=0).stop 2024-03-16T10:16:34.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.500+0000 7fc6af2f9700 1 -- 172.21.15.77:0/661052393 shutdown_connections 2024-03-16T10:16:34.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.500+0000 7fc6af2f9700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc6940a2910 0x7fc6940a4dd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:34.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.500+0000 7fc6af2f9700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6a81005e0 0x7fc6a81984e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:34.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.500+0000 7fc6af2f9700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6a8103f60 0x7fc6a819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:34.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.500+0000 7fc6af2f9700 1 --2- 172.21.15.77:0/661052393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6a81035c0 0x7fc6a8198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:34.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.500+0000 7fc6af2f9700 1 -- 172.21.15.77:0/661052393 >> 172.21.15.77:0/661052393 conn(0x7fc6a80fbf60 msgr2=0x7fc6a8106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:34.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.500+0000 7fc6af2f9700 1 -- 172.21.15.77:0/661052393 shutdown_connections 2024-03-16T10:16:34.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:34.500+0000 7fc6af2f9700 1 -- 172.21.15.77:0/661052393 wait complete. 2024-03-16T10:16:35.207 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-16T10:16:35.452 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:36.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:35 smithi192 ceph-mon[36354]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:36.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:35 smithi192 ceph-mon[36354]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-16T10:16:36.231 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:35 smithi077 ceph-mon[32400]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:36.232 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:35 smithi077 ceph-mon[32400]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-16T10:16:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:35 smithi193 ceph-mon[36311]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:35 smithi193 ceph-mon[36311]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-16T10:16:37.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.151+0000 7f123545d700 1 -- 172.21.15.77:0/2291042435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1230074cd0 msgr2=0x7f1230075150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:37.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.151+0000 7f123545d700 1 --2- 172.21.15.77:0/2291042435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1230074cd0 0x7f1230075150 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1224009990 tx=0x7f1224009ca0 comp rx=0 tx=0).stop 2024-03-16T10:16:37.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.151+0000 7f123545d700 1 -- 172.21.15.77:0/2291042435 shutdown_connections 2024-03-16T10:16:37.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.151+0000 7f123545d700 1 --2- 172.21.15.77:0/2291042435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f123010a220 0x7f123010c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:37.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.151+0000 7f123545d700 1 --2- 172.21.15.77:0/2291042435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12300762a0 0x7f1230074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:37.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.151+0000 7f123545d700 1 --2- 172.21.15.77:0/2291042435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1230074cd0 0x7f1230075150 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:37.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.151+0000 7f123545d700 1 -- 172.21.15.77:0/2291042435 >> 172.21.15.77:0/2291042435 conn(0x7f12300fd920 msgr2=0x7f12300ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.151+0000 7f123545d700 1 -- 172.21.15.77:0/2291042435 shutdown_connections 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.151+0000 7f123545d700 1 -- 172.21.15.77:0/2291042435 wait complete. 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.152+0000 7f123545d700 1 Processor -- start 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.152+0000 7f123545d700 1 -- start start 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.152+0000 7f123545d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1230074cd0 0x7f123019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.152+0000 7f123545d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12300762a0 0x7f123019ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.152+0000 7f123545d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f123010a220 0x7f12301a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.152+0000 7f123545d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1230110110 con 0x7f1230074cd0 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.152+0000 7f123545d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f123010ff90 con 0x7f12300762a0 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.152+0000 7f123545d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1230110290 con 0x7f123010a220 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.152+0000 7f122effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1230074cd0 0x7f123019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.152+0000 7f122effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1230074cd0 0x7f123019a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50226/0 (socket says 172.21.15.77:50226) 2024-03-16T10:16:37.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.152+0000 7f122effd700 1 -- 172.21.15.77:0/4185466946 learned_addr learned my addr 172.21.15.77:0/4185466946 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f122e7fc700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12300762a0 0x7f123019ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f122f7fe700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f123010a220 0x7f12301a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f122effd700 1 -- 172.21.15.77:0/4185466946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f123010a220 msgr2=0x7f12301a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f122effd700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f123010a220 0x7f12301a1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f122effd700 1 -- 172.21.15.77:0/4185466946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12300762a0 msgr2=0x7f123019ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f122effd700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12300762a0 0x7f123019ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f122effd700 1 -- 172.21.15.77:0/4185466946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12301a2320 con 0x7f1230074cd0 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f122effd700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1230074cd0 0x7f123019a6c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f122000e9d0 tx=0x7f122000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f1217fff700 1 -- 172.21.15.77:0/4185466946 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122000cbd0 con 0x7f1230074cd0 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f123545d700 1 -- 172.21.15.77:0/4185466946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12301a2610 con 0x7f1230074cd0 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f1217fff700 1 -- 172.21.15.77:0/4185466946 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f122000cd70 con 0x7f1230074cd0 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f1217fff700 1 -- 172.21.15.77:0/4185466946 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220018680 con 0x7f1230074cd0 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.153+0000 7f123545d700 1 -- 172.21.15.77:0/4185466946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12301a2ba0 con 0x7f1230074cd0 2024-03-16T10:16:37.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.154+0000 7f123545d700 1 -- 172.21.15.77:0/4185466946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1230068f90 con 0x7f1230074cd0 2024-03-16T10:16:37.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.155+0000 7f1217fff700 1 -- 172.21.15.77:0/4185466946 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f12200188e0 con 0x7f1230074cd0 2024-03-16T10:16:37.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.156+0000 7f1217fff700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f12180a2890 0x7f12180a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:37.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.156+0000 7f122e7fc700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f12180a2890 0x7f12180a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:37.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.156+0000 7f1217fff700 1 -- 172.21.15.77:0/4185466946 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f12200cf1c0 con 0x7f1230074cd0 2024-03-16T10:16:37.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.157+0000 7f122e7fc700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f12180a2890 0x7f12180a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f122400b2e0 tx=0x7f1224005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:37.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.160+0000 7f1217fff700 1 -- 172.21.15.77:0/4185466946 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f122009c250 con 0x7f1230074cd0 2024-03-16T10:16:37.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.372+0000 7f123545d700 1 -- 172.21.15.77:0/4185466946 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f1230062710 con 0x7f12180a2890 2024-03-16T10:16:37.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.386+0000 7f1217fff700 1 -- 172.21.15.77:0/4185466946 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f1230062710 con 0x7f12180a2890 2024-03-16T10:16:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-16T10:16:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:mds.a smithi077 running (104s) 18s ago 104s 17.7M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 2c643f6991bd 2024-03-16T10:16:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:mds.b smithi192 running (101s) 34s ago 101s 25.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b ca4655a331f9 2024-03-16T10:16:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:mds.c smithi193 running (116s) 35s ago 115s 17.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b b02aaf5c0f08 2024-03-16T10:16:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:mds.d smithi077 running (2m) 18s ago 2m 13.7M - 19.0.0-2162-g3d78f5ba be86dc8cb22b ffb63e7b892a 2024-03-16T10:16:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:mds.e smithi192 running (2m) 34s ago 2m 19.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 05840130053b 2024-03-16T10:16:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:mds.f smithi193 running (118s) 35s ago 118s 17.9M - 19.0.0-2162-g3d78f5ba be86dc8cb22b f91bda9fc43c 2024-03-16T10:16:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:mds.g smithi077 running (107s) 18s ago 107s 17.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 367a20f0727e 2024-03-16T10:16:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:mds.h smithi192 running (113s) 34s ago 112s 17.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b cf66993ac3c7 2024-03-16T10:16:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:mds.i smithi193 running (110s) 35s ago 109s 19.5M - 19.0.0-2162-g3d78f5ba be86dc8cb22b bb895664766a 2024-03-16T10:16:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:mds.j smithi077 running (2m) 18s ago 2m 15.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 08b26e4e3012 2024-03-16T10:16:37.387 INFO:teuthology.orchestra.run.smithi077.stdout:mds.k smithi192 running (2m) 34s ago 2m 16.9M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 841d624d452a 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:mds.l smithi193 running (2m) 35s ago 2m 16.9M - 19.0.0-2162-g3d78f5ba be86dc8cb22b c1f47e11a6a3 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi077 *:9283,8765 running (9m) 18s ago 9m 513M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 955d6952e8d9 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi192 *:8443,8765 running (7m) 34s ago 7m 454M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 48b8155497ba 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.z smithi193 *:8443,8765 running (7m) 35s ago 7m 451M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 2fdf6e064b0b 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (9m) 18s ago 9m 50.2M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b e223b065e454 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi192 running (7m) 34s ago 7m 42.6M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 632d7ca9fce9 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi193 running (7m) 35s ago 7m 39.7M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b aff55e0c4197 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (7m) 18s ago 7m 46.7M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 4cb67a40b041 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi192 running (6m) 34s ago 6m 57.9M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b d20a9a5694b6 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi193 running (6m) 35s ago 6m 47.0M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b b1fcc1d9d7c0 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (5m) 18s ago 5m 43.1M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 3fd1d99c2fd8 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi192 running (5m) 34s ago 5m 42.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b dcd4be9c88cc 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi193 running (5m) 35s ago 5m 54.0M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b b633f7e1d978 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi077 running (4m) 18s ago 4m 50.8M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b a6120bab28c9 2024-03-16T10:16:37.388 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi192 running (4m) 34s ago 4m 54.1M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 1300011d8b35 2024-03-16T10:16:37.389 INFO:teuthology.orchestra.run.smithi077.stdout:osd.8 smithi193 running (3m) 35s ago 3m 54.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b fcde8b849444 2024-03-16T10:16:37.389 INFO:teuthology.orchestra.run.smithi077.stdout:osd.9 smithi077 running (3m) 18s ago 3m 44.4M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b a1b7b6f74a8e 2024-03-16T10:16:37.389 INFO:teuthology.orchestra.run.smithi077.stdout:osd.10 smithi192 running (3m) 34s ago 3m 43.3M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b baa1f177535e 2024-03-16T10:16:37.389 INFO:teuthology.orchestra.run.smithi077.stdout:osd.11 smithi193 running (2m) 35s ago 2m 50.3M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b a1c1422524f7 2024-03-16T10:16:37.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.388+0000 7f123545d700 1 -- 172.21.15.77:0/4185466946 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f12180a2890 msgr2=0x7f12180a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:37.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.388+0000 7f123545d700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f12180a2890 0x7f12180a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f122400b2e0 tx=0x7f1224005fd0 comp rx=0 tx=0).stop 2024-03-16T10:16:37.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.388+0000 7f123545d700 1 -- 172.21.15.77:0/4185466946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1230074cd0 msgr2=0x7f123019a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:37.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.388+0000 7f123545d700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1230074cd0 0x7f123019a6c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f122000e9d0 tx=0x7f122000ece0 comp rx=0 tx=0).stop 2024-03-16T10:16:37.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.388+0000 7f123545d700 1 -- 172.21.15.77:0/4185466946 shutdown_connections 2024-03-16T10:16:37.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.388+0000 7f123545d700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f12180a2890 0x7f12180a4d50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:37.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.388+0000 7f123545d700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1230074cd0 0x7f123019a6c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:37.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.388+0000 7f123545d700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f123010a220 0x7f12301a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:37.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.388+0000 7f123545d700 1 --2- 172.21.15.77:0/4185466946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12300762a0 0x7f123019ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:37.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.388+0000 7f123545d700 1 -- 172.21.15.77:0/4185466946 >> 172.21.15.77:0/4185466946 conn(0x7f12300fd920 msgr2=0x7f1230106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:37.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.388+0000 7f123545d700 1 -- 172.21.15.77:0/4185466946 shutdown_connections 2024-03-16T10:16:37.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:37.388+0000 7f123545d700 1 -- 172.21.15.77:0/4185466946 wait complete. 2024-03-16T10:16:38.165 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:37 smithi077 ceph-mon[32400]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-16T10:16:38.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:37 smithi192 ceph-mon[36354]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-16T10:16:38.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-16T10:16:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:37 smithi193 ceph-mon[36311]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-16T10:16:38.492 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:39.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:38 smithi192 ceph-mon[36354]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:39.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:38 smithi077 ceph-mon[32400]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:39.271 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:38 smithi193 ceph-mon[36311]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:40.124 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:39 smithi077 ceph-mon[32400]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-16T10:16:40.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:39 smithi192 ceph-mon[36354]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-16T10:16:40.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:39 smithi193 ceph-mon[36311]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-16T10:16:40.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.436+0000 7f8f71eae700 1 -- 172.21.15.77:0/3255219140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f6c1005c0 msgr2=0x7f8f6c1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:40.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.436+0000 7f8f71eae700 1 --2- 172.21.15.77:0/3255219140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f6c1005c0 0x7f8f6c1009e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009990 tx=0x7f8f5c009ca0 comp rx=0 tx=0).stop 2024-03-16T10:16:40.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.436+0000 7f8f71eae700 1 -- 172.21.15.77:0/3255219140 shutdown_connections 2024-03-16T10:16:40.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.436+0000 7f8f71eae700 1 --2- 172.21.15.77:0/3255219140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f6c1005c0 0x7f8f6c1009e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:40.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.436+0000 7f8f71eae700 1 --2- 172.21.15.77:0/3255219140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f6c103f40 0x7f8f6c10a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:40.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.436+0000 7f8f71eae700 1 --2- 172.21.15.77:0/3255219140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f6c1035a0 0x7f8f6c103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:40.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.436+0000 7f8f71eae700 1 -- 172.21.15.77:0/3255219140 >> 172.21.15.77:0/3255219140 conn(0x7f8f6c0fbf40 msgr2=0x7f8f6c0fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:40.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.436+0000 7f8f71eae700 1 -- 172.21.15.77:0/3255219140 shutdown_connections 2024-03-16T10:16:40.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.436+0000 7f8f71eae700 1 -- 172.21.15.77:0/3255219140 wait complete. 2024-03-16T10:16:40.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.437+0000 7f8f71eae700 1 Processor -- start 2024-03-16T10:16:40.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.437+0000 7f8f71eae700 1 -- start start 2024-03-16T10:16:40.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.437+0000 7f8f71eae700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f6c1005c0 0x7f8f6c102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.437+0000 7f8f71eae700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f6c1035a0 0x7f8f6c101550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.437+0000 7f8f71eae700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f6c103f40 0x7f8f6c101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.437+0000 7f8f71eae700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6c10ded0 con 0x7f8f6c103f40 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.437+0000 7f8f71eae700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6c10dd50 con 0x7f8f6c1035a0 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.437+0000 7f8f71eae700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6c10e050 con 0x7f8f6c1005c0 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f6bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f6c103f40 0x7f8f6c101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f6bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f6c103f40 0x7f8f6c101a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50250/0 (socket says 172.21.15.77:50250) 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f6bfff700 1 -- 172.21.15.77:0/3829995061 learned_addr learned my addr 172.21.15.77:0/3829995061 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f6affd700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f6c1035a0 0x7f8f6c101550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f6b7fe700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f6c1005c0 0x7f8f6c102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f6bfff700 1 -- 172.21.15.77:0/3829995061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f6c1005c0 msgr2=0x7f8f6c102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f6bfff700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f6c1005c0 0x7f8f6c102f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f6bfff700 1 -- 172.21.15.77:0/3829995061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f6c1035a0 msgr2=0x7f8f6c101550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f6bfff700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f6c1035a0 0x7f8f6c101550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f6bfff700 1 -- 172.21.15.77:0/3829995061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6c102340 con 0x7f8f6c103f40 2024-03-16T10:16:40.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f6affd700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f6c1035a0 0x7f8f6c101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:40.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f6bfff700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f6c103f40 0x7f8f6c101a90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400ef30 tx=0x7f8f6400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:40.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f68ff9700 1 -- 172.21.15.77:0/3829995061 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f640092e0 con 0x7f8f6c103f40 2024-03-16T10:16:40.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f68ff9700 1 -- 172.21.15.77:0/3829995061 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f64004bb0 con 0x7f8f6c103f40 2024-03-16T10:16:40.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f68ff9700 1 -- 172.21.15.77:0/3829995061 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f64004280 con 0x7f8f6c103f40 2024-03-16T10:16:40.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f71eae700 1 -- 172.21.15.77:0/3829995061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f6c1a2670 con 0x7f8f6c103f40 2024-03-16T10:16:40.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.438+0000 7f8f71eae700 1 -- 172.21.15.77:0/3829995061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f6c1a2bb0 con 0x7f8f6c103f40 2024-03-16T10:16:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.439+0000 7f8f71eae700 1 -- 172.21.15.77:0/3829995061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f6c068f90 con 0x7f8f6c103f40 2024-03-16T10:16:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.440+0000 7f8f68ff9700 1 -- 172.21.15.77:0/3829995061 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8f6401b030 con 0x7f8f6c103f40 2024-03-16T10:16:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.441+0000 7f8f68ff9700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8f540a2910 0x7f8f540a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.441+0000 7f8f6b7fe700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8f540a2910 0x7f8f540a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.441+0000 7f8f68ff9700 1 -- 172.21.15.77:0/3829995061 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8f640cee90 con 0x7f8f6c103f40 2024-03-16T10:16:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.442+0000 7f8f6b7fe700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8f540a2910 0x7f8f540a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00b7b0 tx=0x7f8f5c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.445+0000 7f8f68ff9700 1 -- 172.21.15.77:0/3829995061 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f64022020 con 0x7f8f6c103f40 2024-03-16T10:16:40.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.654+0000 7f8f71eae700 1 -- 172.21.15.77:0/3829995061 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f8f6c062710 con 0x7f8f540a2910 2024-03-16T10:16:40.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.661+0000 7f8f68ff9700 1 -- 172.21.15.77:0/3829995061 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f8f6c062710 con 0x7f8f540a2910 2024-03-16T10:16:40.662 INFO:teuthology.orchestra.run.smithi077.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-16T10:16:40.662 INFO:teuthology.orchestra.run.smithi077.stdout:mds.all 12/12 38s ago 2m smithi077=a;smithi077=d;smithi077=g;smithi077=j;smithi192=b;smithi192=e;smithi192=h;smithi192=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-03-16T10:16:40.662 INFO:teuthology.orchestra.run.smithi077.stdout:mgr 3/3 38s ago 7m smithi077=x;smithi192=y;smithi193=z;count:3 2024-03-16T10:16:40.662 INFO:teuthology.orchestra.run.smithi077.stdout:mon 3/3 38s ago 8m smithi077:172.21.15.77=a;smithi192:172.21.15.192=b;smithi193:172.21.15.193=c;count:3 2024-03-16T10:16:40.662 INFO:teuthology.orchestra.run.smithi077.stdout:osd 12 38s ago - 2024-03-16T10:16:40.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.663+0000 7f8f71eae700 1 -- 172.21.15.77:0/3829995061 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8f540a2910 msgr2=0x7f8f540a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:40.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.663+0000 7f8f71eae700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8f540a2910 0x7f8f540a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00b7b0 tx=0x7f8f5c005fd0 comp rx=0 tx=0).stop 2024-03-16T10:16:40.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.663+0000 7f8f71eae700 1 -- 172.21.15.77:0/3829995061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f6c103f40 msgr2=0x7f8f6c101a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:40.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.663+0000 7f8f71eae700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f6c103f40 0x7f8f6c101a90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400ef30 tx=0x7f8f6400c3a0 comp rx=0 tx=0).stop 2024-03-16T10:16:40.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.663+0000 7f8f71eae700 1 -- 172.21.15.77:0/3829995061 shutdown_connections 2024-03-16T10:16:40.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.663+0000 7f8f71eae700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8f540a2910 0x7f8f540a4dd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:40.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.663+0000 7f8f71eae700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f6c103f40 0x7f8f6c101a90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:40.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.663+0000 7f8f71eae700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f6c1005c0 0x7f8f6c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:40.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.663+0000 7f8f71eae700 1 --2- 172.21.15.77:0/3829995061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f6c1035a0 0x7f8f6c101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:40.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.663+0000 7f8f71eae700 1 -- 172.21.15.77:0/3829995061 >> 172.21.15.77:0/3829995061 conn(0x7f8f6c0fbf40 msgr2=0x7f8f6c1068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:40.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.663+0000 7f8f71eae700 1 -- 172.21.15.77:0/3829995061 shutdown_connections 2024-03-16T10:16:40.665 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:40.663+0000 7f8f71eae700 1 -- 172.21.15.77:0/3829995061 wait complete. 2024-03-16T10:16:40.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:40 smithi077 ceph-mon[32400]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:40.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:40 smithi077 ceph-mon[32400]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-16T10:16:41.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:40 smithi192 ceph-mon[36354]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:41.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:40 smithi192 ceph-mon[36354]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-16T10:16:41.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:40 smithi193 ceph-mon[36311]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:41.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:40 smithi193 ceph-mon[36311]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-16T10:16:41.523 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-16T10:16:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:43.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.386+0000 7f65ef38b700 1 -- 172.21.15.77:0/620948996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e81011d0 msgr2=0x7f65e81015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:43.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.386+0000 7f65ef38b700 1 --2- 172.21.15.77:0/620948996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e81011d0 0x7f65e81015f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f65e40099c0 tx=0x7f65e4009cd0 comp rx=0 tx=0).stop 2024-03-16T10:16:43.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.386+0000 7f65ef38b700 1 -- 172.21.15.77:0/620948996 shutdown_connections 2024-03-16T10:16:43.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.386+0000 7f65ef38b700 1 --2- 172.21.15.77:0/620948996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e81011d0 0x7f65e81015f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:43.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.386+0000 7f65ef38b700 1 --2- 172.21.15.77:0/620948996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e8102d70 0x7f65e8109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.386+0000 7f65ef38b700 1 --2- 172.21.15.77:0/620948996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65e81023d0 0x7f65e8102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.386+0000 7f65ef38b700 1 -- 172.21.15.77:0/620948996 >> 172.21.15.77:0/620948996 conn(0x7f65e80fc770 msgr2=0x7f65e80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.386+0000 7f65ef38b700 1 -- 172.21.15.77:0/620948996 shutdown_connections 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.386+0000 7f65ef38b700 1 -- 172.21.15.77:0/620948996 wait complete. 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ef38b700 1 Processor -- start 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ef38b700 1 -- start start 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ef38b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e81011d0 0x7f65e806ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ef38b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e81023d0 0x7f65e819d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ef38b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65e8102d70 0x7f65e819f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ef38b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f65e810cdc0 con 0x7f65e81023d0 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ef38b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f65e810cc40 con 0x7f65e8102d70 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ef38b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f65e810cf40 con 0x7f65e81011d0 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ec926700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e81023d0 0x7f65e819d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:43.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ed127700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e81011d0 0x7f65e806ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ec926700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e81023d0 0x7f65e819d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35872/0 (socket says 172.21.15.77:35872) 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ed127700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e81011d0 0x7f65e806ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:48302/0 (socket says 172.21.15.77:48302) 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ec926700 1 -- 172.21.15.77:0/1456105061 learned_addr learned my addr 172.21.15.77:0/1456105061 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ed928700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65e8102d70 0x7f65e819f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.387+0000 7f65ed127700 1 -- 172.21.15.77:0/1456105061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65e8102d70 msgr2=0x7f65e819f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.388+0000 7f65ed127700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65e8102d70 0x7f65e819f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.388+0000 7f65ed127700 1 -- 172.21.15.77:0/1456105061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e81023d0 msgr2=0x7f65e819d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.388+0000 7f65ed127700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e81023d0 0x7f65e819d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.388+0000 7f65ed127700 1 -- 172.21.15.77:0/1456105061 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e81a01b0 con 0x7f65e81011d0 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.388+0000 7f65ec926700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e81023d0 0x7f65e819d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.388+0000 7f65ed928700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65e8102d70 0x7f65e819f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.388+0000 7f65ed127700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e81011d0 0x7f65e806ab40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f65e400b580 tx=0x7f65e400f710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.388+0000 7f65de7fc700 1 -- 172.21.15.77:0/1456105061 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4009e60 con 0x7f65e81011d0 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.388+0000 7f65ef38b700 1 -- 172.21.15.77:0/1456105061 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e81a0440 con 0x7f65e81011d0 2024-03-16T10:16:43.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.388+0000 7f65de7fc700 1 -- 172.21.15.77:0/1456105061 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e4003e40 con 0x7f65e81011d0 2024-03-16T10:16:43.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.388+0000 7f65de7fc700 1 -- 172.21.15.77:0/1456105061 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4005490 con 0x7f65e81011d0 2024-03-16T10:16:43.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.388+0000 7f65ef38b700 1 -- 172.21.15.77:0/1456105061 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e81a0920 con 0x7f65e81011d0 2024-03-16T10:16:43.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.389+0000 7f65ef38b700 1 -- 172.21.15.77:0/1456105061 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65e8068f90 con 0x7f65e81011d0 2024-03-16T10:16:43.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.390+0000 7f65de7fc700 1 -- 172.21.15.77:0/1456105061 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f65e400fab0 con 0x7f65e81011d0 2024-03-16T10:16:43.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.391+0000 7f65de7fc700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65d40a2890 0x7f65d40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:43.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.391+0000 7f65ec926700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65d40a2890 0x7f65d40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:43.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.391+0000 7f65de7fc700 1 -- 172.21.15.77:0/1456105061 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f65e40cf650 con 0x7f65e81011d0 2024-03-16T10:16:43.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.391+0000 7f65ec926700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65d40a2890 0x7f65d40a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f65d8011550 tx=0x7f65d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:43.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.394+0000 7f65de7fc700 1 -- 172.21.15.77:0/1456105061 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f65e4013070 con 0x7f65e81011d0 2024-03-16T10:16:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.602+0000 7f65ef38b700 1 -- 172.21.15.77:0/1456105061 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f65e8062710 con 0x7f65d40a2890 2024-03-16T10:16:43.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.603+0000 7f65de7fc700 1 -- 172.21.15.77:0/1456105061 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f65e8062710 con 0x7f65d40a2890 2024-03-16T10:16:43.604 INFO:teuthology.orchestra.run.smithi077.stdout:HOST ADDR LABELS STATUS 2024-03-16T10:16:43.604 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 172.21.15.77 2024-03-16T10:16:43.604 INFO:teuthology.orchestra.run.smithi077.stdout:smithi192 172.21.15.192 2024-03-16T10:16:43.604 INFO:teuthology.orchestra.run.smithi077.stdout:smithi193 172.21.15.193 2024-03-16T10:16:43.604 INFO:teuthology.orchestra.run.smithi077.stdout:3 hosts in cluster 2024-03-16T10:16:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.606+0000 7f65ef38b700 1 -- 172.21.15.77:0/1456105061 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65d40a2890 msgr2=0x7f65d40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.606+0000 7f65ef38b700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65d40a2890 0x7f65d40a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f65d8011550 tx=0x7f65d8009260 comp rx=0 tx=0).stop 2024-03-16T10:16:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.606+0000 7f65ef38b700 1 -- 172.21.15.77:0/1456105061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e81011d0 msgr2=0x7f65e806ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.606+0000 7f65ef38b700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e81011d0 0x7f65e806ab40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f65e400b580 tx=0x7f65e400f710 comp rx=0 tx=0).stop 2024-03-16T10:16:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.606+0000 7f65ef38b700 1 -- 172.21.15.77:0/1456105061 shutdown_connections 2024-03-16T10:16:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.606+0000 7f65ef38b700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65d40a2890 0x7f65d40a4d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.606+0000 7f65ef38b700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65e8102d70 0x7f65e819f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.606+0000 7f65ef38b700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e81011d0 0x7f65e806ab40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.606+0000 7f65ef38b700 1 --2- 172.21.15.77:0/1456105061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65e81023d0 0x7f65e819d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.606+0000 7f65ef38b700 1 -- 172.21.15.77:0/1456105061 >> 172.21.15.77:0/1456105061 conn(0x7f65e80fc770 msgr2=0x7f65e81057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.606+0000 7f65ef38b700 1 -- 172.21.15.77:0/1456105061 shutdown_connections 2024-03-16T10:16:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:43.606+0000 7f65ef38b700 1 -- 172.21.15.77:0/1456105061 wait complete. 2024-03-16T10:16:43.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:43 smithi077 ceph-mon[32400]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-16T10:16:44.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:43 smithi192 ceph-mon[36354]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-16T10:16:44.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:43 smithi193 ceph-mon[36311]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-16T10:16:44.338 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-16T10:16:44.582 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:44.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:44 smithi077 ceph-mon[32400]: from='client.24973 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:45.153 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:44 smithi193 ceph-mon[36311]: from='client.24973 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:45.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:44 smithi192 ceph-mon[36354]: from='client.24973 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:45.982 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:45 smithi077 ceph-mon[32400]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-16T10:16:46.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:45 smithi192 ceph-mon[36354]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-16T10:16:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.217+0000 7f0c6d491700 1 -- 172.21.15.77:0/2081031689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c68102300 msgr2=0x7f0c68102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.217+0000 7f0c6d491700 1 --2- 172.21.15.77:0/2081031689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c68102300 0x7f0c68102720 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0c580099c0 tx=0x7f0c58009cd0 comp rx=0 tx=0).stop 2024-03-16T10:16:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.217+0000 7f0c6d491700 1 -- 172.21.15.77:0/2081031689 shutdown_connections 2024-03-16T10:16:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.217+0000 7f0c6d491700 1 --2- 172.21.15.77:0/2081031689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c68102300 0x7f0c68102720 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:46.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.217+0000 7f0c6d491700 1 --2- 172.21.15.77:0/2081031689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c68103ea0 0x7f0c6810a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:46.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.217+0000 7f0c6d491700 1 --2- 172.21.15.77:0/2081031689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c68103500 0x7f0c68103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:46.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.217+0000 7f0c6d491700 1 -- 172.21.15.77:0/2081031689 >> 172.21.15.77:0/2081031689 conn(0x7f0c680fd8a0 msgr2=0x7f0c680ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:46.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.217+0000 7f0c6d491700 1 -- 172.21.15.77:0/2081031689 shutdown_connections 2024-03-16T10:16:46.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.217+0000 7f0c6d491700 1 -- 172.21.15.77:0/2081031689 wait complete. 2024-03-16T10:16:46.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.218+0000 7f0c6d491700 1 Processor -- start 2024-03-16T10:16:46.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.218+0000 7f0c6d491700 1 -- start start 2024-03-16T10:16:46.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.218+0000 7f0c6d491700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c68102300 0x7f0c681984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:46.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.218+0000 7f0c6d491700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c68103500 0x7f0c681989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:46.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.218+0000 7f0c6d491700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c68103ea0 0x7f0c6819fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:46.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.218+0000 7f0c6d491700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6810dec0 con 0x7f0c68102300 2024-03-16T10:16:46.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.218+0000 7f0c6d491700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6810dd40 con 0x7f0c68103500 2024-03-16T10:16:46.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.218+0000 7f0c6d491700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6810e040 con 0x7f0c68103ea0 2024-03-16T10:16:46.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.218+0000 7f0c66ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c68102300 0x7f0c681984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:46.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.218+0000 7f0c66ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c68102300 0x7f0c681984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35890/0 (socket says 172.21.15.77:35890) 2024-03-16T10:16:46.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.218+0000 7f0c66ffd700 1 -- 172.21.15.77:0/3482570101 learned_addr learned my addr 172.21.15.77:0/3482570101 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:46.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.218+0000 7f0c667fc700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c68103500 0x7f0c681989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c677fe700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c68103ea0 0x7f0c6819fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c667fc700 1 -- 172.21.15.77:0/3482570101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c68103ea0 msgr2=0x7f0c6819fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c667fc700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c68103ea0 0x7f0c6819fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c667fc700 1 -- 172.21.15.77:0/3482570101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c68102300 msgr2=0x7f0c681984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c667fc700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c68102300 0x7f0c681984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c667fc700 1 -- 172.21.15.77:0/3482570101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c681a0100 con 0x7f0c68103500 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c66ffd700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c68102300 0x7f0c681984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c667fc700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c68103500 0x7f0c681989e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00d750 tx=0x7f0c5c00da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c4ffff700 1 -- 172.21.15.77:0/3482570101 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c014030 con 0x7f0c68103500 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c6d491700 1 -- 172.21.15.77:0/3482570101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c681a03f0 con 0x7f0c68103500 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c4ffff700 1 -- 172.21.15.77:0/3482570101 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c5c004500 con 0x7f0c68103500 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c4ffff700 1 -- 172.21.15.77:0/3482570101 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c005480 con 0x7f0c68103500 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.219+0000 7f0c6d491700 1 -- 172.21.15.77:0/3482570101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c681a0930 con 0x7f0c68103500 2024-03-16T10:16:46.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.220+0000 7f0c6d491700 1 -- 172.21.15.77:0/3482570101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c68068f90 con 0x7f0c68103500 2024-03-16T10:16:46.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.221+0000 7f0c4ffff700 1 -- 172.21.15.77:0/3482570101 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0c5c004750 con 0x7f0c68103500 2024-03-16T10:16:46.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.222+0000 7f0c4ffff700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0c500a2890 0x7f0c500a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:46.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.222+0000 7f0c66ffd700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0c500a2890 0x7f0c500a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:46.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.222+0000 7f0c4ffff700 1 -- 172.21.15.77:0/3482570101 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f0c5c0cea00 con 0x7f0c68103500 2024-03-16T10:16:46.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.222+0000 7f0c66ffd700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0c500a2890 0x7f0c500a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0c58000c00 tx=0x7f0c58005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:46.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.230+0000 7f0c4ffff700 1 -- 172.21.15.77:0/3482570101 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0c5c09ba90 con 0x7f0c68103500 2024-03-16T10:16:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:45 smithi193 ceph-mon[36311]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-16T10:16:46.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.442+0000 7f0c6d491700 1 -- 172.21.15.77:0/3482570101 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f0c680627d0 con 0x7f0c500a2890 2024-03-16T10:16:46.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.443+0000 7f0c4ffff700 1 -- 172.21.15.77:0/3482570101 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f0c680627d0 con 0x7f0c500a2890 2024-03-16T10:16:46.444 INFO:teuthology.orchestra.run.smithi077.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-16T10:16:46.444 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-16T10:16:46.444 INFO:teuthology.orchestra.run.smithi077.stdout:smithi192 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500NN400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-16T10:16:46.444 INFO:teuthology.orchestra.run.smithi077.stdout:smithi193 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500HB400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-16T10:16:46.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.445+0000 7f0c6d491700 1 -- 172.21.15.77:0/3482570101 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0c500a2890 msgr2=0x7f0c500a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:46.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.445+0000 7f0c6d491700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0c500a2890 0x7f0c500a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0c58000c00 tx=0x7f0c58005fb0 comp rx=0 tx=0).stop 2024-03-16T10:16:46.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.445+0000 7f0c6d491700 1 -- 172.21.15.77:0/3482570101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c68103500 msgr2=0x7f0c681989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:46.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.445+0000 7f0c6d491700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c68103500 0x7f0c681989e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00d750 tx=0x7f0c5c00da60 comp rx=0 tx=0).stop 2024-03-16T10:16:46.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.446+0000 7f0c6d491700 1 -- 172.21.15.77:0/3482570101 shutdown_connections 2024-03-16T10:16:46.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.446+0000 7f0c6d491700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0c500a2890 0x7f0c500a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:46.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.446+0000 7f0c6d491700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c68102300 0x7f0c681984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:46.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.446+0000 7f0c6d491700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c68103ea0 0x7f0c6819fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:46.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.446+0000 7f0c6d491700 1 --2- 172.21.15.77:0/3482570101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c68103500 0x7f0c681989e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:46.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.446+0000 7f0c6d491700 1 -- 172.21.15.77:0/3482570101 >> 172.21.15.77:0/3482570101 conn(0x7f0c680fd8a0 msgr2=0x7f0c68106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:46.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.446+0000 7f0c6d491700 1 -- 172.21.15.77:0/3482570101 shutdown_connections 2024-03-16T10:16:46.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:46.446+0000 7f0c6d491700 1 -- 172.21.15.77:0/3482570101 wait complete. 2024-03-16T10:16:47.212 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T10:16:47.221 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T10:16:47.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-16T10:16:47.465 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:47.898 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:47 smithi077 ceph-mon[32400]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:47.898 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:47 smithi077 ceph-mon[32400]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:48.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:47 smithi192 ceph-mon[36354]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:48.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:47 smithi192 ceph-mon[36354]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:47 smithi193 ceph-mon[36311]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:16:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:47 smithi193 ceph-mon[36311]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.124+0000 7f45e5724700 1 -- 172.21.15.77:0/1199592133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45e01035a0 msgr2=0x7f45e0103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.124+0000 7f45e5724700 1 --2- 172.21.15.77:0/1199592133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45e01035a0 0x7f45e0103a00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009990 tx=0x7f45c8009ca0 comp rx=0 tx=0).stop 2024-03-16T10:16:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.124+0000 7f45e5724700 1 -- 172.21.15.77:0/1199592133 shutdown_connections 2024-03-16T10:16:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.124+0000 7f45e5724700 1 --2- 172.21.15.77:0/1199592133 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45e0103f40 0x7f45e010a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.124+0000 7f45e5724700 1 --2- 172.21.15.77:0/1199592133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45e01005c0 0x7f45e01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.124+0000 7f45e5724700 1 --2- 172.21.15.77:0/1199592133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45e01035a0 0x7f45e0103a00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.124+0000 7f45e5724700 1 -- 172.21.15.77:0/1199592133 >> 172.21.15.77:0/1199592133 conn(0x7f45e00fbf40 msgr2=0x7f45e00fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.124+0000 7f45e5724700 1 -- 172.21.15.77:0/1199592133 shutdown_connections 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.124+0000 7f45e5724700 1 -- 172.21.15.77:0/1199592133 wait complete. 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45e5724700 1 Processor -- start 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45e5724700 1 -- start start 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45e5724700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45e01005c0 0x7f45e0198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45e5724700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45e01035a0 0x7f45e01989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45e5724700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45e0103f40 0x7f45e019fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45e5724700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f45e010ded0 con 0x7f45e0103f40 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45e5724700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f45e010dd50 con 0x7f45e01035a0 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45e5724700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f45e010e050 con 0x7f45e01005c0 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45df7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45e0103f40 0x7f45e019fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45deffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45e01005c0 0x7f45e0198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45df7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45e0103f40 0x7f45e019fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35908/0 (socket says 172.21.15.77:35908) 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45deffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45e01005c0 0x7f45e0198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:48348/0 (socket says 172.21.15.77:48348) 2024-03-16T10:16:49.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45df7fe700 1 -- 172.21.15.77:0/1865851687 learned_addr learned my addr 172.21.15.77:0/1865851687 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.125+0000 7f45d7fff700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45e01035a0 0x7f45e01989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.126+0000 7f45df7fe700 1 -- 172.21.15.77:0/1865851687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45e01005c0 msgr2=0x7f45e0198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.126+0000 7f45df7fe700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45e01005c0 0x7f45e0198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.126+0000 7f45df7fe700 1 -- 172.21.15.77:0/1865851687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45e01035a0 msgr2=0x7f45e01989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.126+0000 7f45df7fe700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45e01035a0 0x7f45e01989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.126+0000 7f45df7fe700 1 -- 172.21.15.77:0/1865851687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e01a00f0 con 0x7f45e0103f40 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.126+0000 7f45d7fff700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45e01035a0 0x7f45e01989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.126+0000 7f45df7fe700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45e0103f40 0x7f45e019fa50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f45d800ef60 tx=0x7f45d800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.126+0000 7f45dcff9700 1 -- 172.21.15.77:0/1865851687 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d80092e0 con 0x7f45e0103f40 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.126+0000 7f45dcff9700 1 -- 172.21.15.77:0/1865851687 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45d8004bb0 con 0x7f45e0103f40 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.126+0000 7f45e5724700 1 -- 172.21.15.77:0/1865851687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e01a0380 con 0x7f45e0103f40 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.126+0000 7f45dcff9700 1 -- 172.21.15.77:0/1865851687 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8004280 con 0x7f45e0103f40 2024-03-16T10:16:49.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.126+0000 7f45e5724700 1 -- 172.21.15.77:0/1865851687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e01a0910 con 0x7f45e0103f40 2024-03-16T10:16:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.127+0000 7f45e5724700 1 -- 172.21.15.77:0/1865851687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45e0068f90 con 0x7f45e0103f40 2024-03-16T10:16:49.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.129+0000 7f45dcff9700 1 -- 172.21.15.77:0/1865851687 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f45d801b030 con 0x7f45e0103f40 2024-03-16T10:16:49.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.129+0000 7f45dcff9700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f45cc0a2910 0x7f45cc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:49.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.130+0000 7f45deffd700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f45cc0a2910 0x7f45cc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:49.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.130+0000 7f45dcff9700 1 -- 172.21.15.77:0/1865851687 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f45d80ced30 con 0x7f45e0103f40 2024-03-16T10:16:49.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.130+0000 7f45deffd700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f45cc0a2910 0x7f45cc0a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f45e0069d80 tx=0x7f45d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:49.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.134+0000 7f45dcff9700 1 -- 172.21.15.77:0/1865851687 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f45d8022020 con 0x7f45e0103f40 2024-03-16T10:16:49.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.379+0000 7f45e5724700 1 -- 172.21.15.77:0/1865851687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f45e002db70 con 0x7f45e0103f40 2024-03-16T10:16:49.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.379+0000 7f45dcff9700 1 -- 172.21.15.77:0/1865851687 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) ==== 76+0+3094 (secure 0 0 0) 0x7f45d8010040 con 0x7f45e0103f40 2024-03-16T10:16:49.380 INFO:teuthology.orchestra.run.smithi077.stdout:e26 2024-03-16T10:16:49.380 INFO:teuthology.orchestra.run.smithi077.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T10:16:49.380 INFO:teuthology.orchestra.run.smithi077.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T10:16:49.380 INFO:teuthology.orchestra.run.smithi077.stdout:legacy client fscid: 1 2024-03-16T10:16:49.380 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:49.380 INFO:teuthology.orchestra.run.smithi077.stdout:Filesystem 'cephfs' (1) 2024-03-16T10:16:49.380 INFO:teuthology.orchestra.run.smithi077.stdout:fs_name cephfs 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 26 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:created 2024-03-16T10:15:02.344673+0000 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:modified 2024-03-16T10:15:09.227516+0000 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:tableserver 0 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:root 0 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:session_timeout 300 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:session_autoclose 300 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:max_file_size 1099511627776 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:max_xattr_size 65536 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:required_client_features {} 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:last_failure 0 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:last_failure_osd_epoch 0 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:max_mds 5 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:in 0,1,2,3,4 2024-03-16T10:16:49.381 INFO:teuthology.orchestra.run.smithi077.stdout:up {0=24482,1=24475,2=24455,3=24454,4=14658} 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:failed 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:damaged 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:stopped 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:data_pools [3] 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:metadata_pool 2 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:inline_data disabled 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:balancer 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:bal_rank_mask -1 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:standby_count_wanted 0 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:qdb_cluster leader: 24482 members: 14658,24455,24482,24454,24475 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b{0:24482} state up:active seq 3 addr [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.f{0:24442} state up:standby-replay seq 1 addr [v2:172.21.15.193:6834/3768363028,v1:172.21.15.193:6835/3768363028] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.i{1:24475} state up:active seq 6 addr [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.e{1:24425} state up:standby-replay seq 1 addr [v2:172.21.15.192:6834/3288720506,v1:172.21.15.192:6835/3288720506] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T10:16:49.382 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.h{2:24455} state up:active seq 7 addr [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T10:16:49.383 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.l{2:24403} state up:standby-replay seq 1 addr [v2:172.21.15.193:6832/1430295144,v1:172.21.15.193:6833/1430295144] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T10:16:49.383 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c{3:24454} state up:active seq 8 addr [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T10:16:49.383 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.k{3:24395} state up:standby-replay seq 1 addr [v2:172.21.15.192:6832/3214007033,v1:172.21.15.192:6833/3214007033] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T10:16:49.383 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a{4:14658} state up:active seq 5 addr [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T10:16:49.383 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.g{4:14646} state up:standby-replay seq 1 addr [v2:172.21.15.77:6838/3943375297,v1:172.21.15.77:6839/3943375297] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T10:16:49.383 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:49.383 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:49.383 INFO:teuthology.orchestra.run.smithi077.stdout:Standby daemons: 2024-03-16T10:16:49.383 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:49.383 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.j{-1:14568} state up:standby seq 1 addr [v2:172.21.15.77:6834/2545278002,v1:172.21.15.77:6835/2545278002] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T10:16:49.383 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d{-1:14580} state up:standby seq 1 addr [v2:172.21.15.77:6836/1581869083,v1:172.21.15.77:6837/1581869083] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T10:16:49.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.382+0000 7f45e5724700 1 -- 172.21.15.77:0/1865851687 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f45cc0a2910 msgr2=0x7f45cc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:49.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.382+0000 7f45e5724700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f45cc0a2910 0x7f45cc0a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f45e0069d80 tx=0x7f45d0009260 comp rx=0 tx=0).stop 2024-03-16T10:16:49.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.382+0000 7f45e5724700 1 -- 172.21.15.77:0/1865851687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45e0103f40 msgr2=0x7f45e019fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:49.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.382+0000 7f45e5724700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45e0103f40 0x7f45e019fa50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f45d800ef60 tx=0x7f45d800ef90 comp rx=0 tx=0).stop 2024-03-16T10:16:49.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.382+0000 7f45e5724700 1 -- 172.21.15.77:0/1865851687 shutdown_connections 2024-03-16T10:16:49.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.382+0000 7f45e5724700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f45cc0a2910 0x7f45cc0a4dd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:49.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.382+0000 7f45e5724700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45e0103f40 0x7f45e019fa50 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:49.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.382+0000 7f45e5724700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45e01005c0 0x7f45e0198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:49.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.382+0000 7f45e5724700 1 --2- 172.21.15.77:0/1865851687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45e01035a0 0x7f45e01989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:49.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.382+0000 7f45e5724700 1 -- 172.21.15.77:0/1865851687 >> 172.21.15.77:0/1865851687 conn(0x7f45e00fbf40 msgr2=0x7f45e01068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:49.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.382+0000 7f45e5724700 1 -- 172.21.15.77:0/1865851687 shutdown_connections 2024-03-16T10:16:49.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:49.382+0000 7f45e5724700 1 -- 172.21.15.77:0/1865851687 wait complete. 2024-03-16T10:16:49.385 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-16T10:16:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:49 smithi192 ceph-mon[36354]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T10:16:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:49 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1865851687' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T10:16:50.168 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:49 smithi077 ceph-mon[32400]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T10:16:50.168 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:49 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1865851687' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T10:16:50.194 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-16T10:16:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:49 smithi193 ceph-mon[36311]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T10:16:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:49 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1865851687' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T10:16:50.456 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:51.918 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:51 smithi077 ceph-mon[32400]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:52.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:51 smithi192 ceph-mon[36354]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:52.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.173+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/4094080113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b24108030 msgr2=0x7f6b2410a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:52.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.173+0000 7f6b2c3ce700 1 --2- 172.21.15.77:0/4094080113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b24108030 0x7f6b2410a460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00b0a0 tx=0x7f6b1c00b3b0 comp rx=0 tx=0).stop 2024-03-16T10:16:52.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.173+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/4094080113 shutdown_connections 2024-03-16T10:16:52.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.173+0000 7f6b2c3ce700 1 --2- 172.21.15.77:0/4094080113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b24102dd0 0x7f6b241051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:52.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.173+0000 7f6b2c3ce700 1 --2- 172.21.15.77:0/4094080113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b24108030 0x7f6b2410a460 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:52.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.173+0000 7f6b2c3ce700 1 --2- 172.21.15.77:0/4094080113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b24105700 0x7f6b24107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:52.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.173+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/4094080113 >> 172.21.15.77:0/4094080113 conn(0x7f6b240fc7a0 msgr2=0x7f6b240febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:52.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.173+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/4094080113 shutdown_connections 2024-03-16T10:16:52.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.173+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/4094080113 wait complete. 2024-03-16T10:16:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b2c3ce700 1 Processor -- start 2024-03-16T10:16:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b2c3ce700 1 -- start start 2024-03-16T10:16:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b2c3ce700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b24102dd0 0x7f6b24198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:52.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b2c3ce700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b24105700 0x7f6b24198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b2c3ce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b24108030 0x7f6b2419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b2c3ce700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2410ded0 con 0x7f6b24102dd0 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b2c3ce700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2410dd50 con 0x7f6b24108030 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b2c3ce700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2410e050 con 0x7f6b24105700 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b2a16a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b24102dd0 0x7f6b24198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b2a16a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b24102dd0 0x7f6b24198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41576/0 (socket says 172.21.15.77:41576) 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b2a16a700 1 -- 172.21.15.77:0/2833797576 learned_addr learned my addr 172.21.15.77:0/2833797576 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b29969700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b24105700 0x7f6b24198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.174+0000 7f6b2a96b700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b24108030 0x7f6b2419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.175+0000 7f6b2a16a700 1 -- 172.21.15.77:0/2833797576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b24105700 msgr2=0x7f6b24198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.175+0000 7f6b2a16a700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b24105700 0x7f6b24198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.175+0000 7f6b2a16a700 1 -- 172.21.15.77:0/2833797576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b24108030 msgr2=0x7f6b2419fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.175+0000 7f6b2a16a700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b24108030 0x7f6b2419fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:52.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.175+0000 7f6b2a16a700 1 -- 172.21.15.77:0/2833797576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b241a01a0 con 0x7f6b24102dd0 2024-03-16T10:16:52.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.175+0000 7f6b29969700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b24105700 0x7f6b24198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:52.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.175+0000 7f6b2a16a700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b24102dd0 0x7f6b24198540 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6b2000eb60 tx=0x7f6b2000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:52.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.175+0000 7f6b1b7fe700 1 -- 172.21.15.77:0/2833797576 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2000cbd0 con 0x7f6b24102dd0 2024-03-16T10:16:52.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.175+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/2833797576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b241a0490 con 0x7f6b24102dd0 2024-03-16T10:16:52.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.175+0000 7f6b1b7fe700 1 -- 172.21.15.77:0/2833797576 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b20004500 con 0x7f6b24102dd0 2024-03-16T10:16:52.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.176+0000 7f6b1b7fe700 1 -- 172.21.15.77:0/2833797576 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b20010430 con 0x7f6b24102dd0 2024-03-16T10:16:52.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.176+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/2833797576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b241a09d0 con 0x7f6b24102dd0 2024-03-16T10:16:52.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.177+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/2833797576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b24068f90 con 0x7f6b24102dd0 2024-03-16T10:16:52.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.177+0000 7f6b1b7fe700 1 -- 172.21.15.77:0/2833797576 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6b20003680 con 0x7f6b24102dd0 2024-03-16T10:16:52.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.178+0000 7f6b1b7fe700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b100a2890 0x7f6b100a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:52.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.178+0000 7f6b29969700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b100a2890 0x7f6b100a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:52.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.178+0000 7f6b1b7fe700 1 -- 172.21.15.77:0/2833797576 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f6b200cec10 con 0x7f6b24102dd0 2024-03-16T10:16:52.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.178+0000 7f6b29969700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b100a2890 0x7f6b100a4d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6b14011490 tx=0x7f6b14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:52.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.182+0000 7f6b1b7fe700 1 -- 172.21.15.77:0/2833797576 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b2000cd70 con 0x7f6b24102dd0 2024-03-16T10:16:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:51 smithi193 ceph-mon[36311]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:52.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.386+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/2833797576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f6b2402df90 con 0x7f6b24102dd0 2024-03-16T10:16:52.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.387+0000 7f6b1b7fe700 1 -- 172.21.15.77:0/2833797576 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) ==== 56+0+5227 (secure 0 0 0) 0x7f6b20097070 con 0x7f6b24102dd0 2024-03-16T10:16:52.388 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 87 2024-03-16T10:16:52.388 INFO:teuthology.orchestra.run.smithi077.stdout:fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:16:52.388 INFO:teuthology.orchestra.run.smithi077.stdout:created 2024-03-16T10:06:58.028804+0000 2024-03-16T10:16:52.388 INFO:teuthology.orchestra.run.smithi077.stdout:modified 2024-03-16T10:15:59.826259+0000 2024-03-16T10:16:52.388 INFO:teuthology.orchestra.run.smithi077.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-16T10:16:52.388 INFO:teuthology.orchestra.run.smithi077.stdout:crush_version 26 2024-03-16T10:16:52.388 INFO:teuthology.orchestra.run.smithi077.stdout:full_ratio 0.95 2024-03-16T10:16:52.388 INFO:teuthology.orchestra.run.smithi077.stdout:backfillfull_ratio 0.9 2024-03-16T10:16:52.388 INFO:teuthology.orchestra.run.smithi077.stdout:nearfull_ratio 0.85 2024-03-16T10:16:52.388 INFO:teuthology.orchestra.run.smithi077.stdout:require_min_compat_client luminous 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:min_compat_client luminous 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:require_osd_release squid 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:stretch_mode_enabled false 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:max_osd 12 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6802/2169364613,v1:172.21.15.77:6803/2169364613] [v2:172.21.15.77:6804/2169364613,v1:172.21.15.77:6805/2169364613] exists,up b40ed61e-36f0-43e0-8f47-e80bc6001587 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6800/4276469545,v1:172.21.15.192:6801/4276469545] [v2:172.21.15.192:6802/4276469545,v1:172.21.15.192:6803/4276469545] exists,up f7d45429-9695-4b62-a44f-edc170f07838 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 up in weight 1 up_from 18 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6800/2172182947,v1:172.21.15.193:6801/2172182947] [v2:172.21.15.193:6802/2172182947,v1:172.21.15.193:6803/2172182947] exists,up e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6810/2887114423,v1:172.21.15.77:6811/2887114423] [v2:172.21.15.77:6812/2887114423,v1:172.21.15.77:6813/2887114423] exists,up d140b161-7795-4be8-bbb4-8a010ea7707e 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6808/3948604951,v1:172.21.15.192:6809/3948604951] [v2:172.21.15.192:6810/3948604951,v1:172.21.15.192:6811/3948604951] exists,up 5101c752-e69f-4cff-af41-8818a614d0ba 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 up in weight 1 up_from 38 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6808/83797054,v1:172.21.15.193:6809/83797054] [v2:172.21.15.193:6810/83797054,v1:172.21.15.193:6811/83797054] exists,up e66d1765-4379-49e9-9fa8-531107546970 2024-03-16T10:16:52.389 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 up in weight 1 up_from 46 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6818/668716736,v1:172.21.15.77:6819/668716736] [v2:172.21.15.77:6820/668716736,v1:172.21.15.77:6821/668716736] exists,up e3ff0ae8-233d-4840-8cd1-a0050cd4aa43 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 up in weight 1 up_from 51 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6816/384726243,v1:172.21.15.192:6817/384726243] [v2:172.21.15.192:6818/384726243,v1:172.21.15.192:6819/384726243] exists,up 9c422662-cf7a-478c-ae70-309d21cbb34d 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:osd.8 up in weight 1 up_from 59 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6816/3436040269,v1:172.21.15.193:6817/3436040269] [v2:172.21.15.193:6818/3436040269,v1:172.21.15.193:6819/3436040269] exists,up 09fc6b3a-3c26-42b2-a975-51c1c4617605 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:osd.9 up in weight 1 up_from 67 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6826/2903071086,v1:172.21.15.77:6827/2903071086] [v2:172.21.15.77:6828/2903071086,v1:172.21.15.77:6829/2903071086] exists,up fe5dbc58-ce7a-48ac-816f-df30c3560a15 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:osd.10 up in weight 1 up_from 72 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6824/946112377,v1:172.21.15.192:6825/946112377] [v2:172.21.15.192:6826/946112377,v1:172.21.15.192:6827/946112377] exists,up c77ee11a-014b-40fa-bb49-297f4144eec5 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6824/2760387113,v1:172.21.15.193:6825/2760387113] [v2:172.21.15.193:6826/2760387113,v1:172.21.15.193:6827/2760387113] exists,up fafc7e50-31b9-4bb9-9e17-aee4f80a2e70 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 2.8 [2,11] 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 2.a [2,11] 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 3.e [4,8,5,9] 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 3.22 [3,9] 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 3.24 [4,9] 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:pg_upmap_items 3.3a [4,6] 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:0/233716866 expires 2024-03-17T10:07:58.634847+0000 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:0/3665134846 expires 2024-03-17T10:07:58.634847+0000 2024-03-16T10:16:52.390 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:0/3504149086 expires 2024-03-17T10:07:32.924837+0000 2024-03-16T10:16:52.391 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:0/3120915723 expires 2024-03-17T10:07:32.924837+0000 2024-03-16T10:16:52.391 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:0/465332686 expires 2024-03-17T10:07:58.634847+0000 2024-03-16T10:16:52.391 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:6801/1353424963 expires 2024-03-17T10:07:58.634847+0000 2024-03-16T10:16:52.391 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:6800/1353424963 expires 2024-03-17T10:07:58.634847+0000 2024-03-16T10:16:52.391 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:6801/1093017736 expires 2024-03-17T10:07:32.924837+0000 2024-03-16T10:16:52.391 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:6800/1093017736 expires 2024-03-17T10:07:32.924837+0000 2024-03-16T10:16:52.391 INFO:teuthology.orchestra.run.smithi077.stdout:blocklist 172.21.15.77:0/1991017150 expires 2024-03-17T10:07:32.924837+0000 2024-03-16T10:16:52.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.390+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/2833797576 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b100a2890 msgr2=0x7f6b100a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:52.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.390+0000 7f6b2c3ce700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b100a2890 0x7f6b100a4d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6b14011490 tx=0x7f6b14009260 comp rx=0 tx=0).stop 2024-03-16T10:16:52.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.390+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/2833797576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b24102dd0 msgr2=0x7f6b24198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:52.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.390+0000 7f6b2c3ce700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b24102dd0 0x7f6b24198540 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6b2000eb60 tx=0x7f6b2000ee70 comp rx=0 tx=0).stop 2024-03-16T10:16:52.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.390+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/2833797576 shutdown_connections 2024-03-16T10:16:52.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.390+0000 7f6b2c3ce700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b100a2890 0x7f6b100a4d50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:52.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.390+0000 7f6b2c3ce700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b24102dd0 0x7f6b24198540 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:52.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.390+0000 7f6b2c3ce700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b24108030 0x7f6b2419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:52.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.390+0000 7f6b2c3ce700 1 --2- 172.21.15.77:0/2833797576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b24105700 0x7f6b24198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:52.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.390+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/2833797576 >> 172.21.15.77:0/2833797576 conn(0x7f6b240fc7a0 msgr2=0x7f6b241037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:52.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.390+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/2833797576 shutdown_connections 2024-03-16T10:16:52.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:52.390+0000 7f6b2c3ce700 1 -- 172.21.15.77:0/2833797576 wait complete. 2024-03-16T10:16:53.143 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:52 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2833797576' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T10:16:53.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:52 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2833797576' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T10:16:53.169 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-16T10:16:53.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:16:53.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:52 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2833797576' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T10:16:53.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.290+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1406385018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96c8100700 msgr2=0x7f96c8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.290+0000 7f96cfcf1640 1 --2- 172.21.15.77:0/1406385018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96c8100700 0x7f96c8100b20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009900 tx=0x7f96c402bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:53.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.290+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1406385018 shutdown_connections 2024-03-16T10:16:53.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.290+0000 7f96cfcf1640 1 --2- 172.21.15.77:0/1406385018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96c81022b0 0x7f96c8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.290+0000 7f96cfcf1640 1 --2- 172.21.15.77:0/1406385018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96c81018f0 0x7f96c8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.290+0000 7f96cfcf1640 1 --2- 172.21.15.77:0/1406385018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96c8100700 0x7f96c8100b20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.290+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1406385018 >> 172.21.15.77:0/1406385018 conn(0x7f96c80fbc70 msgr2=0x7f96c80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:53.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.291+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1406385018 shutdown_connections 2024-03-16T10:16:53.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.291+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1406385018 wait complete. 2024-03-16T10:16:53.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.291+0000 7f96cfcf1640 1 Processor -- start 2024-03-16T10:16:53.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.291+0000 7f96cfcf1640 1 -- start start 2024-03-16T10:16:53.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.291+0000 7f96cfcf1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96c8100700 0x7f96c810b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:53.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cfcf1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96c81018f0 0x7f96c81096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cfcf1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96c81022b0 0x7f96c8109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cfcf1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f96c810d4a0 con 0x7f96c81018f0 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cfcf1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f96c810d320 con 0x7f96c81022b0 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cfcf1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f96c810d620 con 0x7f96c8100700 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cd28c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96c81018f0 0x7f96c81096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cd28c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96c81018f0 0x7f96c81096c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41590/0 (socket says 172.21.15.77:41590) 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cd28c640 1 -- 172.21.15.77:0/1247946691 learned_addr learned my addr 172.21.15.77:0/1247946691 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cda8d640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96c8100700 0x7f96c810b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96ce28e640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96c81022b0 0x7f96c8109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cd28c640 1 -- 172.21.15.77:0/1247946691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96c8100700 msgr2=0x7f96c810b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cd28c640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96c8100700 0x7f96c810b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cd28c640 1 -- 172.21.15.77:0/1247946691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96c81022b0 msgr2=0x7f96c8109c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cd28c640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96c81022b0 0x7f96c8109c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cd28c640 1 -- 172.21.15.77:0/1247946691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c810a490 con 0x7f96c81018f0 2024-03-16T10:16:53.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cd28c640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96c81018f0 0x7f96c81096c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f96b800e9f0 tx=0x7f96b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:53.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96beffd640 1 -- 172.21.15.77:0/1247946691 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b800cd80 con 0x7f96c81018f0 2024-03-16T10:16:53.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1247946691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96c81a1c40 con 0x7f96c81018f0 2024-03-16T10:16:53.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1247946691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96c81a2190 con 0x7f96c81018f0 2024-03-16T10:16:53.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96beffd640 1 -- 172.21.15.77:0/1247946691 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96b8010910 con 0x7f96c81018f0 2024-03-16T10:16:53.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.292+0000 7f96beffd640 1 -- 172.21.15.77:0/1247946691 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b8018980 con 0x7f96c81018f0 2024-03-16T10:16:53.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.293+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1247946691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96ac0052c0 con 0x7f96c81018f0 2024-03-16T10:16:53.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.294+0000 7f96beffd640 1 -- 172.21.15.77:0/1247946691 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f96b8010ac0 con 0x7f96c81018f0 2024-03-16T10:16:53.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.295+0000 7f96beffd640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f96b40a2800 0x7f96b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:53.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.295+0000 7f96cda8d640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f96b40a2800 0x7f96b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:53.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.295+0000 7f96beffd640 1 -- 172.21.15.77:0/1247946691 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f96b80cec80 con 0x7f96c81018f0 2024-03-16T10:16:53.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.296+0000 7f96cda8d640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f96b40a2800 0x7f96b40a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f96c4000c00 tx=0x7f96c4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:53.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.299+0000 7f96beffd640 1 -- 172.21.15.77:0/1247946691 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f96b809b760 con 0x7f96c81018f0 2024-03-16T10:16:53.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.489+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1247946691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96ac005880 con 0x7f96c81018f0 2024-03-16T10:16:53.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.491+0000 7f96beffd640 1 -- 172.21.15.77:0/1247946691 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22496 (secure 0 0 0) 0x7f96b80bb070 con 0x7f96c81018f0 2024-03-16T10:16:53.492 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:53.492 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":87,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:59.826259+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:16:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.493+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1247946691 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f96b40a2800 msgr2=0x7f96b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.493+0000 7f96cfcf1640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f96b40a2800 0x7f96b40a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f96c4000c00 tx=0x7f96c4005fd0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.493+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1247946691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96c81018f0 msgr2=0x7f96c81096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.493+0000 7f96cfcf1640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96c81018f0 0x7f96c81096c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f96b800e9f0 tx=0x7f96b800eeb0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.493+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1247946691 shutdown_connections 2024-03-16T10:16:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.493+0000 7f96cfcf1640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96c81022b0 0x7f96c8109c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.493+0000 7f96cfcf1640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96c81018f0 0x7f96c81096c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.493+0000 7f96cfcf1640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f96b40a2800 0x7f96b40a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.493+0000 7f96cfcf1640 1 --2- 172.21.15.77:0/1247946691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96c8100700 0x7f96c810b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.493+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1247946691 >> 172.21.15.77:0/1247946691 conn(0x7f96c80fbc70 msgr2=0x7f96c8104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:53.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.493+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1247946691 shutdown_connections 2024-03-16T10:16:53.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.493+0000 7f96cfcf1640 1 -- 172.21.15.77:0/1247946691 wait complete. 2024-03-16T10:16:53.505 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:10:22.965004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T10:15:00.264039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T10:15:01.536067+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T10:16:53.505 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T10:16:53.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.610+0000 7f588aedc640 1 -- 172.21.15.77:0/3169061647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58841006e0 msgr2=0x7f5884100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.610+0000 7f588aedc640 1 --2- 172.21.15.77:0/3169061647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58841006e0 0x7f5884100b00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5878009900 tx=0x7f587802bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.610+0000 7f588aedc640 1 -- 172.21.15.77:0/3169061647 shutdown_connections 2024-03-16T10:16:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.610+0000 7f588aedc640 1 --2- 172.21.15.77:0/3169061647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5884102290 0x7f5884108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.610+0000 7f588aedc640 1 --2- 172.21.15.77:0/3169061647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58841018d0 0x7f5884101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.610+0000 7f588aedc640 1 --2- 172.21.15.77:0/3169061647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58841006e0 0x7f5884100b00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.610+0000 7f588aedc640 1 -- 172.21.15.77:0/3169061647 >> 172.21.15.77:0/3169061647 conn(0x7f58840fbc70 msgr2=0x7f58840fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.610+0000 7f588aedc640 1 -- 172.21.15.77:0/3169061647 shutdown_connections 2024-03-16T10:16:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.610+0000 7f588aedc640 1 -- 172.21.15.77:0/3169061647 wait complete. 2024-03-16T10:16:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f588aedc640 1 Processor -- start 2024-03-16T10:16:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f588aedc640 1 -- start start 2024-03-16T10:16:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f588aedc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58841006e0 0x7f5884196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f588aedc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58841018d0 0x7f5884196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f588aedc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5884102290 0x7f588419dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f588aedc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f588410c360 con 0x7f58841006e0 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f588aedc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f588410c1e0 con 0x7f5884102290 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f588aedc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f588410c4e0 con 0x7f58841018d0 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f5888c78640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58841006e0 0x7f5884196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f5888c78640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58841006e0 0x7f5884196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41610/0 (socket says 172.21.15.77:41610) 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f5888c78640 1 -- 172.21.15.77:0/2703430978 learned_addr learned my addr 172.21.15.77:0/2703430978 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f5883fff640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58841018d0 0x7f5884196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f5889479640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5884102290 0x7f588419dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f5888c78640 1 -- 172.21.15.77:0/2703430978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58841018d0 msgr2=0x7f5884196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f5888c78640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58841018d0 0x7f5884196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f5888c78640 1 -- 172.21.15.77:0/2703430978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5884102290 msgr2=0x7f588419dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f5888c78640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5884102290 0x7f588419dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.611+0000 7f5888c78640 1 -- 172.21.15.77:0/2703430978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f588419e640 con 0x7f58841006e0 2024-03-16T10:16:53.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.612+0000 7f5888c78640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58841006e0 0x7f5884196a20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5878000c00 tx=0x7f587802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:53.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.612+0000 7f5881ffb640 1 -- 172.21.15.77:0/2703430978 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878041070 con 0x7f58841006e0 2024-03-16T10:16:53.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.612+0000 7f588aedc640 1 -- 172.21.15.77:0/2703430978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f588419e8d0 con 0x7f58841006e0 2024-03-16T10:16:53.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.612+0000 7f5881ffb640 1 -- 172.21.15.77:0/2703430978 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5878004450 con 0x7f58841006e0 2024-03-16T10:16:53.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.612+0000 7f5881ffb640 1 -- 172.21.15.77:0/2703430978 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878034440 con 0x7f58841006e0 2024-03-16T10:16:53.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.612+0000 7f588aedc640 1 -- 172.21.15.77:0/2703430978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f588419edc0 con 0x7f58841006e0 2024-03-16T10:16:53.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.613+0000 7f588aedc640 1 -- 172.21.15.77:0/2703430978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5884107ad0 con 0x7f58841006e0 2024-03-16T10:16:53.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.614+0000 7f5881ffb640 1 -- 172.21.15.77:0/2703430978 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5878044070 con 0x7f58841006e0 2024-03-16T10:16:53.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.615+0000 7f5881ffb640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f58740a2800 0x7f58740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:53.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.615+0000 7f5883fff640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f58740a2800 0x7f58740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:53.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.615+0000 7f5881ffb640 1 -- 172.21.15.77:0/2703430978 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f58780f0490 con 0x7f58841006e0 2024-03-16T10:16:53.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.615+0000 7f5883fff640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f58740a2800 0x7f58740a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5884197f30 tx=0x7f5870005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:53.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.619+0000 7f5881ffb640 1 -- 172.21.15.77:0/2703430978 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f58780bd000 con 0x7f58841006e0 2024-03-16T10:16:53.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.800+0000 7f588aedc640 1 -- 172.21.15.77:0/2703430978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f588402d3d0 con 0x7f58841006e0 2024-03-16T10:16:53.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.801+0000 7f5881ffb640 1 -- 172.21.15.77:0/2703430978 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f58780c10a0 con 0x7f58841006e0 2024-03-16T10:16:53.802 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T10:16:53.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.803+0000 7f588aedc640 1 -- 172.21.15.77:0/2703430978 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f58740a2800 msgr2=0x7f58740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.803+0000 7f588aedc640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f58740a2800 0x7f58740a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5884197f30 tx=0x7f5870005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:53.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.803+0000 7f588aedc640 1 -- 172.21.15.77:0/2703430978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58841006e0 msgr2=0x7f5884196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.803+0000 7f588aedc640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58841006e0 0x7f5884196a20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5878000c00 tx=0x7f587802c640 comp rx=0 tx=0).stop 2024-03-16T10:16:53.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.803+0000 7f588aedc640 1 -- 172.21.15.77:0/2703430978 shutdown_connections 2024-03-16T10:16:53.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.803+0000 7f588aedc640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5884102290 0x7f588419dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.803+0000 7f588aedc640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f58740a2800 0x7f58740a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.803+0000 7f588aedc640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58841018d0 0x7f5884196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.803+0000 7f588aedc640 1 --2- 172.21.15.77:0/2703430978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58841006e0 0x7f5884196a20 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.803+0000 7f588aedc640 1 -- 172.21.15.77:0/2703430978 >> 172.21.15.77:0/2703430978 conn(0x7f58840fbc70 msgr2=0x7f5884104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.803+0000 7f588aedc640 1 -- 172.21.15.77:0/2703430978 shutdown_connections 2024-03-16T10:16:53.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.803+0000 7f588aedc640 1 -- 172.21.15.77:0/2703430978 wait complete. 2024-03-16T10:16:53.816 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T10:16:53.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.934+0000 7f4d9fd53640 1 -- 172.21.15.77:0/2014820987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d981018d0 msgr2=0x7f4d98101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.934+0000 7f4d9fd53640 1 --2- 172.21.15.77:0/2014820987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d981018d0 0x7f4d98101d50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4d88009900 tx=0x7f4d8802bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:53.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.934+0000 7f4d9fd53640 1 -- 172.21.15.77:0/2014820987 shutdown_connections 2024-03-16T10:16:53.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.934+0000 7f4d9fd53640 1 --2- 172.21.15.77:0/2014820987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d98102290 0x7f4d98108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.934+0000 7f4d9fd53640 1 --2- 172.21.15.77:0/2014820987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d981018d0 0x7f4d98101d50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.934+0000 7f4d9fd53640 1 --2- 172.21.15.77:0/2014820987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d981006e0 0x7f4d98100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.934+0000 7f4d9fd53640 1 -- 172.21.15.77:0/2014820987 >> 172.21.15.77:0/2014820987 conn(0x7f4d980fbc70 msgr2=0x7f4d980fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:53.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.934+0000 7f4d9fd53640 1 -- 172.21.15.77:0/2014820987 shutdown_connections 2024-03-16T10:16:53.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.934+0000 7f4d9fd53640 1 -- 172.21.15.77:0/2014820987 wait complete. 2024-03-16T10:16:53.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.934+0000 7f4d9fd53640 1 Processor -- start 2024-03-16T10:16:53.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.934+0000 7f4d9fd53640 1 -- start start 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.934+0000 7f4d9fd53640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d981006e0 0x7f4d98069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9fd53640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d981018d0 0x7f4d9819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9fd53640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d98102290 0x7f4d9819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9fd53640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9810c360 con 0x7f4d981006e0 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9fd53640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9810c1e0 con 0x7f4d981018d0 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9fd53640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9810c4e0 con 0x7f4d98102290 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9daef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d981006e0 0x7f4d98069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9daef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d981006e0 0x7f4d98069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41622/0 (socket says 172.21.15.77:41622) 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9daef640 1 -- 172.21.15.77:0/1103415583 learned_addr learned my addr 172.21.15.77:0/1103415583 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9d2ee640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d981018d0 0x7f4d9819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9e2f0640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d98102290 0x7f4d9819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9daef640 1 -- 172.21.15.77:0/1103415583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d98102290 msgr2=0x7f4d9819ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9daef640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d98102290 0x7f4d9819ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9daef640 1 -- 172.21.15.77:0/1103415583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d981018d0 msgr2=0x7f4d9819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:53.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9daef640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d981018d0 0x7f4d9819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:53.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9daef640 1 -- 172.21.15.77:0/1103415583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d9819f750 con 0x7f4d981006e0 2024-03-16T10:16:53.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9daef640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d981006e0 0x7f4d98069510 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4d9000e9c0 tx=0x7f4d9000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:53.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d8effd640 1 -- 172.21.15.77:0/1103415583 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9000cd80 con 0x7f4d981006e0 2024-03-16T10:16:53.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d8effd640 1 -- 172.21.15.77:0/1103415583 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d900035f0 con 0x7f4d981006e0 2024-03-16T10:16:53.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9fd53640 1 -- 172.21.15.77:0/1103415583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d9819fa40 con 0x7f4d981006e0 2024-03-16T10:16:53.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d8effd640 1 -- 172.21.15.77:0/1103415583 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d90010690 con 0x7f4d981006e0 2024-03-16T10:16:53.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.935+0000 7f4d9fd53640 1 -- 172.21.15.77:0/1103415583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d9819ff10 con 0x7f4d981006e0 2024-03-16T10:16:53.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.936+0000 7f4d9fd53640 1 -- 172.21.15.77:0/1103415583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d98107b20 con 0x7f4d981006e0 2024-03-16T10:16:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.937+0000 7f4d8effd640 1 -- 172.21.15.77:0/1103415583 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4d900037a0 con 0x7f4d981006e0 2024-03-16T10:16:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.938+0000 7f4d8effd640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d840a2800 0x7f4d840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.938+0000 7f4d9d2ee640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d840a2800 0x7f4d840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.938+0000 7f4d8effd640 1 -- 172.21.15.77:0/1103415583 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f4d900ced30 con 0x7f4d981006e0 2024-03-16T10:16:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.938+0000 7f4d9d2ee640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d840a2800 0x7f4d840a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4d8802c7b0 tx=0x7f4d88005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:53.942+0000 7f4d8effd640 1 -- 172.21.15.77:0/1103415583 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4d9009be60 con 0x7f4d981006e0 2024-03-16T10:16:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.125+0000 7f4d9fd53640 1 -- 172.21.15.77:0/1103415583 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4d9802d1b0 con 0x7f4d981006e0 2024-03-16T10:16:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.125+0000 7f4d8effd640 1 -- 172.21.15.77:0/1103415583 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f4d9009b760 con 0x7f4d981006e0 2024-03-16T10:16:54.126 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T10:16:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.127+0000 7f4d9fd53640 1 -- 172.21.15.77:0/1103415583 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d840a2800 msgr2=0x7f4d840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.127+0000 7f4d9fd53640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d840a2800 0x7f4d840a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4d8802c7b0 tx=0x7f4d88005fd0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.127+0000 7f4d9fd53640 1 -- 172.21.15.77:0/1103415583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d981006e0 msgr2=0x7f4d98069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.127+0000 7f4d9fd53640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d981006e0 0x7f4d98069510 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4d9000e9c0 tx=0x7f4d9000ee80 comp rx=0 tx=0).stop 2024-03-16T10:16:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.127+0000 7f4d9fd53640 1 -- 172.21.15.77:0/1103415583 shutdown_connections 2024-03-16T10:16:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.127+0000 7f4d9fd53640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d840a2800 0x7f4d840a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.127+0000 7f4d9fd53640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d98102290 0x7f4d9819ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.127+0000 7f4d9fd53640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d981018d0 0x7f4d9819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.127+0000 7f4d9fd53640 1 --2- 172.21.15.77:0/1103415583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d981006e0 0x7f4d98069510 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.127+0000 7f4d9fd53640 1 -- 172.21.15.77:0/1103415583 >> 172.21.15.77:0/1103415583 conn(0x7f4d980fbc70 msgr2=0x7f4d98104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.127+0000 7f4d9fd53640 1 -- 172.21.15.77:0/1103415583 shutdown_connections 2024-03-16T10:16:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.127+0000 7f4d9fd53640 1 -- 172.21.15.77:0/1103415583 wait complete. 2024-03-16T10:16:54.138 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:53 smithi077 ceph-mon[32400]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:16:54.138 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:53 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1247946691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:54.138 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:53 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2703430978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:16:54.139 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T10:16:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:53 smithi192 ceph-mon[36354]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:16:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:53 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1247946691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:53 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2703430978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:16:54.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.243+0000 7f9b582d1640 1 -- 172.21.15.77:0/687283985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b50102290 msgr2=0x7f9b50108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:54.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.243+0000 7f9b582d1640 1 --2- 172.21.15.77:0/687283985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b50102290 0x7f9b50108b20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009900 tx=0x7f9b4802bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:54.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.243+0000 7f9b582d1640 1 -- 172.21.15.77:0/687283985 shutdown_connections 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.243+0000 7f9b582d1640 1 --2- 172.21.15.77:0/687283985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b50102290 0x7f9b50108b20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.243+0000 7f9b582d1640 1 --2- 172.21.15.77:0/687283985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b501018d0 0x7f9b50101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.243+0000 7f9b582d1640 1 --2- 172.21.15.77:0/687283985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b501006e0 0x7f9b50100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.243+0000 7f9b582d1640 1 -- 172.21.15.77:0/687283985 >> 172.21.15.77:0/687283985 conn(0x7f9b500fbc70 msgr2=0x7f9b500fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.243+0000 7f9b582d1640 1 -- 172.21.15.77:0/687283985 shutdown_connections 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.243+0000 7f9b582d1640 1 -- 172.21.15.77:0/687283985 wait complete. 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b582d1640 1 Processor -- start 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b582d1640 1 -- start start 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b582d1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b501006e0 0x7f9b501969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b582d1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b501018d0 0x7f9b50196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b582d1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b50102290 0x7f9b5019df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b582d1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5010c340 con 0x7f9b501006e0 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b582d1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5010c1c0 con 0x7f9b50102290 2024-03-16T10:16:54.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b582d1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5010c4c0 con 0x7f9b501018d0 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b5606d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b501006e0 0x7f9b501969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b5606d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b501006e0 0x7f9b501969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41648/0 (socket says 172.21.15.77:41648) 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b5606d640 1 -- 172.21.15.77:0/220207697 learned_addr learned my addr 172.21.15.77:0/220207697 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b5586c640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b501018d0 0x7f9b50196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b5686e640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b50102290 0x7f9b5019df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b5606d640 1 -- 172.21.15.77:0/220207697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b501018d0 msgr2=0x7f9b50196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b5606d640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b501018d0 0x7f9b50196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b5606d640 1 -- 172.21.15.77:0/220207697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b50102290 msgr2=0x7f9b5019df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b5606d640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b50102290 0x7f9b5019df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.244+0000 7f9b5606d640 1 -- 172.21.15.77:0/220207697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5019e5d0 con 0x7f9b501006e0 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.245+0000 7f9b5606d640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b501006e0 0x7f9b501969b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00ebc0 tx=0x7f9b4c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.245+0000 7f9b477fe640 1 -- 172.21.15.77:0/220207697 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c003b70 con 0x7f9b501006e0 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.245+0000 7f9b477fe640 1 -- 172.21.15.77:0/220207697 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b4c003d10 con 0x7f9b501006e0 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.245+0000 7f9b582d1640 1 -- 172.21.15.77:0/220207697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b5019e8c0 con 0x7f9b501006e0 2024-03-16T10:16:54.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.245+0000 7f9b477fe640 1 -- 172.21.15.77:0/220207697 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c019980 con 0x7f9b501006e0 2024-03-16T10:16:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.245+0000 7f9b582d1640 1 -- 172.21.15.77:0/220207697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b5019ee10 con 0x7f9b501006e0 2024-03-16T10:16:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.246+0000 7f9b582d1640 1 -- 172.21.15.77:0/220207697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b50107a00 con 0x7f9b501006e0 2024-03-16T10:16:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.248+0000 7f9b477fe640 1 -- 172.21.15.77:0/220207697 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9b4c019b20 con 0x7f9b501006e0 2024-03-16T10:16:54.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.248+0000 7f9b477fe640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9b3c0a2800 0x7f9b3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.248+0000 7f9b5586c640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9b3c0a2800 0x7f9b3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.248+0000 7f9b477fe640 1 -- 172.21.15.77:0/220207697 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f9b4c0cf2f0 con 0x7f9b501006e0 2024-03-16T10:16:54.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.249+0000 7f9b5586c640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9b3c0a2800 0x7f9b3c0a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9b50197ec0 tx=0x7f9b40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.251+0000 7f9b477fe640 1 -- 172.21.15.77:0/220207697 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9b4c09c4b0 con 0x7f9b501006e0 2024-03-16T10:16:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:53 smithi193 ceph-mon[36311]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:16:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:53 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1247946691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:53 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2703430978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:16:54.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.432+0000 7f9b582d1640 1 -- 172.21.15.77:0/220207697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9b5002d410 con 0x7f9b501006e0 2024-03-16T10:16:54.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.432+0000 7f9b477fe640 1 -- 172.21.15.77:0/220207697 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f9b4c0a0e20 con 0x7f9b501006e0 2024-03-16T10:16:54.433 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T10:16:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.434+0000 7f9b582d1640 1 -- 172.21.15.77:0/220207697 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9b3c0a2800 msgr2=0x7f9b3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.434+0000 7f9b582d1640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9b3c0a2800 0x7f9b3c0a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9b50197ec0 tx=0x7f9b40005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.434+0000 7f9b582d1640 1 -- 172.21.15.77:0/220207697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b501006e0 msgr2=0x7f9b501969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.434+0000 7f9b582d1640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b501006e0 0x7f9b501969b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00ebc0 tx=0x7f9b4c00c6a0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.434+0000 7f9b582d1640 1 -- 172.21.15.77:0/220207697 shutdown_connections 2024-03-16T10:16:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.434+0000 7f9b582d1640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b50102290 0x7f9b5019df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.434+0000 7f9b582d1640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9b3c0a2800 0x7f9b3c0a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.434+0000 7f9b582d1640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b501018d0 0x7f9b50196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.434+0000 7f9b582d1640 1 --2- 172.21.15.77:0/220207697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b501006e0 0x7f9b501969b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.434+0000 7f9b582d1640 1 -- 172.21.15.77:0/220207697 >> 172.21.15.77:0/220207697 conn(0x7f9b500fbc70 msgr2=0x7f9b50104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.434+0000 7f9b582d1640 1 -- 172.21.15.77:0/220207697 shutdown_connections 2024-03-16T10:16:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.434+0000 7f9b582d1640 1 -- 172.21.15.77:0/220207697 wait complete. 2024-03-16T10:16:54.447 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:16:54.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.550+0000 7f1205881640 1 -- 172.21.15.77:0/817359434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12001074e0 msgr2=0x7f1200109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:54.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.550+0000 7f1205881640 1 --2- 172.21.15.77:0/817359434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12001074e0 0x7f1200109930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009ca0 tx=0x7f11f802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.550+0000 7f1205881640 1 -- 172.21.15.77:0/817359434 shutdown_connections 2024-03-16T10:16:54.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.550+0000 7f1205881640 1 --2- 172.21.15.77:0/817359434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12001074e0 0x7f1200109930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.550+0000 7f1205881640 1 --2- 172.21.15.77:0/817359434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1200104bb0 0x7f1200106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.550+0000 7f1205881640 1 --2- 172.21.15.77:0/817359434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1200102280 0x7f1200104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.550+0000 7f1205881640 1 -- 172.21.15.77:0/817359434 >> 172.21.15.77:0/817359434 conn(0x7f12000fbc70 msgr2=0x7f12000fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.550+0000 7f1205881640 1 -- 172.21.15.77:0/817359434 shutdown_connections 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.550+0000 7f1205881640 1 -- 172.21.15.77:0/817359434 wait complete. 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.550+0000 7f1205881640 1 Processor -- start 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.550+0000 7f1205881640 1 -- start start 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f1205881640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1200102280 0x7f1200101bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f1205881640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1200104bb0 0x7f1200100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f1205881640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12001074e0 0x7f1200100770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f1205881640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f120010d4a0 con 0x7f1200102280 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f1205881640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f120010d320 con 0x7f1200104bb0 2024-03-16T10:16:54.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f1205881640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f120010d620 con 0x7f12001074e0 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f11feffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1200102280 0x7f1200101bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f11feffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1200102280 0x7f1200101bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41660/0 (socket says 172.21.15.77:41660) 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f11feffd640 1 -- 172.21.15.77:0/3551617580 learned_addr learned my addr 172.21.15.77:0/3551617580 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f11f67fc640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1200104bb0 0x7f1200100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f11ff7fe640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12001074e0 0x7f1200100770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f11feffd640 1 -- 172.21.15.77:0/3551617580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12001074e0 msgr2=0x7f1200100770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f11feffd640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12001074e0 0x7f1200100770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f11feffd640 1 -- 172.21.15.77:0/3551617580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1200104bb0 msgr2=0x7f1200100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f11feffd640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1200104bb0 0x7f1200100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f11feffd640 1 -- 172.21.15.77:0/3551617580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1200101000 con 0x7f1200102280 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f11f67fc640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1200104bb0 0x7f1200100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f11feffd640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1200102280 0x7f1200101bf0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f11f000e9f0 tx=0x7f11f000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f120487f640 1 -- 172.21.15.77:0/3551617580 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f000cd80 con 0x7f1200102280 2024-03-16T10:16:54.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f120487f640 1 -- 172.21.15.77:0/3551617580 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11f0010910 con 0x7f1200102280 2024-03-16T10:16:54.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f120487f640 1 -- 172.21.15.77:0/3551617580 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f0018990 con 0x7f1200102280 2024-03-16T10:16:54.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f1205881640 1 -- 172.21.15.77:0/3551617580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12001a1be0 con 0x7f1200102280 2024-03-16T10:16:54.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.551+0000 7f1205881640 1 -- 172.21.15.77:0/3551617580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12001a2180 con 0x7f1200102280 2024-03-16T10:16:54.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.552+0000 7f1205881640 1 -- 172.21.15.77:0/3551617580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12000fd760 con 0x7f1200102280 2024-03-16T10:16:54.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.554+0000 7f120487f640 1 -- 172.21.15.77:0/3551617580 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f11f0010ac0 con 0x7f1200102280 2024-03-16T10:16:54.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.554+0000 7f120487f640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f11ec0a2880 0x7f11ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:54.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.554+0000 7f11f67fc640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f11ec0a2880 0x7f11ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:54.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.554+0000 7f120487f640 1 -- 172.21.15.77:0/3551617580 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f11f00cf040 con 0x7f1200102280 2024-03-16T10:16:54.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.555+0000 7f11f67fc640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f11ec0a2880 0x7f11ec0a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1200101960 tx=0x7f11e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:54.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.558+0000 7f120487f640 1 -- 172.21.15.77:0/3551617580 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f11f009bb20 con 0x7f1200102280 2024-03-16T10:16:54.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.778+0000 7f1205881640 1 -- 172.21.15.77:0/3551617580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f120002d370 con 0x7f1200102280 2024-03-16T10:16:54.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.778+0000 7f120487f640 1 -- 172.21.15.77:0/3551617580 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11938 (secure 0 0 0) 0x7f11f009fbc0 con 0x7f1200102280 2024-03-16T10:16:54.896 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:54.896 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:15:09.227516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[14658,24455,24482,24454,24475]},"id":1}]} 2024-03-16T10:16:54.898 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-16T10:16:54.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.780+0000 7f1205881640 1 -- 172.21.15.77:0/3551617580 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f11ec0a2880 msgr2=0x7f11ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:54.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.780+0000 7f1205881640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f11ec0a2880 0x7f11ec0a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1200101960 tx=0x7f11e8005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:54.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.780+0000 7f1205881640 1 -- 172.21.15.77:0/3551617580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1200102280 msgr2=0x7f1200101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:54.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.780+0000 7f1205881640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1200102280 0x7f1200101bf0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f11f000e9f0 tx=0x7f11f000eeb0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.780+0000 7f1205881640 1 -- 172.21.15.77:0/3551617580 shutdown_connections 2024-03-16T10:16:54.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.780+0000 7f1205881640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f11ec0a2880 0x7f11ec0a4d40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.780+0000 7f1205881640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12001074e0 0x7f1200100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.780+0000 7f1205881640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1200104bb0 0x7f1200100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.780+0000 7f1205881640 1 --2- 172.21.15.77:0/3551617580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1200102280 0x7f1200101bf0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:54.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.780+0000 7f1205881640 1 -- 172.21.15.77:0/3551617580 >> 172.21.15.77:0/3551617580 conn(0x7f12000fbc70 msgr2=0x7f1200102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:54.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.780+0000 7f1205881640 1 -- 172.21.15.77:0/3551617580 shutdown_connections 2024-03-16T10:16:54.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:54.780+0000 7f1205881640 1 -- 172.21.15.77:0/3551617580 wait complete. 2024-03-16T10:16:54.900 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:16:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.002+0000 7f77f91c6640 1 -- 172.21.15.77:0/2127261169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f4100700 msgr2=0x7f77f4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.002+0000 7f77f91c6640 1 --2- 172.21.15.77:0/2127261169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f4100700 0x7f77f4100b20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009ca0 tx=0x7f77e402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.002+0000 7f77f91c6640 1 -- 172.21.15.77:0/2127261169 shutdown_connections 2024-03-16T10:16:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.002+0000 7f77f91c6640 1 --2- 172.21.15.77:0/2127261169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f41022b0 0x7f77f4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.002+0000 7f77f91c6640 1 --2- 172.21.15.77:0/2127261169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f41018f0 0x7f77f4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.002+0000 7f77f91c6640 1 --2- 172.21.15.77:0/2127261169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f4100700 0x7f77f4100b20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.002+0000 7f77f91c6640 1 -- 172.21.15.77:0/2127261169 >> 172.21.15.77:0/2127261169 conn(0x7f77f40fbc70 msgr2=0x7f77f40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.002+0000 7f77f91c6640 1 -- 172.21.15.77:0/2127261169 shutdown_connections 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.002+0000 7f77f91c6640 1 -- 172.21.15.77:0/2127261169 wait complete. 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f91c6640 1 Processor -- start 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f91c6640 1 -- start start 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f91c6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f4100700 0x7f77f4196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f91c6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f41018f0 0x7f77f4196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f91c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f41022b0 0x7f77f419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f91c6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f77f410c370 con 0x7f77f41022b0 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f91c6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f77f410c1f0 con 0x7f77f41018f0 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f91c6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f77f410c4f0 con 0x7f77f4100700 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f41022b0 0x7f77f419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f41022b0 0x7f77f419e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41678/0 (socket says 172.21.15.77:41678) 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f359e640 1 -- 172.21.15.77:0/1516103650 learned_addr learned my addr 172.21.15.77:0/1516103650 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f2d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f4100700 0x7f77f4196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f259c640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f41018f0 0x7f77f4196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:55.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f2d9d640 1 -- 172.21.15.77:0/1516103650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f41018f0 msgr2=0x7f77f4196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f2d9d640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f41018f0 0x7f77f4196fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f2d9d640 1 -- 172.21.15.77:0/1516103650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f41022b0 msgr2=0x7f77f419e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f2d9d640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f41022b0 0x7f77f419e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f2d9d640 1 -- 172.21.15.77:0/1516103650 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f419e690 con 0x7f77f4100700 2024-03-16T10:16:55.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.003+0000 7f77f359e640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f41022b0 0x7f77f419e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:55.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.004+0000 7f77f2d9d640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f4100700 0x7f77f4196a70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77e402c630 tx=0x7f77e4032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:55.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.004+0000 7f77dbfff640 1 -- 172.21.15.77:0/1516103650 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e4041020 con 0x7f77f4100700 2024-03-16T10:16:55.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.004+0000 7f77f91c6640 1 -- 172.21.15.77:0/1516103650 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77f419e920 con 0x7f77f4100700 2024-03-16T10:16:55.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.004+0000 7f77dbfff640 1 -- 172.21.15.77:0/1516103650 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77e4034cd0 con 0x7f77f4100700 2024-03-16T10:16:55.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.004+0000 7f77dbfff640 1 -- 172.21.15.77:0/1516103650 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e4039040 con 0x7f77f4100700 2024-03-16T10:16:55.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.004+0000 7f77f91c6640 1 -- 172.21.15.77:0/1516103650 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77f419ee60 con 0x7f77f4100700 2024-03-16T10:16:55.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.005+0000 7f77f91c6640 1 -- 172.21.15.77:0/1516103650 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77f4107a90 con 0x7f77f4100700 2024-03-16T10:16:55.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.006+0000 7f77dbfff640 1 -- 172.21.15.77:0/1516103650 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f77e4044020 con 0x7f77f4100700 2024-03-16T10:16:55.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.006+0000 7f77dbfff640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f77dc0a2800 0x7f77dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:55.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.006+0000 7f77f259c640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f77dc0a2800 0x7f77dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:55.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.006+0000 7f77dbfff640 1 -- 172.21.15.77:0/1516103650 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f77e40f07d0 con 0x7f77f4100700 2024-03-16T10:16:55.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.007+0000 7f77f259c640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f77dc0a2800 0x7f77dc0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f77f4197f80 tx=0x7f77e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:55.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.011+0000 7f77dbfff640 1 -- 172.21.15.77:0/1516103650 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f77e404b020 con 0x7f77f4100700 2024-03-16T10:16:55.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:54 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1103415583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:16:55.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:54 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/220207697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:16:55.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:54 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3551617580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:55.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.196+0000 7f77f91c6640 1 -- 172.21.15.77:0/1516103650 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77f402d400 con 0x7f77f4100700 2024-03-16T10:16:55.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.197+0000 7f77dbfff640 1 -- 172.21.15.77:0/1516103650 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22496 (secure 0 0 0) 0x7f77e40bd990 con 0x7f77f4100700 2024-03-16T10:16:55.198 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:55.198 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":87,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:59.826259+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:16:55.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.199+0000 7f77f91c6640 1 -- 172.21.15.77:0/1516103650 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f77dc0a2800 msgr2=0x7f77dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.199+0000 7f77f91c6640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f77dc0a2800 0x7f77dc0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f77f4197f80 tx=0x7f77e8005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:55.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.199+0000 7f77f91c6640 1 -- 172.21.15.77:0/1516103650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f4100700 msgr2=0x7f77f4196a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.199+0000 7f77f91c6640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f4100700 0x7f77f4196a70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77e402c630 tx=0x7f77e4032f20 comp rx=0 tx=0).stop 2024-03-16T10:16:55.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.199+0000 7f77f91c6640 1 -- 172.21.15.77:0/1516103650 shutdown_connections 2024-03-16T10:16:55.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.199+0000 7f77f91c6640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f41022b0 0x7f77f419e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.199+0000 7f77f91c6640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f41018f0 0x7f77f4196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.199+0000 7f77f91c6640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f77dc0a2800 0x7f77dc0a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.199+0000 7f77f91c6640 1 --2- 172.21.15.77:0/1516103650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f4100700 0x7f77f4196a70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.199+0000 7f77f91c6640 1 -- 172.21.15.77:0/1516103650 >> 172.21.15.77:0/1516103650 conn(0x7f77f40fbc70 msgr2=0x7f77f4104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:55.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.199+0000 7f77f91c6640 1 -- 172.21.15.77:0/1516103650 shutdown_connections 2024-03-16T10:16:55.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.199+0000 7f77f91c6640 1 -- 172.21.15.77:0/1516103650 wait complete. 2024-03-16T10:16:55.212 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:54 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1103415583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:16:55.212 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:54 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/220207697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:16:55.212 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:54 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3551617580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:55.214 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:10:22.965004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T10:15:00.264039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T10:15:01.536067+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T10:16:55.214 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T10:16:55.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:54 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1103415583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:16:55.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:54 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/220207697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:16:55.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:54 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3551617580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:55.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.344+0000 7f0940bd4640 1 -- 172.21.15.77:0/167195114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f093c107500 msgr2=0x7f093c109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.344+0000 7f0940bd4640 1 --2- 172.21.15.77:0/167195114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f093c107500 0x7f093c109910 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0934009ca0 tx=0x7f093402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.344+0000 7f0940bd4640 1 -- 172.21.15.77:0/167195114 shutdown_connections 2024-03-16T10:16:55.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.344+0000 7f0940bd4640 1 --2- 172.21.15.77:0/167195114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f093c107500 0x7f093c109910 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.344+0000 7f0940bd4640 1 --2- 172.21.15.77:0/167195114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f093c104bd0 0x7f093c106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.344+0000 7f0940bd4640 1 --2- 172.21.15.77:0/167195114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f093c1022a0 0x7f093c104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.344+0000 7f0940bd4640 1 -- 172.21.15.77:0/167195114 >> 172.21.15.77:0/167195114 conn(0x7f093c0fbc70 msgr2=0x7f093c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.344+0000 7f0940bd4640 1 -- 172.21.15.77:0/167195114 shutdown_connections 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.344+0000 7f0940bd4640 1 -- 172.21.15.77:0/167195114 wait complete. 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f0940bd4640 1 Processor -- start 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f0940bd4640 1 -- start start 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f0940bd4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f093c1022a0 0x7f093c199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f0940bd4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f093c104bd0 0x7f093c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f0940bd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f093c107500 0x7f093c1a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f0940bd4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f093c10d460 con 0x7f093c107500 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f0940bd4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f093c10d2e0 con 0x7f093c1022a0 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f0940bd4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f093c10d5e0 con 0x7f093c104bd0 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f093ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f093c107500 0x7f093c1a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:55.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f093ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f093c107500 0x7f093c1a12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41700/0 (socket says 172.21.15.77:41700) 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f093ad9d640 1 -- 172.21.15.77:0/489196755 learned_addr learned my addr 172.21.15.77:0/489196755 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f0939d9b640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f093c104bd0 0x7f093c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.345+0000 7f093a59c640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f093c1022a0 0x7f093c199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.346+0000 7f093ad9d640 1 -- 172.21.15.77:0/489196755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f093c104bd0 msgr2=0x7f093c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.346+0000 7f093ad9d640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f093c104bd0 0x7f093c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.346+0000 7f093ad9d640 1 -- 172.21.15.77:0/489196755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f093c1022a0 msgr2=0x7f093c199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.346+0000 7f093ad9d640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f093c1022a0 0x7f093c199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.346+0000 7f093ad9d640 1 -- 172.21.15.77:0/489196755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f093c1a1960 con 0x7f093c107500 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.346+0000 7f093ad9d640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f093c107500 0x7f093c1a12c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0934037040 tx=0x7f0934032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.346+0000 7f092b7fe640 1 -- 172.21.15.77:0/489196755 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0934041020 con 0x7f093c107500 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.346+0000 7f0940bd4640 1 -- 172.21.15.77:0/489196755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f093c1a1bf0 con 0x7f093c107500 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.346+0000 7f092b7fe640 1 -- 172.21.15.77:0/489196755 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0934034cd0 con 0x7f093c107500 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.346+0000 7f092b7fe640 1 -- 172.21.15.77:0/489196755 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0934039040 con 0x7f093c107500 2024-03-16T10:16:55.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.346+0000 7f0940bd4640 1 -- 172.21.15.77:0/489196755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f093c1a20e0 con 0x7f093c107500 2024-03-16T10:16:55.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.347+0000 7f0940bd4640 1 -- 172.21.15.77:0/489196755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f093c108ff0 con 0x7f093c107500 2024-03-16T10:16:55.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.348+0000 7f092b7fe640 1 -- 172.21.15.77:0/489196755 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0934044020 con 0x7f093c107500 2024-03-16T10:16:55.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.349+0000 7f092b7fe640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09240a2800 0x7f09240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:55.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.349+0000 7f093a59c640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09240a2800 0x7f09240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:55.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.349+0000 7f092b7fe640 1 -- 172.21.15.77:0/489196755 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f09340f0900 con 0x7f093c107500 2024-03-16T10:16:55.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.349+0000 7f093a59c640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09240a2800 0x7f09240a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f093c19b220 tx=0x7f092c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:55.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.353+0000 7f092b7fe640 1 -- 172.21.15.77:0/489196755 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f093404b020 con 0x7f093c107500 2024-03-16T10:16:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.583+0000 7f0940bd4640 1 -- 172.21.15.77:0/489196755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f093c02d3b0 con 0x7f093c107500 2024-03-16T10:16:55.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.583+0000 7f092b7fe640 1 -- 172.21.15.77:0/489196755 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f09340bda30 con 0x7f093c107500 2024-03-16T10:16:55.584 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T10:16:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.586+0000 7f0940bd4640 1 -- 172.21.15.77:0/489196755 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09240a2800 msgr2=0x7f09240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.586+0000 7f0940bd4640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09240a2800 0x7f09240a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f093c19b220 tx=0x7f092c005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.586+0000 7f0940bd4640 1 -- 172.21.15.77:0/489196755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f093c107500 msgr2=0x7f093c1a12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.586+0000 7f0940bd4640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f093c107500 0x7f093c1a12c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0934037040 tx=0x7f0934032f40 comp rx=0 tx=0).stop 2024-03-16T10:16:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.586+0000 7f0940bd4640 1 -- 172.21.15.77:0/489196755 shutdown_connections 2024-03-16T10:16:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.586+0000 7f0940bd4640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f093c107500 0x7f093c1a12c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.586+0000 7f0940bd4640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09240a2800 0x7f09240a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.586+0000 7f0940bd4640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f093c104bd0 0x7f093c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.586+0000 7f0940bd4640 1 --2- 172.21.15.77:0/489196755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f093c1022a0 0x7f093c199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.586+0000 7f0940bd4640 1 -- 172.21.15.77:0/489196755 >> 172.21.15.77:0/489196755 conn(0x7f093c0fbc70 msgr2=0x7f093c1019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.586+0000 7f0940bd4640 1 -- 172.21.15.77:0/489196755 shutdown_connections 2024-03-16T10:16:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.586+0000 7f0940bd4640 1 -- 172.21.15.77:0/489196755 wait complete. 2024-03-16T10:16:55.599 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T10:16:55.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.720+0000 7f7057287640 1 -- 172.21.15.77:0/3650883572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70500ffe90 msgr2=0x7f70501002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.720+0000 7f7057287640 1 --2- 172.21.15.77:0/3650883572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70500ffe90 0x7f70501002b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f704800aa10 tx=0x7f704802cf70 comp rx=0 tx=0).stop 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.720+0000 7f7057287640 1 -- 172.21.15.77:0/3650883572 shutdown_connections 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.720+0000 7f7057287640 1 --2- 172.21.15.77:0/3650883572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70501019a0 0x7f70501082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.720+0000 7f7057287640 1 --2- 172.21.15.77:0/3650883572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7050100fe0 0x7f7050101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.720+0000 7f7057287640 1 --2- 172.21.15.77:0/3650883572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70500ffe90 0x7f70501002b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.720+0000 7f7057287640 1 -- 172.21.15.77:0/3650883572 >> 172.21.15.77:0/3650883572 conn(0x7f70500fb400 msgr2=0x7f70500fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.720+0000 7f7057287640 1 -- 172.21.15.77:0/3650883572 shutdown_connections 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.720+0000 7f7057287640 1 -- 172.21.15.77:0/3650883572 wait complete. 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7057287640 1 Processor -- start 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7057287640 1 -- start start 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7057287640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70500ffe90 0x7f7050195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7057287640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7050100fe0 0x7f7050195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7057287640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70501019a0 0x7f705019cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7057287640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f705010b290 con 0x7f7050100fe0 2024-03-16T10:16:55.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7057287640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f705010b110 con 0x7f70501019a0 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7057287640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f705010b410 con 0x7f70500ffe90 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7054822640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7050100fe0 0x7f7050195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7054822640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7050100fe0 0x7f7050195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41718/0 (socket says 172.21.15.77:41718) 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7054822640 1 -- 172.21.15.77:0/346948678 learned_addr learned my addr 172.21.15.77:0/346948678 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7055023640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70500ffe90 0x7f7050195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7055824640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70501019a0 0x7f705019cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7054822640 1 -- 172.21.15.77:0/346948678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70500ffe90 msgr2=0x7f7050195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7054822640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70500ffe90 0x7f7050195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7054822640 1 -- 172.21.15.77:0/346948678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70501019a0 msgr2=0x7f705019cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7054822640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70501019a0 0x7f705019cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.721+0000 7f7054822640 1 -- 172.21.15.77:0/346948678 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705019d580 con 0x7f7050100fe0 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.722+0000 7f7054822640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7050100fe0 0x7f7050195ea0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f704400e9c0 tx=0x7f704400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.722+0000 7f704e7fc640 1 -- 172.21.15.77:0/346948678 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704400cde0 con 0x7f7050100fe0 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.722+0000 7f704e7fc640 1 -- 172.21.15.77:0/346948678 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7044010910 con 0x7f7050100fe0 2024-03-16T10:16:55.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.722+0000 7f7057287640 1 -- 172.21.15.77:0/346948678 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f705019d870 con 0x7f7050100fe0 2024-03-16T10:16:55.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.722+0000 7f704e7fc640 1 -- 172.21.15.77:0/346948678 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044018a00 con 0x7f7050100fe0 2024-03-16T10:16:55.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.722+0000 7f7057287640 1 -- 172.21.15.77:0/346948678 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f705019ddc0 con 0x7f7050100fe0 2024-03-16T10:16:55.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.723+0000 7f7057287640 1 -- 172.21.15.77:0/346948678 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70501071c0 con 0x7f7050100fe0 2024-03-16T10:16:55.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.724+0000 7f704e7fc640 1 -- 172.21.15.77:0/346948678 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f704401f020 con 0x7f7050100fe0 2024-03-16T10:16:55.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.724+0000 7f704e7fc640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f70380a2800 0x7f70380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:55.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.725+0000 7f7055023640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f70380a2800 0x7f70380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:55.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.725+0000 7f704e7fc640 1 -- 172.21.15.77:0/346948678 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f70440cf180 con 0x7f7050100fe0 2024-03-16T10:16:55.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.725+0000 7f7055023640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f70380a2800 0x7f70380a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7048004d40 tx=0x7f7048005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:55.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.728+0000 7f704e7fc640 1 -- 172.21.15.77:0/346948678 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f704409c2b0 con 0x7f7050100fe0 2024-03-16T10:16:55.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.937+0000 7f7057287640 1 -- 172.21.15.77:0/346948678 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f705002d340 con 0x7f7050100fe0 2024-03-16T10:16:55.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.937+0000 7f704e7fc640 1 -- 172.21.15.77:0/346948678 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f704409bbb0 con 0x7f7050100fe0 2024-03-16T10:16:55.938 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T10:16:55.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.939+0000 7f7057287640 1 -- 172.21.15.77:0/346948678 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f70380a2800 msgr2=0x7f70380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.939+0000 7f7057287640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f70380a2800 0x7f70380a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7048004d40 tx=0x7f7048005fd0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.939+0000 7f7057287640 1 -- 172.21.15.77:0/346948678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7050100fe0 msgr2=0x7f7050195ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:55.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.939+0000 7f7057287640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7050100fe0 0x7f7050195ea0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f704400e9c0 tx=0x7f704400ee80 comp rx=0 tx=0).stop 2024-03-16T10:16:55.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.939+0000 7f7057287640 1 -- 172.21.15.77:0/346948678 shutdown_connections 2024-03-16T10:16:55.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.939+0000 7f7057287640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70501019a0 0x7f705019cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.939+0000 7f7057287640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7050100fe0 0x7f7050195ea0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.939+0000 7f7057287640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f70380a2800 0x7f70380a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.939+0000 7f7057287640 1 --2- 172.21.15.77:0/346948678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70500ffe90 0x7f7050195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:55.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.939+0000 7f7057287640 1 -- 172.21.15.77:0/346948678 >> 172.21.15.77:0/346948678 conn(0x7f70500fb400 msgr2=0x7f7050104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:55.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.939+0000 7f7057287640 1 -- 172.21.15.77:0/346948678 shutdown_connections 2024-03-16T10:16:55.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:55.939+0000 7f7057287640 1 -- 172.21.15.77:0/346948678 wait complete. 2024-03-16T10:16:55.950 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:55 smithi077 ceph-mon[32400]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T10:16:55.950 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:55 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1516103650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:55.950 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:55 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/489196755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:16:55.951 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T10:16:56.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.056+0000 7fca5eaa1640 1 -- 172.21.15.77:0/2575452512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca58104bf0 msgr2=0x7fca58106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.056+0000 7fca5eaa1640 1 --2- 172.21.15.77:0/2575452512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca58104bf0 0x7fca58106fe0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fca44009900 tx=0x7fca4402bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:56.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.056+0000 7fca5eaa1640 1 -- 172.21.15.77:0/2575452512 shutdown_connections 2024-03-16T10:16:56.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.056+0000 7fca5eaa1640 1 --2- 172.21.15.77:0/2575452512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca58107520 0x7fca58109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.056+0000 7fca5eaa1640 1 --2- 172.21.15.77:0/2575452512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca58104bf0 0x7fca58106fe0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.056+0000 7fca5eaa1640 1 --2- 172.21.15.77:0/2575452512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca581022c0 0x7fca581046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.056+0000 7fca5eaa1640 1 -- 172.21.15.77:0/2575452512 >> 172.21.15.77:0/2575452512 conn(0x7fca580fbc70 msgr2=0x7fca580fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.056+0000 7fca5eaa1640 1 -- 172.21.15.77:0/2575452512 shutdown_connections 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.056+0000 7fca5eaa1640 1 -- 172.21.15.77:0/2575452512 wait complete. 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5eaa1640 1 Processor -- start 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5eaa1640 1 -- start start 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5eaa1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca581022c0 0x7fca58197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5eaa1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca58104bf0 0x7fca58198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5eaa1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca58107520 0x7fca5819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5eaa1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fca5810d450 con 0x7fca581022c0 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5eaa1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fca5810d2d0 con 0x7fca58107520 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5eaa1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fca5810d5d0 con 0x7fca58104bf0 2024-03-16T10:16:56.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5c83d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca581022c0 0x7fca58197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5c83d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca581022c0 0x7fca58197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41738/0 (socket says 172.21.15.77:41738) 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5c83d640 1 -- 172.21.15.77:0/957198482 learned_addr learned my addr 172.21.15.77:0/957198482 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca57fff640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca58104bf0 0x7fca58198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5d03e640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca58107520 0x7fca5819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5c83d640 1 -- 172.21.15.77:0/957198482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca58104bf0 msgr2=0x7fca58198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5c83d640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca58104bf0 0x7fca58198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5c83d640 1 -- 172.21.15.77:0/957198482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca58107520 msgr2=0x7fca5819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5c83d640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca58107520 0x7fca5819f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.057+0000 7fca5c83d640 1 -- 172.21.15.77:0/957198482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca5819f770 con 0x7fca581022c0 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.058+0000 7fca5c83d640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca581022c0 0x7fca58197b50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00e9f0 tx=0x7fca4c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.058+0000 7fca55ffb640 1 -- 172.21.15.77:0/957198482 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c00cd80 con 0x7fca581022c0 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.058+0000 7fca55ffb640 1 -- 172.21.15.77:0/957198482 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca4c005500 con 0x7fca581022c0 2024-03-16T10:16:56.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.058+0000 7fca5eaa1640 1 -- 172.21.15.77:0/957198482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca5819fa60 con 0x7fca581022c0 2024-03-16T10:16:56.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.058+0000 7fca55ffb640 1 -- 172.21.15.77:0/957198482 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c010690 con 0x7fca581022c0 2024-03-16T10:16:56.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.058+0000 7fca5eaa1640 1 -- 172.21.15.77:0/957198482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca5819ff30 con 0x7fca581022c0 2024-03-16T10:16:56.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.059+0000 7fca5eaa1640 1 -- 172.21.15.77:0/957198482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca580fd760 con 0x7fca581022c0 2024-03-16T10:16:56.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.060+0000 7fca55ffb640 1 -- 172.21.15.77:0/957198482 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fca4c0046c0 con 0x7fca581022c0 2024-03-16T10:16:56.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.060+0000 7fca55ffb640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fca480a2800 0x7fca480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:56.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.060+0000 7fca57fff640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fca480a2800 0x7fca480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:56.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.060+0000 7fca55ffb640 1 -- 172.21.15.77:0/957198482 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fca4c0cedb0 con 0x7fca581022c0 2024-03-16T10:16:56.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.061+0000 7fca57fff640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fca480a2800 0x7fca480a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fca44000c00 tx=0x7fca44005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:56.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.065+0000 7fca55ffb640 1 -- 172.21.15.77:0/957198482 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fca4c09bee0 con 0x7fca581022c0 2024-03-16T10:16:56.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:55 smithi192 ceph-mon[36354]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T10:16:56.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:55 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1516103650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:56.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:55 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/489196755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:16:56.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.244+0000 7fca5eaa1640 1 -- 172.21.15.77:0/957198482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fca5802d440 con 0x7fca581022c0 2024-03-16T10:16:56.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.244+0000 7fca55ffb640 1 -- 172.21.15.77:0/957198482 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fca4c09b7e0 con 0x7fca581022c0 2024-03-16T10:16:56.245 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T10:16:56.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.246+0000 7fca5eaa1640 1 -- 172.21.15.77:0/957198482 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fca480a2800 msgr2=0x7fca480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.246+0000 7fca5eaa1640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fca480a2800 0x7fca480a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fca44000c00 tx=0x7fca44005fd0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.246+0000 7fca5eaa1640 1 -- 172.21.15.77:0/957198482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca581022c0 msgr2=0x7fca58197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.246+0000 7fca5eaa1640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca581022c0 0x7fca58197b50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00e9f0 tx=0x7fca4c00edf0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.246+0000 7fca5eaa1640 1 -- 172.21.15.77:0/957198482 shutdown_connections 2024-03-16T10:16:56.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.246+0000 7fca5eaa1640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fca58107520 0x7fca5819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.246+0000 7fca5eaa1640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fca480a2800 0x7fca480a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.246+0000 7fca5eaa1640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca58104bf0 0x7fca58198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.246+0000 7fca5eaa1640 1 --2- 172.21.15.77:0/957198482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca581022c0 0x7fca58197b50 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.246+0000 7fca5eaa1640 1 -- 172.21.15.77:0/957198482 >> 172.21.15.77:0/957198482 conn(0x7fca580fbc70 msgr2=0x7fca58102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:56.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.247+0000 7fca5eaa1640 1 -- 172.21.15.77:0/957198482 shutdown_connections 2024-03-16T10:16:56.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.247+0000 7fca5eaa1640 1 -- 172.21.15.77:0/957198482 wait complete. 2024-03-16T10:16:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:55 smithi193 ceph-mon[36311]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T10:16:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:55 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1516103650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:55 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/489196755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:16:56.258 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:16:56.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.360+0000 7fe35d027640 1 -- 172.21.15.77:0/3635932110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3581022b0 msgr2=0x7fe358108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.360+0000 7fe35d027640 1 --2- 172.21.15.77:0/3635932110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3581022b0 0x7fe358108b40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe350009ca0 tx=0x7fe35002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.360+0000 7fe35d027640 1 -- 172.21.15.77:0/3635932110 shutdown_connections 2024-03-16T10:16:56.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.360+0000 7fe35d027640 1 --2- 172.21.15.77:0/3635932110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3581022b0 0x7fe358108b40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.360+0000 7fe35d027640 1 --2- 172.21.15.77:0/3635932110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3581018f0 0x7fe358101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.360+0000 7fe35d027640 1 --2- 172.21.15.77:0/3635932110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe358100700 0x7fe358100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.360+0000 7fe35d027640 1 -- 172.21.15.77:0/3635932110 >> 172.21.15.77:0/3635932110 conn(0x7fe3580fbc70 msgr2=0x7fe3580fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.360+0000 7fe35d027640 1 -- 172.21.15.77:0/3635932110 shutdown_connections 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.360+0000 7fe35d027640 1 -- 172.21.15.77:0/3635932110 wait complete. 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe35d027640 1 Processor -- start 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe35d027640 1 -- start start 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe35d027640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe358100700 0x7fe358197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe35d027640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3581018f0 0x7fe358198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe35d027640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3581022b0 0x7fe35819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe35d027640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe35810c380 con 0x7fe358100700 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe35d027640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe35810c200 con 0x7fe3581022b0 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe35d027640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe35810c500 con 0x7fe3581018f0 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe356d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe358100700 0x7fe358197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe356d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe358100700 0x7fe358197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41756/0 (socket says 172.21.15.77:41756) 2024-03-16T10:16:56.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe356d9d640 1 -- 172.21.15.77:0/297691863 learned_addr learned my addr 172.21.15.77:0/297691863 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe34e59c640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3581018f0 0x7fe358198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe35759e640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3581022b0 0x7fe35819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe356d9d640 1 -- 172.21.15.77:0/297691863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3581018f0 msgr2=0x7fe358198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe356d9d640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3581018f0 0x7fe358198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe356d9d640 1 -- 172.21.15.77:0/297691863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3581022b0 msgr2=0x7fe35819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe356d9d640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3581022b0 0x7fe35819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.361+0000 7fe356d9d640 1 -- 172.21.15.77:0/297691863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35819f700 con 0x7fe358100700 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.362+0000 7fe356d9d640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe358100700 0x7fe358197ae0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe34800e9c0 tx=0x7fe34800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.362+0000 7fe354d99640 1 -- 172.21.15.77:0/297691863 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34800cd80 con 0x7fe358100700 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.362+0000 7fe35d027640 1 -- 172.21.15.77:0/297691863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe35819f9f0 con 0x7fe358100700 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.362+0000 7fe354d99640 1 -- 172.21.15.77:0/297691863 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3480035f0 con 0x7fe358100700 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.362+0000 7fe354d99640 1 -- 172.21.15.77:0/297691863 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348010690 con 0x7fe358100700 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.362+0000 7fe35d027640 1 -- 172.21.15.77:0/297691863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe35819ff40 con 0x7fe358100700 2024-03-16T10:16:56.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.363+0000 7fe35d027640 1 -- 172.21.15.77:0/297691863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3581a0170 con 0x7fe358100700 2024-03-16T10:16:56.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.364+0000 7fe354d99640 1 -- 172.21.15.77:0/297691863 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe3480037a0 con 0x7fe358100700 2024-03-16T10:16:56.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.365+0000 7fe354d99640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3440a2800 0x7fe3440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:56.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.365+0000 7fe34e59c640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3440a2800 0x7fe3440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:56.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.365+0000 7fe354d99640 1 -- 172.21.15.77:0/297691863 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe3480cee00 con 0x7fe358100700 2024-03-16T10:16:56.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.365+0000 7fe34e59c640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3440a2800 0x7fe3440a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe358198ff0 tx=0x7fe34000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:56.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.368+0000 7fe354d99640 1 -- 172.21.15.77:0/297691863 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe34809bf30 con 0x7fe358100700 2024-03-16T10:16:56.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.605+0000 7fe35d027640 1 -- 172.21.15.77:0/297691863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3581a0340 con 0x7fe358100700 2024-03-16T10:16:56.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.606+0000 7fe354d99640 1 -- 172.21.15.77:0/297691863 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11938 (secure 0 0 0) 0x7fe34809b830 con 0x7fe358100700 2024-03-16T10:16:56.610 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:56.610 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:15:09.227516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[14658,24455,24482,24454,24475]},"id":1}]} 2024-03-16T10:16:56.611 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2024-03-16T10:16:56.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.608+0000 7fe35d027640 1 -- 172.21.15.77:0/297691863 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3440a2800 msgr2=0x7fe3440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.608+0000 7fe35d027640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3440a2800 0x7fe3440a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe358198ff0 tx=0x7fe34000b410 comp rx=0 tx=0).stop 2024-03-16T10:16:56.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.608+0000 7fe35d027640 1 -- 172.21.15.77:0/297691863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe358100700 msgr2=0x7fe358197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.608+0000 7fe35d027640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe358100700 0x7fe358197ae0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe34800e9c0 tx=0x7fe34800ee80 comp rx=0 tx=0).stop 2024-03-16T10:16:56.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.608+0000 7fe35d027640 1 -- 172.21.15.77:0/297691863 shutdown_connections 2024-03-16T10:16:56.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.608+0000 7fe35d027640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3581022b0 0x7fe35819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.608+0000 7fe35d027640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe3440a2800 0x7fe3440a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.608+0000 7fe35d027640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3581018f0 0x7fe358198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.608+0000 7fe35d027640 1 --2- 172.21.15.77:0/297691863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe358100700 0x7fe358197ae0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.608+0000 7fe35d027640 1 -- 172.21.15.77:0/297691863 >> 172.21.15.77:0/297691863 conn(0x7fe3580fbc70 msgr2=0x7fe3580fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:56.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.608+0000 7fe35d027640 1 -- 172.21.15.77:0/297691863 shutdown_connections 2024-03-16T10:16:56.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.608+0000 7fe35d027640 1 -- 172.21.15.77:0/297691863 wait complete. 2024-03-16T10:16:56.620 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:16:56.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 -- 172.21.15.77:0/2197060191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c1033d0 msgr2=0x7f7b4c109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 --2- 172.21.15.77:0/2197060191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c1033d0 0x7f7b4c109c60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7b44009ca0 tx=0x7f7b4402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 -- 172.21.15.77:0/2197060191 shutdown_connections 2024-03-16T10:16:56.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 --2- 172.21.15.77:0/2197060191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c1033d0 0x7f7b4c109c60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 --2- 172.21.15.77:0/2197060191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b4c102a10 0x7f7b4c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 --2- 172.21.15.77:0/2197060191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b4c101820 0x7f7b4c101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 -- 172.21.15.77:0/2197060191 >> 172.21.15.77:0/2197060191 conn(0x7f7b4c0fcd90 msgr2=0x7f7b4c0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:56.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 -- 172.21.15.77:0/2197060191 shutdown_connections 2024-03-16T10:16:56.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 -- 172.21.15.77:0/2197060191 wait complete. 2024-03-16T10:16:56.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 Processor -- start 2024-03-16T10:16:56.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 -- start start 2024-03-16T10:16:56.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b4c101820 0x7f7b4c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b4c102a10 0x7f7b4c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c1033d0 0x7f7b4c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c10d4a0 con 0x7f7b4c1033d0 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c10d320 con 0x7f7b4c101820 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b51480640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4c10d620 con 0x7f7b4c102a10 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.722+0000 7f7b4b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c1033d0 0x7f7b4c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b4b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c1033d0 0x7f7b4c19f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41782/0 (socket says 172.21.15.77:41782) 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b4b7fe640 1 -- 172.21.15.77:0/850004730 learned_addr learned my addr 172.21.15.77:0/850004730 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b43fff640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b4c102a10 0x7f7b4c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b4affd640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b4c101820 0x7f7b4c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b4b7fe640 1 -- 172.21.15.77:0/850004730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b4c102a10 msgr2=0x7f7b4c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b4b7fe640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b4c102a10 0x7f7b4c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b4b7fe640 1 -- 172.21.15.77:0/850004730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b4c101820 msgr2=0x7f7b4c197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b4b7fe640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b4c101820 0x7f7b4c197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b4b7fe640 1 -- 172.21.15.77:0/850004730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b4c19f6d0 con 0x7f7b4c1033d0 2024-03-16T10:16:56.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b4b7fe640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c1033d0 0x7f7b4c19f060 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7b44037040 tx=0x7f7b44032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:56.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b48ff9640 1 -- 172.21.15.77:0/850004730 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b44041020 con 0x7f7b4c1033d0 2024-03-16T10:16:56.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b51480640 1 -- 172.21.15.77:0/850004730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b4c19f960 con 0x7f7b4c1033d0 2024-03-16T10:16:56.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b48ff9640 1 -- 172.21.15.77:0/850004730 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b44034cd0 con 0x7f7b4c1033d0 2024-03-16T10:16:56.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b48ff9640 1 -- 172.21.15.77:0/850004730 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b44039040 con 0x7f7b4c1033d0 2024-03-16T10:16:56.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.723+0000 7f7b51480640 1 -- 172.21.15.77:0/850004730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b4c19fe50 con 0x7f7b4c1033d0 2024-03-16T10:16:56.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.724+0000 7f7b51480640 1 -- 172.21.15.77:0/850004730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b4c108b40 con 0x7f7b4c1033d0 2024-03-16T10:16:56.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.726+0000 7f7b48ff9640 1 -- 172.21.15.77:0/850004730 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7b44044020 con 0x7f7b4c1033d0 2024-03-16T10:16:56.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.727+0000 7f7b48ff9640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7b380a2800 0x7f7b380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:56.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.727+0000 7f7b4affd640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7b380a2800 0x7f7b380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:56.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.727+0000 7f7b48ff9640 1 -- 172.21.15.77:0/850004730 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7b440f0c30 con 0x7f7b4c1033d0 2024-03-16T10:16:56.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.727+0000 7f7b4affd640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7b380a2800 0x7f7b380a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c102870 tx=0x7f7b3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:56.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.729+0000 7f7b48ff9640 1 -- 172.21.15.77:0/850004730 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7b4404b020 con 0x7f7b4c1033d0 2024-03-16T10:16:56.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.964+0000 7f7b51480640 1 -- 172.21.15.77:0/850004730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b4c02d370 con 0x7f7b4c1033d0 2024-03-16T10:16:56.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.965+0000 7f7b48ff9640 1 -- 172.21.15.77:0/850004730 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22496 (secure 0 0 0) 0x7f7b440bddf0 con 0x7f7b4c1033d0 2024-03-16T10:16:56.966 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:16:56.967 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":87,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:59.826259+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:16:56.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.967+0000 7f7b51480640 1 -- 172.21.15.77:0/850004730 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7b380a2800 msgr2=0x7f7b380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.967+0000 7f7b51480640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7b380a2800 0x7f7b380a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c102870 tx=0x7f7b3c005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:56.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.967+0000 7f7b51480640 1 -- 172.21.15.77:0/850004730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c1033d0 msgr2=0x7f7b4c19f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:56.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.967+0000 7f7b51480640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c1033d0 0x7f7b4c19f060 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7b44037040 tx=0x7f7b44032f40 comp rx=0 tx=0).stop 2024-03-16T10:16:56.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.967+0000 7f7b51480640 1 -- 172.21.15.77:0/850004730 shutdown_connections 2024-03-16T10:16:56.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.967+0000 7f7b51480640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b4c1033d0 0x7f7b4c19f060 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.967+0000 7f7b51480640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7b380a2800 0x7f7b380a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.967+0000 7f7b51480640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b4c102a10 0x7f7b4c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.967+0000 7f7b51480640 1 --2- 172.21.15.77:0/850004730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b4c101820 0x7f7b4c197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:56.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.967+0000 7f7b51480640 1 -- 172.21.15.77:0/850004730 >> 172.21.15.77:0/850004730 conn(0x7f7b4c0fcd90 msgr2=0x7f7b4c105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:56.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.968+0000 7f7b51480640 1 -- 172.21.15.77:0/850004730 shutdown_connections 2024-03-16T10:16:56.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:56.968+0000 7f7b51480640 1 -- 172.21.15.77:0/850004730 wait complete. 2024-03-16T10:16:56.978 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:56 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/346948678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:16:56.978 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:56 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/957198482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:16:56.978 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:56 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/297691863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:56.980 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-16T10:15:02.344673+0000', 'modified': '2024-03-16T10:15:09.227516+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24482, 'mds_1': 24475, 'mds_2': 24455, 'mds_3': 24454, 'mds_4': 14658}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14646': {'gid': 14646, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6839/3943375297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 3943375297}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 3943375297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14658': {'gid': 14658, 'name': 'a', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.77:6841/857863693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6840', 'nonce': 857863693}, {'type': 'v1', 'addr': '172.21.15.77:6841', 'nonce': 857863693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24395': {'gid': 24395, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6833/3214007033', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6832', 'nonce': 3214007033}, {'type': 'v1', 'addr': '172.21.15.192:6833', 'nonce': 3214007033}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24403': {'gid': 24403, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6833/1430295144', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6832', 'nonce': 1430295144}, {'type': 'v1', 'addr': '172.21.15.193:6833', 'nonce': 1430295144}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6835/3288720506', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6834', 'nonce': 3288720506}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 3288720506}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/3768363028', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 3768363028}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 3768363028}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.193:6837/3018320989', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 3018320989}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 3018320989}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.192:6837/440836568', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6836', 'nonce': 440836568}, {'type': 'v1', 'addr': '172.21.15.192:6837', 'nonce': 440836568}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.193:6839/3991082280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 3991082280}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 3991082280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24482': {'gid': 24482, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.192:6839/3223311954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6838', 'nonce': 3223311954}, {'type': 'v1', 'addr': '172.21.15.192:6839', 'nonce': 3223311954}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24482, 'qdb_cluster': [14658, 24455, 24482, 24454, 24475]} 2024-03-16T10:16:56.980 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-16T10:16:56.981 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-16T10:16:57.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.084+0000 7fefc34cc640 1 -- 172.21.15.77:0/3156731261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc1006e0 msgr2=0x7fefbc100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.084+0000 7fefc34cc640 1 --2- 172.21.15.77:0/3156731261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc1006e0 0x7fefbc100b00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009900 tx=0x7fefb802bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:57.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.084+0000 7fefc34cc640 1 -- 172.21.15.77:0/3156731261 shutdown_connections 2024-03-16T10:16:57.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.084+0000 7fefc34cc640 1 --2- 172.21.15.77:0/3156731261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefbc102290 0x7fefbc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.084+0000 7fefc34cc640 1 --2- 172.21.15.77:0/3156731261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefbc1018d0 0x7fefbc101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.084+0000 7fefc34cc640 1 --2- 172.21.15.77:0/3156731261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc1006e0 0x7fefbc100b00 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.084+0000 7fefc34cc640 1 -- 172.21.15.77:0/3156731261 >> 172.21.15.77:0/3156731261 conn(0x7fefbc0fbc70 msgr2=0x7fefbc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.084+0000 7fefc34cc640 1 -- 172.21.15.77:0/3156731261 shutdown_connections 2024-03-16T10:16:57.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.084+0000 7fefc34cc640 1 -- 172.21.15.77:0/3156731261 wait complete. 2024-03-16T10:16:57.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.084+0000 7fefc34cc640 1 Processor -- start 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc34cc640 1 -- start start 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc34cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc1006e0 0x7fefbc0694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc34cc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefbc1018d0 0x7fefbc19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc34cc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefbc102290 0x7fefbc19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc34cc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc10c350 con 0x7fefbc1006e0 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc34cc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc10c1d0 con 0x7fefbc1018d0 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc34cc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc10c4d0 con 0x7fefbc102290 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc1268640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc1006e0 0x7fefbc0694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc1268640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc1006e0 0x7fefbc0694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41790/0 (socket says 172.21.15.77:41790) 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc1268640 1 -- 172.21.15.77:0/2757846954 learned_addr learned my addr 172.21.15.77:0/2757846954 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc0a67640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefbc1018d0 0x7fefbc19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc1a69640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefbc102290 0x7fefbc19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc1268640 1 -- 172.21.15.77:0/2757846954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefbc102290 msgr2=0x7fefbc19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc1268640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefbc102290 0x7fefbc19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc1268640 1 -- 172.21.15.77:0/2757846954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefbc1018d0 msgr2=0x7fefbc19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc1268640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefbc1018d0 0x7fefbc19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc1268640 1 -- 172.21.15.77:0/2757846954 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefbc19f780 con 0x7fefbc1006e0 2024-03-16T10:16:57.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc1268640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc1006e0 0x7fefbc0694d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fefb8000c00 tx=0x7fefb802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefb27fc640 1 -- 172.21.15.77:0/2757846954 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb8041070 con 0x7fefbc1006e0 2024-03-16T10:16:57.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefb27fc640 1 -- 172.21.15.77:0/2757846954 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb8004450 con 0x7fefbc1006e0 2024-03-16T10:16:57.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefc34cc640 1 -- 172.21.15.77:0/2757846954 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefbc19fa10 con 0x7fefbc1006e0 2024-03-16T10:16:57.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.085+0000 7fefb27fc640 1 -- 172.21.15.77:0/2757846954 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb8034440 con 0x7fefbc1006e0 2024-03-16T10:16:57.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.086+0000 7fefc34cc640 1 -- 172.21.15.77:0/2757846954 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefbc19ff00 con 0x7fefbc1006e0 2024-03-16T10:16:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.088+0000 7fefc618a740 1 Processor -- start 2024-03-16T10:16:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.088+0000 7fefb27fc640 1 -- 172.21.15.77:0/2757846954 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fefb8044070 con 0x7fefbc1006e0 2024-03-16T10:16:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.088+0000 7fefc618a740 1 -- start start 2024-03-16T10:16:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc618a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 0x561100673ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc618a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611006745c0 0x5611006790c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc618a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100679730 0x56110067bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc618a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5611005d7c80 con 0x561100673bd0 2024-03-16T10:16:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc618a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561100580aa0 con 0x5611006745c0 2024-03-16T10:16:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc618a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5611005d3d20 con 0x561100679730 2024-03-16T10:16:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc1a69640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 0x561100673ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc1a69640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 0x561100673ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41804/0 (socket says 172.21.15.77:41804) 2024-03-16T10:16:57.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc1268640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611006745c0 0x5611006790c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc1a69640 1 -- 172.21.15.77:0/2318215801 learned_addr learned my addr 172.21.15.77:0/2318215801 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc0a67640 1 --2- 172.21.15.77:0/2318215801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100679730 0x56110067bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc1a69640 1 -- 172.21.15.77:0/2318215801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100679730 msgr2=0x56110067bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc1a69640 1 --2- 172.21.15.77:0/2318215801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100679730 0x56110067bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc1a69640 1 -- 172.21.15.77:0/2318215801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611006745c0 msgr2=0x5611006790c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc1a69640 1 --2- 172.21.15.77:0/2318215801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611006745c0 0x5611006790c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefb27fc640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fefa80a2800 0x7fefa80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc1a69640 1 -- 172.21.15.77:0/2318215801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611005a2ff0 con 0x561100673bd0 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc1268640 1 --2- 172.21.15.77:0/2318215801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611006745c0 0x5611006790c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc0a67640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fefa80a2800 0x7fefa80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefc1a69640 1 --2- 172.21.15.77:0/2318215801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 0x561100673ff0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009fd0 tx=0x7fefb40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af7e2a5a4a2d34d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefb27fc640 1 -- 172.21.15.77:0/2757846954 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fefb80f0490 con 0x7fefbc1006e0 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.089+0000 7fefa77fe640 1 -- 172.21.15.77:0/2318215801 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb4004d20 con 0x561100673bd0 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.090+0000 7fefa77fe640 1 -- 172.21.15.77:0/2318215801 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb4004050 con 0x561100673bd0 2024-03-16T10:16:57.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.090+0000 7fefa77fe640 1 -- 172.21.15.77:0/2318215801 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb4005280 con 0x561100673bd0 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.090+0000 7fefc618a740 1 -- 172.21.15.77:0/2318215801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 msgr2=0x561100673ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.090+0000 7fefc0a67640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fefa80a2800 0x7fefa80a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fefbc069240 tx=0x7fefac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.090+0000 7fefc618a740 1 --2- 172.21.15.77:0/2318215801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 0x561100673ff0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009fd0 tx=0x7fefb40118d0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.090+0000 7fefc618a740 1 -- 172.21.15.77:0/2318215801 shutdown_connections 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.090+0000 7fefc618a740 1 --2- 172.21.15.77:0/2318215801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100679730 0x56110067bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.090+0000 7fefc618a740 1 --2- 172.21.15.77:0/2318215801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611006745c0 0x5611006790c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.090+0000 7fefc618a740 1 --2- 172.21.15.77:0/2318215801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 0x561100673ff0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.090+0000 7fefc618a740 1 -- 172.21.15.77:0/2318215801 >> 172.21.15.77:0/2318215801 conn(0x56110066f140 msgr2=0x56110066f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.090+0000 7fefc618a740 1 -- 172.21.15.77:0/2318215801 shutdown_connections 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.090+0000 7fefc618a740 1 -- 172.21.15.77:0/2318215801 wait complete. 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc618a740 1 -- start start 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc618a740 1 Processor -- start 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc618a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100673bd0 0x56110067ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc618a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611006745c0 0x56110067e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc618a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100679730 0x561100685340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc618a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56110067c6e0 con 0x561100679730 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc618a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561100593ef0 con 0x5611006745c0 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc618a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56110067c860 con 0x561100673bd0 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc618a740 10 client.0 _ll_register_callbacks cb 0x56110058edc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc1a69640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100679730 0x561100685340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc1268640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100673bd0 0x56110067ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc1a69640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100679730 0x561100685340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41818/0 (socket says 172.21.15.77:41818) 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc1a69640 1 -- 172.21.15.77:0/1452061465 learned_addr learned my addr 172.21.15.77:0/1452061465 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.091+0000 7fefc0a67640 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611006745c0 0x56110067e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefc1268640 1 -- 172.21.15.77:0/1452061465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611006745c0 msgr2=0x56110067e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefc1268640 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611006745c0 0x56110067e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefc1268640 1 -- 172.21.15.77:0/1452061465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100679730 msgr2=0x561100685340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefc1268640 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100679730 0x561100685340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefc1268640 1 -- 172.21.15.77:0/1452061465 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611005a0a30 con 0x561100673bd0 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefc1a69640 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100679730 0x561100685340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefc0a67640 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611006745c0 0x56110067e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefc1268640 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100673bd0 0x56110067ddc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fefb80ee070 tx=0x7fefb8005670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5071a2d4e72d69af server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefa4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefa4ff9640 1 -- 172.21.15.77:0/1452061465 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb804b070 con 0x561100673bd0 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefa4ff9640 1 -- 172.21.15.77:0/1452061465 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fefb80c51d0 con 0x561100673bd0 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefc618a740 1 -- 172.21.15.77:0/1452061465 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefb4004050 con 0x561100673bd0 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefa4ff9640 1 -- 172.21.15.77:0/1452061465 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fefb80b92a0 con 0x561100673bd0 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefc618a740 10 client.25069 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefc618a740 1 -- 172.21.15.77:0/1452061465 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5611002ec2d0 con 0x561100673bd0 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefa4ff9640 1 -- 172.21.15.77:0/1452061465 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb4004050 con 0x561100673bd0 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefa4ff9640 1 -- 172.21.15.77:0/1452061465 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fefb80f00d0 con 0x561100673bd0 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.092+0000 7fefc618a740 10 client.25069 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:57.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.093+0000 7fefc618a740 10 client.25069 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.093+0000 7fefc618a740 20 client.25069 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.093+0000 7fefc618a740 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56110068a7b0 0x56110068bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.093+0000 7fefc618a740 4 client.25069 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.093+0000 7fefc618a740 1 -- 172.21.15.77:0/1452061465 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611005e51c0 con 0x56110068a7b0 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.093+0000 7fefc0a67640 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56110068a7b0 0x56110068bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.093+0000 7fefa4ff9640 10 client.25069 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.093+0000 7fefc0a67640 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56110068a7b0 0x56110068bc20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8f49a30099b4be2a server_cookie=d067ff5e75c09314 in_seq=0 out_seq=0 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefa4ff9640 1 -- 172.21.15.77:0/1452061465 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5611005e51c0 con 0x56110068a7b0 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefa4ff9640 10 client.25069 handle_command_reply: tid=0 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefc618a740 1 client.25069 shutdown 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefc618a740 1 -- 172.21.15.77:0/1452061465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100673bd0 msgr2=0x56110067ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefc618a740 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100673bd0 0x56110067ddc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fefb80ee070 tx=0x7fefb8005670 comp rx=0 tx=0).stop 2024-03-16T10:16:57.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefc618a740 1 -- 172.21.15.77:0/1452061465 shutdown_connections 2024-03-16T10:16:57.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefc618a740 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100679730 0x561100685340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefc618a740 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611006745c0 0x56110067e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefc618a740 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56110068a7b0 0x56110068bc20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefc618a740 1 --2- 172.21.15.77:0/1452061465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100673bd0 0x56110067ddc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefa4ff9640 0 client.25069 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:16:57.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefc618a740 1 -- 172.21.15.77:0/1452061465 >> 172.21.15.77:0/1452061465 conn(0x56110066f140 msgr2=0x56110066e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefc618a740 1 -- 172.21.15.77:0/1452061465 shutdown_connections 2024-03-16T10:16:57.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefc618a740 1 -- 172.21.15.77:0/1452061465 wait complete. 2024-03-16T10:16:57.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.094+0000 7fefc618a740 20 client.25069 trim_cache size 0 max 16384 2024-03-16T10:16:57.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc618a740 1 Processor -- start 2024-03-16T10:16:57.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc618a740 1 -- start start 2024-03-16T10:16:57.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc618a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 0x56110067d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc618a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611006745c0 0x5611006852d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc618a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561100679730 0x56110067df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc618a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561100689f60 con 0x561100673bd0 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc618a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56110068a630 con 0x561100679730 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc618a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56110068ad00 con 0x5611006745c0 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc1268640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 0x56110067d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc1268640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 0x56110067d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41834/0 (socket says 172.21.15.77:41834) 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc1268640 1 -- 172.21.15.77:0/709293132 learned_addr learned my addr 172.21.15.77:0/709293132 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc1a69640 1 --2- 172.21.15.77:0/709293132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561100679730 0x56110067df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc0a67640 1 --2- 172.21.15.77:0/709293132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611006745c0 0x5611006852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc1268640 1 -- 172.21.15.77:0/709293132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611006745c0 msgr2=0x5611006852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc1268640 1 --2- 172.21.15.77:0/709293132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611006745c0 0x5611006852d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc1268640 1 -- 172.21.15.77:0/709293132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561100679730 msgr2=0x56110067df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc1268640 1 --2- 172.21.15.77:0/709293132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561100679730 0x56110067df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.112+0000 7fefc1268640 1 -- 172.21.15.77:0/709293132 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611003b66b0 con 0x561100673bd0 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.113+0000 7fefc0a67640 1 --2- 172.21.15.77:0/709293132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611006745c0 0x5611006852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.113+0000 7fefc1268640 1 --2- 172.21.15.77:0/709293132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 0x56110067d9d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fefb80bcf90 tx=0x7fefb80f3cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4197869b7fb10da server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.113+0000 7fefa6ffd640 1 -- 172.21.15.77:0/709293132 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb804b070 con 0x561100673bd0 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.113+0000 7fefa6ffd640 1 -- 172.21.15.77:0/709293132 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb80bd000 con 0x561100673bd0 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.113+0000 7fefa6ffd640 1 -- 172.21.15.77:0/709293132 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb80bc960 con 0x561100673bd0 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.113+0000 7fefc618a740 1 -- 172.21.15.77:0/709293132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 msgr2=0x56110067d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.113+0000 7fefc618a740 1 --2- 172.21.15.77:0/709293132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 0x56110067d9d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fefb80bcf90 tx=0x7fefb80f3cc0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.113+0000 7fefc618a740 1 -- 172.21.15.77:0/709293132 shutdown_connections 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.113+0000 7fefc618a740 1 --2- 172.21.15.77:0/709293132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561100679730 0x56110067df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.113+0000 7fefc618a740 1 --2- 172.21.15.77:0/709293132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611006745c0 0x5611006852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.113+0000 7fefc618a740 1 --2- 172.21.15.77:0/709293132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561100673bd0 0x56110067d9d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.113+0000 7fefc618a740 1 -- 172.21.15.77:0/709293132 >> 172.21.15.77:0/709293132 conn(0x56110066f140 msgr2=0x5611006894a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.114+0000 7fefc618a740 1 -- 172.21.15.77:0/709293132 shutdown_connections 2024-03-16T10:16:57.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.114+0000 7fefc618a740 1 -- 172.21.15.77:0/709293132 wait complete. 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.114+0000 7fefc618a740 1 -- start start 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.114+0000 7fefc618a740 1 Processor -- start 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.114+0000 7fefc618a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100673bd0 0x5611006801d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.114+0000 7fefc618a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611006745c0 0x561100680710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.114+0000 7fefc618a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561100679730 0x5611006ee510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.114+0000 7fefc618a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56110068baa0 con 0x5611006745c0 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.114+0000 7fefc618a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56110068b3d0 con 0x561100679730 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.114+0000 7fefc618a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56110068c170 con 0x561100673bd0 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.114+0000 7fefc618a740 10 client.0 _ll_register_callbacks cb 0x7fefb8005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc0a67640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611006745c0 0x561100680710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc0a67640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611006745c0 0x561100680710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41838/0 (socket says 172.21.15.77:41838) 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc0a67640 1 -- 172.21.15.77:0/1674991960 learned_addr learned my addr 172.21.15.77:0/1674991960 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc1a69640 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561100679730 0x5611006ee510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc1268640 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100673bd0 0x5611006801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc0a67640 1 -- 172.21.15.77:0/1674991960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100673bd0 msgr2=0x5611006801d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc0a67640 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100673bd0 0x5611006801d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc0a67640 1 -- 172.21.15.77:0/1674991960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561100679730 msgr2=0x5611006ee510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc0a67640 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561100679730 0x5611006ee510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc0a67640 1 -- 172.21.15.77:0/1674991960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611006eea50 con 0x5611006745c0 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc0a67640 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611006745c0 0x561100680710 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fefac009b80 tx=0x7fefac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4c06c401732dd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefa4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefa4ff9640 1 -- 172.21.15.77:0/1674991960 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac0042c0 con 0x5611006745c0 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefa4ff9640 1 -- 172.21.15.77:0/1674991960 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fefac004460 con 0x5611006745c0 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc618a740 1 -- 172.21.15.77:0/1674991960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefb80bd000 con 0x5611006745c0 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefa4ff9640 1 -- 172.21.15.77:0/1674991960 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fefac02e0e0 con 0x5611006745c0 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc618a740 10 client.15285 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefc618a740 1 -- 172.21.15.77:0/1674991960 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5611006efd60 con 0x5611006745c0 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.115+0000 7fefa4ff9640 1 -- 172.21.15.77:0/1674991960 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb80bd000 con 0x5611006745c0 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefa4ff9640 1 -- 172.21.15.77:0/1674991960 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fefac003ca0 con 0x5611006745c0 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 10 client.15285 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 10 client.15285 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-16T10:16:57.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 20 client.15285 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x5611006f4940 0x5611006f3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 4 client.15285 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 1 -- 172.21.15.77:0/1674991960 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "status"}) -- 0x5611005df510 con 0x5611006f4940 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc1268640 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x5611006f4940 0x5611006f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefa4ff9640 10 client.15285 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc1268640 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x5611006f4940 0x5611006f3650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fd2215a19cf03e5 server_cookie=4557e2fd049061dc in_seq=0 out_seq=0 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefa4ff9640 1 -- 172.21.15.77:0/1674991960 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+320 (crc 0 0 0) 0x5611005df510 con 0x5611006f4940 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefa4ff9640 10 client.15285 handle_command_reply: tid=0 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 1 client.15285 shutdown 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 1 -- 172.21.15.77:0/1674991960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611006745c0 msgr2=0x561100680710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611006745c0 0x561100680710 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fefac009b80 tx=0x7fefac0046c0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 1 -- 172.21.15.77:0/1674991960 shutdown_connections 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561100679730 0x5611006ee510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611006745c0 0x561100680710 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.116+0000 7fefc618a740 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x5611006f4940 0x5611006f3650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc618a740 1 --2- 172.21.15.77:0/1674991960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561100673bd0 0x5611006801d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefa4ff9640 0 client.15285 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:16:57.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc618a740 1 -- 172.21.15.77:0/1674991960 >> 172.21.15.77:0/1674991960 conn(0x56110066f140 msgr2=0x5611006723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc618a740 1 -- 172.21.15.77:0/1674991960 shutdown_connections 2024-03-16T10:16:57.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc618a740 1 -- 172.21.15.77:0/1674991960 wait complete. 2024-03-16T10:16:57.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc618a740 20 client.15285 trim_cache size 0 max 16384 2024-03-16T10:16:57.130 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:16:57.130 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:16:57.130 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 4, 2024-03-16T10:16:57.131 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 14658, 2024-03-16T10:16:57.131 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T10:16:57.131 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T10:16:57.131 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T10:16:57.131 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 108.8953746, 2024-03-16T10:16:57.131 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 26, 2024-03-16T10:16:57.131 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 87, 2024-03-16T10:16:57.131 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T10:16:57.131 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 124.064108084 2024-03-16T10:16:57.131 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:16:57.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc34cc640 1 -- 172.21.15.77:0/2757846954 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fefa80a2800 msgr2=0x7fefa80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc34cc640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fefa80a2800 0x7fefa80a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fefbc069240 tx=0x7fefac005f90 comp rx=0 tx=0).stop 2024-03-16T10:16:57.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc34cc640 1 -- 172.21.15.77:0/2757846954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc1006e0 msgr2=0x7fefbc0694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc34cc640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc1006e0 0x7fefbc0694d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fefb8000c00 tx=0x7fefb802c640 comp rx=0 tx=0).stop 2024-03-16T10:16:57.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc34cc640 1 -- 172.21.15.77:0/2757846954 shutdown_connections 2024-03-16T10:16:57.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc34cc640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fefa80a2800 0x7fefa80a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc34cc640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefbc102290 0x7fefbc19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc34cc640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefbc1018d0 0x7fefbc19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc34cc640 1 --2- 172.21.15.77:0/2757846954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc1006e0 0x7fefbc0694d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc34cc640 1 -- 172.21.15.77:0/2757846954 >> 172.21.15.77:0/2757846954 conn(0x7fefbc0fbc70 msgr2=0x7fefbc104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc34cc640 1 -- 172.21.15.77:0/2757846954 shutdown_connections 2024-03-16T10:16:57.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.117+0000 7fefc34cc640 1 -- 172.21.15.77:0/2757846954 wait complete. 2024-03-16T10:16:57.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-16T10:16:57.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:56 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/346948678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:16:57.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:56 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/957198482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:16:57.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:56 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/297691863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:57.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.236+0000 7f091fd5b640 1 -- 172.21.15.77:0/1055561442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09181006e0 msgr2=0x7f0918100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.236+0000 7f091fd5b640 1 --2- 172.21.15.77:0/1055561442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09181006e0 0x7f0918100b00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0914009900 tx=0x7f091402bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:57.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.236+0000 7f091fd5b640 1 -- 172.21.15.77:0/1055561442 shutdown_connections 2024-03-16T10:16:57.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.236+0000 7f091fd5b640 1 --2- 172.21.15.77:0/1055561442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0918102290 0x7f0918108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.236+0000 7f091fd5b640 1 --2- 172.21.15.77:0/1055561442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09181018d0 0x7f0918101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.236+0000 7f091fd5b640 1 --2- 172.21.15.77:0/1055561442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09181006e0 0x7f0918100b00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.236+0000 7f091fd5b640 1 -- 172.21.15.77:0/1055561442 >> 172.21.15.77:0/1055561442 conn(0x7f09180fbc70 msgr2=0x7f09180fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.236+0000 7f091fd5b640 1 -- 172.21.15.77:0/1055561442 shutdown_connections 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.236+0000 7f091fd5b640 1 -- 172.21.15.77:0/1055561442 wait complete. 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091fd5b640 1 Processor -- start 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091fd5b640 1 -- start start 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091fd5b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09181006e0 0x7f0918197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091fd5b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09181018d0 0x7f09181980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091fd5b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0918102290 0x7f091819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091fd5b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f091810c380 con 0x7f09181006e0 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091fd5b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f091810c200 con 0x7f09181018d0 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091fd5b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f091810c500 con 0x7f0918102290 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091daf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09181006e0 0x7f0918197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091daf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09181006e0 0x7f0918197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41858/0 (socket says 172.21.15.77:41858) 2024-03-16T10:16:57.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091daf7640 1 -- 172.21.15.77:0/2078449007 learned_addr learned my addr 172.21.15.77:0/2078449007 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091e2f8640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0918102290 0x7f091819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091d2f6640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09181018d0 0x7f09181980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091daf7640 1 -- 172.21.15.77:0/2078449007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0918102290 msgr2=0x7f091819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091daf7640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0918102290 0x7f091819f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091daf7640 1 -- 172.21.15.77:0/2078449007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09181018d0 msgr2=0x7f09181980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.237+0000 7f091daf7640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09181018d0 0x7f09181980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.238+0000 7f091daf7640 1 -- 172.21.15.77:0/2078449007 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f091819f770 con 0x7f09181006e0 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.238+0000 7f091daf7640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09181006e0 0x7f0918197ba0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0914000c00 tx=0x7f091402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.238+0000 7f090effd640 1 -- 172.21.15.77:0/2078449007 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914043020 con 0x7f09181006e0 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.238+0000 7f091fd5b640 1 -- 172.21.15.77:0/2078449007 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f091819fa00 con 0x7f09181006e0 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.238+0000 7f090effd640 1 -- 172.21.15.77:0/2078449007 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09140054e0 con 0x7f09181006e0 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.238+0000 7f090effd640 1 -- 172.21.15.77:0/2078449007 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914003c40 con 0x7f09181006e0 2024-03-16T10:16:57.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.238+0000 7f091fd5b640 1 -- 172.21.15.77:0/2078449007 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f091819ffa0 con 0x7f09181006e0 2024-03-16T10:16:57.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.240+0000 7f090effd640 1 -- 172.21.15.77:0/2078449007 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0914046020 con 0x7f09181006e0 2024-03-16T10:16:57.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.240+0000 7f0922a27740 1 Processor -- start 2024-03-16T10:16:57.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f0922a27740 1 -- start start 2024-03-16T10:16:57.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f0922a27740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 0x556e2aa85ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f0922a27740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa865c0 0x556e2aa8b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f0922a27740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa8b730 0x556e2aa8dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f0922a27740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556e2a9e9c80 con 0x556e2aa85bd0 2024-03-16T10:16:57.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f0922a27740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x556e2a992aa0 con 0x556e2aa8b730 2024-03-16T10:16:57.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f0922a27740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556e2a9e5d20 con 0x556e2aa865c0 2024-03-16T10:16:57.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f091e2f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 0x556e2aa85ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f091e2f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 0x556e2aa85ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41864/0 (socket says 172.21.15.77:41864) 2024-03-16T10:16:57.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f091e2f8640 1 -- 172.21.15.77:0/1061623002 learned_addr learned my addr 172.21.15.77:0/1061623002 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f090effd640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09040a2800 0x7f09040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f091daf7640 1 --2- 172.21.15.77:0/1061623002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa865c0 0x556e2aa8b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f091d2f6640 1 --2- 172.21.15.77:0/1061623002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa8b730 0x556e2aa8dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f091e2f8640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09040a2800 0x7f09040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f091e2f8640 1 -- 172.21.15.77:0/1061623002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa865c0 msgr2=0x556e2aa8b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f091e2f8640 1 --2- 172.21.15.77:0/1061623002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa865c0 0x556e2aa8b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f090effd640 1 -- 172.21.15.77:0/2078449007 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f09140f0990 con 0x7f09181006e0 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f091e2f8640 1 -- 172.21.15.77:0/1061623002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa8b730 msgr2=0x556e2aa8dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f091e2f8640 1 --2- 172.21.15.77:0/1061623002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa8b730 0x556e2aa8dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.241+0000 7f091e2f8640 1 -- 172.21.15.77:0/1061623002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e2a9b4ff0 con 0x556e2aa85bd0 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.242+0000 7f091e2f8640 1 --2- 172.21.15.77:0/1061623002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 0x556e2aa85ff0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0910005c40 tx=0x7f091000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b225de4ddfb6f079 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.242+0000 7f0903fff640 1 -- 172.21.15.77:0/1061623002 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0910022020 con 0x556e2aa85bd0 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.242+0000 7f091e2f8640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09040a2800 0x7f09040a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0910023000 tx=0x7f0910021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.242+0000 7f0903fff640 1 -- 172.21.15.77:0/1061623002 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0910013170 con 0x556e2aa85bd0 2024-03-16T10:16:57.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.242+0000 7f0903fff640 1 -- 172.21.15.77:0/1061623002 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0910010790 con 0x556e2aa85bd0 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.243+0000 7f0922a27740 1 -- 172.21.15.77:0/1061623002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 msgr2=0x556e2aa85ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.243+0000 7f0922a27740 1 --2- 172.21.15.77:0/1061623002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 0x556e2aa85ff0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0910005c40 tx=0x7f091000f3c0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.243+0000 7f0922a27740 1 -- 172.21.15.77:0/1061623002 shutdown_connections 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.243+0000 7f0922a27740 1 --2- 172.21.15.77:0/1061623002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa8b730 0x556e2aa8dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.243+0000 7f0922a27740 1 --2- 172.21.15.77:0/1061623002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa865c0 0x556e2aa8b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.243+0000 7f0922a27740 1 --2- 172.21.15.77:0/1061623002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 0x556e2aa85ff0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.243+0000 7f0922a27740 1 -- 172.21.15.77:0/1061623002 >> 172.21.15.77:0/1061623002 conn(0x556e2aa81140 msgr2=0x556e2aa81540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.243+0000 7f0922a27740 1 -- 172.21.15.77:0/1061623002 shutdown_connections 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.243+0000 7f0922a27740 1 -- 172.21.15.77:0/1061623002 wait complete. 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.243+0000 7f0922a27740 1 -- start start 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.243+0000 7f0922a27740 1 Processor -- start 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f0922a27740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa85bd0 0x556e2aa8fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f0922a27740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa865c0 0x556e2aa902f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f0922a27740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa8b730 0x556e2aa97330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f0922a27740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556e2aa8e6e0 con 0x556e2aa8b730 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f0922a27740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x556e2a9a5ef0 con 0x556e2aa85bd0 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f0922a27740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556e2aa8e860 con 0x556e2aa865c0 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f0922a27740 10 client.0 _ll_register_callbacks cb 0x556e2a9a0dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f091daf7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa85bd0 0x556e2aa8fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f091daf7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa85bd0 0x556e2aa8fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:39474/0 (socket says 172.21.15.77:39474) 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f091daf7640 1 -- 172.21.15.77:0/3549788787 learned_addr learned my addr 172.21.15.77:0/3549788787 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f091d2f6640 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa865c0 0x556e2aa902f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f091e2f8640 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa8b730 0x556e2aa97330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f091daf7640 1 -- 172.21.15.77:0/3549788787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa865c0 msgr2=0x556e2aa902f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f091daf7640 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa865c0 0x556e2aa902f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f091daf7640 1 -- 172.21.15.77:0/3549788787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa8b730 msgr2=0x556e2aa97330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f091daf7640 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa8b730 0x556e2aa97330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f091daf7640 1 -- 172.21.15.77:0/3549788787 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e2a9b2a30 con 0x556e2aa85bd0 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f091daf7640 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa85bd0 0x556e2aa8fdb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f09140bd350 tx=0x7f09140b9050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f9a70cbca5db319 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f09017fa640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f09017fa640 1 -- 172.21.15.77:0/3549788787 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09140bcf00 con 0x556e2aa85bd0 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f09017fa640 1 -- 172.21.15.77:0/3549788787 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f09140bb6c0 con 0x556e2aa85bd0 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.244+0000 7f0922a27740 1 -- 172.21.15.77:0/3549788787 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0910013170 con 0x556e2aa85bd0 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f09017fa640 1 -- 172.21.15.77:0/3549788787 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f09140bcd20 con 0x556e2aa85bd0 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f0922a27740 10 client.25103 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f0922a27740 1 -- 172.21.15.77:0/3549788787 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556e2a6fe2d0 con 0x556e2aa85bd0 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f09017fa640 1 -- 172.21.15.77:0/3549788787 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0910013170 con 0x556e2aa85bd0 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f09017fa640 1 -- 172.21.15.77:0/3549788787 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f09140c1140 con 0x556e2aa85bd0 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f0922a27740 10 client.25103 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f0922a27740 10 client.25103 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f0922a27740 20 client.25103 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f0922a27740 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x556e2aa9c790 0x556e2aa9dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f0922a27740 4 client.25103 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f0922a27740 1 -- 172.21.15.77:0/3549788787 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e2a9f71c0 con 0x556e2aa9c790 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f091d2f6640 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x556e2aa9c790 0x556e2aa9dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f09017fa640 10 client.25103 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.245+0000 7f091d2f6640 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x556e2aa9c790 0x556e2aa9dc00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8443f69182d40dc6 server_cookie=6db66edc11785860 in_seq=0 out_seq=0 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f09017fa640 1 -- 172.21.15.77:0/3549788787 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556e2a9f71c0 con 0x556e2aa9c790 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f09017fa640 10 client.25103 handle_command_reply: tid=0 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f0922a27740 1 client.25103 shutdown 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f0922a27740 1 -- 172.21.15.77:0/3549788787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa85bd0 msgr2=0x556e2aa8fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f0922a27740 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa85bd0 0x556e2aa8fdb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f09140bd350 tx=0x7f09140b9050 comp rx=0 tx=0).stop 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f0922a27740 1 -- 172.21.15.77:0/3549788787 shutdown_connections 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f0922a27740 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x556e2aa9c790 0x556e2aa9dc00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f0922a27740 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa8b730 0x556e2aa97330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f09017fa640 0 client.25103 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f0922a27740 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa865c0 0x556e2aa902f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f0922a27740 1 --2- 172.21.15.77:0/3549788787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa85bd0 0x556e2aa8fdb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f0922a27740 1 -- 172.21.15.77:0/3549788787 >> 172.21.15.77:0/3549788787 conn(0x556e2aa81140 msgr2=0x556e2aa80380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f0922a27740 1 -- 172.21.15.77:0/3549788787 shutdown_connections 2024-03-16T10:16:57.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.246+0000 7f0922a27740 1 -- 172.21.15.77:0/3549788787 wait complete. 2024-03-16T10:16:57.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.247+0000 7f0922a27740 20 client.25103 trim_cache size 0 max 16384 2024-03-16T10:16:57.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:56 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/346948678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:16:57.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:56 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/957198482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:16:57.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:56 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/297691863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:16:57.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.264+0000 7f0922a27740 1 Processor -- start 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f0922a27740 1 -- start start 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f0922a27740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 0x556e2aa8f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f0922a27740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa865c0 0x556e2aa972c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f0922a27740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa8b730 0x556e2aa8ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f0922a27740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556e2aa9bf40 con 0x556e2aa85bd0 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f0922a27740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x556e2aa9c610 con 0x556e2aa865c0 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f0922a27740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556e2aa9cce0 con 0x556e2aa8b730 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091daf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 0x556e2aa8f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091daf7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 0x556e2aa8f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41882/0 (socket says 172.21.15.77:41882) 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091daf7640 1 -- 172.21.15.77:0/3991388723 learned_addr learned my addr 172.21.15.77:0/3991388723 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091d2f6640 1 --2- 172.21.15.77:0/3991388723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa865c0 0x556e2aa972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091e2f8640 1 --2- 172.21.15.77:0/3991388723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa8b730 0x556e2aa8ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091daf7640 1 -- 172.21.15.77:0/3991388723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa8b730 msgr2=0x556e2aa8ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091daf7640 1 --2- 172.21.15.77:0/3991388723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa8b730 0x556e2aa8ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091daf7640 1 -- 172.21.15.77:0/3991388723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa865c0 msgr2=0x556e2aa972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091daf7640 1 --2- 172.21.15.77:0/3991388723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa865c0 0x556e2aa972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091daf7640 1 -- 172.21.15.77:0/3991388723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e2a7c86b0 con 0x556e2aa85bd0 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091d2f6640 1 --2- 172.21.15.77:0/3991388723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa865c0 0x556e2aa972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091e2f8640 1 --2- 172.21.15.77:0/3991388723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa8b730 0x556e2aa8ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f091daf7640 1 --2- 172.21.15.77:0/3991388723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 0x556e2aa8f9c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0914009340 tx=0x7f0914009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee512328688719f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f09037fe640 1 -- 172.21.15.77:0/3991388723 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09140bcf00 con 0x556e2aa85bd0 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f09037fe640 1 -- 172.21.15.77:0/3991388723 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0914005690 con 0x556e2aa85bd0 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.265+0000 7f09037fe640 1 -- 172.21.15.77:0/3991388723 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09140b9660 con 0x556e2aa85bd0 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.266+0000 7f0922a27740 1 -- 172.21.15.77:0/3991388723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 msgr2=0x556e2aa8f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.266+0000 7f0922a27740 1 --2- 172.21.15.77:0/3991388723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 0x556e2aa8f9c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0914009340 tx=0x7f0914009620 comp rx=0 tx=0).stop 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.266+0000 7f0922a27740 1 -- 172.21.15.77:0/3991388723 shutdown_connections 2024-03-16T10:16:57.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.266+0000 7f0922a27740 1 --2- 172.21.15.77:0/3991388723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa8b730 0x556e2aa8ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.266+0000 7f0922a27740 1 --2- 172.21.15.77:0/3991388723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa865c0 0x556e2aa972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.266+0000 7f0922a27740 1 --2- 172.21.15.77:0/3991388723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa85bd0 0x556e2aa8f9c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.266+0000 7f0922a27740 1 -- 172.21.15.77:0/3991388723 >> 172.21.15.77:0/3991388723 conn(0x556e2aa81140 msgr2=0x556e2aa8ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.266+0000 7f0922a27740 1 -- 172.21.15.77:0/3991388723 shutdown_connections 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.266+0000 7f0922a27740 1 -- 172.21.15.77:0/3991388723 wait complete. 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.266+0000 7f0922a27740 1 -- start start 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.266+0000 7f0922a27740 1 Processor -- start 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f0922a27740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa85bd0 0x556e2aa95490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f0922a27740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa865c0 0x556e2aa959d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f0922a27740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa8b730 0x556e2aa95f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f0922a27740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556e2aa9da80 con 0x556e2aa865c0 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f0922a27740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x556e2aa9d3b0 con 0x556e2aa85bd0 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f0922a27740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556e2aa9e150 con 0x556e2aa8b730 2024-03-16T10:16:57.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f0922a27740 10 client.0 _ll_register_callbacks cb 0x7f0914005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f091d2f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa865c0 0x556e2aa959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f091d2f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa865c0 0x556e2aa959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41886/0 (socket says 172.21.15.77:41886) 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f091d2f6640 1 -- 172.21.15.77:0/3825143546 learned_addr learned my addr 172.21.15.77:0/3825143546 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f091daf7640 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa85bd0 0x556e2aa95490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f091e2f8640 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa8b730 0x556e2aa95f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f091d2f6640 1 -- 172.21.15.77:0/3825143546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa8b730 msgr2=0x556e2aa95f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f091d2f6640 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa8b730 0x556e2aa95f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f091d2f6640 1 -- 172.21.15.77:0/3825143546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa85bd0 msgr2=0x556e2aa95490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f091d2f6640 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa85bd0 0x556e2aa95490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f091d2f6640 1 -- 172.21.15.77:0/3825143546 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e2aa96780 con 0x556e2aa865c0 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f091d2f6640 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa865c0 0x556e2aa959d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0908000c00 tx=0x7f0908005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25877999ac635caa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f09017fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f09017fa640 1 -- 172.21.15.77:0/3825143546 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0908023020 con 0x556e2aa865c0 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f09017fa640 1 -- 172.21.15.77:0/3825143546 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f090801f270 con 0x556e2aa865c0 2024-03-16T10:16:57.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.267+0000 7f0922a27740 1 -- 172.21.15.77:0/3825143546 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556e2a9f1510 con 0x556e2aa865c0 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.268+0000 7f09017fa640 1 -- 172.21.15.77:0/3825143546 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f090802e050 con 0x556e2aa865c0 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.268+0000 7f0922a27740 10 client.15303 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.268+0000 7f0922a27740 1 -- 172.21.15.77:0/3825143546 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556e2ab027b0 con 0x556e2aa865c0 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.268+0000 7f09017fa640 1 -- 172.21.15.77:0/3825143546 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556e2a9f1510 con 0x556e2aa865c0 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.268+0000 7f09017fa640 1 -- 172.21.15.77:0/3825143546 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f0908010b60 con 0x556e2aa865c0 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.268+0000 7f0922a27740 10 client.15303 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.268+0000 7f0922a27740 10 client.15303 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.268+0000 7f0922a27740 20 client.15303 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.268+0000 7f0922a27740 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x556e2ab073c0 0x556e2ab060d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.268+0000 7f0922a27740 4 client.15303 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.268+0000 7f0922a27740 1 -- 172.21.15.77:0/3825143546 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "status"}) -- 0x556e2a96fa00 con 0x556e2ab073c0 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.268+0000 7f091daf7640 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x556e2ab073c0 0x556e2ab060d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f09017fa640 10 client.15303 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f091daf7640 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x556e2ab073c0 0x556e2ab060d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6a0f5ada83efc7b2 server_cookie=16153ababf31035c in_seq=0 out_seq=0 2024-03-16T10:16:57.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f09017fa640 1 -- 172.21.15.77:0/3825143546 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556e2a96fa00 con 0x556e2ab073c0 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f09017fa640 10 client.15303 handle_command_reply: tid=0 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f0922a27740 1 client.15303 shutdown 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f0922a27740 1 -- 172.21.15.77:0/3825143546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa865c0 msgr2=0x556e2aa959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f0922a27740 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa865c0 0x556e2aa959d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0908000c00 tx=0x7f0908005db0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f0922a27740 1 -- 172.21.15.77:0/3825143546 shutdown_connections 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f0922a27740 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x556e2ab073c0 0x556e2ab060d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f0922a27740 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556e2aa8b730 0x556e2aa95f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f09017fa640 0 client.15303 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f0922a27740 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556e2aa865c0 0x556e2aa959d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f0922a27740 1 --2- 172.21.15.77:0/3825143546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556e2aa85bd0 0x556e2aa95490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f0922a27740 1 -- 172.21.15.77:0/3825143546 >> 172.21.15.77:0/3825143546 conn(0x556e2aa81140 msgr2=0x556e2aa8f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f0922a27740 1 -- 172.21.15.77:0/3825143546 shutdown_connections 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f0922a27740 1 -- 172.21.15.77:0/3825143546 wait complete. 2024-03-16T10:16:57.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.269+0000 7f0922a27740 20 client.15303 trim_cache size 0 max 16384 2024-03-16T10:16:57.278 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:16:57.278 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:16:57.278 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 3, 2024-03-16T10:16:57.278 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24454, 2024-03-16T10:16:57.278 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T10:16:57.278 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T10:16:57.278 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T10:16:57.278 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 110.014417682, 2024-03-16T10:16:57.278 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 26, 2024-03-16T10:16:57.279 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 87, 2024-03-16T10:16:57.279 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T10:16:57.279 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 135.56745642600001 2024-03-16T10:16:57.279 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:16:57.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.270+0000 7f091fd5b640 1 -- 172.21.15.77:0/2078449007 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09040a2800 msgr2=0x7f09040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.270+0000 7f091fd5b640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09040a2800 0x7f09040a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0910023000 tx=0x7f0910021000 comp rx=0 tx=0).stop 2024-03-16T10:16:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.270+0000 7f091fd5b640 1 -- 172.21.15.77:0/2078449007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09181006e0 msgr2=0x7f0918197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.270+0000 7f091fd5b640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09181006e0 0x7f0918197ba0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0914000c00 tx=0x7f091402c650 comp rx=0 tx=0).stop 2024-03-16T10:16:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.270+0000 7f091fd5b640 1 -- 172.21.15.77:0/2078449007 shutdown_connections 2024-03-16T10:16:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.270+0000 7f091fd5b640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09040a2800 0x7f09040a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.270+0000 7f091fd5b640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0918102290 0x7f091819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.270+0000 7f091fd5b640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09181018d0 0x7f09181980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.270+0000 7f091fd5b640 1 --2- 172.21.15.77:0/2078449007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09181006e0 0x7f0918197ba0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.270+0000 7f091fd5b640 1 -- 172.21.15.77:0/2078449007 >> 172.21.15.77:0/2078449007 conn(0x7f09180fbc70 msgr2=0x7f09180fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.270+0000 7f091fd5b640 1 -- 172.21.15.77:0/2078449007 shutdown_connections 2024-03-16T10:16:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.270+0000 7f091fd5b640 1 -- 172.21.15.77:0/2078449007 wait complete. 2024-03-16T10:16:57.281 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-16T10:16:57.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 -- 172.21.15.77:0/3616258903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7aec1018d0 msgr2=0x7f7aec101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 --2- 172.21.15.77:0/3616258903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7aec1018d0 0x7f7aec101d50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009860 tx=0x7f7ad402bb70 comp rx=0 tx=0).stop 2024-03-16T10:16:57.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 -- 172.21.15.77:0/3616258903 shutdown_connections 2024-03-16T10:16:57.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 --2- 172.21.15.77:0/3616258903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aec102290 0x7f7aec108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 --2- 172.21.15.77:0/3616258903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7aec1018d0 0x7f7aec101d50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 --2- 172.21.15.77:0/3616258903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec1006e0 0x7f7aec100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 -- 172.21.15.77:0/3616258903 >> 172.21.15.77:0/3616258903 conn(0x7f7aec0fbc70 msgr2=0x7f7aec0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 -- 172.21.15.77:0/3616258903 shutdown_connections 2024-03-16T10:16:57.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 -- 172.21.15.77:0/3616258903 wait complete. 2024-03-16T10:16:57.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 Processor -- start 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 -- start start 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aec1006e0 0x7f7aec104670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec1018d0 0x7f7aec102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7aec102290 0x7f7aec1031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec10c360 con 0x7f7aec1018d0 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec10c1e0 con 0x7f7aec1006e0 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.385+0000 7f7af15a9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec10c4e0 con 0x7f7aec102290 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7ae3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec1018d0 0x7f7aec102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7ae3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec1018d0 0x7f7aec102cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41890/0 (socket says 172.21.15.77:41890) 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7ae3fff640 1 -- 172.21.15.77:0/1115727547 learned_addr learned my addr 172.21.15.77:0/1115727547 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aec1006e0 0x7f7aec104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7aeb7fe640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7aec102290 0x7f7aec1031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7ae3fff640 1 -- 172.21.15.77:0/1115727547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7aec102290 msgr2=0x7f7aec1031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7ae3fff640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7aec102290 0x7f7aec1031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7ae3fff640 1 -- 172.21.15.77:0/1115727547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aec1006e0 msgr2=0x7f7aec104670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7ae3fff640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aec1006e0 0x7f7aec104670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7ae3fff640 1 -- 172.21.15.77:0/1115727547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aec103a80 con 0x7f7aec1018d0 2024-03-16T10:16:57.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7ae3fff640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec1018d0 0x7f7aec102cb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4005940 tx=0x7f7ad4032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7ae8ff9640 1 -- 172.21.15.77:0/1115727547 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4041020 con 0x7f7aec1018d0 2024-03-16T10:16:57.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7af15a9640 1 -- 172.21.15.77:0/1115727547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aec1a1c10 con 0x7f7aec1018d0 2024-03-16T10:16:57.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7ae8ff9640 1 -- 172.21.15.77:0/1115727547 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ad402c9f0 con 0x7f7aec1018d0 2024-03-16T10:16:57.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7ae8ff9640 1 -- 172.21.15.77:0/1115727547 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4039040 con 0x7f7aec1018d0 2024-03-16T10:16:57.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.386+0000 7f7af15a9640 1 -- 172.21.15.77:0/1115727547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aec1a2160 con 0x7f7aec1018d0 2024-03-16T10:16:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.388+0000 7f7ae8ff9640 1 -- 172.21.15.77:0/1115727547 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7ad4044020 con 0x7f7aec1018d0 2024-03-16T10:16:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7af4193740 1 Processor -- start 2024-03-16T10:16:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7af4193740 1 -- start start 2024-03-16T10:16:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7af4193740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 0x55ec33e9dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7af4193740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33ea30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7af4193740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33ea3730 0x55ec33ea5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7af4193740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ec33e01c80 con 0x55ec33e9dbd0 2024-03-16T10:16:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7af4193740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ec33daaaa0 con 0x55ec33ea3730 2024-03-16T10:16:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7af4193740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ec33dfdd20 con 0x55ec33e9e5c0 2024-03-16T10:16:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 0x55ec33e9dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 0x55ec33e9dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41900/0 (socket says 172.21.15.77:41900) 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7aeb7fe640 1 -- 172.21.15.77:0/3047200913 learned_addr learned my addr 172.21.15.77:0/3047200913 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/3047200913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33ea30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7ae8ff9640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7ad80a2800 0x7f7ad80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7ae3fff640 1 --2- 172.21.15.77:0/3047200913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33ea3730 0x55ec33ea5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7aeb7fe640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7ad80a2800 0x7f7ad80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7aeaffd640 1 -- 172.21.15.77:0/3047200913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33ea3730 msgr2=0x55ec33ea5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/3047200913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33ea3730 0x55ec33ea5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7aeaffd640 1 -- 172.21.15.77:0/3047200913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 msgr2=0x55ec33e9dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/3047200913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 0x55ec33e9dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7aeaffd640 1 -- 172.21.15.77:0/3047200913 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec33dccff0 con 0x55ec33e9e5c0 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7ae3fff640 1 --2- 172.21.15.77:0/3047200913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33ea3730 0x55ec33ea5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7aeb7fe640 1 --2- 172.21.15.77:0/3047200913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 0x55ec33e9dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.389+0000 7f7ae8ff9640 1 -- 172.21.15.77:0/1115727547 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7ad40f0310 con 0x7f7aec1018d0 2024-03-16T10:16:57.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.390+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/3047200913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33ea30c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7aec101730 tx=0x7f7adc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54c92b3173422e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.390+0000 7f7aeb7fe640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7ad80a2800 0x7f7ad80a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x55ec33e9ee20 tx=0x7f7ae4004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.390+0000 7f7ae17fa640 1 -- 172.21.15.77:0/3047200913 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc01e070 con 0x55ec33e9e5c0 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.390+0000 7f7ae17fa640 1 -- 172.21.15.77:0/3047200913 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7adc019910 con 0x55ec33e9e5c0 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.390+0000 7f7ae17fa640 1 -- 172.21.15.77:0/3047200913 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc00e9d0 con 0x55ec33e9e5c0 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.390+0000 7f7af4193740 1 -- 172.21.15.77:0/3047200913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 msgr2=0x55ec33ea30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.390+0000 7f7af4193740 1 --2- 172.21.15.77:0/3047200913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33ea30c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7aec101730 tx=0x7f7adc011810 comp rx=0 tx=0).stop 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 -- 172.21.15.77:0/3047200913 shutdown_connections 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 --2- 172.21.15.77:0/3047200913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33ea3730 0x55ec33ea5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 --2- 172.21.15.77:0/3047200913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33ea30c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 --2- 172.21.15.77:0/3047200913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 0x55ec33e9dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 -- 172.21.15.77:0/3047200913 >> 172.21.15.77:0/3047200913 conn(0x55ec33e99140 msgr2=0x55ec33e99540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 -- 172.21.15.77:0/3047200913 shutdown_connections 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 -- 172.21.15.77:0/3047200913 wait complete. 2024-03-16T10:16:57.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 -- start start 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 Processor -- start 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33ea8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 0x55ec33eaf390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ec33ea66e0 con 0x55ec33ea3730 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ec33dbdef0 con 0x55ec33e9dbd0 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ec33ea6860 con 0x55ec33e9e5c0 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7af4193740 10 client.0 _ll_register_callbacks cb 0x55ec33db8dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 0x55ec33eaf390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 0x55ec33eaf390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41908/0 (socket says 172.21.15.77:41908) 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7aeb7fe640 1 -- 172.21.15.77:0/3469306740 learned_addr learned my addr 172.21.15.77:0/3469306740 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7ae3fff640 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33ea8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.391+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7aeb7fe640 1 -- 172.21.15.77:0/3469306740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 msgr2=0x55ec33ea8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7aeb7fe640 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33ea8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7aeb7fe640 1 -- 172.21.15.77:0/3469306740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9dbd0 msgr2=0x55ec33ea7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7aeb7fe640 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7aeb7fe640 1 -- 172.21.15.77:0/3469306740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec33dcaa30 con 0x55ec33ea3730 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7aeb7fe640 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 0x55ec33eaf390 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40133f0 tx=0x7f7ae4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15dd6e7aa1b4821 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7ad27fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7ad27fc640 1 -- 172.21.15.77:0/3469306740 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4026070 con 0x55ec33ea3730 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7ad27fc640 1 -- 172.21.15.77:0/3469306740 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7ae4010d40 con 0x55ec33ea3730 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7af4193740 1 -- 172.21.15.77:0/3469306740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7adc019910 con 0x55ec33ea3730 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7ad27fc640 1 -- 172.21.15.77:0/3469306740 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7ae4031050 con 0x55ec33ea3730 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7af4193740 10 client.15315 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7af4193740 1 -- 172.21.15.77:0/3469306740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ec33b162d0 con 0x55ec33ea3730 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7ad27fc640 1 -- 172.21.15.77:0/3469306740 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7adc019910 con 0x55ec33ea3730 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.392+0000 7f7ad27fc640 1 -- 172.21.15.77:0/3469306740 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f7ae401f050 con 0x55ec33ea3730 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.393+0000 7f7af4193740 10 client.15315 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:57.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.393+0000 7f7af4193740 10 client.15315 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.393+0000 7f7af4193740 20 client.15315 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.393+0000 7f7af4193740 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ec33eb4910 0x55ec33eb5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.393+0000 7f7af4193740 4 client.15315 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.393+0000 7f7af4193740 1 -- 172.21.15.77:0/3469306740 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec33e0f1c0 con 0x55ec33eb4910 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.393+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ec33eb4910 0x55ec33eb5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.393+0000 7f7ad27fc640 10 client.15315 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.393+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ec33eb4910 0x55ec33eb5d80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ddbe0c1ed926d1e7 server_cookie=a9ea8a50a295e840 in_seq=0 out_seq=0 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7ad27fc640 1 -- 172.21.15.77:0/3469306740 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ec33e0f1c0 con 0x55ec33eb4910 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7ad27fc640 10 client.15315 handle_command_reply: tid=0 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7af4193740 1 client.15315 shutdown 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7af4193740 1 -- 172.21.15.77:0/3469306740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 msgr2=0x55ec33eaf390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7af4193740 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 0x55ec33eaf390 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40133f0 tx=0x7f7ae4004a40 comp rx=0 tx=0).stop 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7af4193740 1 -- 172.21.15.77:0/3469306740 shutdown_connections 2024-03-16T10:16:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7af4193740 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 0x55ec33eaf390 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7af4193740 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33ea8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7af4193740 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ec33eb4910 0x55ec33eb5d80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7af4193740 1 --2- 172.21.15.77:0/3469306740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7af4193740 1 -- 172.21.15.77:0/3469306740 >> 172.21.15.77:0/3469306740 conn(0x55ec33e99140 msgr2=0x55ec33e98380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7ad27fc640 0 client.15315 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:16:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7af4193740 1 -- 172.21.15.77:0/3469306740 shutdown_connections 2024-03-16T10:16:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.394+0000 7f7af4193740 1 -- 172.21.15.77:0/3469306740 wait complete. 2024-03-16T10:16:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.395+0000 7f7af4193740 20 client.15315 trim_cache size 0 max 16384 2024-03-16T10:16:57.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.412+0000 7f7af4193740 1 Processor -- start 2024-03-16T10:16:57.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.412+0000 7f7af4193740 1 -- start start 2024-03-16T10:16:57.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.412+0000 7f7af4193740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.412+0000 7f7af4193740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9e5c0 0x55ec33e99520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.412+0000 7f7af4193740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33ea3730 0x55ec33eaf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.412+0000 7f7af4193740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ec33eb40c0 con 0x55ec33e9dbd0 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.412+0000 7f7af4193740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ec33eb4790 con 0x55ec33e9e5c0 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.412+0000 7f7af4193740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ec33eb4e60 con 0x55ec33ea3730 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.412+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.412+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41922/0 (socket says 172.21.15.77:41922) 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.412+0000 7f7aeaffd640 1 -- 172.21.15.77:0/1582739559 learned_addr learned my addr 172.21.15.77:0/1582739559 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.412+0000 7f7aeb7fe640 1 --2- 172.21.15.77:0/1582739559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33ea3730 0x55ec33eaf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.413+0000 7f7ae3fff640 1 --2- 172.21.15.77:0/1582739559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9e5c0 0x55ec33e99520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.413+0000 7f7aeaffd640 1 -- 172.21.15.77:0/1582739559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33ea3730 msgr2=0x55ec33eaf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.413+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/1582739559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33ea3730 0x55ec33eaf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.413+0000 7f7aeaffd640 1 -- 172.21.15.77:0/1582739559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9e5c0 msgr2=0x55ec33e99520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.413+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/1582739559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9e5c0 0x55ec33e99520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.413+0000 7f7aeaffd640 1 -- 172.21.15.77:0/1582739559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec33be06b0 con 0x55ec33e9dbd0 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.413+0000 7f7aeb7fe640 1 --2- 172.21.15.77:0/1582739559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33ea3730 0x55ec33eaf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.413+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/1582739559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea2dd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7adc01c040 tx=0x7f7adc009db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3596af539b24e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.413+0000 7f7ae0ff9640 1 -- 172.21.15.77:0/1582739559 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc02f070 con 0x55ec33e9dbd0 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.413+0000 7f7ae0ff9640 1 -- 172.21.15.77:0/1582739559 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7adc004640 con 0x55ec33e9dbd0 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.413+0000 7f7ae0ff9640 1 -- 172.21.15.77:0/1582739559 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc01e040 con 0x55ec33e9dbd0 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 -- 172.21.15.77:0/1582739559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 msgr2=0x55ec33ea2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 --2- 172.21.15.77:0/1582739559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea2dd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7adc01c040 tx=0x7f7adc009db0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 -- 172.21.15.77:0/1582739559 shutdown_connections 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 --2- 172.21.15.77:0/1582739559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33ea3730 0x55ec33eaf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 --2- 172.21.15.77:0/1582739559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9e5c0 0x55ec33e99520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 --2- 172.21.15.77:0/1582739559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea2dd0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 -- 172.21.15.77:0/1582739559 >> 172.21.15.77:0/1582739559 conn(0x55ec33e99140 msgr2=0x55ec33eafa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 -- 172.21.15.77:0/1582739559 shutdown_connections 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 -- 172.21.15.77:0/1582739559 wait complete. 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 -- start start 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 Processor -- start 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33eaf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 0x55ec33eaa340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ec33eb5c00 con 0x55ec33ea3730 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ec33eb5530 con 0x55ec33e9dbd0 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ec33eb62d0 con 0x55ec33e9e5c0 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7af4193740 10 client.0 _ll_register_callbacks cb 0x7f7ae4003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 0x55ec33eaa340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 0x55ec33eaa340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41934/0 (socket says 172.21.15.77:41934) 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7aeb7fe640 1 -- 172.21.15.77:0/1349847894 learned_addr learned my addr 172.21.15.77:0/1349847894 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.414+0000 7f7ae3fff640 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33eaf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7aeb7fe640 1 -- 172.21.15.77:0/1349847894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 msgr2=0x55ec33eaf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7aeb7fe640 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33eaf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7aeb7fe640 1 -- 172.21.15.77:0/1349847894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9dbd0 msgr2=0x55ec33ea9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7aeb7fe640 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7aeb7fe640 1 -- 172.21.15.77:0/1349847894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec33f18670 con 0x55ec33ea3730 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7aeb7fe640 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 0x55ec33eaa340 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009c20 tx=0x7f7ae4009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22c9c3e74dc3b95 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7ad27fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7ad27fc640 1 -- 172.21.15.77:0/1349847894 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4039020 con 0x55ec33ea3730 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7ad27fc640 1 -- 172.21.15.77:0/1349847894 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7ae40217d0 con 0x55ec33ea3730 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7af4193740 1 -- 172.21.15.77:0/1349847894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7adc004640 con 0x55ec33ea3730 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7ad27fc640 1 -- 172.21.15.77:0/1349847894 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7ae4044050 con 0x55ec33ea3730 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7af4193740 10 client.15321 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:57.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7af4193740 1 -- 172.21.15.77:0/1349847894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ec33f19960 con 0x55ec33ea3730 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7ad27fc640 1 -- 172.21.15.77:0/1349847894 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7adc004640 con 0x55ec33ea3730 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7ad27fc640 1 -- 172.21.15.77:0/1349847894 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f7ae401f050 con 0x55ec33ea3730 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7af4193740 10 client.15321 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7af4193740 10 client.15321 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.415+0000 7f7af4193740 20 client.15321 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ec33f1e610 0x55ec33f1d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 4 client.15321 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 1 -- 172.21.15.77:0/1349847894 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "status"}) -- 0x55ec33e09510 con 0x55ec33f1e610 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ec33f1e610 0x55ec33f1d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7aeaffd640 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ec33f1e610 0x55ec33f1d320 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2e9a1678520adb6f server_cookie=af8b4220811a8026 in_seq=0 out_seq=0 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7ad27fc640 10 client.15321 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7ad27fc640 1 -- 172.21.15.77:0/1349847894 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55ec33e09510 con 0x55ec33f1e610 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7ad27fc640 10 client.15321 handle_command_reply: tid=0 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 1 client.15321 shutdown 2024-03-16T10:16:57.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 1 -- 172.21.15.77:0/1349847894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 msgr2=0x55ec33eaa340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 0x55ec33eaa340 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009c20 tx=0x7f7ae4009ac0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 1 -- 172.21.15.77:0/1349847894 shutdown_connections 2024-03-16T10:16:57.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ec33ea3730 0x55ec33eaa340 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec33e9e5c0 0x55ec33eaf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ec33f1e610 0x55ec33f1d320 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 1 --2- 172.21.15.77:0/1349847894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ec33e9dbd0 0x55ec33ea9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 1 -- 172.21.15.77:0/1349847894 >> 172.21.15.77:0/1349847894 conn(0x55ec33e99140 msgr2=0x55ec33e9c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7ad27fc640 0 client.15321 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:16:57.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.416+0000 7f7af4193740 1 -- 172.21.15.77:0/1349847894 shutdown_connections 2024-03-16T10:16:57.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af4193740 1 -- 172.21.15.77:0/1349847894 wait complete. 2024-03-16T10:16:57.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af4193740 20 client.15321 trim_cache size 0 max 16384 2024-03-16T10:16:57.425 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:16:57.426 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:16:57.426 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 2, 2024-03-16T10:16:57.426 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24455, 2024-03-16T10:16:57.426 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T10:16:57.426 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T10:16:57.426 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T10:16:57.426 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 111.173741591, 2024-03-16T10:16:57.426 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 26, 2024-03-16T10:16:57.426 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 87, 2024-03-16T10:16:57.426 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T10:16:57.426 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 132.766514554 2024-03-16T10:16:57.426 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:16:57.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af15a9640 1 -- 172.21.15.77:0/1115727547 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7ad80a2800 msgr2=0x7f7ad80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af15a9640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7ad80a2800 0x7f7ad80a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x55ec33e9ee20 tx=0x7f7ae4004ab0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af15a9640 1 -- 172.21.15.77:0/1115727547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec1018d0 msgr2=0x7f7aec102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af15a9640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec1018d0 0x7f7aec102cb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4005940 tx=0x7f7ad4032470 comp rx=0 tx=0).stop 2024-03-16T10:16:57.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af15a9640 1 -- 172.21.15.77:0/1115727547 shutdown_connections 2024-03-16T10:16:57.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af15a9640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7ad80a2800 0x7f7ad80a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af15a9640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7aec102290 0x7f7aec1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af15a9640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7aec1018d0 0x7f7aec102cb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af15a9640 1 --2- 172.21.15.77:0/1115727547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7aec1006e0 0x7f7aec104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af15a9640 1 -- 172.21.15.77:0/1115727547 >> 172.21.15.77:0/1115727547 conn(0x7f7aec0fbc70 msgr2=0x7f7aec0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af15a9640 1 -- 172.21.15.77:0/1115727547 shutdown_connections 2024-03-16T10:16:57.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.417+0000 7f7af15a9640 1 -- 172.21.15.77:0/1115727547 wait complete. 2024-03-16T10:16:57.429 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T10:16:57.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.535+0000 7f201cd60640 1 -- 172.21.15.77:0/4034622350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2018102290 msgr2=0x7f2018108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.535+0000 7f201cd60640 1 --2- 172.21.15.77:0/4034622350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2018102290 0x7f2018108b20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2008009900 tx=0x7f200802bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:57.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.535+0000 7f201cd60640 1 -- 172.21.15.77:0/4034622350 shutdown_connections 2024-03-16T10:16:57.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.535+0000 7f201cd60640 1 --2- 172.21.15.77:0/4034622350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2018102290 0x7f2018108b20 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.535+0000 7f201cd60640 1 --2- 172.21.15.77:0/4034622350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20181018d0 0x7f2018101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.535+0000 7f201cd60640 1 --2- 172.21.15.77:0/4034622350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20181006e0 0x7f2018100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.535+0000 7f201cd60640 1 -- 172.21.15.77:0/4034622350 >> 172.21.15.77:0/4034622350 conn(0x7f20180fbc70 msgr2=0x7f20180fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.535+0000 7f201cd60640 1 -- 172.21.15.77:0/4034622350 shutdown_connections 2024-03-16T10:16:57.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.535+0000 7f201cd60640 1 -- 172.21.15.77:0/4034622350 wait complete. 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f201cd60640 1 Processor -- start 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f201cd60640 1 -- start start 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f201cd60640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20181006e0 0x7f2018197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f201cd60640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20181018d0 0x7f2018197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f201cd60640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2018102290 0x7f201819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f201cd60640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f201810c340 con 0x7f2018102290 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f201cd60640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f201810c1c0 con 0x7f20181006e0 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f201cd60640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f201810c4c0 con 0x7f20181018d0 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f2016d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2018102290 0x7f201819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f2016d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2018102290 0x7f201819f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41946/0 (socket says 172.21.15.77:41946) 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f2016d9d640 1 -- 172.21.15.77:0/945356987 learned_addr learned my addr 172.21.15.77:0/945356987 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f2015d9b640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20181018d0 0x7f2018197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f201659c640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20181006e0 0x7f2018197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f2016d9d640 1 -- 172.21.15.77:0/945356987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20181018d0 msgr2=0x7f2018197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f2016d9d640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20181018d0 0x7f2018197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f2016d9d640 1 -- 172.21.15.77:0/945356987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20181006e0 msgr2=0x7f2018197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f2016d9d640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20181006e0 0x7f2018197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f2016d9d640 1 -- 172.21.15.77:0/945356987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201819f6d0 con 0x7f2018102290 2024-03-16T10:16:57.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.536+0000 7f2016d9d640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2018102290 0x7f201819f060 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2008005ab0 tx=0x7f200802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.537+0000 7f20077fe640 1 -- 172.21.15.77:0/945356987 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008041070 con 0x7f2018102290 2024-03-16T10:16:57.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.537+0000 7f201cd60640 1 -- 172.21.15.77:0/945356987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201819f960 con 0x7f2018102290 2024-03-16T10:16:57.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.537+0000 7f20077fe640 1 -- 172.21.15.77:0/945356987 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2008004450 con 0x7f2018102290 2024-03-16T10:16:57.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.537+0000 7f20077fe640 1 -- 172.21.15.77:0/945356987 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008003e60 con 0x7f2018102290 2024-03-16T10:16:57.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.537+0000 7f201cd60640 1 -- 172.21.15.77:0/945356987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f201819fea0 con 0x7f2018102290 2024-03-16T10:16:57.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.539+0000 7f20077fe640 1 -- 172.21.15.77:0/945356987 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2008044070 con 0x7f2018102290 2024-03-16T10:16:57.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.539+0000 7f201fa20740 1 Processor -- start 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.539+0000 7f201fa20740 1 -- start start 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.539+0000 7f20077fe640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f20000a2800 0x7f20000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201fa20740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b4bd0 0x5599055b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201fa20740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055b55c0 0x5599055ba0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201fa20740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055ba730 0x5599055bcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f2015d9b640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f20000a2800 0x7f20000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201fa20740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559905518c80 con 0x5599055b55c0 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201fa20740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5599054c1aa0 con 0x5599055b4bd0 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201fa20740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559905514d20 con 0x5599055ba730 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f20077fe640 1 -- 172.21.15.77:0/945356987 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f20080f05f0 con 0x7f2018102290 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b4bd0 0x5599055b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b4bd0 0x5599055b4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:39572/0 (socket says 172.21.15.77:39572) 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201659c640 1 -- 172.21.15.77:0/1687746626 learned_addr learned my addr 172.21.15.77:0/1687746626 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f2016d9d640 1 --2- 172.21.15.77:0/1687746626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055ba730 0x5599055bcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f2015d9b640 1 --2- 172.21.15.77:0/1687746626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055b55c0 0x5599055ba0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201659c640 1 -- 172.21.15.77:0/1687746626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055ba730 msgr2=0x5599055bcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201659c640 1 --2- 172.21.15.77:0/1687746626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055ba730 0x5599055bcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201659c640 1 -- 172.21.15.77:0/1687746626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055b55c0 msgr2=0x5599055ba0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201659c640 1 --2- 172.21.15.77:0/1687746626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055b55c0 0x5599055ba0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201659c640 1 -- 172.21.15.77:0/1687746626 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599054e3ff0 con 0x5599055b4bd0 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f2016d9d640 1 --2- 172.21.15.77:0/1687746626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055ba730 0x5599055bcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f2015d9b640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f20000a2800 0x7f20000a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2018198fc0 tx=0x7f200c013be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f201659c640 1 --2- 172.21.15.77:0/1687746626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b4bd0 0x5599055b4ff0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2010009ca0 tx=0x7f2010011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54f0d6144b2bbd27 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f2004ff9640 1 -- 172.21.15.77:0/1687746626 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2010009e50 con 0x5599055b4bd0 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f2004ff9640 1 -- 172.21.15.77:0/1687746626 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20100035f0 con 0x5599055b4bd0 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.540+0000 7f2004ff9640 1 -- 172.21.15.77:0/1687746626 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2010019650 con 0x5599055b4bd0 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.541+0000 7f201fa20740 1 -- 172.21.15.77:0/1687746626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b4bd0 msgr2=0x5599055b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.541+0000 7f201fa20740 1 --2- 172.21.15.77:0/1687746626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b4bd0 0x5599055b4ff0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2010009ca0 tx=0x7f2010011810 comp rx=0 tx=0).stop 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.541+0000 7f201fa20740 1 -- 172.21.15.77:0/1687746626 shutdown_connections 2024-03-16T10:16:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.541+0000 7f201fa20740 1 --2- 172.21.15.77:0/1687746626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055ba730 0x5599055bcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.541+0000 7f201fa20740 1 --2- 172.21.15.77:0/1687746626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055b55c0 0x5599055ba0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.541+0000 7f201fa20740 1 --2- 172.21.15.77:0/1687746626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b4bd0 0x5599055b4ff0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.541+0000 7f201fa20740 1 -- 172.21.15.77:0/1687746626 >> 172.21.15.77:0/1687746626 conn(0x5599055b0140 msgr2=0x5599055b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.541+0000 7f201fa20740 1 -- 172.21.15.77:0/1687746626 shutdown_connections 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.541+0000 7f201fa20740 1 -- 172.21.15.77:0/1687746626 wait complete. 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.541+0000 7f201fa20740 1 -- start start 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f201fa20740 1 Processor -- start 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f201fa20740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b4bd0 0x5599055bed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f201fa20740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b55c0 0x5599055bf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f201fa20740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 0x5599055c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f201fa20740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5599055bd6e0 con 0x5599055ba730 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f201fa20740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5599054d4ef0 con 0x5599055b55c0 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f201fa20740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5599055bd860 con 0x5599055b4bd0 2024-03-16T10:16:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f201fa20740 10 client.0 _ll_register_callbacks cb 0x5599054cfdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f2016d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 0x5599055c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f201659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b4bd0 0x5599055bed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f2016d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 0x5599055c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41974/0 (socket says 172.21.15.77:41974) 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f2016d9d640 1 -- 172.21.15.77:0/3840195566 learned_addr learned my addr 172.21.15.77:0/3840195566 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f2015d9b640 1 --2- 172.21.15.77:0/3840195566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b55c0 0x5599055bf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f2016d9d640 1 -- 172.21.15.77:0/3840195566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b4bd0 msgr2=0x5599055bed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f2016d9d640 1 --2- 172.21.15.77:0/3840195566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b4bd0 0x5599055bed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f2016d9d640 1 -- 172.21.15.77:0/3840195566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b55c0 msgr2=0x5599055bf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f2016d9d640 1 --2- 172.21.15.77:0/3840195566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b55c0 0x5599055bf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f2016d9d640 1 -- 172.21.15.77:0/3840195566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599054e1a30 con 0x5599055ba730 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f2016d9d640 1 --2- 172.21.15.77:0/3840195566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 0x5599055c62d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f20080b9530 tx=0x7f20080bcaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e897b658dfb076 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f1ffdffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f1ffdffb640 1 -- 172.21.15.77:0/3840195566 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200804b040 con 0x5599055ba730 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f1ffdffb640 1 -- 172.21.15.77:0/3840195566 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f20080c5330 con 0x5599055ba730 2024-03-16T10:16:57.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.542+0000 7f201fa20740 1 -- 172.21.15.77:0/3840195566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20100035f0 con 0x5599055ba730 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.543+0000 7f1ffdffb640 1 -- 172.21.15.77:0/3840195566 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f20080c57c0 con 0x5599055ba730 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.543+0000 7f201fa20740 10 client.15333 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.543+0000 7f201fa20740 1 -- 172.21.15.77:0/3840195566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55990522d2d0 con 0x5599055ba730 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.543+0000 7f1ffdffb640 1 -- 172.21.15.77:0/3840195566 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20100035f0 con 0x5599055ba730 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.543+0000 7f1ffdffb640 1 -- 172.21.15.77:0/3840195566 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f20080f00f0 con 0x5599055ba730 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.543+0000 7f201fa20740 10 client.15333 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.543+0000 7f201fa20740 10 client.15333 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.543+0000 7f201fa20740 20 client.15333 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.543+0000 7f201fa20740 1 --2- 172.21.15.77:0/3840195566 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x5599055cb7a0 0x5599055ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.543+0000 7f201fa20740 4 client.15333 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.543+0000 7f201fa20740 1 -- 172.21.15.77:0/3840195566 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599055261c0 con 0x5599055cb7a0 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.543+0000 7f201659c640 1 --2- 172.21.15.77:0/3840195566 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x5599055cb7a0 0x5599055ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201659c640 1 --2- 172.21.15.77:0/3840195566 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x5599055cb7a0 0x5599055ccc10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ff96a5da786e98f7 server_cookie=8a15683a5e886998 in_seq=0 out_seq=0 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f1ffdffb640 10 client.15333 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:16:57.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f1ffdffb640 1 -- 172.21.15.77:0/3840195566 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5599055261c0 con 0x5599055cb7a0 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f1ffdffb640 10 client.15333 handle_command_reply: tid=0 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201fa20740 1 client.15333 shutdown 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201fa20740 1 -- 172.21.15.77:0/3840195566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 msgr2=0x5599055c62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201fa20740 1 --2- 172.21.15.77:0/3840195566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 0x5599055c62d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f20080b9530 tx=0x7f20080bcaf0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201fa20740 1 -- 172.21.15.77:0/3840195566 shutdown_connections 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201fa20740 1 --2- 172.21.15.77:0/3840195566 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x5599055cb7a0 0x5599055ccc10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201fa20740 1 --2- 172.21.15.77:0/3840195566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 0x5599055c62d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f1ffdffb640 0 client.15333 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201fa20740 1 --2- 172.21.15.77:0/3840195566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b55c0 0x5599055bf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201fa20740 1 --2- 172.21.15.77:0/3840195566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b4bd0 0x5599055bed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201fa20740 1 -- 172.21.15.77:0/3840195566 >> 172.21.15.77:0/3840195566 conn(0x5599055b0140 msgr2=0x5599055af380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201fa20740 1 -- 172.21.15.77:0/3840195566 shutdown_connections 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201fa20740 1 -- 172.21.15.77:0/3840195566 wait complete. 2024-03-16T10:16:57.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.544+0000 7f201fa20740 20 client.15333 trim_cache size 0 max 16384 2024-03-16T10:16:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f201fa20740 1 Processor -- start 2024-03-16T10:16:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f201fa20740 1 -- start start 2024-03-16T10:16:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f201fa20740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b4bd0 0x5599055ca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f201fa20740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b55c0 0x5599055ca970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f201fa20740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 0x5599055c6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f201fa20740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5599055caf50 con 0x5599055ba730 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f201fa20740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5599055cb620 con 0x5599055b4bd0 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f201fa20740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5599055cbcf0 con 0x5599055b55c0 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f2016d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 0x5599055c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f2016d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 0x5599055c6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41984/0 (socket says 172.21.15.77:41984) 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f2016d9d640 1 -- 172.21.15.77:0/1444294471 learned_addr learned my addr 172.21.15.77:0/1444294471 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f201659c640 1 --2- 172.21.15.77:0/1444294471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b4bd0 0x5599055ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.565+0000 7f2015d9b640 1 --2- 172.21.15.77:0/1444294471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b55c0 0x5599055ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f2016d9d640 1 -- 172.21.15.77:0/1444294471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b55c0 msgr2=0x5599055ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f2016d9d640 1 --2- 172.21.15.77:0/1444294471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b55c0 0x5599055ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f2016d9d640 1 -- 172.21.15.77:0/1444294471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b4bd0 msgr2=0x5599055ca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f2016d9d640 1 --2- 172.21.15.77:0/1444294471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b4bd0 0x5599055ca430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f2016d9d640 1 -- 172.21.15.77:0/1444294471 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599052f76b0 con 0x5599055ba730 2024-03-16T10:16:57.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f2016d9d640 1 --2- 172.21.15.77:0/1444294471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 0x5599055c6290 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f20080f0910 tx=0x7f20080c12f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a352488c5100dbb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f1ffffff640 1 -- 172.21.15.77:0/1444294471 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200804b070 con 0x5599055ba730 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f1ffffff640 1 -- 172.21.15.77:0/1444294471 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20080f00f0 con 0x5599055ba730 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f1ffffff640 1 -- 172.21.15.77:0/1444294471 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20080c5590 con 0x5599055ba730 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f201fa20740 1 -- 172.21.15.77:0/1444294471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 msgr2=0x5599055c6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f201fa20740 1 --2- 172.21.15.77:0/1444294471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 0x5599055c6290 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f20080f0910 tx=0x7f20080c12f0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f201fa20740 1 -- 172.21.15.77:0/1444294471 shutdown_connections 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f201fa20740 1 --2- 172.21.15.77:0/1444294471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055ba730 0x5599055c6290 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f201fa20740 1 --2- 172.21.15.77:0/1444294471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b55c0 0x5599055ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f201fa20740 1 --2- 172.21.15.77:0/1444294471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055b4bd0 0x5599055ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.566+0000 7f201fa20740 1 -- 172.21.15.77:0/1444294471 >> 172.21.15.77:0/1444294471 conn(0x5599055b0140 msgr2=0x5599055bed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f201fa20740 1 -- 172.21.15.77:0/1444294471 shutdown_connections 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f201fa20740 1 -- 172.21.15.77:0/1444294471 wait complete. 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f201fa20740 1 -- start start 2024-03-16T10:16:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f201fa20740 1 Processor -- start 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f201fa20740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b4bd0 0x5599055c4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f201fa20740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055b55c0 0x55990562d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f201fa20740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055ba730 0x55990562fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f201fa20740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5599055cca90 con 0x5599055b55c0 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f201fa20740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5599055cc3c0 con 0x5599055ba730 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f201fa20740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5599055cd160 con 0x5599055b4bd0 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f201fa20740 10 client.0 _ll_register_callbacks cb 0x7f20080b9460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f2015d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055b55c0 0x55990562d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f201659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b4bd0 0x5599055c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f2015d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055b55c0 0x55990562d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42000/0 (socket says 172.21.15.77:42000) 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f2015d9b640 1 -- 172.21.15.77:0/1277687698 learned_addr learned my addr 172.21.15.77:0/1277687698 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f2016d9d640 1 --2- 172.21.15.77:0/1277687698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055ba730 0x55990562fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f2015d9b640 1 -- 172.21.15.77:0/1277687698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b4bd0 msgr2=0x5599055c4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f2015d9b640 1 --2- 172.21.15.77:0/1277687698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b4bd0 0x5599055c4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f2015d9b640 1 -- 172.21.15.77:0/1277687698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055ba730 msgr2=0x55990562fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f2015d9b640 1 --2- 172.21.15.77:0/1277687698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055ba730 0x55990562fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.567+0000 7f2015d9b640 1 -- 172.21.15.77:0/1277687698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599055b9500 con 0x5599055b55c0 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f2015d9b640 1 --2- 172.21.15.77:0/1277687698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055b55c0 0x55990562d4f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f200c01d000 tx=0x7f200c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44026314e1656dc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f1ffdffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f1ffdffb640 1 -- 172.21.15.77:0/1277687698 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c004720 con 0x5599055b55c0 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f1ffdffb640 1 -- 172.21.15.77:0/1277687698 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f200c02a480 con 0x5599055b55c0 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f201fa20740 1 -- 172.21.15.77:0/1277687698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20080f00f0 con 0x5599055b55c0 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f1ffdffb640 1 -- 172.21.15.77:0/1277687698 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f200c02a680 con 0x5599055b55c0 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f201fa20740 10 client.15339 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f201fa20740 1 -- 172.21.15.77:0/1277687698 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559905631100 con 0x5599055b55c0 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f1ffdffb640 1 -- 172.21.15.77:0/1277687698 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20080f00f0 con 0x5599055b55c0 2024-03-16T10:16:57.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f1ffdffb640 1 -- 172.21.15.77:0/1277687698 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f200c01f070 con 0x5599055b55c0 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f201fa20740 10 client.15339 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f201fa20740 10 client.15339 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f201fa20740 20 client.15339 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f201fa20740 1 --2- 172.21.15.77:0/1277687698 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x559905635dd0 0x559905632210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f201fa20740 4 client.15339 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f201fa20740 1 -- 172.21.15.77:0/1277687698 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "status"}) -- 0x559905520510 con 0x559905635dd0 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.568+0000 7f201659c640 1 --2- 172.21.15.77:0/1277687698 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x559905635dd0 0x559905632210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201659c640 1 --2- 172.21.15.77:0/1277687698 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x559905635dd0 0x559905632210 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ee20576a62188aae server_cookie=dad737c863c52a6c in_seq=0 out_seq=0 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f1ffdffb640 10 client.15339 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f1ffdffb640 1 -- 172.21.15.77:0/1277687698 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x559905520510 con 0x559905635dd0 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f1ffdffb640 10 client.15339 handle_command_reply: tid=0 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201fa20740 1 client.15339 shutdown 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201fa20740 1 -- 172.21.15.77:0/1277687698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055b55c0 msgr2=0x55990562d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201fa20740 1 --2- 172.21.15.77:0/1277687698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055b55c0 0x55990562d4f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f200c01d000 tx=0x7f200c004ab0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201fa20740 1 -- 172.21.15.77:0/1277687698 shutdown_connections 2024-03-16T10:16:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201fa20740 1 --2- 172.21.15.77:0/1277687698 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x559905635dd0 0x559905632210 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201fa20740 1 --2- 172.21.15.77:0/1277687698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599055ba730 0x55990562fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f1ffdffb640 0 client.15339 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:16:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201fa20740 1 --2- 172.21.15.77:0/1277687698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599055b55c0 0x55990562d4f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201fa20740 1 --2- 172.21.15.77:0/1277687698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599055b4bd0 0x5599055c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201fa20740 1 -- 172.21.15.77:0/1277687698 >> 172.21.15.77:0/1277687698 conn(0x5599055b0140 msgr2=0x5599055bed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201fa20740 1 -- 172.21.15.77:0/1277687698 shutdown_connections 2024-03-16T10:16:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201fa20740 1 -- 172.21.15.77:0/1277687698 wait complete. 2024-03-16T10:16:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.569+0000 7f201fa20740 20 client.15339 trim_cache size 0 max 16384 2024-03-16T10:16:57.577 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:16:57.577 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:16:57.577 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 1, 2024-03-16T10:16:57.578 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24475, 2024-03-16T10:16:57.578 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T10:16:57.578 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T10:16:57.578 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T10:16:57.578 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 112.33132680200001, 2024-03-16T10:16:57.578 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 26, 2024-03-16T10:16:57.578 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 87, 2024-03-16T10:16:57.578 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T10:16:57.578 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 130.11269380600001 2024-03-16T10:16:57.578 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:16:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.570+0000 7f201cd60640 1 -- 172.21.15.77:0/945356987 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f20000a2800 msgr2=0x7f20000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.570+0000 7f201cd60640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f20000a2800 0x7f20000a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2018198fc0 tx=0x7f200c013be0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.570+0000 7f201cd60640 1 -- 172.21.15.77:0/945356987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2018102290 msgr2=0x7f201819f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.570+0000 7f201cd60640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2018102290 0x7f201819f060 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2008005ab0 tx=0x7f200802c6c0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.570+0000 7f201cd60640 1 -- 172.21.15.77:0/945356987 shutdown_connections 2024-03-16T10:16:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.570+0000 7f201cd60640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2018102290 0x7f201819f060 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.570+0000 7f201cd60640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f20000a2800 0x7f20000a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.570+0000 7f201cd60640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20181018d0 0x7f2018197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.570+0000 7f201cd60640 1 --2- 172.21.15.77:0/945356987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20181006e0 0x7f2018197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.570+0000 7f201cd60640 1 -- 172.21.15.77:0/945356987 >> 172.21.15.77:0/945356987 conn(0x7f20180fbc70 msgr2=0x7f20180fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.570+0000 7f201cd60640 1 -- 172.21.15.77:0/945356987 shutdown_connections 2024-03-16T10:16:57.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.570+0000 7f201cd60640 1 -- 172.21.15.77:0/945356987 wait complete. 2024-03-16T10:16:57.581 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T10:16:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.686+0000 7f94a4510640 1 -- 172.21.15.77:0/3465924097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949c1018d0 msgr2=0x7f949c101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.686+0000 7f94a4510640 1 --2- 172.21.15.77:0/3465924097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949c1018d0 0x7f949c101d50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9498009900 tx=0x7f949802bc30 comp rx=0 tx=0).stop 2024-03-16T10:16:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.686+0000 7f94a4510640 1 -- 172.21.15.77:0/3465924097 shutdown_connections 2024-03-16T10:16:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.686+0000 7f94a4510640 1 --2- 172.21.15.77:0/3465924097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f949c102290 0x7f949c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.686+0000 7f94a4510640 1 --2- 172.21.15.77:0/3465924097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949c1018d0 0x7f949c101d50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a4510640 1 --2- 172.21.15.77:0/3465924097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f949c1006e0 0x7f949c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a4510640 1 -- 172.21.15.77:0/3465924097 >> 172.21.15.77:0/3465924097 conn(0x7f949c0fbc70 msgr2=0x7f949c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a4510640 1 -- 172.21.15.77:0/3465924097 shutdown_connections 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a4510640 1 -- 172.21.15.77:0/3465924097 wait complete. 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a4510640 1 Processor -- start 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a4510640 1 -- start start 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a4510640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f949c1006e0 0x7f949c0694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a4510640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f949c1018d0 0x7f949c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a4510640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949c102290 0x7f949c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a4510640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f949c10c360 con 0x7f949c102290 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a4510640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f949c10c1e0 con 0x7f949c1006e0 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a4510640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f949c10c4e0 con 0x7f949c1018d0 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a2aad640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949c102290 0x7f949c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a22ac640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f949c1006e0 0x7f949c0694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a2aad640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949c102290 0x7f949c19eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42030/0 (socket says 172.21.15.77:42030) 2024-03-16T10:16:57.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a2aad640 1 -- 172.21.15.77:0/3758952435 learned_addr learned my addr 172.21.15.77:0/3758952435 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.687+0000 7f94a1aab640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f949c1018d0 0x7f949c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.688+0000 7f94a22ac640 1 -- 172.21.15.77:0/3758952435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f949c1018d0 msgr2=0x7f949c19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.688+0000 7f94a22ac640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f949c1018d0 0x7f949c19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.688+0000 7f94a22ac640 1 -- 172.21.15.77:0/3758952435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949c102290 msgr2=0x7f949c19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.688+0000 7f94a22ac640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949c102290 0x7f949c19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.688+0000 7f94a22ac640 1 -- 172.21.15.77:0/3758952435 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949c19f780 con 0x7f949c1006e0 2024-03-16T10:16:57.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.688+0000 7f94a2aad640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949c102290 0x7f949c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:57.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.688+0000 7f94a22ac640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f949c1006e0 0x7f949c0694d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f949000e9c0 tx=0x7f949000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.688+0000 7f948f7fe640 1 -- 172.21.15.77:0/3758952435 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949000ccc0 con 0x7f949c1006e0 2024-03-16T10:16:57.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.688+0000 7f948f7fe640 1 -- 172.21.15.77:0/3758952435 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94900035f0 con 0x7f949c1006e0 2024-03-16T10:16:57.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.688+0000 7f94a4510640 1 -- 172.21.15.77:0/3758952435 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949c19fa70 con 0x7f949c1006e0 2024-03-16T10:16:57.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.688+0000 7f948f7fe640 1 -- 172.21.15.77:0/3758952435 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949000ce60 con 0x7f949c1006e0 2024-03-16T10:16:57.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.688+0000 7f94a4510640 1 -- 172.21.15.77:0/3758952435 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f949c19ff40 con 0x7f949c1006e0 2024-03-16T10:16:57.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a71da740 1 Processor -- start 2024-03-16T10:16:57.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a71da740 1 -- start start 2024-03-16T10:16:57.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a71da740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e493400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a71da740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e4939d0 0x55b54e4984d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a71da740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 0x55b54e49afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a71da740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b54e3f2c80 con 0x55b54e498b40 2024-03-16T10:16:57.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a71da740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55b54e39baa0 con 0x55b54e492fe0 2024-03-16T10:16:57.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a71da740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b54e3eed20 con 0x55b54e4939d0 2024-03-16T10:16:57.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f948f7fe640 1 -- 172.21.15.77:0/3758952435 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f94900037a0 con 0x7f949c1006e0 2024-03-16T10:16:57.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a1aab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 0x55b54e49afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a1aab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 0x55b54e49afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42034/0 (socket says 172.21.15.77:42034) 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a1aab640 1 -- 172.21.15.77:0/4118294611 learned_addr learned my addr 172.21.15.77:0/4118294611 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a2aad640 1 --2- 172.21.15.77:0/4118294611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e493400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a22ac640 1 --2- 172.21.15.77:0/4118294611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e4939d0 0x55b54e4984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a1aab640 1 -- 172.21.15.77:0/4118294611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e4939d0 msgr2=0x55b54e4984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a1aab640 1 --2- 172.21.15.77:0/4118294611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e4939d0 0x55b54e4984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a1aab640 1 -- 172.21.15.77:0/4118294611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 msgr2=0x55b54e493400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a1aab640 1 --2- 172.21.15.77:0/4118294611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e493400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a1aab640 1 -- 172.21.15.77:0/4118294611 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54e3bdff0 con 0x55b54e498b40 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a22ac640 1 --2- 172.21.15.77:0/4118294611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e4939d0 0x55b54e4984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.691+0000 7f94a2aad640 1 --2- 172.21.15.77:0/4118294611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e493400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.692+0000 7f94a1aab640 1 --2- 172.21.15.77:0/4118294611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 0x55b54e49afd0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9498009900 tx=0x7f9498009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca8768e9d60f594 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.692+0000 7f948f7fe640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94880a2800 0x7f94880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.692+0000 7f948cbfe640 1 -- 172.21.15.77:0/4118294611 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949802cb80 con 0x55b54e498b40 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.692+0000 7f948cbfe640 1 -- 172.21.15.77:0/4118294611 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9498005040 con 0x55b54e498b40 2024-03-16T10:16:57.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.692+0000 7f948cbfe640 1 -- 172.21.15.77:0/4118294611 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498004920 con 0x55b54e498b40 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.692+0000 7f94a2aad640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94880a2800 0x7f94880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.692+0000 7f948f7fe640 1 -- 172.21.15.77:0/3758952435 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f94900cef10 con 0x7f949c1006e0 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 -- 172.21.15.77:0/4118294611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 msgr2=0x55b54e49afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 --2- 172.21.15.77:0/4118294611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 0x55b54e49afd0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9498009900 tx=0x7f9498009560 comp rx=0 tx=0).stop 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a2aad640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94880a2800 0x7f94880a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x55b54e494230 tx=0x7f9494009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 -- 172.21.15.77:0/4118294611 shutdown_connections 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 --2- 172.21.15.77:0/4118294611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 0x55b54e49afd0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 --2- 172.21.15.77:0/4118294611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e4939d0 0x55b54e4984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 --2- 172.21.15.77:0/4118294611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e493400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 -- 172.21.15.77:0/4118294611 >> 172.21.15.77:0/4118294611 conn(0x55b54e48a140 msgr2=0x55b54e48a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 -- 172.21.15.77:0/4118294611 shutdown_connections 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 -- 172.21.15.77:0/4118294611 wait complete. 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 -- start start 2024-03-16T10:16:57.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 Processor -- start 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e492fe0 0x55b54e49d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e4939d0 0x55b54e49d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e498b40 0x55b54e4a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b54e49ba90 con 0x55b54e492fe0 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55b54e3aeef0 con 0x55b54e4939d0 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b54e49bc10 con 0x55b54e498b40 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a71da740 10 client.0 _ll_register_callbacks cb 0x55b54e3a9dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a22ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e492fe0 0x55b54e49d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a22ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e492fe0 0x55b54e49d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42044/0 (socket says 172.21.15.77:42044) 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.693+0000 7f94a22ac640 1 -- 172.21.15.77:0/2468504062 learned_addr learned my addr 172.21.15.77:0/2468504062 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f94a2aad640 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e498b40 0x55b54e4a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f94a1aab640 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e4939d0 0x55b54e49d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f94a22ac640 1 -- 172.21.15.77:0/2468504062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e498b40 msgr2=0x55b54e4a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f94a22ac640 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e498b40 0x55b54e4a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f94a22ac640 1 -- 172.21.15.77:0/2468504062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e4939d0 msgr2=0x55b54e49d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f94a22ac640 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e4939d0 0x55b54e49d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f94a22ac640 1 -- 172.21.15.77:0/2468504062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b54e3bba30 con 0x55b54e492fe0 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f94a22ac640 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e492fe0 0x55b54e49d160 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f949009b640 tx=0x7f9490097cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a5347d63a02830 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f9485ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f9485ffb640 1 -- 172.21.15.77:0/2468504062 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949009fa90 con 0x55b54e492fe0 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f9485ffb640 1 -- 172.21.15.77:0/2468504062 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f94900860b0 con 0x55b54e492fe0 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f94a71da740 1 -- 172.21.15.77:0/2468504062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9498005040 con 0x55b54e492fe0 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f9485ffb640 1 -- 172.21.15.77:0/2468504062 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f94900a40e0 con 0x55b54e492fe0 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f94a71da740 10 client.15351 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f94a71da740 1 -- 172.21.15.77:0/2468504062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b54e1072d0 con 0x55b54e492fe0 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f9485ffb640 1 -- 172.21.15.77:0/2468504062 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9498005040 con 0x55b54e492fe0 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.694+0000 7f9485ffb640 1 -- 172.21.15.77:0/2468504062 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f94900977b0 con 0x55b54e492fe0 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.695+0000 7f94a71da740 10 client.15351 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.695+0000 7f94a71da740 10 client.15351 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T10:16:57.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.695+0000 7f94a71da740 20 client.15351 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.695+0000 7f94a71da740 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b54e4a9b50 0x55b54e4aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.695+0000 7f94a71da740 4 client.15351 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.695+0000 7f94a71da740 1 -- 172.21.15.77:0/2468504062 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b54e4001c0 con 0x55b54e4a9b50 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.695+0000 7f94a1aab640 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b54e4a9b50 0x55b54e4aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.695+0000 7f94a1aab640 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b54e4a9b50 0x55b54e4aafc0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3089d88d91a9b67d server_cookie=787340484b377241 in_seq=0 out_seq=0 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.695+0000 7f9485ffb640 10 client.15351 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f9485ffb640 1 -- 172.21.15.77:0/2468504062 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b54e4001c0 con 0x55b54e4a9b50 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f9485ffb640 10 client.15351 handle_command_reply: tid=0 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f94a71da740 1 client.15351 shutdown 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f94a71da740 1 -- 172.21.15.77:0/2468504062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e492fe0 msgr2=0x55b54e49d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f94a71da740 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e492fe0 0x55b54e49d160 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f949009b640 tx=0x7f9490097cf0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f94a71da740 1 -- 172.21.15.77:0/2468504062 shutdown_connections 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f94a71da740 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e498b40 0x55b54e4a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f94a71da740 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e4939d0 0x55b54e49d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f94a71da740 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b54e4a9b50 0x55b54e4aafc0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f94a71da740 1 --2- 172.21.15.77:0/2468504062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e492fe0 0x55b54e49d160 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f9485ffb640 0 client.15351 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:16:57.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f94a71da740 1 -- 172.21.15.77:0/2468504062 >> 172.21.15.77:0/2468504062 conn(0x55b54e48a140 msgr2=0x55b54e489380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f94a71da740 1 -- 172.21.15.77:0/2468504062 shutdown_connections 2024-03-16T10:16:57.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f94a71da740 1 -- 172.21.15.77:0/2468504062 wait complete. 2024-03-16T10:16:57.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.696+0000 7f94a71da740 20 client.15351 trim_cache size 0 max 16384 2024-03-16T10:16:57.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a71da740 1 Processor -- start 2024-03-16T10:16:57.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a71da740 1 -- start start 2024-03-16T10:16:57.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a71da740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e49cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a71da740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e4939d0 0x55b54e4a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a71da740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 0x55b54e49d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a71da740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b54e4a9300 con 0x55b54e498b40 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a71da740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55b54e4a99d0 con 0x55b54e492fe0 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a71da740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b54e4aa0a0 con 0x55b54e4939d0 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a2aad640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 0x55b54e49d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a2aad640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 0x55b54e49d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42052/0 (socket says 172.21.15.77:42052) 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a2aad640 1 -- 172.21.15.77:0/1183703844 learned_addr learned my addr 172.21.15.77:0/1183703844 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a22ac640 1 --2- 172.21.15.77:0/1183703844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e49cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a1aab640 1 --2- 172.21.15.77:0/1183703844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e4939d0 0x55b54e4a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a2aad640 1 -- 172.21.15.77:0/1183703844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e4939d0 msgr2=0x55b54e4a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a2aad640 1 --2- 172.21.15.77:0/1183703844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e4939d0 0x55b54e4a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a2aad640 1 -- 172.21.15.77:0/1183703844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 msgr2=0x55b54e49cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a2aad640 1 --2- 172.21.15.77:0/1183703844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e49cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.714+0000 7f94a2aad640 1 -- 172.21.15.77:0/1183703844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54e1d16b0 con 0x55b54e498b40 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f94a2aad640 1 --2- 172.21.15.77:0/1183703844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 0x55b54e49d2b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f949400ff00 tx=0x7f949401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e38bd0505388d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f9487fff640 1 -- 172.21.15.77:0/1183703844 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494011a40 con 0x55b54e498b40 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f9487fff640 1 -- 172.21.15.77:0/1183703844 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9494011be0 con 0x55b54e498b40 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f9487fff640 1 -- 172.21.15.77:0/1183703844 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949401f690 con 0x55b54e498b40 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f94a71da740 1 -- 172.21.15.77:0/1183703844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 msgr2=0x55b54e49d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f94a71da740 1 --2- 172.21.15.77:0/1183703844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 0x55b54e49d2b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f949400ff00 tx=0x7f949401b000 comp rx=0 tx=0).stop 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f94a71da740 1 -- 172.21.15.77:0/1183703844 shutdown_connections 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f94a71da740 1 --2- 172.21.15.77:0/1183703844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e498b40 0x55b54e49d2b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f94a71da740 1 --2- 172.21.15.77:0/1183703844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e4939d0 0x55b54e4a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f94a71da740 1 --2- 172.21.15.77:0/1183703844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e49cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f94a71da740 1 -- 172.21.15.77:0/1183703844 >> 172.21.15.77:0/1183703844 conn(0x55b54e48a140 msgr2=0x55b54e4981f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f94a71da740 1 -- 172.21.15.77:0/1183703844 shutdown_connections 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.715+0000 7f94a71da740 1 -- 172.21.15.77:0/1183703844 wait complete. 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a71da740 1 -- start start 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a71da740 1 Processor -- start 2024-03-16T10:16:57.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a71da740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e497520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a71da740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e4939d0 0x55b54e4a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a71da740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e498b40 0x55b54e50d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a71da740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b54e4aae40 con 0x55b54e4939d0 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a71da740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55b54e4aa770 con 0x55b54e492fe0 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a71da740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b54e4ab510 con 0x55b54e498b40 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a71da740 10 client.0 _ll_register_callbacks cb 0x7f9490003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a1aab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e4939d0 0x55b54e4a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a1aab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e4939d0 0x55b54e4a46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42056/0 (socket says 172.21.15.77:42056) 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a1aab640 1 -- 172.21.15.77:0/2159396483 learned_addr learned my addr 172.21.15.77:0/2159396483 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a22ac640 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e497520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a2aad640 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e498b40 0x55b54e50d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a1aab640 1 -- 172.21.15.77:0/2159396483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e498b40 msgr2=0x55b54e50d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a1aab640 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e498b40 0x55b54e50d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a1aab640 1 -- 172.21.15.77:0/2159396483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 msgr2=0x55b54e497520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a1aab640 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e497520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a1aab640 1 -- 172.21.15.77:0/2159396483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b54e50e060 con 0x55b54e4939d0 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f94a1aab640 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e4939d0 0x55b54e4a46a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9498009900 tx=0x7f9498005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc9870a505ede93 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.716+0000 7f9485ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f9485ffb640 1 -- 172.21.15.77:0/2159396483 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498054020 con 0x55b54e4939d0 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f9485ffb640 1 -- 172.21.15.77:0/2159396483 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f9498050d90 con 0x55b54e4939d0 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f94a71da740 1 -- 172.21.15.77:0/2159396483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9494011be0 con 0x55b54e4939d0 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f9485ffb640 1 -- 172.21.15.77:0/2159396483 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f949805f050 con 0x55b54e4939d0 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f94a71da740 10 client.15357 fetch_fsmap learned FSMap version 26 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f94a71da740 1 -- 172.21.15.77:0/2159396483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b54e50f360 con 0x55b54e4939d0 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f9485ffb640 1 -- 172.21.15.77:0/2159396483 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9494011be0 con 0x55b54e4939d0 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f9485ffb640 1 -- 172.21.15.77:0/2159396483 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f94980046c0 con 0x55b54e4939d0 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f94a71da740 10 client.15357 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f94a71da740 10 client.15357 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f94a71da740 20 client.15357 populate_metadata read hostname 'smithi077' 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f94a71da740 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b54e513ba0 0x55b54e5128b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:57.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f94a71da740 4 client.15357 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f94a71da740 1 -- 172.21.15.77:0/2159396483 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "status"}) -- 0x55b54e3fa510 con 0x55b54e513ba0 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.717+0000 7f94a22ac640 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b54e513ba0 0x55b54e5128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f9485ffb640 10 client.15357 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a22ac640 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b54e513ba0 0x55b54e5128b0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89b34d0f8e458dc5 server_cookie=480346097dca6bb6 in_seq=0 out_seq=0 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f9485ffb640 1 -- 172.21.15.77:0/2159396483 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x55b54e3fa510 con 0x55b54e513ba0 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f9485ffb640 10 client.15357 handle_command_reply: tid=0 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a71da740 1 client.15357 shutdown 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a71da740 1 -- 172.21.15.77:0/2159396483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e4939d0 msgr2=0x55b54e4a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a71da740 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e4939d0 0x55b54e4a46a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9498009900 tx=0x7f9498005e40 comp rx=0 tx=0).stop 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a71da740 1 -- 172.21.15.77:0/2159396483 shutdown_connections 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a71da740 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b54e498b40 0x55b54e50d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a71da740 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b54e513ba0 0x55b54e5128b0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a71da740 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b54e4939d0 0x55b54e4a46a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a71da740 1 --2- 172.21.15.77:0/2159396483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b54e492fe0 0x55b54e497520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f9485ffb640 0 client.15357 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:16:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a71da740 1 -- 172.21.15.77:0/2159396483 >> 172.21.15.77:0/2159396483 conn(0x55b54e48a140 msgr2=0x55b54e49cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a71da740 1 -- 172.21.15.77:0/2159396483 shutdown_connections 2024-03-16T10:16:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a71da740 1 -- 172.21.15.77:0/2159396483 wait complete. 2024-03-16T10:16:57.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.718+0000 7f94a71da740 20 client.15357 trim_cache size 0 max 16384 2024-03-16T10:16:57.726 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:16:57.726 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "c6ee6c60-e37c-11ee-95c9-87774f69a715", 2024-03-16T10:16:57.726 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2024-03-16T10:16:57.726 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 24482, 2024-03-16T10:16:57.726 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2024-03-16T10:16:57.726 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2024-03-16T10:16:57.727 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2024-03-16T10:16:57.727 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 114.709364686, 2024-03-16T10:16:57.727 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 26, 2024-03-16T10:16:57.727 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 87, 2024-03-16T10:16:57.727 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T10:16:57.727 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 121.68291692 2024-03-16T10:16:57.727 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:16:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.719+0000 7f94a4510640 1 -- 172.21.15.77:0/3758952435 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94880a2800 msgr2=0x7f94880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.719+0000 7f94a4510640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94880a2800 0x7f94880a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x55b54e494230 tx=0x7f9494009500 comp rx=0 tx=0).stop 2024-03-16T10:16:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.719+0000 7f94a4510640 1 -- 172.21.15.77:0/3758952435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f949c1006e0 msgr2=0x7f949c0694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.719+0000 7f94a4510640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f949c1006e0 0x7f949c0694d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f949000e9c0 tx=0x7f949000ee80 comp rx=0 tx=0).stop 2024-03-16T10:16:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.719+0000 7f94a4510640 1 -- 172.21.15.77:0/3758952435 shutdown_connections 2024-03-16T10:16:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.719+0000 7f94a4510640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949c102290 0x7f949c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.719+0000 7f94a4510640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94880a2800 0x7f94880a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.719+0000 7f94a4510640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f949c1018d0 0x7f949c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.719+0000 7f94a4510640 1 --2- 172.21.15.77:0/3758952435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f949c1006e0 0x7f949c0694d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.719+0000 7f94a4510640 1 -- 172.21.15.77:0/3758952435 >> 172.21.15.77:0/3758952435 conn(0x7f949c0fbc70 msgr2=0x7f949c104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.719+0000 7f94a4510640 1 -- 172.21.15.77:0/3758952435 shutdown_connections 2024-03-16T10:16:57.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:57.719+0000 7f94a4510640 1 -- 172.21.15.77:0/3758952435 wait complete. 2024-03-16T10:16:57.730 INFO:teuthology.run_tasks:Running task kclient... 2024-03-16T10:16:57.758 INFO:tasks.kclient:Mounting kernel clients... 2024-03-16T10:16:57.759 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-16T10:16:57.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:16:57.759 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-16T10:16:57.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:16:57.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-03-16T10:16:57.837 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:57 smithi077 ceph-mon[32400]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:57.837 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:57 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/850004730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:57.842 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2024-03-16T10:16:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:16:57.844 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-16T10:16:57.844 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-16T10:16:57.844 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T10:16:57.844 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-16T10:16:58.140 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:16:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:57 smithi192 ceph-mon[36354]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:57 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/850004730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:57 smithi193 ceph-mon[36311]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:16:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:57 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/850004730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:16:59.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.947+0000 7f2e7f41e700 1 -- 172.21.15.77:0/643310495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e781011d0 msgr2=0x7f2e78101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.947+0000 7f2e7f41e700 1 --2- 172.21.15.77:0/643310495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e781011d0 0x7f2e78101630 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009990 tx=0x7f2e68009ca0 comp rx=0 tx=0).stop 2024-03-16T10:16:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.947+0000 7f2e7f41e700 1 -- 172.21.15.77:0/643310495 shutdown_connections 2024-03-16T10:16:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.947+0000 7f2e7f41e700 1 --2- 172.21.15.77:0/643310495 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e78101b70 0x7f2e7810e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.947+0000 7f2e7f41e700 1 --2- 172.21.15.77:0/643310495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e78107210 0x7f2e781075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.947+0000 7f2e7f41e700 1 --2- 172.21.15.77:0/643310495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e781011d0 0x7f2e78101630 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.947+0000 7f2e7f41e700 1 -- 172.21.15.77:0/643310495 >> 172.21.15.77:0/643310495 conn(0x7f2e78078100 msgr2=0x7f2e78078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:16:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.947+0000 7f2e7f41e700 1 -- 172.21.15.77:0/643310495 shutdown_connections 2024-03-16T10:16:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.947+0000 7f2e7f41e700 1 -- 172.21.15.77:0/643310495 wait complete. 2024-03-16T10:16:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7f41e700 1 Processor -- start 2024-03-16T10:16:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7f41e700 1 -- start start 2024-03-16T10:16:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7f41e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e781011d0 0x7f2e7819c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7f41e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e78101b70 0x7f2e7819caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7f41e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e78107210 0x7f2e78196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7f41e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2e781103e0 con 0x7f2e78101b70 2024-03-16T10:16:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7f41e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2e78110260 con 0x7f2e781011d0 2024-03-16T10:16:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7f41e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e78110560 con 0x7f2e78107210 2024-03-16T10:16:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7c9b9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e78101b70 0x7f2e7819caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7c9b9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e78101b70 0x7f2e7819caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42080/0 (socket says 172.21.15.77:42080) 2024-03-16T10:16:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7c9b9700 1 -- 172.21.15.77:0/2196449888 learned_addr learned my addr 172.21.15.77:0/2196449888 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:16:59.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7d9bb700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e78107210 0x7f2e78196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7d1ba700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e781011d0 0x7f2e7819c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7c9b9700 1 -- 172.21.15.77:0/2196449888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e78107210 msgr2=0x7f2e78196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7c9b9700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e78107210 0x7f2e78196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7c9b9700 1 -- 172.21.15.77:0/2196449888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e781011d0 msgr2=0x7f2e7819c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7c9b9700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e781011d0 0x7f2e7819c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7c9b9700 1 -- 172.21.15.77:0/2196449888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e78196f60 con 0x7f2e78101b70 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.948+0000 7f2e7c9b9700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e78101b70 0x7f2e7819caf0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2e68006010 tx=0x7f2e68003c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.949+0000 7f2e6e7fc700 1 -- 172.21.15.77:0/2196449888 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e68023070 con 0x7f2e78101b70 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.949+0000 7f2e7f41e700 1 -- 172.21.15.77:0/2196449888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e781971f0 con 0x7f2e78101b70 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.949+0000 7f2e6e7fc700 1 -- 172.21.15.77:0/2196449888 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e6800dd00 con 0x7f2e78101b70 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.949+0000 7f2e6e7fc700 1 -- 172.21.15.77:0/2196449888 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6801a070 con 0x7f2e78101b70 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.949+0000 7f2e7f41e700 1 -- 172.21.15.77:0/2196449888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e781a3530 con 0x7f2e78101b70 2024-03-16T10:16:59.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.950+0000 7f2e7f41e700 1 -- 172.21.15.77:0/2196449888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e78197550 con 0x7f2e78101b70 2024-03-16T10:16:59.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.951+0000 7f2e6e7fc700 1 -- 172.21.15.77:0/2196449888 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2e68026070 con 0x7f2e78101b70 2024-03-16T10:16:59.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.952+0000 7f2e6e7fc700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e640a2910 0x7f2e640a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:16:59.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.952+0000 7f2e7d1ba700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e640a2910 0x7f2e640a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:16:59.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.952+0000 7f2e6e7fc700 1 -- 172.21.15.77:0/2196449888 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2e680cf4e0 con 0x7f2e78101b70 2024-03-16T10:16:59.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.952+0000 7f2e7d1ba700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e640a2910 0x7f2e640a4dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2e74010d00 tx=0x7f2e74012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:16:59.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:16:59.955+0000 7f2e6e7fc700 1 -- 172.21.15.77:0/2196449888 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2e6802d070 con 0x7f2e78101b70 2024-03-16T10:17:00.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:16:59 smithi192 ceph-mon[36354]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T10:17:00.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.203+0000 7f2e7f41e700 1 -- 172.21.15.77:0/2196449888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2e78197bb0 con 0x7f2e78101b70 2024-03-16T10:17:00.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.204+0000 7f2e6e7fc700 1 -- 172.21.15.77:0/2196449888 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+73 (secure 0 0 0) 0x7f2e6809c6a0 con 0x7f2e78101b70 2024-03-16T10:17:00.205 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-16T10:17:00.205 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:16:59 smithi077 ceph-mon[32400]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T10:17:00.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.206+0000 7f2e7f41e700 1 -- 172.21.15.77:0/2196449888 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e640a2910 msgr2=0x7f2e640a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:00.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.206+0000 7f2e7f41e700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e640a2910 0x7f2e640a4dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2e74010d00 tx=0x7f2e74012040 comp rx=0 tx=0).stop 2024-03-16T10:17:00.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.206+0000 7f2e7f41e700 1 -- 172.21.15.77:0/2196449888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e78101b70 msgr2=0x7f2e7819caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:00.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.206+0000 7f2e7f41e700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e78101b70 0x7f2e7819caf0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2e68006010 tx=0x7f2e68003c90 comp rx=0 tx=0).stop 2024-03-16T10:17:00.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.206+0000 7f2e7f41e700 1 -- 172.21.15.77:0/2196449888 shutdown_connections 2024-03-16T10:17:00.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.206+0000 7f2e7f41e700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e640a2910 0x7f2e640a4dd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:00.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.206+0000 7f2e7f41e700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e78107210 0x7f2e78196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:00.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.206+0000 7f2e7f41e700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e781011d0 0x7f2e7819c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:00.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.206+0000 7f2e7f41e700 1 --2- 172.21.15.77:0/2196449888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e78101b70 0x7f2e7819caf0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:00.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.206+0000 7f2e7f41e700 1 -- 172.21.15.77:0/2196449888 >> 172.21.15.77:0/2196449888 conn(0x7f2e78078100 msgr2=0x7f2e78100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:00.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.206+0000 7f2e7f41e700 1 -- 172.21.15.77:0/2196449888 shutdown_connections 2024-03-16T10:17:00.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00.206+0000 7f2e7f41e700 1 -- 172.21.15.77:0/2196449888 wait complete. 2024-03-16T10:17:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:16:59 smithi193 ceph-mon[36311]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T10:17:00.801 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-16T10:17:00.802 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-16T10:17:00.802 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-03-16T10:17:00.802 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-16T10:17:00.802 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T10:17:00.802 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-16T10:17:00.802 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-16T10:17:00.802 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-16T10:17:00.802 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-16T10:17:00.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:17:00.803 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-03-16T10:17:00.819 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-16T10:17:00.819 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-16T10:17:00.819 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-03-16T10:17:00.819 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-16T10:17:00.819 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-03-16T10:17:00.819 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-16T10:17:00.819 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T10:17:00.819 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-03-16T10:17:00.819 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-03-16T10:17:00.820 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T10:17:00.820 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-03-16T10:17:00.820 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-03-16T10:17:00.820 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T10:17:00.820 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-03-16T10:17:00.820 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-16T10:17:00.820 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-03-16T10:17:00.820 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-16T10:17:00.820 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42140sec preferred_lft 42140sec 2024-03-16T10:17:00.820 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-03-16T10:17:00.820 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-03-16T10:17:00.821 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-16T10:17:00.821 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T10:17:00.821 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-16T10:17:00.821 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add name ceph-brx type bridge 2024-03-16T10:17:00.821 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr flush dev ceph-brx 2024-03-16T10:17:00.821 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set ceph-brx up 2024-03-16T10:17:00.821 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-16T10:17:00.821 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-16T10:17:00.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T10:17:01.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T10:17:01.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:17:01.022 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-16T10:17:01.039 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:00 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2196449888' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T10:17:01.051 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-03-16T10:17:01.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:17:01.053 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-03-16T10:17:01.129 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-03-16T10:17:01.129 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-16T10:17:01.130 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-16T10:17:01.130 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-16T10:17:01.130 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-16T10:17:01.130 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T10:17:01.130 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-16T10:17:01.130 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-16T10:17:01.130 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-16T10:17:01.130 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-16T10:17:01.131 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-16T10:17:01.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:00 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2196449888' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T10:17:01.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T10:17:01.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:00 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2196449888' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T10:17:01.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T10:17:01.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:17:01.448 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-16T10:17:01.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:17:01.463 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2024-03-16T10:17:01.518 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T10:17:01.518 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-16T10:17:01.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T10:17:01.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-16T10:17:01.518 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-16T10:17:01.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T10:17:01.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T10:17:01.641 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-16T10:17:01.641 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T10:17:01.641 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-16T10:17:01.641 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-16T10:17:01.641 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-16T10:17:01.641 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-16T10:17:01.641 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-16T10:17:01.641 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-16T10:17:01.641 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-16T10:17:01.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T10:17:01.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T10:17:01.815 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T10:17:01.815 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-03-16T10:17:01.815 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-03-16T10:17:01.815 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.0 master ceph-brx 2024-03-16T10:17:01.815 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-03-16T10:17:01.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T10:17:01.874 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:01 smithi077 ceph-mon[32400]: pgmap v348: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:17:01.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T10:17:01.899 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T10:17:01.899 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-16T10:17:01.913 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-16T10:17:01.913 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T10:17:01.913 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-16T10:17:01.973 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-16T10:17:01.973 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-16T10:17:01.973 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/7eff604f-057e-4c66-8c0e-f7c4c1eee205 2024-03-16T10:17:01.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:17:01.973 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/7eff604f-057e-4c66-8c0e-f7c4c1eee205 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-16T10:17:02.057 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-16T10:17:02.057 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-16T10:17:02.057 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-03-16T10:17:02.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:01 smithi192 ceph-mon[36354]: pgmap v348: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:17:02.196 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-16T10:17:02.196 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-16T10:17:02.196 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-03-16T10:17:02.196 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.192:6789,172.21.15.193:6789" 2024-03-16T10:17:02.196 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.192:6789,172.21.15.193:6789:/volumes/qa/sv_1/7eff604f-057e-4c66-8c0e-f7c4c1eee205 2024-03-16T10:17:02.196 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=c6ee6c60-e37c-11ee-95c9-87774f69a715" will pass to kernel 2024-03-16T10:17:02.197 INFO:teuthology.orchestra.run.smithi077.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-16T10:17:02.197 INFO:teuthology.orchestra.run.smithi077.stdout: You just mounted a file system that supports labels which does not 2024-03-16T10:17:02.197 INFO:teuthology.orchestra.run.smithi077.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-16T10:17:02.197 INFO:teuthology.orchestra.run.smithi077.stdout: applications will generate AVC messages and not be allowed access to 2024-03-16T10:17:02.197 INFO:teuthology.orchestra.run.smithi077.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-16T10:17:02.199 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-16T10:17:02.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:17:02.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-16T10:17:02.228 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T10:17:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:01 smithi193 ceph-mon[36311]: pgmap v348: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:17:02.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-16T10:17:02.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T10:17:02.325 INFO:teuthology.orchestra.run.smithi077.stdout:client25156 2024-03-16T10:17:02.325 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:17:02.326 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T10:17:02.341 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T10:17:02.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-16T10:17:02.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T10:17:02.436 INFO:teuthology.orchestra.run.smithi077.stdout:client25156 2024-03-16T10:17:02.436 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T10:17:02.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-16T10:17:02.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T10:17:02.492 INFO:teuthology.orchestra.run.smithi077.stdout:client25156 2024-03-16T10:17:02.493 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715.client25156/status') 2024-03-16T10:17:02.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T10:17:02.553 INFO:teuthology.orchestra.run.smithi077.stdout:instance: client.25156 (3)172.21.15.77:0/3346219351 2024-03-16T10:17:02.553 INFO:teuthology.orchestra.run.smithi077.stdout:blocklisted: false 2024-03-16T10:17:02.553 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records in 2024-03-16T10:17:02.553 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records out 2024-03-16T10:17:02.553 INFO:teuthology.orchestra.run.smithi077.stderr:71 bytes copied, 2.8126e-05 s, 2.5 MB/s 2024-03-16T10:17:02.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T10:17:02.560 INFO:teuthology.run_tasks:Running task exec... 2024-03-16T10:17:02.571 INFO:teuthology.task.exec:Executing custom commands... 2024-03-16T10:17:02.571 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T10:17:02.571 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-16T10:17:02.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.674+0000 7f50be590640 1 -- 172.21.15.77:0/957834512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50b81018f0 msgr2=0x7f50b8101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:02.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.674+0000 7f50be590640 1 --2- 172.21.15.77:0/957834512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50b81018f0 0x7f50b8101d70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009900 tx=0x7f50ac02bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:02.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.674+0000 7f50be590640 1 -- 172.21.15.77:0/957834512 shutdown_connections 2024-03-16T10:17:02.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.674+0000 7f50be590640 1 --2- 172.21.15.77:0/957834512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b81022b0 0x7f50b8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:02.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.674+0000 7f50be590640 1 --2- 172.21.15.77:0/957834512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50b81018f0 0x7f50b8101d70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:02.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.674+0000 7f50be590640 1 --2- 172.21.15.77:0/957834512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50b8100700 0x7f50b8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:02.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.674+0000 7f50be590640 1 -- 172.21.15.77:0/957834512 >> 172.21.15.77:0/957834512 conn(0x7f50b80fbc70 msgr2=0x7f50b80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:02.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.674+0000 7f50be590640 1 -- 172.21.15.77:0/957834512 shutdown_connections 2024-03-16T10:17:02.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.674+0000 7f50be590640 1 -- 172.21.15.77:0/957834512 wait complete. 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50be590640 1 Processor -- start 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50be590640 1 -- start start 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50be590640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b8100700 0x7f50b80699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50be590640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50b81018f0 0x7f50b819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50be590640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50b81022b0 0x7f50b819ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50be590640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f50b810c380 con 0x7f50b81018f0 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50be590640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f50b810c200 con 0x7f50b8100700 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50be590640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50b810c500 con 0x7f50b81022b0 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50b77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50b81018f0 0x7f50b819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50b77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50b81018f0 0x7f50b819c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55962/0 (socket says 172.21.15.77:55962) 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50b77fe640 1 -- 172.21.15.77:0/1584035225 learned_addr learned my addr 172.21.15.77:0/1584035225 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50b7fff640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b8100700 0x7f50b80699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:02.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50bcb2d640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50b81022b0 0x7f50b819ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:02.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50b77fe640 1 -- 172.21.15.77:0/1584035225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50b81022b0 msgr2=0x7f50b819ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:02.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50b77fe640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50b81022b0 0x7f50b819ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:02.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50b77fe640 1 -- 172.21.15.77:0/1584035225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b8100700 msgr2=0x7f50b80699d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:02.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50b77fe640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b8100700 0x7f50b80699d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:02.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50b77fe640 1 -- 172.21.15.77:0/1584035225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b819f740 con 0x7f50b81018f0 2024-03-16T10:17:02.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.675+0000 7f50b7fff640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b8100700 0x7f50b80699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:02.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.676+0000 7f50b77fe640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50b81018f0 0x7f50b819c810 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009230 tx=0x7f50ac02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:02.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.676+0000 7f50b57fa640 1 -- 172.21.15.77:0/1584035225 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac041070 con 0x7f50b81018f0 2024-03-16T10:17:02.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.676+0000 7f50b57fa640 1 -- 172.21.15.77:0/1584035225 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ac004450 con 0x7f50b81018f0 2024-03-16T10:17:02.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.676+0000 7f50be590640 1 -- 172.21.15.77:0/1584035225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50b819f970 con 0x7f50b81018f0 2024-03-16T10:17:02.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.676+0000 7f50b57fa640 1 -- 172.21.15.77:0/1584035225 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac003e20 con 0x7f50b81018f0 2024-03-16T10:17:02.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.676+0000 7f50be590640 1 -- 172.21.15.77:0/1584035225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50b819fe60 con 0x7f50b81018f0 2024-03-16T10:17:02.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.677+0000 7f50be590640 1 -- 172.21.15.77:0/1584035225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50b8107aa0 con 0x7f50b81018f0 2024-03-16T10:17:02.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.678+0000 7f50b57fa640 1 -- 172.21.15.77:0/1584035225 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f50ac044070 con 0x7f50b81018f0 2024-03-16T10:17:02.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.678+0000 7f50b57fa640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50a00a28a0 0x7f50a00a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:02.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.679+0000 7f50b7fff640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50a00a28a0 0x7f50a00a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:02.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.679+0000 7f50b57fa640 1 -- 172.21.15.77:0/1584035225 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f50ac0f08d0 con 0x7f50b81018f0 2024-03-16T10:17:02.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.679+0000 7f50b7fff640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50a00a28a0 0x7f50a00a4d60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f50b8101750 tx=0x7f50a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:02.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.682+0000 7f50b57fa640 1 -- 172.21.15.77:0/1584035225 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f50ac0bd3b0 con 0x7f50b81018f0 2024-03-16T10:17:02.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:02.899+0000 7f50be590640 1 -- 172.21.15.77:0/1584035225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f50b802d3c0 con 0x7f50b81018f0 2024-03-16T10:17:03.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.829+0000 7f50b57fa640 1 -- 172.21.15.77:0/1584035225 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v27) ==== 118+0+0 (secure 0 0 0) 0x7f50ac0c1450 con 0x7f50b81018f0 2024-03-16T10:17:03.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.831+0000 7f50be590640 1 -- 172.21.15.77:0/1584035225 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50a00a28a0 msgr2=0x7f50a00a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:03.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.831+0000 7f50be590640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50a00a28a0 0x7f50a00a4d60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f50b8101750 tx=0x7f50a8005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:03.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.831+0000 7f50be590640 1 -- 172.21.15.77:0/1584035225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50b81018f0 msgr2=0x7f50b819c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:03.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.831+0000 7f50be590640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50b81018f0 0x7f50b819c810 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009230 tx=0x7f50ac02c650 comp rx=0 tx=0).stop 2024-03-16T10:17:03.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.831+0000 7f50be590640 1 -- 172.21.15.77:0/1584035225 shutdown_connections 2024-03-16T10:17:03.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.831+0000 7f50be590640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50a00a28a0 0x7f50a00a4d60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:03.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.831+0000 7f50be590640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50b81022b0 0x7f50b819ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:03.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.831+0000 7f50be590640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50b81018f0 0x7f50b819c810 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.831+0000 7f50be590640 1 --2- 172.21.15.77:0/1584035225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b8100700 0x7f50b80699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.831+0000 7f50be590640 1 -- 172.21.15.77:0/1584035225 >> 172.21.15.77:0/1584035225 conn(0x7f50b80fbc70 msgr2=0x7f50b8104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.831+0000 7f50be590640 1 -- 172.21.15.77:0/1584035225 shutdown_connections 2024-03-16T10:17:03.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.831+0000 7f50be590640 1 -- 172.21.15.77:0/1584035225 wait complete. 2024-03-16T10:17:03.844 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-16T10:17:03.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.964+0000 7f2cfb729640 1 -- 172.21.15.77:0/2169707758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cf4102a00 msgr2=0x7f2cf4102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:03.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.964+0000 7f2cfb729640 1 --2- 172.21.15.77:0/2169707758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cf4102a00 0x7f2cf4102e80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009900 tx=0x7f2cf002bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:03.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.964+0000 7f2cfb729640 1 -- 172.21.15.77:0/2169707758 shutdown_connections 2024-03-16T10:17:03.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.964+0000 7f2cfb729640 1 --2- 172.21.15.77:0/2169707758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cf41033c0 0x7f2cf4109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:03.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.964+0000 7f2cfb729640 1 --2- 172.21.15.77:0/2169707758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cf4102a00 0x7f2cf4102e80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:03.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.964+0000 7f2cfb729640 1 --2- 172.21.15.77:0/2169707758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cf4101810 0x7f2cf4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:03.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.964+0000 7f2cfb729640 1 -- 172.21.15.77:0/2169707758 >> 172.21.15.77:0/2169707758 conn(0x7f2cf40fcda0 msgr2=0x7f2cf40ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:03.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.964+0000 7f2cfb729640 1 -- 172.21.15.77:0/2169707758 shutdown_connections 2024-03-16T10:17:03.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.964+0000 7f2cfb729640 1 -- 172.21.15.77:0/2169707758 wait complete. 2024-03-16T10:17:03.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cfb729640 1 Processor -- start 2024-03-16T10:17:03.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cfb729640 1 -- start start 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cfb729640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cf4101810 0x7f2cf4197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cfb729640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cf4102a00 0x7f2cf4197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cfb729640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cf41033c0 0x7f2cf419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cfb729640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf410d4c0 con 0x7f2cf4101810 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cfb729640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf410d340 con 0x7f2cf41033c0 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cfb729640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf410d640 con 0x7f2cf4102a00 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cf94c5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cf4101810 0x7f2cf4197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cf94c5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cf4101810 0x7f2cf4197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55972/0 (socket says 172.21.15.77:55972) 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cf94c5640 1 -- 172.21.15.77:0/2092032772 learned_addr learned my addr 172.21.15.77:0/2092032772 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cf8cc4640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cf4102a00 0x7f2cf4197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cf9cc6640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cf41033c0 0x7f2cf419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cf94c5640 1 -- 172.21.15.77:0/2092032772 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cf4102a00 msgr2=0x7f2cf4197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cf94c5640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cf4102a00 0x7f2cf4197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:03.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cf94c5640 1 -- 172.21.15.77:0/2092032772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cf41033c0 msgr2=0x7f2cf419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:03.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cf94c5640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cf41033c0 0x7f2cf419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:03.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cf94c5640 1 -- 172.21.15.77:0/2092032772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf419f630 con 0x7f2cf4101810 2024-03-16T10:17:03.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.965+0000 7f2cf94c5640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cf4101810 0x7f2cf4197a10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400e9c0 tx=0x7f2ce400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:03.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.966+0000 7f2cea7fc640 1 -- 172.21.15.77:0/2092032772 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce400cd80 con 0x7f2cf4101810 2024-03-16T10:17:03.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.966+0000 7f2cea7fc640 1 -- 172.21.15.77:0/2092032772 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ce40035f0 con 0x7f2cf4101810 2024-03-16T10:17:03.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.966+0000 7f2cea7fc640 1 -- 172.21.15.77:0/2092032772 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4010690 con 0x7f2cf4101810 2024-03-16T10:17:03.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.966+0000 7f2cfb729640 1 -- 172.21.15.77:0/2092032772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf419f920 con 0x7f2cf4101810 2024-03-16T10:17:03.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.966+0000 7f2cfb729640 1 -- 172.21.15.77:0/2092032772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf419fe40 con 0x7f2cf4101810 2024-03-16T10:17:03.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.967+0000 7f2cfb729640 1 -- 172.21.15.77:0/2092032772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf404f540 con 0x7f2cf4101810 2024-03-16T10:17:03.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.968+0000 7f2cea7fc640 1 -- 172.21.15.77:0/2092032772 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2ce40037a0 con 0x7f2cf4101810 2024-03-16T10:17:03.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.969+0000 7f2cea7fc640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ce00a28a0 0x7f2ce00a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:03.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.969+0000 7f2cf8cc4640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ce00a28a0 0x7f2ce00a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:03.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.969+0000 7f2cea7fc640 1 -- 172.21.15.77:0/2092032772 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2ce40cee60 con 0x7f2cf4101810 2024-03-16T10:17:03.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.969+0000 7f2cf8cc4640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ce00a28a0 0x7f2ce00a4d60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2cf002c240 tx=0x7f2cf002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:03.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:03.972+0000 7f2cea7fc640 1 -- 172.21.15.77:0/2092032772 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2ce409c020 con 0x7f2cf4101810 2024-03-16T10:17:04.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:03 smithi192 ceph-mon[36354]: pgmap v349: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:17:04.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:03 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1584035225' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-16T10:17:04.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.207+0000 7f2cfb729640 1 -- 172.21.15.77:0/2092032772 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f2cf4061c20 con 0x7f2ce00a28a0 2024-03-16T10:17:04.213 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:03 smithi077 ceph-mon[32400]: pgmap v349: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:17:04.213 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:03 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1584035225' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-16T10:17:04.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.224+0000 7f2cea7fc640 1 -- 172.21.15.77:0/2092032772 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f2cf4061c20 con 0x7f2ce00a28a0 2024-03-16T10:17:04.225 INFO:teuthology.orchestra.run.smithi077.stdout:{} 2024-03-16T10:17:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.227+0000 7f2cfb729640 1 -- 172.21.15.77:0/2092032772 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ce00a28a0 msgr2=0x7f2ce00a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.227+0000 7f2cfb729640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ce00a28a0 0x7f2ce00a4d60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2cf002c240 tx=0x7f2cf002df90 comp rx=0 tx=0).stop 2024-03-16T10:17:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.227+0000 7f2cfb729640 1 -- 172.21.15.77:0/2092032772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cf4101810 msgr2=0x7f2cf4197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.227+0000 7f2cfb729640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cf4101810 0x7f2cf4197a10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400e9c0 tx=0x7f2ce400ee80 comp rx=0 tx=0).stop 2024-03-16T10:17:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.227+0000 7f2cfb729640 1 -- 172.21.15.77:0/2092032772 shutdown_connections 2024-03-16T10:17:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.227+0000 7f2cfb729640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cf41033c0 0x7f2cf419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.227+0000 7f2cfb729640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ce00a28a0 0x7f2ce00a4d60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.227+0000 7f2cfb729640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cf4102a00 0x7f2cf4197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.227+0000 7f2cfb729640 1 --2- 172.21.15.77:0/2092032772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cf4101810 0x7f2cf4197a10 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.227+0000 7f2cfb729640 1 -- 172.21.15.77:0/2092032772 >> 172.21.15.77:0/2092032772 conn(0x7f2cf40fcda0 msgr2=0x7f2cf4105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.227+0000 7f2cfb729640 1 -- 172.21.15.77:0/2092032772 shutdown_connections 2024-03-16T10:17:04.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.227+0000 7f2cfb729640 1 -- 172.21.15.77:0/2092032772 wait complete. 2024-03-16T10:17:04.239 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-16T10:17:04.253 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-16T10:17:04.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:17:04.264 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:03 smithi193 ceph-mon[36311]: pgmap v349: 97 pgs: 97 active+clean; 642 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T10:17:04.264 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:03 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1584035225' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-16T10:17:04.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.366+0000 7f2878885640 1 -- 172.21.15.77:0/3030916777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28741022b0 msgr2=0x7f2874108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.366+0000 7f2878885640 1 --2- 172.21.15.77:0/3030916777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28741022b0 0x7f2874108b40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2864009ca0 tx=0x7f286402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.367+0000 7f2878885640 1 -- 172.21.15.77:0/3030916777 shutdown_connections 2024-03-16T10:17:04.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.367+0000 7f2878885640 1 --2- 172.21.15.77:0/3030916777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28741022b0 0x7f2874108b40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.367+0000 7f2878885640 1 --2- 172.21.15.77:0/3030916777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28741018f0 0x7f2874101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.367+0000 7f2878885640 1 --2- 172.21.15.77:0/3030916777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2874100700 0x7f2874100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.367+0000 7f2878885640 1 -- 172.21.15.77:0/3030916777 >> 172.21.15.77:0/3030916777 conn(0x7f28740fbc70 msgr2=0x7f28740fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:04.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.367+0000 7f2878885640 1 -- 172.21.15.77:0/3030916777 shutdown_connections 2024-03-16T10:17:04.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.367+0000 7f2878885640 1 -- 172.21.15.77:0/3030916777 wait complete. 2024-03-16T10:17:04.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.367+0000 7f2878885640 1 Processor -- start 2024-03-16T10:17:04.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f2878885640 1 -- start start 2024-03-16T10:17:04.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f2878885640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2874100700 0x7f28740694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f2878885640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28741018f0 0x7f287419c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f2878885640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28741022b0 0x7f287419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f2878885640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f287410c360 con 0x7f2874100700 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f2878885640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f287410c1e0 con 0x7f28741018f0 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f2878885640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f287410c4e0 con 0x7f28741022b0 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f287259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2874100700 0x7f28740694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f287259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2874100700 0x7f28740694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56014/0 (socket says 172.21.15.77:56014) 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f287259c640 1 -- 172.21.15.77:0/687637223 learned_addr learned my addr 172.21.15.77:0/687637223 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f2871d9b640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28741018f0 0x7f287419c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f2872d9d640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28741022b0 0x7f287419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f287259c640 1 -- 172.21.15.77:0/687637223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28741022b0 msgr2=0x7f287419edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f287259c640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28741022b0 0x7f287419edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f287259c640 1 -- 172.21.15.77:0/687637223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28741018f0 msgr2=0x7f287419c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f287259c640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28741018f0 0x7f287419c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f287259c640 1 -- 172.21.15.77:0/687637223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f287419f6b0 con 0x7f2874100700 2024-03-16T10:17:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f287259c640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2874100700 0x7f28740694d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f286c00e9c0 tx=0x7f286c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f28637fe640 1 -- 172.21.15.77:0/687637223 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286c00cd80 con 0x7f2874100700 2024-03-16T10:17:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f28637fe640 1 -- 172.21.15.77:0/687637223 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f286c0035f0 con 0x7f2874100700 2024-03-16T10:17:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f2878885640 1 -- 172.21.15.77:0/687637223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f287419f9a0 con 0x7f2874100700 2024-03-16T10:17:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f28637fe640 1 -- 172.21.15.77:0/687637223 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286c010690 con 0x7f2874100700 2024-03-16T10:17:04.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.368+0000 7f2878885640 1 -- 172.21.15.77:0/687637223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f287419fef0 con 0x7f2874100700 2024-03-16T10:17:04.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.370+0000 7f2878885640 1 -- 172.21.15.77:0/687637223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28741079c0 con 0x7f2874100700 2024-03-16T10:17:04.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.371+0000 7f28637fe640 1 -- 172.21.15.77:0/687637223 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f286c0037a0 con 0x7f2874100700 2024-03-16T10:17:04.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.371+0000 7f28637fe640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f285c0a2800 0x7f285c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:04.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.372+0000 7f2871d9b640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f285c0a2800 0x7f285c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:04.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.372+0000 7f28637fe640 1 -- 172.21.15.77:0/687637223 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f286c0ced80 con 0x7f2874100700 2024-03-16T10:17:04.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.372+0000 7f2871d9b640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f285c0a2800 0x7f285c0a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2874069240 tx=0x7f2868005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:04.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.379+0000 7f28637fe640 1 -- 172.21.15.77:0/687637223 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f286c09beb0 con 0x7f2874100700 2024-03-16T10:17:04.495 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:17:04 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:17:04.211+0000 7f645e2f7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T10:17:04.495 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:17:04 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:17:04.211+0000 7f645e2f7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T10:17:04.495 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:17:04 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:17:04.211+0000 7f645e2f7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T10:17:04.495 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:17:04 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:17:04.211+0000 7f645e2f7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T10:17:04.495 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:17:04 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x[32634]: 2024-03-16T10:17:04.211+0000 7f645e2f7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T10:17:04.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.560+0000 7f2878885640 1 -- 172.21.15.77:0/687637223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f287402d3d0 con 0x7f2874100700 2024-03-16T10:17:04.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.561+0000 7f28637fe640 1 -- 172.21.15.77:0/687637223 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22496 (secure 0 0 0) 0x7f286c09b7b0 con 0x7f2874100700 2024-03-16T10:17:04.563 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:17:04.563 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":87,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:59.826259+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:17:04.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.563+0000 7f2878885640 1 -- 172.21.15.77:0/687637223 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f285c0a2800 msgr2=0x7f285c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.563+0000 7f2878885640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f285c0a2800 0x7f285c0a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2874069240 tx=0x7f2868005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:04.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.563+0000 7f2878885640 1 -- 172.21.15.77:0/687637223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2874100700 msgr2=0x7f28740694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.563+0000 7f2878885640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2874100700 0x7f28740694d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f286c00e9c0 tx=0x7f286c00ee80 comp rx=0 tx=0).stop 2024-03-16T10:17:04.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.563+0000 7f2878885640 1 -- 172.21.15.77:0/687637223 shutdown_connections 2024-03-16T10:17:04.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.563+0000 7f2878885640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f285c0a2800 0x7f285c0a4cc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.563+0000 7f2878885640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28741022b0 0x7f287419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.563+0000 7f2878885640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28741018f0 0x7f287419c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.563+0000 7f2878885640 1 --2- 172.21.15.77:0/687637223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2874100700 0x7f28740694d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.563+0000 7f2878885640 1 -- 172.21.15.77:0/687637223 >> 172.21.15.77:0/687637223 conn(0x7f28740fbc70 msgr2=0x7f2874104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:04.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.564+0000 7f2878885640 1 -- 172.21.15.77:0/687637223 shutdown_connections 2024-03-16T10:17:04.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.564+0000 7f2878885640 1 -- 172.21.15.77:0/687637223 wait complete. 2024-03-16T10:17:04.576 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:10:22.965004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T10:15:00.264039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T10:15:01.536067+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T10:17:04.576 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T10:17:04.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.686+0000 7fc003737640 1 -- 172.21.15.77:0/387428118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbffc1018d0 msgr2=0x7fbffc101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.689 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.686+0000 7fc003737640 1 --2- 172.21.15.77:0/387428118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbffc1018d0 0x7fbffc101d50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009900 tx=0x7fbfec02bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:04.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.686+0000 7fc003737640 1 -- 172.21.15.77:0/387428118 shutdown_connections 2024-03-16T10:17:04.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.686+0000 7fc003737640 1 --2- 172.21.15.77:0/387428118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbffc102290 0x7fbffc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.686+0000 7fc003737640 1 --2- 172.21.15.77:0/387428118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbffc1018d0 0x7fbffc101d50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.686+0000 7fc003737640 1 --2- 172.21.15.77:0/387428118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbffc1006e0 0x7fbffc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.686+0000 7fc003737640 1 -- 172.21.15.77:0/387428118 >> 172.21.15.77:0/387428118 conn(0x7fbffc0fbc70 msgr2=0x7fbffc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:04.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.686+0000 7fc003737640 1 -- 172.21.15.77:0/387428118 shutdown_connections 2024-03-16T10:17:04.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.686+0000 7fc003737640 1 -- 172.21.15.77:0/387428118 wait complete. 2024-03-16T10:17:04.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc003737640 1 Processor -- start 2024-03-16T10:17:04.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc003737640 1 -- start start 2024-03-16T10:17:04.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc003737640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbffc1006e0 0x7fbffc196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc003737640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbffc1018d0 0x7fbffc196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc003737640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbffc102290 0x7fbffc19e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc003737640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc10c360 con 0x7fbffc102290 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc003737640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc10c1e0 con 0x7fbffc1006e0 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc003737640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc10c4e0 con 0x7fbffc1018d0 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc001cd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbffc102290 0x7fbffc19e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc001cd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbffc102290 0x7fbffc19e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56040/0 (socket says 172.21.15.77:56040) 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc001cd4640 1 -- 172.21.15.77:0/1277163328 learned_addr learned my addr 172.21.15.77:0/1277163328 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc0014d3640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbffc1006e0 0x7fbffc196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc000cd2640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbffc1018d0 0x7fbffc196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc001cd4640 1 -- 172.21.15.77:0/1277163328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbffc1018d0 msgr2=0x7fbffc196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc001cd4640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbffc1018d0 0x7fbffc196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc001cd4640 1 -- 172.21.15.77:0/1277163328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbffc1006e0 msgr2=0x7fbffc196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc001cd4640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbffc1006e0 0x7fbffc196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc001cd4640 1 -- 172.21.15.77:0/1277163328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbffc19e670 con 0x7fbffc102290 2024-03-16T10:17:04.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc0014d3640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbffc1006e0 0x7fbffc196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:04.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.687+0000 7fc001cd4640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbffc102290 0x7fbffc19e000 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbff800e970 tx=0x7fbff800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:04.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.688+0000 7fbff27fc640 1 -- 172.21.15.77:0/1277163328 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff800ccc0 con 0x7fbffc102290 2024-03-16T10:17:04.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.688+0000 7fbff27fc640 1 -- 172.21.15.77:0/1277163328 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff800ce60 con 0x7fbffc102290 2024-03-16T10:17:04.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.688+0000 7fbff27fc640 1 -- 172.21.15.77:0/1277163328 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8010490 con 0x7fbffc102290 2024-03-16T10:17:04.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.688+0000 7fc003737640 1 -- 172.21.15.77:0/1277163328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbffc19e960 con 0x7fbffc102290 2024-03-16T10:17:04.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.688+0000 7fc003737640 1 -- 172.21.15.77:0/1277163328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbffc19ee80 con 0x7fbffc102290 2024-03-16T10:17:04.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.689+0000 7fc003737640 1 -- 172.21.15.77:0/1277163328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbffc107ad0 con 0x7fbffc102290 2024-03-16T10:17:04.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.690+0000 7fbff27fc640 1 -- 172.21.15.77:0/1277163328 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbff8005000 con 0x7fbffc102290 2024-03-16T10:17:04.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.690+0000 7fbff27fc640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbfe80a2880 0x7fbfe80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:04.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.690+0000 7fc0014d3640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbfe80a2880 0x7fbfe80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:04.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.690+0000 7fbff27fc640 1 -- 172.21.15.77:0/1277163328 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fbff80cef80 con 0x7fbffc102290 2024-03-16T10:17:04.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.691+0000 7fc0014d3640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbfe80a2880 0x7fbfe80a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbffc101730 tx=0x7fbff4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:04.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.694+0000 7fbff27fc640 1 -- 172.21.15.77:0/1277163328 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbff8053740 con 0x7fbffc102290 2024-03-16T10:17:04.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.877+0000 7fc003737640 1 -- 172.21.15.77:0/1277163328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbffc02d340 con 0x7fbffc102290 2024-03-16T10:17:04.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.878+0000 7fbff27fc640 1 -- 172.21.15.77:0/1277163328 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fbff809c0b0 con 0x7fbffc102290 2024-03-16T10:17:04.879 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T10:17:04.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.880+0000 7fc003737640 1 -- 172.21.15.77:0/1277163328 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbfe80a2880 msgr2=0x7fbfe80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.880+0000 7fc003737640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbfe80a2880 0x7fbfe80a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbffc101730 tx=0x7fbff4009500 comp rx=0 tx=0).stop 2024-03-16T10:17:04.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.880+0000 7fc003737640 1 -- 172.21.15.77:0/1277163328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbffc102290 msgr2=0x7fbffc19e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.880+0000 7fc003737640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbffc102290 0x7fbffc19e000 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbff800e970 tx=0x7fbff800ed70 comp rx=0 tx=0).stop 2024-03-16T10:17:04.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.880+0000 7fc003737640 1 -- 172.21.15.77:0/1277163328 shutdown_connections 2024-03-16T10:17:04.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.880+0000 7fc003737640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbffc102290 0x7fbffc19e000 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.880+0000 7fc003737640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbfe80a2880 0x7fbfe80a4d40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.880+0000 7fc003737640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbffc1018d0 0x7fbffc196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.880+0000 7fc003737640 1 --2- 172.21.15.77:0/1277163328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbffc1006e0 0x7fbffc196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.880+0000 7fc003737640 1 -- 172.21.15.77:0/1277163328 >> 172.21.15.77:0/1277163328 conn(0x7fbffc0fbc70 msgr2=0x7fbffc104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:04.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.880+0000 7fc003737640 1 -- 172.21.15.77:0/1277163328 shutdown_connections 2024-03-16T10:17:04.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.880+0000 7fc003737640 1 -- 172.21.15.77:0/1277163328 wait complete. 2024-03-16T10:17:04.891 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T10:17:04.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.996+0000 7f9a16d13640 1 -- 172.21.15.77:0/206101006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a101018d0 msgr2=0x7f9a10101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:04.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.996+0000 7f9a16d13640 1 --2- 172.21.15.77:0/206101006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a101018d0 0x7f9a10101d50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f99fc009900 tx=0x7f99fc02bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:04.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.996+0000 7f9a16d13640 1 -- 172.21.15.77:0/206101006 shutdown_connections 2024-03-16T10:17:04.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.996+0000 7f9a16d13640 1 --2- 172.21.15.77:0/206101006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a10102290 0x7f9a10108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.996+0000 7f9a16d13640 1 --2- 172.21.15.77:0/206101006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a101018d0 0x7f9a10101d50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.996+0000 7f9a16d13640 1 --2- 172.21.15.77:0/206101006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a101006e0 0x7f9a10100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:04.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.996+0000 7f9a16d13640 1 -- 172.21.15.77:0/206101006 >> 172.21.15.77:0/206101006 conn(0x7f9a100fbc70 msgr2=0x7f9a100fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:04.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.996+0000 7f9a16d13640 1 -- 172.21.15.77:0/206101006 shutdown_connections 2024-03-16T10:17:04.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.996+0000 7f9a16d13640 1 -- 172.21.15.77:0/206101006 wait complete. 2024-03-16T10:17:04.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a16d13640 1 Processor -- start 2024-03-16T10:17:04.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a16d13640 1 -- start start 2024-03-16T10:17:04.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a16d13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a101006e0 0x7f9a100694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a16d13640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a101018d0 0x7f9a1019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a16d13640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a10102290 0x7f9a1019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a16d13640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1010c360 con 0x7f9a101006e0 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a16d13640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1010c1e0 con 0x7f9a10102290 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a16d13640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1010c4e0 con 0x7f9a101018d0 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a14aaf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a101006e0 0x7f9a100694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a14aaf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a101006e0 0x7f9a100694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56060/0 (socket says 172.21.15.77:56060) 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a14aaf640 1 -- 172.21.15.77:0/3583108321 learned_addr learned my addr 172.21.15.77:0/3583108321 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a0ffff640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a101018d0 0x7f9a1019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a152b0640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a10102290 0x7f9a1019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a14aaf640 1 -- 172.21.15.77:0/3583108321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a101018d0 msgr2=0x7f9a1019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a14aaf640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a101018d0 0x7f9a1019c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a14aaf640 1 -- 172.21.15.77:0/3583108321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a10102290 msgr2=0x7f9a1019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a14aaf640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a10102290 0x7f9a1019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a14aaf640 1 -- 172.21.15.77:0/3583108321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1019f7d0 con 0x7f9a101006e0 2024-03-16T10:17:05.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.997+0000 7f9a14aaf640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a101006e0 0x7f9a100694b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9a0400e9c0 tx=0x7f9a0400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:05.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.998+0000 7f9a0dffb640 1 -- 172.21.15.77:0/3583108321 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0400cd80 con 0x7f9a101006e0 2024-03-16T10:17:05.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.998+0000 7f9a0dffb640 1 -- 172.21.15.77:0/3583108321 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a040035f0 con 0x7f9a101006e0 2024-03-16T10:17:05.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.998+0000 7f9a0dffb640 1 -- 172.21.15.77:0/3583108321 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a04010690 con 0x7f9a101006e0 2024-03-16T10:17:05.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.998+0000 7f9a16d13640 1 -- 172.21.15.77:0/3583108321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a1019fa60 con 0x7f9a101006e0 2024-03-16T10:17:05.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.998+0000 7f9a16d13640 1 -- 172.21.15.77:0/3583108321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a1019ff30 con 0x7f9a101006e0 2024-03-16T10:17:05.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:04.999+0000 7f9a16d13640 1 -- 172.21.15.77:0/3583108321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a10107ad0 con 0x7f9a101006e0 2024-03-16T10:17:05.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.000+0000 7f9a0dffb640 1 -- 172.21.15.77:0/3583108321 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9a040037a0 con 0x7f9a101006e0 2024-03-16T10:17:05.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.000+0000 7f9a0dffb640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9a000a2800 0x7f9a000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.000+0000 7f9a0ffff640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9a000a2800 0x7f9a000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.000+0000 7f9a0dffb640 1 -- 172.21.15.77:0/3583108321 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f9a040cedd0 con 0x7f9a101006e0 2024-03-16T10:17:05.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.001+0000 7f9a0ffff640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9a000a2800 0x7f9a000a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f99fc02c7b0 tx=0x7f99fc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:05.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.004+0000 7f9a0dffb640 1 -- 172.21.15.77:0/3583108321 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a0409bf00 con 0x7f9a101006e0 2024-03-16T10:17:05.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:04 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1584035225' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-16T10:17:05.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:04 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:17:05.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:04 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/687637223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:17:05.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.185+0000 7f9a16d13640 1 -- 172.21.15.77:0/3583108321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9a1002d360 con 0x7f9a101006e0 2024-03-16T10:17:05.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.185+0000 7f9a0dffb640 1 -- 172.21.15.77:0/3583108321 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f9a0409b800 con 0x7f9a101006e0 2024-03-16T10:17:05.186 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T10:17:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.187+0000 7f9a16d13640 1 -- 172.21.15.77:0/3583108321 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9a000a2800 msgr2=0x7f9a000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.187+0000 7f9a16d13640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9a000a2800 0x7f9a000a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f99fc02c7b0 tx=0x7f99fc005fd0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.187+0000 7f9a16d13640 1 -- 172.21.15.77:0/3583108321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a101006e0 msgr2=0x7f9a100694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.187+0000 7f9a16d13640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a101006e0 0x7f9a100694b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9a0400e9c0 tx=0x7f9a0400ee80 comp rx=0 tx=0).stop 2024-03-16T10:17:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.187+0000 7f9a16d13640 1 -- 172.21.15.77:0/3583108321 shutdown_connections 2024-03-16T10:17:05.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.187+0000 7f9a16d13640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a10102290 0x7f9a1019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.187+0000 7f9a16d13640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9a000a2800 0x7f9a000a4cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.187+0000 7f9a16d13640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a101018d0 0x7f9a1019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.187+0000 7f9a16d13640 1 --2- 172.21.15.77:0/3583108321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a101006e0 0x7f9a100694b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.187+0000 7f9a16d13640 1 -- 172.21.15.77:0/3583108321 >> 172.21.15.77:0/3583108321 conn(0x7f9a100fbc70 msgr2=0x7f9a10104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:05.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.187+0000 7f9a16d13640 1 -- 172.21.15.77:0/3583108321 shutdown_connections 2024-03-16T10:17:05.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.187+0000 7f9a16d13640 1 -- 172.21.15.77:0/3583108321 wait complete. 2024-03-16T10:17:05.198 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:04 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1584035225' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-16T10:17:05.198 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:04 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:17:05.198 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:04 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/687637223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:17:05.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T10:17:05.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:04 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1584035225' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-16T10:17:05.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:04 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:17:05.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:04 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/687637223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:17:05.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 -- 172.21.15.77:0/4013155911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32e8107500 msgr2=0x7f32e8109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 --2- 172.21.15.77:0/4013155911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32e8107500 0x7f32e8109910 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f32e0009ca0 tx=0x7f32e002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 -- 172.21.15.77:0/4013155911 shutdown_connections 2024-03-16T10:17:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 --2- 172.21.15.77:0/4013155911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32e8107500 0x7f32e8109910 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 --2- 172.21.15.77:0/4013155911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f32e8104bd0 0x7f32e8106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 --2- 172.21.15.77:0/4013155911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e81022a0 0x7f32e8104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 -- 172.21.15.77:0/4013155911 >> 172.21.15.77:0/4013155911 conn(0x7f32e80fbc70 msgr2=0x7f32e80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 -- 172.21.15.77:0/4013155911 shutdown_connections 2024-03-16T10:17:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 -- 172.21.15.77:0/4013155911 wait complete. 2024-03-16T10:17:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 Processor -- start 2024-03-16T10:17:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 -- start start 2024-03-16T10:17:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32e81022a0 0x7f32e8197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e8104bd0 0x7f32e8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f32e8107500 0x7f32e819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f32e810d460 con 0x7f32e8104bd0 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f32e810d2e0 con 0x7f32e81022a0 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.315+0000 7f32eed59640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f32e810d5e0 con 0x7f32e8107500 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32e7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e8104bd0 0x7f32e8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32e7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e8104bd0 0x7f32e8198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56082/0 (socket says 172.21.15.77:56082) 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32e7fff640 1 -- 172.21.15.77:0/1800447173 learned_addr learned my addr 172.21.15.77:0/1800447173 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32ed2f6640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f32e8107500 0x7f32e819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32ecaf5640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32e81022a0 0x7f32e8197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32e7fff640 1 -- 172.21.15.77:0/1800447173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f32e8107500 msgr2=0x7f32e819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32e7fff640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f32e8107500 0x7f32e819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32e7fff640 1 -- 172.21.15.77:0/1800447173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32e81022a0 msgr2=0x7f32e8197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32e7fff640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32e81022a0 0x7f32e8197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32e7fff640 1 -- 172.21.15.77:0/1800447173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e819f720 con 0x7f32e8104bd0 2024-03-16T10:17:05.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32ed2f6640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f32e8107500 0x7f32e819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:05.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32e7fff640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e8104bd0 0x7f32e8198040 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f32d400e9c0 tx=0x7f32d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:05.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32e5ffb640 1 -- 172.21.15.77:0/1800447173 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d400cd80 con 0x7f32e8104bd0 2024-03-16T10:17:05.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32eed59640 1 -- 172.21.15.77:0/1800447173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e819fa10 con 0x7f32e8104bd0 2024-03-16T10:17:05.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32e5ffb640 1 -- 172.21.15.77:0/1800447173 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32d4005500 con 0x7f32e8104bd0 2024-03-16T10:17:05.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32e5ffb640 1 -- 172.21.15.77:0/1800447173 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d4010690 con 0x7f32e8104bd0 2024-03-16T10:17:05.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.316+0000 7f32eed59640 1 -- 172.21.15.77:0/1800447173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e819ff60 con 0x7f32e8104bd0 2024-03-16T10:17:05.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.318+0000 7f32eed59640 1 -- 172.21.15.77:0/1800447173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32e80fd760 con 0x7f32e8104bd0 2024-03-16T10:17:05.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.319+0000 7f32e5ffb640 1 -- 172.21.15.77:0/1800447173 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f32d40046c0 con 0x7f32e8104bd0 2024-03-16T10:17:05.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.320+0000 7f32e5ffb640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f32d80a2800 0x7f32d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.320+0000 7f32ecaf5640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f32d80a2800 0x7f32d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.320+0000 7f32e5ffb640 1 -- 172.21.15.77:0/1800447173 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f32d40ced30 con 0x7f32e8104bd0 2024-03-16T10:17:05.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.321+0000 7f32ecaf5640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f32d80a2800 0x7f32d80a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f32dc0095f0 tx=0x7f32dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:05.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.325+0000 7f32e5ffb640 1 -- 172.21.15.77:0/1800447173 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f32d409be60 con 0x7f32e8104bd0 2024-03-16T10:17:05.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.502+0000 7f32eed59640 1 -- 172.21.15.77:0/1800447173 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f32e802d390 con 0x7f32e8104bd0 2024-03-16T10:17:05.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.502+0000 7f32e5ffb640 1 -- 172.21.15.77:0/1800447173 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f32d409b760 con 0x7f32e8104bd0 2024-03-16T10:17:05.503 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T10:17:05.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.504+0000 7f32eed59640 1 -- 172.21.15.77:0/1800447173 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f32d80a2800 msgr2=0x7f32d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.504+0000 7f32eed59640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f32d80a2800 0x7f32d80a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f32dc0095f0 tx=0x7f32dc005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:05.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.504+0000 7f32eed59640 1 -- 172.21.15.77:0/1800447173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e8104bd0 msgr2=0x7f32e8198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.504+0000 7f32eed59640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e8104bd0 0x7f32e8198040 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f32d400e9c0 tx=0x7f32d400ee80 comp rx=0 tx=0).stop 2024-03-16T10:17:05.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.504+0000 7f32eed59640 1 -- 172.21.15.77:0/1800447173 shutdown_connections 2024-03-16T10:17:05.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.504+0000 7f32eed59640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f32d80a2800 0x7f32d80a4cc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.504+0000 7f32eed59640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f32e8107500 0x7f32e819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.504+0000 7f32eed59640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e8104bd0 0x7f32e8198040 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.504+0000 7f32eed59640 1 --2- 172.21.15.77:0/1800447173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32e81022a0 0x7f32e8197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.504+0000 7f32eed59640 1 -- 172.21.15.77:0/1800447173 >> 172.21.15.77:0/1800447173 conn(0x7f32e80fbc70 msgr2=0x7f32e8102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:05.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.504+0000 7f32eed59640 1 -- 172.21.15.77:0/1800447173 shutdown_connections 2024-03-16T10:17:05.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.504+0000 7f32eed59640 1 -- 172.21.15.77:0/1800447173 wait complete. 2024-03-16T10:17:05.516 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:17:05.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.618+0000 7fe52e4ea640 1 -- 172.21.15.77:0/4226557684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5281018f0 msgr2=0x7fe528101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.618+0000 7fe52e4ea640 1 --2- 172.21.15.77:0/4226557684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5281018f0 0x7fe528101d70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009860 tx=0x7fe51c02bb70 comp rx=0 tx=0).stop 2024-03-16T10:17:05.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.618+0000 7fe52e4ea640 1 -- 172.21.15.77:0/4226557684 shutdown_connections 2024-03-16T10:17:05.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.618+0000 7fe52e4ea640 1 --2- 172.21.15.77:0/4226557684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5281022b0 0x7fe528108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.618+0000 7fe52e4ea640 1 --2- 172.21.15.77:0/4226557684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5281018f0 0x7fe528101d70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.618+0000 7fe52e4ea640 1 --2- 172.21.15.77:0/4226557684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe528100700 0x7fe528100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.618+0000 7fe52e4ea640 1 -- 172.21.15.77:0/4226557684 >> 172.21.15.77:0/4226557684 conn(0x7fe5280fbc70 msgr2=0x7fe5280fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:05.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.618+0000 7fe52e4ea640 1 -- 172.21.15.77:0/4226557684 shutdown_connections 2024-03-16T10:17:05.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.618+0000 7fe52e4ea640 1 -- 172.21.15.77:0/4226557684 wait complete. 2024-03-16T10:17:05.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.619+0000 7fe52e4ea640 1 Processor -- start 2024-03-16T10:17:05.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.619+0000 7fe52e4ea640 1 -- start start 2024-03-16T10:17:05.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.619+0000 7fe52e4ea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe528100700 0x7fe528196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.619+0000 7fe52e4ea640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5281018f0 0x7fe528196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.619+0000 7fe52e4ea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5281022b0 0x7fe52819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.619+0000 7fe52e4ea640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe52810c380 con 0x7fe5281022b0 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.619+0000 7fe52e4ea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe52810c200 con 0x7fe528100700 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.619+0000 7fe52e4ea640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe52810c500 con 0x7fe5281018f0 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.619+0000 7fe52ca87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5281022b0 0x7fe52819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.619+0000 7fe5277fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5281018f0 0x7fe528196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.619+0000 7fe52ca87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5281022b0 0x7fe52819e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56096/0 (socket says 172.21.15.77:56096) 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe52ca87640 1 -- 172.21.15.77:0/1132689322 learned_addr learned my addr 172.21.15.77:0/1132689322 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe5277fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5281018f0 0x7fe528196f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:41756/0 (socket says 172.21.15.77:41756) 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe527fff640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe528100700 0x7fe528196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe52ca87640 1 -- 172.21.15.77:0/1132689322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5281018f0 msgr2=0x7fe528196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe52ca87640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5281018f0 0x7fe528196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe52ca87640 1 -- 172.21.15.77:0/1132689322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe528100700 msgr2=0x7fe528196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe52ca87640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe528100700 0x7fe528196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe52ca87640 1 -- 172.21.15.77:0/1132689322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52819e670 con 0x7fe5281022b0 2024-03-16T10:17:05.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe5277fe640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5281018f0 0x7fe528196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:05.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe52ca87640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5281022b0 0x7fe52819e000 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe52000b5e0 tx=0x7fe52000baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:05.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe5257fa640 1 -- 172.21.15.77:0/1132689322 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe520003970 con 0x7fe5281022b0 2024-03-16T10:17:05.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe5257fa640 1 -- 172.21.15.77:0/1132689322 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe52000a680 con 0x7fe5281022b0 2024-03-16T10:17:05.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe52e4ea640 1 -- 172.21.15.77:0/1132689322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5280689a0 con 0x7fe5281022b0 2024-03-16T10:17:05.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe5257fa640 1 -- 172.21.15.77:0/1132689322 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5200044c0 con 0x7fe5281022b0 2024-03-16T10:17:05.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.620+0000 7fe52e4ea640 1 -- 172.21.15.77:0/1132689322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe528068e70 con 0x7fe5281022b0 2024-03-16T10:17:05.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.621+0000 7fe52e4ea640 1 -- 172.21.15.77:0/1132689322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5280685d0 con 0x7fe5281022b0 2024-03-16T10:17:05.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.622+0000 7fe5257fa640 1 -- 172.21.15.77:0/1132689322 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe520004660 con 0x7fe5281022b0 2024-03-16T10:17:05.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.623+0000 7fe5257fa640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5100a2800 0x7fe5100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.623+0000 7fe527fff640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5100a2800 0x7fe5100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.623+0000 7fe5257fa640 1 -- 172.21.15.77:0/1132689322 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe5200d0880 con 0x7fe5281022b0 2024-03-16T10:17:05.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.623+0000 7fe527fff640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5100a2800 0x7fe5100a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe528101750 tx=0x7fe518005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:05.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.627+0000 7fe5257fa640 1 -- 172.21.15.77:0/1132689322 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe52009d360 con 0x7fe5281022b0 2024-03-16T10:17:05.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.813+0000 7fe52e4ea640 1 -- 172.21.15.77:0/1132689322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5281076a0 con 0x7fe5281022b0 2024-03-16T10:17:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.814+0000 7fe5257fa640 1 -- 172.21.15.77:0/1132689322 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22496 (secure 0 0 0) 0x7fe5200a1400 con 0x7fe5281022b0 2024-03-16T10:17:05.815 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:17:05.815 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":87,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:59.826259+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:17:05.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.817+0000 7fe52e4ea640 1 -- 172.21.15.77:0/1132689322 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5100a2800 msgr2=0x7fe5100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.817+0000 7fe52e4ea640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5100a2800 0x7fe5100a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe528101750 tx=0x7fe518005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:05.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.817+0000 7fe52e4ea640 1 -- 172.21.15.77:0/1132689322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5281022b0 msgr2=0x7fe52819e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.817+0000 7fe52e4ea640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5281022b0 0x7fe52819e000 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe52000b5e0 tx=0x7fe52000baa0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.817+0000 7fe52e4ea640 1 -- 172.21.15.77:0/1132689322 shutdown_connections 2024-03-16T10:17:05.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.817+0000 7fe52e4ea640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5281022b0 0x7fe52819e000 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.817+0000 7fe52e4ea640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5100a2800 0x7fe5100a4cc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.817+0000 7fe52e4ea640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5281018f0 0x7fe528196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.817+0000 7fe52e4ea640 1 --2- 172.21.15.77:0/1132689322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe528100700 0x7fe528196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.817+0000 7fe52e4ea640 1 -- 172.21.15.77:0/1132689322 >> 172.21.15.77:0/1132689322 conn(0x7fe5280fbc70 msgr2=0x7fe528104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:05.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.817+0000 7fe52e4ea640 1 -- 172.21.15.77:0/1132689322 shutdown_connections 2024-03-16T10:17:05.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.817+0000 7fe52e4ea640 1 -- 172.21.15.77:0/1132689322 wait complete. 2024-03-16T10:17:05.830 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:10:22.965004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T10:15:00.264039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T10:15:01.536067+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T10:17:05.831 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T10:17:05.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.939+0000 7f2e460e8640 1 -- 172.21.15.77:0/1613530252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e401018d0 msgr2=0x7f2e40101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.939+0000 7f2e460e8640 1 --2- 172.21.15.77:0/1613530252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e401018d0 0x7f2e40101d50 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2e30009900 tx=0x7f2e3002bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:05.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.939+0000 7f2e460e8640 1 -- 172.21.15.77:0/1613530252 shutdown_connections 2024-03-16T10:17:05.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.939+0000 7f2e460e8640 1 --2- 172.21.15.77:0/1613530252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e40102290 0x7f2e40108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.939+0000 7f2e460e8640 1 --2- 172.21.15.77:0/1613530252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e401018d0 0x7f2e40101d50 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.939+0000 7f2e460e8640 1 --2- 172.21.15.77:0/1613530252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e401006e0 0x7f2e40100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.939+0000 7f2e460e8640 1 -- 172.21.15.77:0/1613530252 >> 172.21.15.77:0/1613530252 conn(0x7f2e400fbc70 msgr2=0x7f2e400fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:05.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.940+0000 7f2e460e8640 1 -- 172.21.15.77:0/1613530252 shutdown_connections 2024-03-16T10:17:05.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.940+0000 7f2e460e8640 1 -- 172.21.15.77:0/1613530252 wait complete. 2024-03-16T10:17:05.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.940+0000 7f2e460e8640 1 Processor -- start 2024-03-16T10:17:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.940+0000 7f2e460e8640 1 -- start start 2024-03-16T10:17:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e460e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e401006e0 0x7f2e400694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e460e8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e401018d0 0x7f2e4019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e460e8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e40102290 0x7f2e4019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e460e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4010c360 con 0x7f2e401006e0 2024-03-16T10:17:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e460e8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4010c1e0 con 0x7f2e40102290 2024-03-16T10:17:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e460e8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4010c4e0 con 0x7f2e401018d0 2024-03-16T10:17:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e401006e0 0x7f2e400694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e401006e0 0x7f2e400694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56112/0 (socket says 172.21.15.77:56112) 2024-03-16T10:17:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3f7fe640 1 -- 172.21.15.77:0/763255744 learned_addr learned my addr 172.21.15.77:0/763255744 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3effd640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e401018d0 0x7f2e4019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3ffff640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e40102290 0x7f2e4019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3f7fe640 1 -- 172.21.15.77:0/763255744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e401018d0 msgr2=0x7f2e4019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3f7fe640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e401018d0 0x7f2e4019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3f7fe640 1 -- 172.21.15.77:0/763255744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e40102290 msgr2=0x7f2e4019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:05.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3f7fe640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e40102290 0x7f2e4019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:05.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3f7fe640 1 -- 172.21.15.77:0/763255744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4019f780 con 0x7f2e401006e0 2024-03-16T10:17:05.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3ffff640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e40102290 0x7f2e4019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:17:05.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3f7fe640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e401006e0 0x7f2e400694d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400e9c0 tx=0x7f2e3400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:05.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3cff9640 1 -- 172.21.15.77:0/763255744 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3400cd80 con 0x7f2e401006e0 2024-03-16T10:17:05.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3cff9640 1 -- 172.21.15.77:0/763255744 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e340035f0 con 0x7f2e401006e0 2024-03-16T10:17:05.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e460e8640 1 -- 172.21.15.77:0/763255744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e4019fa70 con 0x7f2e401006e0 2024-03-16T10:17:05.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.941+0000 7f2e3cff9640 1 -- 172.21.15.77:0/763255744 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e34010690 con 0x7f2e401006e0 2024-03-16T10:17:05.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.942+0000 7f2e460e8640 1 -- 172.21.15.77:0/763255744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e4019ff40 con 0x7f2e401006e0 2024-03-16T10:17:05.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.942+0000 7f2e460e8640 1 -- 172.21.15.77:0/763255744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e40107ad0 con 0x7f2e401006e0 2024-03-16T10:17:05.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.944+0000 7f2e3cff9640 1 -- 172.21.15.77:0/763255744 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2e340037a0 con 0x7f2e401006e0 2024-03-16T10:17:05.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.944+0000 7f2e3cff9640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e280a2800 0x7f2e280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:05.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.945+0000 7f2e3effd640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e280a2800 0x7f2e280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:05.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.945+0000 7f2e3cff9640 1 -- 172.21.15.77:0/763255744 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2e340cee80 con 0x7f2e401006e0 2024-03-16T10:17:05.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.945+0000 7f2e3effd640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e280a2800 0x7f2e280a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2e3002c7b0 tx=0x7f2e30005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:05.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:05.948+0000 7f2e3cff9640 1 -- 172.21.15.77:0/763255744 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2e3409bfb0 con 0x7f2e401006e0 2024-03-16T10:17:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.129+0000 7f2e460e8640 1 -- 172.21.15.77:0/763255744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2e4002d340 con 0x7f2e401006e0 2024-03-16T10:17:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.129+0000 7f2e3cff9640 1 -- 172.21.15.77:0/763255744 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f2e3409b8b0 con 0x7f2e401006e0 2024-03-16T10:17:06.130 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T10:17:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.131+0000 7f2e460e8640 1 -- 172.21.15.77:0/763255744 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e280a2800 msgr2=0x7f2e280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.131+0000 7f2e460e8640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e280a2800 0x7f2e280a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2e3002c7b0 tx=0x7f2e30005fd0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.131+0000 7f2e460e8640 1 -- 172.21.15.77:0/763255744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e401006e0 msgr2=0x7f2e400694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.131+0000 7f2e460e8640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e401006e0 0x7f2e400694d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400e9c0 tx=0x7f2e3400ee80 comp rx=0 tx=0).stop 2024-03-16T10:17:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.131+0000 7f2e460e8640 1 -- 172.21.15.77:0/763255744 shutdown_connections 2024-03-16T10:17:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.131+0000 7f2e460e8640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e40102290 0x7f2e4019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.131+0000 7f2e460e8640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e280a2800 0x7f2e280a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.131+0000 7f2e460e8640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e401018d0 0x7f2e4019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.131+0000 7f2e460e8640 1 --2- 172.21.15.77:0/763255744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e401006e0 0x7f2e400694d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.131+0000 7f2e460e8640 1 -- 172.21.15.77:0/763255744 >> 172.21.15.77:0/763255744 conn(0x7f2e400fbc70 msgr2=0x7f2e40104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.131+0000 7f2e460e8640 1 -- 172.21.15.77:0/763255744 shutdown_connections 2024-03-16T10:17:06.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.131+0000 7f2e460e8640 1 -- 172.21.15.77:0/763255744 wait complete. 2024-03-16T10:17:06.148 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:05 smithi077 ceph-mon[32400]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:17:06.148 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:05 smithi077 ceph-mon[32400]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-16T10:17:06.148 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:05 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1277163328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:17:06.148 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:05 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3583108321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:17:06.148 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:05 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1800447173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:17:06.148 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:05 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1132689322' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:17:06.150 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T10:17:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:05 smithi192 ceph-mon[36354]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:17:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:05 smithi192 ceph-mon[36354]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-16T10:17:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:05 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1277163328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:17:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:05 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3583108321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:17:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:05 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1800447173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:17:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:05 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1132689322' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:17:06.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:05 smithi193 ceph-mon[36311]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T10:17:06.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:05 smithi193 ceph-mon[36311]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-16T10:17:06.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:05 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1277163328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:17:06.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:05 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3583108321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:17:06.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:05 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1800447173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:17:06.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:05 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1132689322' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:17:06.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.254+0000 7f323288c640 1 -- 172.21.15.77:0/3664823918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f322c102290 msgr2=0x7f322c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.254+0000 7f323288c640 1 --2- 172.21.15.77:0/3664823918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f322c102290 0x7f322c108b20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3224009ca0 tx=0x7f322402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.254+0000 7f323288c640 1 -- 172.21.15.77:0/3664823918 shutdown_connections 2024-03-16T10:17:06.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.254+0000 7f323288c640 1 --2- 172.21.15.77:0/3664823918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f322c102290 0x7f322c108b20 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.254+0000 7f323288c640 1 --2- 172.21.15.77:0/3664823918 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f322c1018d0 0x7f322c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.254+0000 7f323288c640 1 --2- 172.21.15.77:0/3664823918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f322c1006e0 0x7f322c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.254+0000 7f323288c640 1 -- 172.21.15.77:0/3664823918 >> 172.21.15.77:0/3664823918 conn(0x7f322c0fbc70 msgr2=0x7f322c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.254+0000 7f323288c640 1 -- 172.21.15.77:0/3664823918 shutdown_connections 2024-03-16T10:17:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.254+0000 7f323288c640 1 -- 172.21.15.77:0/3664823918 wait complete. 2024-03-16T10:17:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f323288c640 1 Processor -- start 2024-03-16T10:17:06.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f323288c640 1 -- start start 2024-03-16T10:17:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f323288c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f322c1006e0 0x7f322c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f323288c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f322c1018d0 0x7f322c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f323288c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f322c102290 0x7f322c19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f323288c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f322c10c360 con 0x7f322c1006e0 2024-03-16T10:17:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f323288c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f322c10c1e0 con 0x7f322c102290 2024-03-16T10:17:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f323288c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f322c10c4e0 con 0x7f322c1018d0 2024-03-16T10:17:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f322bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f322c1006e0 0x7f322c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f322bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f322c1006e0 0x7f322c197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56140/0 (socket says 172.21.15.77:56140) 2024-03-16T10:17:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f322bfff640 1 -- 172.21.15.77:0/22140844 learned_addr learned my addr 172.21.15.77:0/22140844 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:06.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f3230e29640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f322c102290 0x7f322c19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f322b7fe640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f322c1018d0 0x7f322c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f322bfff640 1 -- 172.21.15.77:0/22140844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f322c1018d0 msgr2=0x7f322c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f322bfff640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f322c1018d0 0x7f322c198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f322bfff640 1 -- 172.21.15.77:0/22140844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f322c102290 msgr2=0x7f322c19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f322bfff640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f322c102290 0x7f322c19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f322bfff640 1 -- 172.21.15.77:0/22140844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322c19f6f0 con 0x7f322c1006e0 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.255+0000 7f322bfff640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f322c1006e0 0x7f322c197ad0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f321c00e9c0 tx=0x7f321c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.256+0000 7f32297fa640 1 -- 172.21.15.77:0/22140844 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c00cd80 con 0x7f322c1006e0 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.256+0000 7f32297fa640 1 -- 172.21.15.77:0/22140844 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f321c0035f0 con 0x7f322c1006e0 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.256+0000 7f323288c640 1 -- 172.21.15.77:0/22140844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f322c19f9e0 con 0x7f322c1006e0 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.256+0000 7f32297fa640 1 -- 172.21.15.77:0/22140844 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c010690 con 0x7f322c1006e0 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.256+0000 7f323288c640 1 -- 172.21.15.77:0/22140844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f322c19fee0 con 0x7f322c1006e0 2024-03-16T10:17:06.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.257+0000 7f323288c640 1 -- 172.21.15.77:0/22140844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f322c1a0110 con 0x7f322c1006e0 2024-03-16T10:17:06.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.258+0000 7f32297fa640 1 -- 172.21.15.77:0/22140844 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f321c0037a0 con 0x7f322c1006e0 2024-03-16T10:17:06.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.259+0000 7f32297fa640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f32140a2800 0x7f32140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:06.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.259+0000 7f322b7fe640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f32140a2800 0x7f32140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:06.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.259+0000 7f32297fa640 1 -- 172.21.15.77:0/22140844 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f321c0cee00 con 0x7f322c1006e0 2024-03-16T10:17:06.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.259+0000 7f322b7fe640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f32140a2800 0x7f32140a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f322c198fe0 tx=0x7f3220005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:06.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.262+0000 7f32297fa640 1 -- 172.21.15.77:0/22140844 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f321c09bf30 con 0x7f322c1006e0 2024-03-16T10:17:06.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.445+0000 7f323288c640 1 -- 172.21.15.77:0/22140844 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f322c1a02e0 con 0x7f322c1006e0 2024-03-16T10:17:06.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.446+0000 7f32297fa640 1 -- 172.21.15.77:0/22140844 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f321c09b830 con 0x7f322c1006e0 2024-03-16T10:17:06.446 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T10:17:06.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.448+0000 7f323288c640 1 -- 172.21.15.77:0/22140844 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f32140a2800 msgr2=0x7f32140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.448+0000 7f323288c640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f32140a2800 0x7f32140a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f322c198fe0 tx=0x7f3220005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:06.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.448+0000 7f323288c640 1 -- 172.21.15.77:0/22140844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f322c1006e0 msgr2=0x7f322c197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.448+0000 7f323288c640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f322c1006e0 0x7f322c197ad0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f321c00e9c0 tx=0x7f321c00ee80 comp rx=0 tx=0).stop 2024-03-16T10:17:06.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.448+0000 7f323288c640 1 -- 172.21.15.77:0/22140844 shutdown_connections 2024-03-16T10:17:06.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.448+0000 7f323288c640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f322c102290 0x7f322c19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.448+0000 7f323288c640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f32140a2800 0x7f32140a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.448+0000 7f323288c640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f322c1018d0 0x7f322c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.448+0000 7f323288c640 1 --2- 172.21.15.77:0/22140844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f322c1006e0 0x7f322c197ad0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.448+0000 7f323288c640 1 -- 172.21.15.77:0/22140844 >> 172.21.15.77:0/22140844 conn(0x7f322c0fbc70 msgr2=0x7f322c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:06.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.448+0000 7f323288c640 1 -- 172.21.15.77:0/22140844 shutdown_connections 2024-03-16T10:17:06.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.448+0000 7f323288c640 1 -- 172.21.15.77:0/22140844 wait complete. 2024-03-16T10:17:06.460 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T10:17:06.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.566+0000 7f556596f640 1 -- 172.21.15.77:0/3239488401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f55601018d0 msgr2=0x7f5560101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.566+0000 7f556596f640 1 --2- 172.21.15.77:0/3239488401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f55601018d0 0x7f5560101d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5554009860 tx=0x7f555402bb70 comp rx=0 tx=0).stop 2024-03-16T10:17:06.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.566+0000 7f556596f640 1 -- 172.21.15.77:0/3239488401 shutdown_connections 2024-03-16T10:17:06.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.566+0000 7f556596f640 1 --2- 172.21.15.77:0/3239488401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5560102290 0x7f5560108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.566+0000 7f556596f640 1 --2- 172.21.15.77:0/3239488401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f55601018d0 0x7f5560101d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.566+0000 7f556596f640 1 --2- 172.21.15.77:0/3239488401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55601006e0 0x7f5560100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.566+0000 7f556596f640 1 -- 172.21.15.77:0/3239488401 >> 172.21.15.77:0/3239488401 conn(0x7f55600fbc70 msgr2=0x7f55600fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.566+0000 7f556596f640 1 -- 172.21.15.77:0/3239488401 shutdown_connections 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.566+0000 7f556596f640 1 -- 172.21.15.77:0/3239488401 wait complete. 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.566+0000 7f556596f640 1 Processor -- start 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f556596f640 1 -- start start 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f556596f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f55601006e0 0x7f5560199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f556596f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55601018d0 0x7f556019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f556596f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5560102290 0x7f55601a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f556596f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f556010c340 con 0x7f5560102290 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f556596f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f556010c1c0 con 0x7f55601018d0 2024-03-16T10:17:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f556596f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f556010c4c0 con 0x7f55601006e0 2024-03-16T10:17:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f555f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5560102290 0x7f55601a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f555f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5560102290 0x7f55601a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56162/0 (socket says 172.21.15.77:56162) 2024-03-16T10:17:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f555effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f55601006e0 0x7f5560199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f555f7fe640 1 -- 172.21.15.77:0/2792082616 learned_addr learned my addr 172.21.15.77:0/2792082616 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f555e7fc640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55601018d0 0x7f556019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f555effd640 1 -- 172.21.15.77:0/2792082616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55601018d0 msgr2=0x7f556019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f555effd640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55601018d0 0x7f556019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f555effd640 1 -- 172.21.15.77:0/2792082616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5560102290 msgr2=0x7f55601a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f555effd640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5560102290 0x7f55601a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f555effd640 1 -- 172.21.15.77:0/2792082616 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55601a1990 con 0x7f55601006e0 2024-03-16T10:17:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.567+0000 7f555effd640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f55601006e0 0x7f5560199d70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5550005fd0 tx=0x7f555000ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.568+0000 7f556496d640 1 -- 172.21.15.77:0/2792082616 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555000ccc0 con 0x7f55601006e0 2024-03-16T10:17:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.568+0000 7f556496d640 1 -- 172.21.15.77:0/2792082616 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f555000ee70 con 0x7f55601006e0 2024-03-16T10:17:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.568+0000 7f556596f640 1 -- 172.21.15.77:0/2792082616 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55601a1c80 con 0x7f55601006e0 2024-03-16T10:17:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.568+0000 7f556496d640 1 -- 172.21.15.77:0/2792082616 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550010430 con 0x7f55601006e0 2024-03-16T10:17:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.568+0000 7f556596f640 1 -- 172.21.15.77:0/2792082616 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55601a21a0 con 0x7f55601006e0 2024-03-16T10:17:06.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.569+0000 7f556596f640 1 -- 172.21.15.77:0/2792082616 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55601a23d0 con 0x7f55601006e0 2024-03-16T10:17:06.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.570+0000 7f556496d640 1 -- 172.21.15.77:0/2792082616 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5550003ef0 con 0x7f55601006e0 2024-03-16T10:17:06.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.570+0000 7f556496d640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f55480a2880 0x7f55480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:06.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.570+0000 7f555e7fc640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f55480a2880 0x7f55480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:06.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.570+0000 7f556496d640 1 -- 172.21.15.77:0/2792082616 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f55500cebd0 con 0x7f55601006e0 2024-03-16T10:17:06.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.571+0000 7f555e7fc640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f55480a2880 0x7f55480a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f555402c180 tx=0x7f5554034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:06.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.575+0000 7f556496d640 1 -- 172.21.15.77:0/2792082616 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f555009bd00 con 0x7f55601006e0 2024-03-16T10:17:06.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.811+0000 7f556596f640 1 -- 172.21.15.77:0/2792082616 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f55601a25f0 con 0x7f55601006e0 2024-03-16T10:17:06.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.812+0000 7f556496d640 1 -- 172.21.15.77:0/2792082616 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f555009b600 con 0x7f55601006e0 2024-03-16T10:17:06.812 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T10:17:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.814+0000 7f556596f640 1 -- 172.21.15.77:0/2792082616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f55480a2880 msgr2=0x7f55480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.814+0000 7f556596f640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f55480a2880 0x7f55480a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f555402c180 tx=0x7f5554034ad0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.814+0000 7f556596f640 1 -- 172.21.15.77:0/2792082616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f55601006e0 msgr2=0x7f5560199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.814+0000 7f556596f640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f55601006e0 0x7f5560199d70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5550005fd0 tx=0x7f555000ec20 comp rx=0 tx=0).stop 2024-03-16T10:17:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.814+0000 7f556596f640 1 -- 172.21.15.77:0/2792082616 shutdown_connections 2024-03-16T10:17:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.814+0000 7f556596f640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5560102290 0x7f55601a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.814+0000 7f556596f640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55601018d0 0x7f556019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.814+0000 7f556596f640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f55480a2880 0x7f55480a4d40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.814+0000 7f556596f640 1 --2- 172.21.15.77:0/2792082616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f55601006e0 0x7f5560199d70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.814+0000 7f556596f640 1 -- 172.21.15.77:0/2792082616 >> 172.21.15.77:0/2792082616 conn(0x7f55600fbc70 msgr2=0x7f55600fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.814+0000 7f556596f640 1 -- 172.21.15.77:0/2792082616 shutdown_connections 2024-03-16T10:17:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.814+0000 7f556596f640 1 -- 172.21.15.77:0/2792082616 wait complete. 2024-03-16T10:17:06.827 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-16T10:17:06.827 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:17:06.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.946+0000 7f2e3e497640 1 -- 172.21.15.77:0/2740211556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e38107520 msgr2=0x7f2e38109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.946+0000 7f2e3e497640 1 --2- 172.21.15.77:0/2740211556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e38107520 0x7f2e38109930 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2e30009ca0 tx=0x7f2e3002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.946+0000 7f2e3e497640 1 -- 172.21.15.77:0/2740211556 shutdown_connections 2024-03-16T10:17:06.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.946+0000 7f2e3e497640 1 --2- 172.21.15.77:0/2740211556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e38107520 0x7f2e38109930 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.946+0000 7f2e3e497640 1 --2- 172.21.15.77:0/2740211556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e38104bf0 0x7f2e38106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.946+0000 7f2e3e497640 1 --2- 172.21.15.77:0/2740211556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e381022c0 0x7f2e381046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.946+0000 7f2e3e497640 1 -- 172.21.15.77:0/2740211556 >> 172.21.15.77:0/2740211556 conn(0x7f2e380fbc70 msgr2=0x7f2e380fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:06.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.947+0000 7f2e3e497640 1 -- 172.21.15.77:0/2740211556 shutdown_connections 2024-03-16T10:17:06.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.947+0000 7f2e3e497640 1 -- 172.21.15.77:0/2740211556 wait complete. 2024-03-16T10:17:06.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.947+0000 7f2e3e497640 1 Processor -- start 2024-03-16T10:17:06.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.947+0000 7f2e3e497640 1 -- start start 2024-03-16T10:17:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e3e497640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e381022c0 0x7f2e38197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e3e497640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e38104bf0 0x7f2e38198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e3e497640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e38107520 0x7f2e3819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e3e497640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3810d4a0 con 0x7f2e38104bf0 2024-03-16T10:17:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e3e497640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3810d320 con 0x7f2e38107520 2024-03-16T10:17:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e3e497640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3810d620 con 0x7f2e381022c0 2024-03-16T10:17:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e377fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e38104bf0 0x7f2e38198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e37fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e381022c0 0x7f2e38197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e377fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e38104bf0 0x7f2e38198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56186/0 (socket says 172.21.15.77:56186) 2024-03-16T10:17:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e377fe640 1 -- 172.21.15.77:0/2309530245 learned_addr learned my addr 172.21.15.77:0/2309530245 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:06.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e3ca34640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e38107520 0x7f2e3819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e377fe640 1 -- 172.21.15.77:0/2309530245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e381022c0 msgr2=0x7f2e38197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e377fe640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e381022c0 0x7f2e38197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e377fe640 1 -- 172.21.15.77:0/2309530245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e38107520 msgr2=0x7f2e3819f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e377fe640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e38107520 0x7f2e3819f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e377fe640 1 -- 172.21.15.77:0/2309530245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e3819f710 con 0x7f2e38104bf0 2024-03-16T10:17:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e377fe640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e38104bf0 0x7f2e38198030 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00e9c0 tx=0x7f2e2c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e357fa640 1 -- 172.21.15.77:0/2309530245 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c00cd80 con 0x7f2e38104bf0 2024-03-16T10:17:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e3e497640 1 -- 172.21.15.77:0/2309530245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e3819fa00 con 0x7f2e38104bf0 2024-03-16T10:17:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e357fa640 1 -- 172.21.15.77:0/2309530245 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e2c005500 con 0x7f2e38104bf0 2024-03-16T10:17:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.948+0000 7f2e357fa640 1 -- 172.21.15.77:0/2309530245 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c010690 con 0x7f2e38104bf0 2024-03-16T10:17:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.949+0000 7f2e3e497640 1 -- 172.21.15.77:0/2309530245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e3819ffa0 con 0x7f2e38104bf0 2024-03-16T10:17:06.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.950+0000 7f2e3e497640 1 -- 172.21.15.77:0/2309530245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e380fd760 con 0x7f2e38104bf0 2024-03-16T10:17:06.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.951+0000 7f2e357fa640 1 -- 172.21.15.77:0/2309530245 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f2e2c0046c0 con 0x7f2e38104bf0 2024-03-16T10:17:06.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.952+0000 7f2e357fa640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e200a2890 0x7f2e200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:06.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.952+0000 7f2e37fff640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e200a2890 0x7f2e200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:06.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.952+0000 7f2e357fa640 1 -- 172.21.15.77:0/2309530245 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2e2c0cee60 con 0x7f2e38104bf0 2024-03-16T10:17:06.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.952+0000 7f2e37fff640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e200a2890 0x7f2e200a4d50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2e28009bd0 tx=0x7f2e28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:06.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:06.956+0000 7f2e357fa640 1 -- 172.21.15.77:0/2309530245 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2e2c09be90 con 0x7f2e38104bf0 2024-03-16T10:17:07.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:06 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/763255744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:17:07.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:06 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/22140844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:17:07.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:06 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2792082616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:17:07.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.188+0000 7f2e3e497640 1 -- 172.21.15.77:0/2309530245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e3802d340 con 0x7f2e38104bf0 2024-03-16T10:17:07.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.189+0000 7f2e357fa640 1 -- 172.21.15.77:0/2309530245 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11938 (secure 0 0 0) 0x7f2e2c09b790 con 0x7f2e38104bf0 2024-03-16T10:17:07.190 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:17:07.190 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:17:02.900684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24475,24454,24482,14658,24455]},"id":1}]} 2024-03-16T10:17:07.190 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 27 2024-03-16T10:17:07.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.191+0000 7f2e3e497640 1 -- 172.21.15.77:0/2309530245 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e200a2890 msgr2=0x7f2e200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:07.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.191+0000 7f2e3e497640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e200a2890 0x7f2e200a4d50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2e28009bd0 tx=0x7f2e28005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:07.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.191+0000 7f2e3e497640 1 -- 172.21.15.77:0/2309530245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e38104bf0 msgr2=0x7f2e38198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:07.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.191+0000 7f2e3e497640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e38104bf0 0x7f2e38198030 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00e9c0 tx=0x7f2e2c00ee80 comp rx=0 tx=0).stop 2024-03-16T10:17:07.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.191+0000 7f2e3e497640 1 -- 172.21.15.77:0/2309530245 shutdown_connections 2024-03-16T10:17:07.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.191+0000 7f2e3e497640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e38107520 0x7f2e3819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.191+0000 7f2e3e497640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e38104bf0 0x7f2e38198030 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.191+0000 7f2e3e497640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2e200a2890 0x7f2e200a4d50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.191+0000 7f2e3e497640 1 --2- 172.21.15.77:0/2309530245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e381022c0 0x7f2e38197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.191+0000 7f2e3e497640 1 -- 172.21.15.77:0/2309530245 >> 172.21.15.77:0/2309530245 conn(0x7f2e380fbc70 msgr2=0x7f2e38102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:07.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.191+0000 7f2e3e497640 1 -- 172.21.15.77:0/2309530245 shutdown_connections 2024-03-16T10:17:07.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.191+0000 7f2e3e497640 1 -- 172.21.15.77:0/2309530245 wait complete. 2024-03-16T10:17:07.202 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:06 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/763255744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:17:07.202 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:06 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/22140844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:17:07.202 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:06 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2792082616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:17:07.203 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-16T10:17:07.203 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-16T10:17:07.203 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T10:17:07.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:06 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/763255744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:17:07.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:06 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/22140844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:17:07.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:06 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2792082616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:17:07.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.306+0000 7f37787eb640 1 -- 172.21.15.77:0/3951997115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37701018f0 msgr2=0x7f3770101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.306+0000 7f37787eb640 1 --2- 172.21.15.77:0/3951997115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37701018f0 0x7f3770101d70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f376c009900 tx=0x7f376c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.306+0000 7f37787eb640 1 -- 172.21.15.77:0/3951997115 shutdown_connections 2024-03-16T10:17:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.306+0000 7f37787eb640 1 --2- 172.21.15.77:0/3951997115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37701022b0 0x7f3770108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.306+0000 7f37787eb640 1 --2- 172.21.15.77:0/3951997115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37701018f0 0x7f3770101d70 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.306+0000 7f37787eb640 1 --2- 172.21.15.77:0/3951997115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3770100700 0x7f3770100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.306+0000 7f37787eb640 1 -- 172.21.15.77:0/3951997115 >> 172.21.15.77:0/3951997115 conn(0x7f37700fbc70 msgr2=0x7f37700fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.306+0000 7f37787eb640 1 -- 172.21.15.77:0/3951997115 shutdown_connections 2024-03-16T10:17:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f37787eb640 1 -- 172.21.15.77:0/3951997115 wait complete. 2024-03-16T10:17:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f37787eb640 1 Processor -- start 2024-03-16T10:17:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f37787eb640 1 -- start start 2024-03-16T10:17:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f37787eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3770100700 0x7f3770197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f37787eb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37701018f0 0x7f3770198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f37787eb640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37701022b0 0x7f377019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f37787eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f377010c380 con 0x7f3770100700 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f37787eb640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f377010c200 con 0x7f37701022b0 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f37787eb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f377010c500 con 0x7f37701018f0 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f3776587640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3770100700 0x7f3770197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f3776587640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3770100700 0x7f3770197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56196/0 (socket says 172.21.15.77:56196) 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f3776587640 1 -- 172.21.15.77:0/2419356837 learned_addr learned my addr 172.21.15.77:0/2419356837 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f3776d88640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37701022b0 0x7f377019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.307+0000 7f3775d86640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37701018f0 0x7f3770198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.308+0000 7f3776587640 1 -- 172.21.15.77:0/2419356837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37701018f0 msgr2=0x7f3770198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.308+0000 7f3776587640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37701018f0 0x7f3770198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.308+0000 7f3776587640 1 -- 172.21.15.77:0/2419356837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37701022b0 msgr2=0x7f377019f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.308+0000 7f3776587640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37701022b0 0x7f377019f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.308+0000 7f3776587640 1 -- 172.21.15.77:0/2419356837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f377019f740 con 0x7f3770100700 2024-03-16T10:17:07.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.308+0000 7f3776d88640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37701022b0 0x7f377019f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:07.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.308+0000 7f3776587640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3770100700 0x7f3770197b20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f376000e9c0 tx=0x7f376000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:07.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.308+0000 7f37677fe640 1 -- 172.21.15.77:0/2419356837 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376000cd80 con 0x7f3770100700 2024-03-16T10:17:07.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.308+0000 7f37677fe640 1 -- 172.21.15.77:0/2419356837 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37600035f0 con 0x7f3770100700 2024-03-16T10:17:07.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.308+0000 7f37787eb640 1 -- 172.21.15.77:0/2419356837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f377019fa30 con 0x7f3770100700 2024-03-16T10:17:07.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.308+0000 7f37677fe640 1 -- 172.21.15.77:0/2419356837 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760010690 con 0x7f3770100700 2024-03-16T10:17:07.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.308+0000 7f37787eb640 1 -- 172.21.15.77:0/2419356837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f377019ff50 con 0x7f3770100700 2024-03-16T10:17:07.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.309+0000 7f37787eb640 1 -- 172.21.15.77:0/2419356837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37700685d0 con 0x7f3770100700 2024-03-16T10:17:07.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.311+0000 7f37677fe640 1 -- 172.21.15.77:0/2419356837 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f37600037a0 con 0x7f3770100700 2024-03-16T10:17:07.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.311+0000 7f37677fe640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f375c0a2890 0x7f375c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:07.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.311+0000 7f37677fe640 1 -- 172.21.15.77:0/2419356837 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f37600cef90 con 0x7f3770100700 2024-03-16T10:17:07.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.311+0000 7f3775d86640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f375c0a2890 0x7f375c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:07.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.312+0000 7f3775d86640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f375c0a2890 0x7f375c0a4d50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f376c02c7b0 tx=0x7f376c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:07.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.315+0000 7f37677fe640 1 -- 172.21.15.77:0/2419356837 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f376009c050 con 0x7f3770100700 2024-03-16T10:17:07.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.561+0000 7f37787eb640 1 -- 172.21.15.77:0/2419356837 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f37700fde20 con 0x7f375c0a2890 2024-03-16T10:17:07.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.569+0000 7f37677fe640 1 -- 172.21.15.77:0/2419356837 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259101 (secure 0 0 0) 0x7f37700fde20 con 0x7f375c0a2890 2024-03-16T10:17:07.572 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:17:07.574 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T10:17:07.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.572+0000 7f37787eb640 1 -- 172.21.15.77:0/2419356837 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f375c0a2890 msgr2=0x7f375c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:07.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.572+0000 7f37787eb640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f375c0a2890 0x7f375c0a4d50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f376c02c7b0 tx=0x7f376c005fd0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.572+0000 7f37787eb640 1 -- 172.21.15.77:0/2419356837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3770100700 msgr2=0x7f3770197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:07.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.572+0000 7f37787eb640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3770100700 0x7f3770197b20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f376000e9c0 tx=0x7f376000ee80 comp rx=0 tx=0).stop 2024-03-16T10:17:07.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.572+0000 7f37787eb640 1 -- 172.21.15.77:0/2419356837 shutdown_connections 2024-03-16T10:17:07.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.572+0000 7f37787eb640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37701022b0 0x7f377019f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.572+0000 7f37787eb640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f375c0a2890 0x7f375c0a4d50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.572+0000 7f37787eb640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37701018f0 0x7f3770198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.572+0000 7f37787eb640 1 --2- 172.21.15.77:0/2419356837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3770100700 0x7f3770197b20 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.573+0000 7f37787eb640 1 -- 172.21.15.77:0/2419356837 >> 172.21.15.77:0/2419356837 conn(0x7f37700fbc70 msgr2=0x7f37700fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:07.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.573+0000 7f37787eb640 1 -- 172.21.15.77:0/2419356837 shutdown_connections 2024-03-16T10:17:07.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.573+0000 7f37787eb640 1 -- 172.21.15.77:0/2419356837 wait complete. 2024-03-16T10:17:07.585 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-03-16T10:17:06.743397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658527,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1323,"num_read_kb":784,"num_write":436,"num_write_kb":3084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":858,"ondisk_log_size":858,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360852,"kb_used_data":11700,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580972,"statfs":{"total":1151940427776,"available":1151570915328,"internally_reserved":0,"allocated":11980800,"data_stored":7440687,"data_compressed":112533,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":699,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":52,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002662"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.757174+0000","last_change":"2024-03-16T10:15:03.016880+0000","last_active":"2024-03-16T10:16:28.757174+0000","last_peered":"2024-03-16T10:16:28.757174+0000","last_clean":"2024-03-16T10:16:28.757174+0000","last_became_active":"2024-03-16T10:15:03.016513+0000","last_became_peered":"2024-03-16T10:15:03.016513+0000","last_unstale":"2024-03-16T10:16:28.757174+0000","last_undegraded":"2024-03-16T10:16:28.757174+0000","last_fullsized":"2024-03-16T10:16:28.757174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:39.834015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.753595+0000","last_change":"2024-03-16T10:15:03.015467+0000","last_active":"2024-03-16T10:16:28.753595+0000","last_peered":"2024-03-16T10:16:28.753595+0000","last_clean":"2024-03-16T10:16:28.753595+0000","last_became_active":"2024-03-16T10:15:03.015152+0000","last_became_peered":"2024-03-16T10:15:03.015152+0000","last_unstale":"2024-03-16T10:16:28.753595+0000","last_undegraded":"2024-03-16T10:16:28.753595+0000","last_fullsized":"2024-03-16T10:16:28.753595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:17:29.988569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.753318+0000","last_change":"2024-03-16T10:15:03.023836+0000","last_active":"2024-03-16T10:16:28.753318+0000","last_peered":"2024-03-16T10:16:28.753318+0000","last_clean":"2024-03-16T10:16:28.753318+0000","last_became_active":"2024-03-16T10:15:03.022774+0000","last_became_peered":"2024-03-16T10:15:03.022774+0000","last_unstale":"2024-03-16T10:16:28.753318+0000","last_undegraded":"2024-03-16T10:16:28.753318+0000","last_fullsized":"2024-03-16T10:16:28.753318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:49.837157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.759581+0000","last_change":"2024-03-16T10:15:03.030891+0000","last_active":"2024-03-16T10:16:28.759581+0000","last_peered":"2024-03-16T10:16:28.759581+0000","last_clean":"2024-03-16T10:16:28.759581+0000","last_became_active":"2024-03-16T10:15:03.030278+0000","last_became_peered":"2024-03-16T10:15:03.030278+0000","last_unstale":"2024-03-16T10:16:28.759581+0000","last_undegraded":"2024-03-16T10:16:28.759581+0000","last_fullsized":"2024-03-16T10:16:28.759581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:16.166885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.742976+0000","last_change":"2024-03-16T10:15:03.017610+0000","last_active":"2024-03-16T10:16:28.742976+0000","last_peered":"2024-03-16T10:16:28.742976+0000","last_clean":"2024-03-16T10:16:28.742976+0000","last_became_active":"2024-03-16T10:15:03.017250+0000","last_became_peered":"2024-03-16T10:15:03.017250+0000","last_unstale":"2024-03-16T10:16:28.742976+0000","last_undegraded":"2024-03-16T10:16:28.742976+0000","last_fullsized":"2024-03-16T10:16:28.742976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:07.899425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.753472+0000","last_change":"2024-03-16T10:15:59.856216+0000","last_active":"2024-03-16T10:16:28.753472+0000","last_peered":"2024-03-16T10:16:28.753472+0000","last_clean":"2024-03-16T10:16:28.753472+0000","last_became_active":"2024-03-16T10:15:59.855882+0000","last_became_peered":"2024-03-16T10:15:59.855882+0000","last_unstale":"2024-03-16T10:16:28.753472+0000","last_undegraded":"2024-03-16T10:16:28.753472+0000","last_fullsized":"2024-03-16T10:16:28.753472+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:27:42.682580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762588+0000","last_change":"2024-03-16T10:15:03.017686+0000","last_active":"2024-03-16T10:16:28.762588+0000","last_peered":"2024-03-16T10:16:28.762588+0000","last_clean":"2024-03-16T10:16:28.762588+0000","last_became_active":"2024-03-16T10:15:03.017262+0000","last_became_peered":"2024-03-16T10:15:03.017262+0000","last_unstale":"2024-03-16T10:16:28.762588+0000","last_undegraded":"2024-03-16T10:16:28.762588+0000","last_fullsized":"2024-03-16T10:16:28.762588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:36:22.270651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.759952+0000","last_change":"2024-03-16T10:15:03.020130+0000","last_active":"2024-03-16T10:16:28.759952+0000","last_peered":"2024-03-16T10:16:28.759952+0000","last_clean":"2024-03-16T10:16:28.759952+0000","last_became_active":"2024-03-16T10:15:03.019742+0000","last_became_peered":"2024-03-16T10:15:03.019742+0000","last_unstale":"2024-03-16T10:16:28.759952+0000","last_undegraded":"2024-03-16T10:16:28.759952+0000","last_fullsized":"2024-03-16T10:16:28.759952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:52.450792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.757769+0000","last_change":"2024-03-16T10:15:03.027589+0000","last_active":"2024-03-16T10:16:28.757769+0000","last_peered":"2024-03-16T10:16:28.757769+0000","last_clean":"2024-03-16T10:16:28.757769+0000","last_became_active":"2024-03-16T10:15:03.027226+0000","last_became_peered":"2024-03-16T10:15:03.027226+0000","last_unstale":"2024-03-16T10:16:28.757769+0000","last_undegraded":"2024-03-16T10:16:28.757769+0000","last_fullsized":"2024-03-16T10:16:28.757769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:14:35.354629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.761345+0000","last_change":"2024-03-16T10:15:03.021163+0000","last_active":"2024-03-16T10:16:28.761345+0000","last_peered":"2024-03-16T10:16:28.761345+0000","last_clean":"2024-03-16T10:16:28.761345+0000","last_became_active":"2024-03-16T10:15:03.020483+0000","last_became_peered":"2024-03-16T10:15:03.020483+0000","last_unstale":"2024-03-16T10:16:28.761345+0000","last_undegraded":"2024-03-16T10:16:28.761345+0000","last_fullsized":"2024-03-16T10:16:28.761345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:01:00.873471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.765112+0000","last_change":"2024-03-16T10:15:03.026111+0000","last_active":"2024-03-16T10:16:28.765112+0000","last_peered":"2024-03-16T10:16:28.765112+0000","last_clean":"2024-03-16T10:16:28.765112+0000","last_became_active":"2024-03-16T10:15:03.025296+0000","last_became_peered":"2024-03-16T10:15:03.025296+0000","last_unstale":"2024-03-16T10:16:28.765112+0000","last_undegraded":"2024-03-16T10:16:28.765112+0000","last_fullsized":"2024-03-16T10:16:28.765112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:33:29.606876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.746441+0000","last_change":"2024-03-16T10:15:03.023013+0000","last_active":"2024-03-16T10:16:28.746441+0000","last_peered":"2024-03-16T10:16:28.746441+0000","last_clean":"2024-03-16T10:16:28.746441+0000","last_became_active":"2024-03-16T10:15:03.022251+0000","last_became_peered":"2024-03-16T10:15:03.022251+0000","last_unstale":"2024-03-16T10:16:28.746441+0000","last_undegraded":"2024-03-16T10:16:28.746441+0000","last_fullsized":"2024-03-16T10:16:28.746441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:33.544709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.750180+0000","last_change":"2024-03-16T10:15:03.009548+0000","last_active":"2024-03-16T10:16:28.750180+0000","last_peered":"2024-03-16T10:16:28.750180+0000","last_clean":"2024-03-16T10:16:28.750180+0000","last_became_active":"2024-03-16T10:15:03.007675+0000","last_became_peered":"2024-03-16T10:15:03.007675+0000","last_unstale":"2024-03-16T10:16:28.750180+0000","last_undegraded":"2024-03-16T10:16:28.750180+0000","last_fullsized":"2024-03-16T10:16:28.750180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:08:02.561602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756541+0000","last_change":"2024-03-16T10:15:03.020146+0000","last_active":"2024-03-16T10:16:28.756541+0000","last_peered":"2024-03-16T10:16:28.756541+0000","last_clean":"2024-03-16T10:16:28.756541+0000","last_became_active":"2024-03-16T10:15:03.019823+0000","last_became_peered":"2024-03-16T10:15:03.019823+0000","last_unstale":"2024-03-16T10:16:28.756541+0000","last_undegraded":"2024-03-16T10:16:28.756541+0000","last_fullsized":"2024-03-16T10:16:28.756541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:52.276506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.760668+0000","last_change":"2024-03-16T10:15:03.019150+0000","last_active":"2024-03-16T10:16:28.760668+0000","last_peered":"2024-03-16T10:16:28.760668+0000","last_clean":"2024-03-16T10:16:28.760668+0000","last_became_active":"2024-03-16T10:15:03.018830+0000","last_became_peered":"2024-03-16T10:15:03.018830+0000","last_unstale":"2024-03-16T10:16:28.760668+0000","last_undegraded":"2024-03-16T10:16:28.760668+0000","last_fullsized":"2024-03-16T10:16:28.760668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:46.207187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762615+0000","last_change":"2024-03-16T10:15:03.033321+0000","last_active":"2024-03-16T10:16:28.762615+0000","last_peered":"2024-03-16T10:16:28.762615+0000","last_clean":"2024-03-16T10:16:28.762615+0000","last_became_active":"2024-03-16T10:15:03.032944+0000","last_became_peered":"2024-03-16T10:15:03.032944+0000","last_unstale":"2024-03-16T10:16:28.762615+0000","last_undegraded":"2024-03-16T10:16:28.762615+0000","last_fullsized":"2024-03-16T10:16:28.762615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:14.345488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.757536+0000","last_change":"2024-03-16T10:15:03.019897+0000","last_active":"2024-03-16T10:16:28.757536+0000","last_peered":"2024-03-16T10:16:28.757536+0000","last_clean":"2024-03-16T10:16:28.757536+0000","last_became_active":"2024-03-16T10:15:03.019555+0000","last_became_peered":"2024-03-16T10:15:03.019555+0000","last_unstale":"2024-03-16T10:16:28.757536+0000","last_undegraded":"2024-03-16T10:16:28.757536+0000","last_fullsized":"2024-03-16T10:16:28.757536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:58.307285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.758416+0000","last_change":"2024-03-16T10:15:03.017482+0000","last_active":"2024-03-16T10:16:28.758416+0000","last_peered":"2024-03-16T10:16:28.758416+0000","last_clean":"2024-03-16T10:16:28.758416+0000","last_became_active":"2024-03-16T10:15:03.016521+0000","last_became_peered":"2024-03-16T10:15:03.016521+0000","last_unstale":"2024-03-16T10:16:28.758416+0000","last_undegraded":"2024-03-16T10:16:28.758416+0000","last_fullsized":"2024-03-16T10:16:28.758416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:11.889111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.747191+0000","last_change":"2024-03-16T10:15:03.018625+0000","last_active":"2024-03-16T10:16:28.747191+0000","last_peered":"2024-03-16T10:16:28.747191+0000","last_clean":"2024-03-16T10:16:28.747191+0000","last_became_active":"2024-03-16T10:15:03.018313+0000","last_became_peered":"2024-03-16T10:15:03.018313+0000","last_unstale":"2024-03-16T10:16:28.747191+0000","last_undegraded":"2024-03-16T10:16:28.747191+0000","last_fullsized":"2024-03-16T10:16:28.747191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:35:58.533559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'14","reported_seq":149,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:02.357489+0000","last_change":"2024-03-16T10:15:02.016545+0000","last_active":"2024-03-16T10:17:02.357489+0000","last_peered":"2024-03-16T10:17:02.357489+0000","last_clean":"2024-03-16T10:17:02.357489+0000","last_became_active":"2024-03-16T10:15:02.012088+0000","last_became_peered":"2024-03-16T10:15:02.012088+0000","last_unstale":"2024-03-16T10:17:02.357489+0000","last_undegraded":"2024-03-16T10:17:02.357489+0000","last_fullsized":"2024-03-16T10:17:02.357489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:50.633539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.740676+0000","last_change":"2024-03-16T10:15:03.026565+0000","last_active":"2024-03-16T10:16:28.740676+0000","last_peered":"2024-03-16T10:16:28.740676+0000","last_clean":"2024-03-16T10:16:28.740676+0000","last_became_active":"2024-03-16T10:15:03.025386+0000","last_became_peered":"2024-03-16T10:15:03.025386+0000","last_unstale":"2024-03-16T10:16:28.740676+0000","last_undegraded":"2024-03-16T10:16:28.740676+0000","last_fullsized":"2024-03-16T10:16:28.740676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:28.831764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838352+0000","last_change":"2024-03-16T10:15:02.018097+0000","last_active":"2024-03-16T10:15:59.838352+0000","last_peered":"2024-03-16T10:15:59.838352+0000","last_clean":"2024-03-16T10:15:59.838352+0000","last_became_active":"2024-03-16T10:15:02.017777+0000","last_became_peered":"2024-03-16T10:15:02.017777+0000","last_unstale":"2024-03-16T10:15:59.838352+0000","last_undegraded":"2024-03-16T10:15:59.838352+0000","last_fullsized":"2024-03-16T10:15:59.838352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:04:58.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.753086+0000","last_change":"2024-03-16T10:15:03.023384+0000","last_active":"2024-03-16T10:16:28.753086+0000","last_peered":"2024-03-16T10:16:28.753086+0000","last_clean":"2024-03-16T10:16:28.753086+0000","last_became_active":"2024-03-16T10:15:03.022996+0000","last_became_peered":"2024-03-16T10:15:03.022996+0000","last_unstale":"2024-03-16T10:16:28.753086+0000","last_undegraded":"2024-03-16T10:16:28.753086+0000","last_fullsized":"2024-03-16T10:16:28.753086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:29:20.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834807+0000","last_change":"2024-03-16T10:15:02.014409+0000","last_active":"2024-03-16T10:15:59.834807+0000","last_peered":"2024-03-16T10:15:59.834807+0000","last_clean":"2024-03-16T10:15:59.834807+0000","last_became_active":"2024-03-16T10:15:02.014015+0000","last_became_peered":"2024-03-16T10:15:02.014015+0000","last_unstale":"2024-03-16T10:15:59.834807+0000","last_undegraded":"2024-03-16T10:15:59.834807+0000","last_fullsized":"2024-03-16T10:15:59.834807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:25:53.133466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.759810+0000","last_change":"2024-03-16T10:15:03.031328+0000","last_active":"2024-03-16T10:16:28.759810+0000","last_peered":"2024-03-16T10:16:28.759810+0000","last_clean":"2024-03-16T10:16:28.759810+0000","last_became_active":"2024-03-16T10:15:03.029787+0000","last_became_peered":"2024-03-16T10:15:03.029787+0000","last_unstale":"2024-03-16T10:16:28.759810+0000","last_undegraded":"2024-03-16T10:16:28.759810+0000","last_fullsized":"2024-03-16T10:16:28.759810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:34.321883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834594+0000","last_change":"2024-03-16T10:15:02.018032+0000","last_active":"2024-03-16T10:15:59.834594+0000","last_peered":"2024-03-16T10:15:59.834594+0000","last_clean":"2024-03-16T10:15:59.834594+0000","last_became_active":"2024-03-16T10:15:02.017677+0000","last_became_peered":"2024-03-16T10:15:02.017677+0000","last_unstale":"2024-03-16T10:15:59.834594+0000","last_undegraded":"2024-03-16T10:15:59.834594+0000","last_fullsized":"2024-03-16T10:15:59.834594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:58:07.179129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762172+0000","last_change":"2024-03-16T10:15:03.024958+0000","last_active":"2024-03-16T10:16:28.762172+0000","last_peered":"2024-03-16T10:16:28.762172+0000","last_clean":"2024-03-16T10:16:28.762172+0000","last_became_active":"2024-03-16T10:15:03.023964+0000","last_became_peered":"2024-03-16T10:15:03.023964+0000","last_unstale":"2024-03-16T10:16:28.762172+0000","last_undegraded":"2024-03-16T10:16:28.762172+0000","last_fullsized":"2024-03-16T10:16:28.762172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:55:00.953258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":146,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:04.348190+0000","last_change":"2024-03-16T10:15:02.015062+0000","last_active":"2024-03-16T10:17:04.348190+0000","last_peered":"2024-03-16T10:17:04.348190+0000","last_clean":"2024-03-16T10:17:04.348190+0000","last_became_active":"2024-03-16T10:15:02.014714+0000","last_became_peered":"2024-03-16T10:15:02.014714+0000","last_unstale":"2024-03-16T10:17:04.348190+0000","last_undegraded":"2024-03-16T10:17:04.348190+0000","last_fullsized":"2024-03-16T10:17:04.348190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:23:55.313009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.743988+0000","last_change":"2024-03-16T10:15:03.013956+0000","last_active":"2024-03-16T10:16:28.743988+0000","last_peered":"2024-03-16T10:16:28.743988+0000","last_clean":"2024-03-16T10:16:28.743988+0000","last_became_active":"2024-03-16T10:15:03.013607+0000","last_became_peered":"2024-03-16T10:15:03.013607+0000","last_unstale":"2024-03-16T10:16:28.743988+0000","last_undegraded":"2024-03-16T10:16:28.743988+0000","last_fullsized":"2024-03-16T10:16:28.743988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:40:55.445395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.841646+0000","last_change":"2024-03-16T10:15:02.015651+0000","last_active":"2024-03-16T10:15:59.841646+0000","last_peered":"2024-03-16T10:15:59.841646+0000","last_clean":"2024-03-16T10:15:59.841646+0000","last_became_active":"2024-03-16T10:15:02.015254+0000","last_became_peered":"2024-03-16T10:15:02.015254+0000","last_unstale":"2024-03-16T10:15:59.841646+0000","last_undegraded":"2024-03-16T10:15:59.841646+0000","last_fullsized":"2024-03-16T10:15:59.841646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:41:59.013606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756364+0000","last_change":"2024-03-16T10:15:03.023074+0000","last_active":"2024-03-16T10:16:28.756364+0000","last_peered":"2024-03-16T10:16:28.756364+0000","last_clean":"2024-03-16T10:16:28.756364+0000","last_became_active":"2024-03-16T10:15:03.022700+0000","last_became_peered":"2024-03-16T10:15:03.022700+0000","last_unstale":"2024-03-16T10:16:28.756364+0000","last_undegraded":"2024-03-16T10:16:28.756364+0000","last_fullsized":"2024-03-16T10:16:28.756364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:22.047929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835321+0000","last_change":"2024-03-16T10:15:02.014793+0000","last_active":"2024-03-16T10:15:59.835321+0000","last_peered":"2024-03-16T10:15:59.835321+0000","last_clean":"2024-03-16T10:15:59.835321+0000","last_became_active":"2024-03-16T10:15:02.014459+0000","last_became_peered":"2024-03-16T10:15:02.014459+0000","last_unstale":"2024-03-16T10:15:59.835321+0000","last_undegraded":"2024-03-16T10:15:59.835321+0000","last_fullsized":"2024-03-16T10:15:59.835321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:59:45.035045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.755311+0000","last_change":"2024-03-16T10:15:03.012713+0000","last_active":"2024-03-16T10:16:28.755311+0000","last_peered":"2024-03-16T10:16:28.755311+0000","last_clean":"2024-03-16T10:16:28.755311+0000","last_became_active":"2024-03-16T10:15:03.012407+0000","last_became_peered":"2024-03-16T10:15:03.012407+0000","last_unstale":"2024-03-16T10:16:28.755311+0000","last_undegraded":"2024-03-16T10:16:28.755311+0000","last_fullsized":"2024-03-16T10:16:28.755311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:23:14.592810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898544+0000","last_change":"2024-03-16T10:15:02.015582+0000","last_active":"2024-03-16T10:15:59.898544+0000","last_peered":"2024-03-16T10:15:59.898544+0000","last_clean":"2024-03-16T10:15:59.898544+0000","last_became_active":"2024-03-16T10:15:02.015073+0000","last_became_peered":"2024-03-16T10:15:02.015073+0000","last_unstale":"2024-03-16T10:15:59.898544+0000","last_undegraded":"2024-03-16T10:15:59.898544+0000","last_fullsized":"2024-03-16T10:15:59.898544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:22:48.212448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.750647+0000","last_change":"2024-03-16T10:15:03.016839+0000","last_active":"2024-03-16T10:16:28.750647+0000","last_peered":"2024-03-16T10:16:28.750647+0000","last_clean":"2024-03-16T10:16:28.750647+0000","last_became_active":"2024-03-16T10:15:03.016463+0000","last_became_peered":"2024-03-16T10:15:03.016463+0000","last_unstale":"2024-03-16T10:16:28.750647+0000","last_undegraded":"2024-03-16T10:16:28.750647+0000","last_fullsized":"2024-03-16T10:16:28.750647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:43:34.259049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'5","reported_seq":256,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:03.346162+0000","last_change":"2024-03-16T10:15:02.017884+0000","last_active":"2024-03-16T10:17:03.346162+0000","last_peered":"2024-03-16T10:17:03.346162+0000","last_clean":"2024-03-16T10:17:03.346162+0000","last_became_active":"2024-03-16T10:15:02.017511+0000","last_became_peered":"2024-03-16T10:15:02.017511+0000","last_unstale":"2024-03-16T10:17:03.346162+0000","last_undegraded":"2024-03-16T10:17:03.346162+0000","last_fullsized":"2024-03-16T10:17:03.346162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:31.281631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":232,"num_read_kb":232,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.754815+0000","last_change":"2024-03-16T10:15:03.032259+0000","last_active":"2024-03-16T10:16:28.754815+0000","last_peered":"2024-03-16T10:16:28.754815+0000","last_clean":"2024-03-16T10:16:28.754815+0000","last_became_active":"2024-03-16T10:15:03.031884+0000","last_became_peered":"2024-03-16T10:15:03.031884+0000","last_unstale":"2024-03-16T10:16:28.754815+0000","last_undegraded":"2024-03-16T10:16:28.754815+0000","last_fullsized":"2024-03-16T10:16:28.754815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:37:48.464051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839326+0000","last_change":"2024-03-16T10:15:02.015372+0000","last_active":"2024-03-16T10:15:59.839326+0000","last_peered":"2024-03-16T10:15:59.839326+0000","last_clean":"2024-03-16T10:15:59.839326+0000","last_became_active":"2024-03-16T10:15:02.015022+0000","last_became_peered":"2024-03-16T10:15:02.015022+0000","last_unstale":"2024-03-16T10:15:59.839326+0000","last_undegraded":"2024-03-16T10:15:59.839326+0000","last_fullsized":"2024-03-16T10:15:59.839326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:42.651903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.760728+0000","last_change":"2024-03-16T10:15:03.018661+0000","last_active":"2024-03-16T10:16:28.760728+0000","last_peered":"2024-03-16T10:16:28.760728+0000","last_clean":"2024-03-16T10:16:28.760728+0000","last_became_active":"2024-03-16T10:15:03.018351+0000","last_became_peered":"2024-03-16T10:15:03.018351+0000","last_unstale":"2024-03-16T10:16:28.760728+0000","last_undegraded":"2024-03-16T10:16:28.760728+0000","last_fullsized":"2024-03-16T10:16:28.760728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:21:33.939811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.899049+0000","last_change":"2024-03-16T10:15:02.014343+0000","last_active":"2024-03-16T10:15:59.899049+0000","last_peered":"2024-03-16T10:15:59.899049+0000","last_clean":"2024-03-16T10:15:59.899049+0000","last_became_active":"2024-03-16T10:15:02.013983+0000","last_became_peered":"2024-03-16T10:15:02.013983+0000","last_unstale":"2024-03-16T10:15:59.899049+0000","last_undegraded":"2024-03-16T10:15:59.899049+0000","last_fullsized":"2024-03-16T10:15:59.899049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:35:04.768079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.761372+0000","last_change":"2024-03-16T10:15:03.031055+0000","last_active":"2024-03-16T10:16:28.761372+0000","last_peered":"2024-03-16T10:16:28.761372+0000","last_clean":"2024-03-16T10:16:28.761372+0000","last_became_active":"2024-03-16T10:15:03.029656+0000","last_became_peered":"2024-03-16T10:15:03.029656+0000","last_unstale":"2024-03-16T10:16:28.761372+0000","last_undegraded":"2024-03-16T10:16:28.761372+0000","last_fullsized":"2024-03-16T10:16:28.761372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:27:35.684774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'37","reported_seq":299,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:03.346714+0000","last_change":"2024-03-16T10:15:02.014752+0000","last_active":"2024-03-16T10:17:03.346714+0000","last_peered":"2024-03-16T10:17:03.346714+0000","last_clean":"2024-03-16T10:17:03.346714+0000","last_became_active":"2024-03-16T10:15:02.013763+0000","last_became_peered":"2024-03-16T10:15:02.013763+0000","last_unstale":"2024-03-16T10:17:03.346714+0000","last_undegraded":"2024-03-16T10:17:03.346714+0000","last_fullsized":"2024-03-16T10:17:03.346714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:12:12.759643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60580,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":243,"num_read_kb":182,"num_write":39,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762495+0000","last_change":"2024-03-16T10:15:03.018838+0000","last_active":"2024-03-16T10:16:28.762495+0000","last_peered":"2024-03-16T10:16:28.762495+0000","last_clean":"2024-03-16T10:16:28.762495+0000","last_became_active":"2024-03-16T10:15:03.017713+0000","last_became_peered":"2024-03-16T10:15:03.017713+0000","last_unstale":"2024-03-16T10:16:28.762495+0000","last_undegraded":"2024-03-16T10:16:28.762495+0000","last_fullsized":"2024-03-16T10:16:28.762495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:03:56.936729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835759+0000","last_change":"2024-03-16T10:15:02.016996+0000","last_active":"2024-03-16T10:15:59.835759+0000","last_peered":"2024-03-16T10:15:59.835759+0000","last_clean":"2024-03-16T10:15:59.835759+0000","last_became_active":"2024-03-16T10:15:02.016421+0000","last_became_peered":"2024-03-16T10:15:02.016421+0000","last_unstale":"2024-03-16T10:15:59.835759+0000","last_undegraded":"2024-03-16T10:15:59.835759+0000","last_fullsized":"2024-03-16T10:15:59.835759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:25:09.866730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.750159+0000","last_change":"2024-03-16T10:15:03.017842+0000","last_active":"2024-03-16T10:16:28.750159+0000","last_peered":"2024-03-16T10:16:28.750159+0000","last_clean":"2024-03-16T10:16:28.750159+0000","last_became_active":"2024-03-16T10:15:03.015119+0000","last_became_peered":"2024-03-16T10:15:03.015119+0000","last_unstale":"2024-03-16T10:16:28.750159+0000","last_undegraded":"2024-03-16T10:16:28.750159+0000","last_fullsized":"2024-03-16T10:16:28.750159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:14:34.474519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898795+0000","last_change":"2024-03-16T10:15:02.021407+0000","last_active":"2024-03-16T10:15:59.898795+0000","last_peered":"2024-03-16T10:15:59.898795+0000","last_clean":"2024-03-16T10:15:59.898795+0000","last_became_active":"2024-03-16T10:15:02.020884+0000","last_became_peered":"2024-03-16T10:15:02.020884+0000","last_unstale":"2024-03-16T10:15:59.898795+0000","last_undegraded":"2024-03-16T10:15:59.898795+0000","last_fullsized":"2024-03-16T10:15:59.898795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:24.858281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.766085+0000","last_change":"2024-03-16T10:15:03.028607+0000","last_active":"2024-03-16T10:16:28.766085+0000","last_peered":"2024-03-16T10:16:28.766085+0000","last_clean":"2024-03-16T10:16:28.766085+0000","last_became_active":"2024-03-16T10:15:03.028280+0000","last_became_peered":"2024-03-16T10:15:03.028280+0000","last_unstale":"2024-03-16T10:16:28.766085+0000","last_undegraded":"2024-03-16T10:16:28.766085+0000","last_fullsized":"2024-03-16T10:16:28.766085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:39.905921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'5","reported_seq":141,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:01.356857+0000","last_change":"2024-03-16T10:15:02.008558+0000","last_active":"2024-03-16T10:17:01.356857+0000","last_peered":"2024-03-16T10:17:01.356857+0000","last_clean":"2024-03-16T10:17:01.356857+0000","last_became_active":"2024-03-16T10:15:02.008014+0000","last_became_peered":"2024-03-16T10:15:02.008014+0000","last_unstale":"2024-03-16T10:17:01.356857+0000","last_undegraded":"2024-03-16T10:17:01.356857+0000","last_fullsized":"2024-03-16T10:17:01.356857+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:40:09.217991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.763448+0000","last_change":"2024-03-16T10:15:03.020847+0000","last_active":"2024-03-16T10:16:28.763448+0000","last_peered":"2024-03-16T10:16:28.763448+0000","last_clean":"2024-03-16T10:16:28.763448+0000","last_became_active":"2024-03-16T10:15:03.020512+0000","last_became_peered":"2024-03-16T10:15:03.020512+0000","last_unstale":"2024-03-16T10:16:28.763448+0000","last_undegraded":"2024-03-16T10:16:28.763448+0000","last_fullsized":"2024-03-16T10:16:28.763448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:04.436195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834281+0000","last_change":"2024-03-16T10:15:02.010690+0000","last_active":"2024-03-16T10:15:59.834281+0000","last_peered":"2024-03-16T10:15:59.834281+0000","last_clean":"2024-03-16T10:15:59.834281+0000","last_became_active":"2024-03-16T10:15:02.010228+0000","last_became_peered":"2024-03-16T10:15:02.010228+0000","last_unstale":"2024-03-16T10:15:59.834281+0000","last_undegraded":"2024-03-16T10:15:59.834281+0000","last_fullsized":"2024-03-16T10:15:59.834281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:30:07.513225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756488+0000","last_change":"2024-03-16T10:15:03.025257+0000","last_active":"2024-03-16T10:16:28.756488+0000","last_peered":"2024-03-16T10:16:28.756488+0000","last_clean":"2024-03-16T10:16:28.756488+0000","last_became_active":"2024-03-16T10:15:03.024498+0000","last_became_peered":"2024-03-16T10:15:03.024498+0000","last_unstale":"2024-03-16T10:16:28.756488+0000","last_undegraded":"2024-03-16T10:16:28.756488+0000","last_fullsized":"2024-03-16T10:16:28.756488+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:27:55.894728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834712+0000","last_change":"2024-03-16T10:15:02.014136+0000","last_active":"2024-03-16T10:15:59.834712+0000","last_peered":"2024-03-16T10:15:59.834712+0000","last_clean":"2024-03-16T10:15:59.834712+0000","last_became_active":"2024-03-16T10:15:02.013721+0000","last_became_peered":"2024-03-16T10:15:02.013721+0000","last_unstale":"2024-03-16T10:15:59.834712+0000","last_undegraded":"2024-03-16T10:15:59.834712+0000","last_fullsized":"2024-03-16T10:15:59.834712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:28:12.301450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.744259+0000","last_change":"2024-03-16T10:15:59.856566+0000","last_active":"2024-03-16T10:16:28.744259+0000","last_peered":"2024-03-16T10:16:28.744259+0000","last_clean":"2024-03-16T10:16:28.744259+0000","last_became_active":"2024-03-16T10:15:59.856152+0000","last_became_peered":"2024-03-16T10:15:59.856152+0000","last_unstale":"2024-03-16T10:16:28.744259+0000","last_undegraded":"2024-03-16T10:16:28.744259+0000","last_fullsized":"2024-03-16T10:16:28.744259+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:42:39.104928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.855367+0000","last_change":"2024-03-16T10:15:02.005754+0000","last_active":"2024-03-16T10:15:59.855367+0000","last_peered":"2024-03-16T10:15:59.855367+0000","last_clean":"2024-03-16T10:15:59.855367+0000","last_became_active":"2024-03-16T10:15:02.005344+0000","last_became_peered":"2024-03-16T10:15:02.005344+0000","last_unstale":"2024-03-16T10:15:59.855367+0000","last_undegraded":"2024-03-16T10:15:59.855367+0000","last_fullsized":"2024-03-16T10:15:59.855367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:37:37.918861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.752059+0000","last_change":"2024-03-16T10:15:03.012887+0000","last_active":"2024-03-16T10:16:28.752059+0000","last_peered":"2024-03-16T10:16:28.752059+0000","last_clean":"2024-03-16T10:16:28.752059+0000","last_became_active":"2024-03-16T10:15:03.012496+0000","last_became_peered":"2024-03-16T10:15:03.012496+0000","last_unstale":"2024-03-16T10:16:28.752059+0000","last_undegraded":"2024-03-16T10:16:28.752059+0000","last_fullsized":"2024-03-16T10:16:28.752059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:11.111049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":138,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:00.355736+0000","last_change":"2024-03-16T10:15:02.014334+0000","last_active":"2024-03-16T10:17:00.355736+0000","last_peered":"2024-03-16T10:17:00.355736+0000","last_clean":"2024-03-16T10:17:00.355736+0000","last_became_active":"2024-03-16T10:15:02.013832+0000","last_became_peered":"2024-03-16T10:15:02.013832+0000","last_unstale":"2024-03-16T10:17:00.355736+0000","last_undegraded":"2024-03-16T10:17:00.355736+0000","last_fullsized":"2024-03-16T10:17:00.355736+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:20.391594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.750566+0000","last_change":"2024-03-16T10:15:03.021276+0000","last_active":"2024-03-16T10:16:28.750566+0000","last_peered":"2024-03-16T10:16:28.750566+0000","last_clean":"2024-03-16T10:16:28.750566+0000","last_became_active":"2024-03-16T10:15:03.020927+0000","last_became_peered":"2024-03-16T10:15:03.020927+0000","last_unstale":"2024-03-16T10:16:28.750566+0000","last_undegraded":"2024-03-16T10:16:28.750566+0000","last_fullsized":"2024-03-16T10:16:28.750566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:53:54.127506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.746363+0000","last_change":"2024-03-16T10:15:03.022083+0000","last_active":"2024-03-16T10:16:28.746363+0000","last_peered":"2024-03-16T10:16:28.746363+0000","last_clean":"2024-03-16T10:16:28.746363+0000","last_became_active":"2024-03-16T10:15:03.021609+0000","last_became_peered":"2024-03-16T10:15:03.021609+0000","last_unstale":"2024-03-16T10:16:28.746363+0000","last_undegraded":"2024-03-16T10:16:28.746363+0000","last_fullsized":"2024-03-16T10:16:28.746363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:53.864402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":139,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:00.355523+0000","last_change":"2024-03-16T10:15:02.015497+0000","last_active":"2024-03-16T10:17:00.355523+0000","last_peered":"2024-03-16T10:17:00.355523+0000","last_clean":"2024-03-16T10:17:00.355523+0000","last_became_active":"2024-03-16T10:15:02.015183+0000","last_became_peered":"2024-03-16T10:15:02.015183+0000","last_unstale":"2024-03-16T10:17:00.355523+0000","last_undegraded":"2024-03-16T10:17:00.355523+0000","last_fullsized":"2024-03-16T10:17:00.355523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:44:05.913030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.755036+0000","last_change":"2024-03-16T10:15:03.020046+0000","last_active":"2024-03-16T10:16:28.755036+0000","last_peered":"2024-03-16T10:16:28.755036+0000","last_clean":"2024-03-16T10:16:28.755036+0000","last_became_active":"2024-03-16T10:15:03.019647+0000","last_became_peered":"2024-03-16T10:15:03.019647+0000","last_unstale":"2024-03-16T10:16:28.755036+0000","last_undegraded":"2024-03-16T10:16:28.755036+0000","last_fullsized":"2024-03-16T10:16:28.755036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:18.343495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134445+0000","last_change":"2024-03-16T10:15:02.008560+0000","last_active":"2024-03-16T10:16:00.134445+0000","last_peered":"2024-03-16T10:16:00.134445+0000","last_clean":"2024-03-16T10:16:00.134445+0000","last_became_active":"2024-03-16T10:15:02.008094+0000","last_became_peered":"2024-03-16T10:15:02.008094+0000","last_unstale":"2024-03-16T10:16:00.134445+0000","last_undegraded":"2024-03-16T10:16:00.134445+0000","last_fullsized":"2024-03-16T10:16:00.134445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:21:43.555120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":342,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838687+0000","last_change":"2024-03-16T10:13:47.211815+0000","last_active":"2024-03-16T10:15:59.838687+0000","last_peered":"2024-03-16T10:15:59.838687+0000","last_clean":"2024-03-16T10:15:59.838687+0000","last_became_active":"2024-03-16T10:13:47.211353+0000","last_became_peered":"2024-03-16T10:13:47.211353+0000","last_unstale":"2024-03-16T10:15:59.838687+0000","last_undegraded":"2024-03-16T10:15:59.838687+0000","last_fullsized":"2024-03-16T10:15:59.838687+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_clean_scrub_stamp":"2024-03-16T10:10:22.965421+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:11:20.607686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.751691+0000","last_change":"2024-03-16T10:15:03.020181+0000","last_active":"2024-03-16T10:16:28.751691+0000","last_peered":"2024-03-16T10:16:28.751691+0000","last_clean":"2024-03-16T10:16:28.751691+0000","last_became_active":"2024-03-16T10:15:03.019781+0000","last_became_peered":"2024-03-16T10:15:03.019781+0000","last_unstale":"2024-03-16T10:16:28.751691+0000","last_undegraded":"2024-03-16T10:16:28.751691+0000","last_fullsized":"2024-03-16T10:16:28.751691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:04:49.369484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838833+0000","last_change":"2024-03-16T10:15:02.014461+0000","last_active":"2024-03-16T10:15:59.838833+0000","last_peered":"2024-03-16T10:15:59.838833+0000","last_clean":"2024-03-16T10:15:59.838833+0000","last_became_active":"2024-03-16T10:15:02.014137+0000","last_became_peered":"2024-03-16T10:15:02.014137+0000","last_unstale":"2024-03-16T10:15:59.838833+0000","last_undegraded":"2024-03-16T10:15:59.838833+0000","last_fullsized":"2024-03-16T10:15:59.838833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:36:41.830893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834705+0000","last_change":"2024-03-16T10:15:02.019441+0000","last_active":"2024-03-16T10:15:59.834705+0000","last_peered":"2024-03-16T10:15:59.834705+0000","last_clean":"2024-03-16T10:15:59.834705+0000","last_became_active":"2024-03-16T10:15:02.019091+0000","last_became_peered":"2024-03-16T10:15:02.019091+0000","last_unstale":"2024-03-16T10:15:59.834705+0000","last_undegraded":"2024-03-16T10:15:59.834705+0000","last_fullsized":"2024-03-16T10:15:59.834705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:04:51.758722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.745121+0000","last_change":"2024-03-16T10:15:03.012141+0000","last_active":"2024-03-16T10:16:28.745121+0000","last_peered":"2024-03-16T10:16:28.745121+0000","last_clean":"2024-03-16T10:16:28.745121+0000","last_became_active":"2024-03-16T10:15:03.011771+0000","last_became_peered":"2024-03-16T10:15:03.011771+0000","last_unstale":"2024-03-16T10:16:28.745121+0000","last_undegraded":"2024-03-16T10:16:28.745121+0000","last_fullsized":"2024-03-16T10:16:28.745121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:02:28.522530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'15","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:29.030048+0000","last_change":"2024-03-16T10:15:02.011898+0000","last_active":"2024-03-16T10:16:29.030048+0000","last_peered":"2024-03-16T10:16:29.030048+0000","last_clean":"2024-03-16T10:16:29.030048+0000","last_became_active":"2024-03-16T10:15:02.011578+0000","last_became_peered":"2024-03-16T10:15:02.011578+0000","last_unstale":"2024-03-16T10:16:29.030048+0000","last_undegraded":"2024-03-16T10:16:29.030048+0000","last_fullsized":"2024-03-16T10:16:29.030048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:27:35.756031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762502+0000","last_change":"2024-03-16T10:15:03.024923+0000","last_active":"2024-03-16T10:16:28.762502+0000","last_peered":"2024-03-16T10:16:28.762502+0000","last_clean":"2024-03-16T10:16:28.762502+0000","last_became_active":"2024-03-16T10:15:03.024577+0000","last_became_peered":"2024-03-16T10:15:03.024577+0000","last_unstale":"2024-03-16T10:16:28.762502+0000","last_undegraded":"2024-03-16T10:16:28.762502+0000","last_fullsized":"2024-03-16T10:16:28.762502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:04.650499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134719+0000","last_change":"2024-03-16T10:15:02.006842+0000","last_active":"2024-03-16T10:16:00.134719+0000","last_peered":"2024-03-16T10:16:00.134719+0000","last_clean":"2024-03-16T10:16:00.134719+0000","last_became_active":"2024-03-16T10:15:02.006483+0000","last_became_peered":"2024-03-16T10:15:02.006483+0000","last_unstale":"2024-03-16T10:16:00.134719+0000","last_undegraded":"2024-03-16T10:16:00.134719+0000","last_fullsized":"2024-03-16T10:16:00.134719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:30:55.856007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.760743+0000","last_change":"2024-03-16T10:15:03.018691+0000","last_active":"2024-03-16T10:16:28.760743+0000","last_peered":"2024-03-16T10:16:28.760743+0000","last_clean":"2024-03-16T10:16:28.760743+0000","last_became_active":"2024-03-16T10:15:03.018304+0000","last_became_peered":"2024-03-16T10:15:03.018304+0000","last_unstale":"2024-03-16T10:16:28.760743+0000","last_undegraded":"2024-03-16T10:16:28.760743+0000","last_fullsized":"2024-03-16T10:16:28.760743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:15.446759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.841306+0000","last_change":"2024-03-16T10:15:59.841306+0000","last_active":"2024-03-16T10:15:59.841306+0000","last_peered":"2024-03-16T10:15:59.841306+0000","last_clean":"2024-03-16T10:15:59.841306+0000","last_became_active":"2024-03-16T10:15:59.838434+0000","last_became_peered":"2024-03-16T10:15:59.838434+0000","last_unstale":"2024-03-16T10:15:59.841306+0000","last_undegraded":"2024-03-16T10:15:59.841306+0000","last_fullsized":"2024-03-16T10:15:59.841306+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:14:21.702809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.719194+0000","last_change":"2024-03-16T10:15:03.017262+0000","last_active":"2024-03-16T10:16:28.719194+0000","last_peered":"2024-03-16T10:16:28.719194+0000","last_clean":"2024-03-16T10:16:28.719194+0000","last_became_active":"2024-03-16T10:15:03.016909+0000","last_became_peered":"2024-03-16T10:15:03.016909+0000","last_unstale":"2024-03-16T10:16:28.719194+0000","last_undegraded":"2024-03-16T10:16:28.719194+0000","last_fullsized":"2024-03-16T10:16:28.719194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:53:11.385236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837856+0000","last_change":"2024-03-16T10:15:02.011325+0000","last_active":"2024-03-16T10:15:59.837856+0000","last_peered":"2024-03-16T10:15:59.837856+0000","last_clean":"2024-03-16T10:15:59.837856+0000","last_became_active":"2024-03-16T10:15:02.010999+0000","last_became_peered":"2024-03-16T10:15:02.010999+0000","last_unstale":"2024-03-16T10:15:59.837856+0000","last_undegraded":"2024-03-16T10:15:59.837856+0000","last_fullsized":"2024-03-16T10:15:59.837856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:30:08.137693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.757020+0000","last_change":"2024-03-16T10:15:03.012146+0000","last_active":"2024-03-16T10:16:28.757020+0000","last_peered":"2024-03-16T10:16:28.757020+0000","last_clean":"2024-03-16T10:16:28.757020+0000","last_became_active":"2024-03-16T10:15:03.011790+0000","last_became_peered":"2024-03-16T10:15:03.011790+0000","last_unstale":"2024-03-16T10:16:28.757020+0000","last_undegraded":"2024-03-16T10:16:28.757020+0000","last_fullsized":"2024-03-16T10:16:28.757020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:45:22.642925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.842829+0000","last_change":"2024-03-16T10:15:59.842829+0000","last_active":"2024-03-16T10:15:59.842829+0000","last_peered":"2024-03-16T10:15:59.842829+0000","last_clean":"2024-03-16T10:15:59.842829+0000","last_became_active":"2024-03-16T10:15:59.838787+0000","last_became_peered":"2024-03-16T10:15:59.838787+0000","last_unstale":"2024-03-16T10:15:59.842829+0000","last_undegraded":"2024-03-16T10:15:59.842829+0000","last_fullsized":"2024-03-16T10:15:59.842829+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:47:27.573657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.759857+0000","last_change":"2024-03-16T10:15:03.021094+0000","last_active":"2024-03-16T10:16:28.759857+0000","last_peered":"2024-03-16T10:16:28.759857+0000","last_clean":"2024-03-16T10:16:28.759857+0000","last_became_active":"2024-03-16T10:15:03.017640+0000","last_became_peered":"2024-03-16T10:15:03.017640+0000","last_unstale":"2024-03-16T10:16:28.759857+0000","last_undegraded":"2024-03-16T10:16:28.759857+0000","last_fullsized":"2024-03-16T10:16:28.759857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:02.030112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834498+0000","last_change":"2024-03-16T10:15:02.015652+0000","last_active":"2024-03-16T10:15:59.834498+0000","last_peered":"2024-03-16T10:15:59.834498+0000","last_clean":"2024-03-16T10:15:59.834498+0000","last_became_active":"2024-03-16T10:15:02.014046+0000","last_became_peered":"2024-03-16T10:15:02.014046+0000","last_unstale":"2024-03-16T10:15:59.834498+0000","last_undegraded":"2024-03-16T10:15:59.834498+0000","last_fullsized":"2024-03-16T10:15:59.834498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.765255+0000","last_change":"2024-03-16T10:15:03.023233+0000","last_active":"2024-03-16T10:16:28.765255+0000","last_peered":"2024-03-16T10:16:28.765255+0000","last_clean":"2024-03-16T10:16:28.765255+0000","last_became_active":"2024-03-16T10:15:03.022906+0000","last_became_peered":"2024-03-16T10:15:03.022906+0000","last_unstale":"2024-03-16T10:16:28.765255+0000","last_undegraded":"2024-03-16T10:16:28.765255+0000","last_fullsized":"2024-03-16T10:16:28.765255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:22.943087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'7","reported_seq":144,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:01.344825+0000","last_change":"2024-03-16T10:15:02.010755+0000","last_active":"2024-03-16T10:17:01.344825+0000","last_peered":"2024-03-16T10:17:01.344825+0000","last_clean":"2024-03-16T10:17:01.344825+0000","last_became_active":"2024-03-16T10:15:02.010422+0000","last_became_peered":"2024-03-16T10:15:02.010422+0000","last_unstale":"2024-03-16T10:17:01.344825+0000","last_undegraded":"2024-03-16T10:17:01.344825+0000","last_fullsized":"2024-03-16T10:17:01.344825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:17.606346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.765240+0000","last_change":"2024-03-16T10:15:03.025789+0000","last_active":"2024-03-16T10:16:28.765240+0000","last_peered":"2024-03-16T10:16:28.765240+0000","last_clean":"2024-03-16T10:16:28.765240+0000","last_became_active":"2024-03-16T10:15:03.024812+0000","last_became_peered":"2024-03-16T10:15:03.024812+0000","last_unstale":"2024-03-16T10:16:28.765240+0000","last_undegraded":"2024-03-16T10:16:28.765240+0000","last_fullsized":"2024-03-16T10:16:28.765240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:53.041234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898933+0000","last_change":"2024-03-16T10:15:02.020599+0000","last_active":"2024-03-16T10:15:59.898933+0000","last_peered":"2024-03-16T10:15:59.898933+0000","last_clean":"2024-03-16T10:15:59.898933+0000","last_became_active":"2024-03-16T10:15:02.020203+0000","last_became_peered":"2024-03-16T10:15:02.020203+0000","last_unstale":"2024-03-16T10:15:59.898933+0000","last_undegraded":"2024-03-16T10:15:59.898933+0000","last_fullsized":"2024-03-16T10:15:59.898933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:46:33.032310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.764526+0000","last_change":"2024-03-16T10:15:03.016223+0000","last_active":"2024-03-16T10:16:28.764526+0000","last_peered":"2024-03-16T10:16:28.764526+0000","last_clean":"2024-03-16T10:16:28.764526+0000","last_became_active":"2024-03-16T10:15:03.015857+0000","last_became_peered":"2024-03-16T10:15:03.015857+0000","last_unstale":"2024-03-16T10:16:28.764526+0000","last_undegraded":"2024-03-16T10:16:28.764526+0000","last_fullsized":"2024-03-16T10:16:28.764526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:40:10.329722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835797+0000","last_change":"2024-03-16T10:15:02.015690+0000","last_active":"2024-03-16T10:15:59.835797+0000","last_peered":"2024-03-16T10:15:59.835797+0000","last_clean":"2024-03-16T10:15:59.835797+0000","last_became_active":"2024-03-16T10:15:02.015375+0000","last_became_peered":"2024-03-16T10:15:02.015375+0000","last_unstale":"2024-03-16T10:15:59.835797+0000","last_undegraded":"2024-03-16T10:15:59.835797+0000","last_fullsized":"2024-03-16T10:15:59.835797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:39.118343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762466+0000","last_change":"2024-03-16T10:15:03.012702+0000","last_active":"2024-03-16T10:16:28.762466+0000","last_peered":"2024-03-16T10:16:28.762466+0000","last_clean":"2024-03-16T10:16:28.762466+0000","last_became_active":"2024-03-16T10:15:03.012387+0000","last_became_peered":"2024-03-16T10:15:03.012387+0000","last_unstale":"2024-03-16T10:16:28.762466+0000","last_undegraded":"2024-03-16T10:16:28.762466+0000","last_fullsized":"2024-03-16T10:16:28.762466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:16.938372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756885+0000","last_change":"2024-03-16T10:15:03.015290+0000","last_active":"2024-03-16T10:16:28.756885+0000","last_peered":"2024-03-16T10:16:28.756885+0000","last_clean":"2024-03-16T10:16:28.756885+0000","last_became_active":"2024-03-16T10:15:03.013846+0000","last_became_peered":"2024-03-16T10:15:03.013846+0000","last_unstale":"2024-03-16T10:16:28.756885+0000","last_undegraded":"2024-03-16T10:16:28.756885+0000","last_fullsized":"2024-03-16T10:16:28.756885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:33.377793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.744383+0000","last_change":"2024-03-16T10:15:03.023733+0000","last_active":"2024-03-16T10:16:28.744383+0000","last_peered":"2024-03-16T10:16:28.744383+0000","last_clean":"2024-03-16T10:16:28.744383+0000","last_became_active":"2024-03-16T10:15:03.023409+0000","last_became_peered":"2024-03-16T10:15:03.023409+0000","last_unstale":"2024-03-16T10:16:28.744383+0000","last_undegraded":"2024-03-16T10:16:28.744383+0000","last_fullsized":"2024-03-16T10:16:28.744383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:44.748152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756641+0000","last_change":"2024-03-16T10:15:59.856644+0000","last_active":"2024-03-16T10:16:28.756641+0000","last_peered":"2024-03-16T10:16:28.756641+0000","last_clean":"2024-03-16T10:16:28.756641+0000","last_became_active":"2024-03-16T10:15:59.856243+0000","last_became_peered":"2024-03-16T10:15:59.856243+0000","last_unstale":"2024-03-16T10:16:28.756641+0000","last_undegraded":"2024-03-16T10:16:28.756641+0000","last_fullsized":"2024-03-16T10:16:28.756641+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:22.831073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762122+0000","last_change":"2024-03-16T10:15:03.029277+0000","last_active":"2024-03-16T10:16:28.762122+0000","last_peered":"2024-03-16T10:16:28.762122+0000","last_clean":"2024-03-16T10:16:28.762122+0000","last_became_active":"2024-03-16T10:15:03.028960+0000","last_became_peered":"2024-03-16T10:15:03.028960+0000","last_unstale":"2024-03-16T10:16:28.762122+0000","last_undegraded":"2024-03-16T10:16:28.762122+0000","last_fullsized":"2024-03-16T10:16:28.762122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:50:48.362914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756920+0000","last_change":"2024-03-16T10:15:59.856671+0000","last_active":"2024-03-16T10:16:28.756920+0000","last_peered":"2024-03-16T10:16:28.756920+0000","last_clean":"2024-03-16T10:16:28.756920+0000","last_became_active":"2024-03-16T10:15:59.856287+0000","last_became_peered":"2024-03-16T10:15:59.856287+0000","last_unstale":"2024-03-16T10:16:28.756920+0000","last_undegraded":"2024-03-16T10:16:28.756920+0000","last_fullsized":"2024-03-16T10:16:28.756920+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:15:58.409554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756231+0000","last_change":"2024-03-16T10:15:03.017499+0000","last_active":"2024-03-16T10:16:28.756231+0000","last_peered":"2024-03-16T10:16:28.756231+0000","last_clean":"2024-03-16T10:16:28.756231+0000","last_became_active":"2024-03-16T10:15:03.017085+0000","last_became_peered":"2024-03-16T10:15:03.017085+0000","last_unstale":"2024-03-16T10:16:28.756231+0000","last_undegraded":"2024-03-16T10:16:28.756231+0000","last_fullsized":"2024-03-16T10:16:28.756231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:57:09.592420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.752712+0000","last_change":"2024-03-16T10:15:03.018943+0000","last_active":"2024-03-16T10:16:28.752712+0000","last_peered":"2024-03-16T10:16:28.752712+0000","last_clean":"2024-03-16T10:16:28.752712+0000","last_became_active":"2024-03-16T10:15:03.017519+0000","last_became_peered":"2024-03-16T10:15:03.017519+0000","last_unstale":"2024-03-16T10:16:28.752712+0000","last_undegraded":"2024-03-16T10:16:28.752712+0000","last_fullsized":"2024-03-16T10:16:28.752712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:29.026988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.754110+0000","last_change":"2024-03-16T10:15:03.024697+0000","last_active":"2024-03-16T10:16:28.754110+0000","last_peered":"2024-03-16T10:16:28.754110+0000","last_clean":"2024-03-16T10:16:28.754110+0000","last_became_active":"2024-03-16T10:15:03.024305+0000","last_became_peered":"2024-03-16T10:15:03.024305+0000","last_unstale":"2024-03-16T10:16:28.754110+0000","last_undegraded":"2024-03-16T10:16:28.754110+0000","last_fullsized":"2024-03-16T10:16:28.754110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:07:35.160746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.755394+0000","last_change":"2024-03-16T10:15:03.017504+0000","last_active":"2024-03-16T10:16:28.755394+0000","last_peered":"2024-03-16T10:16:28.755394+0000","last_clean":"2024-03-16T10:16:28.755394+0000","last_became_active":"2024-03-16T10:15:03.017100+0000","last_became_peered":"2024-03-16T10:15:03.017100+0000","last_unstale":"2024-03-16T10:16:28.755394+0000","last_undegraded":"2024-03-16T10:16:28.755394+0000","last_fullsized":"2024-03-16T10:16:28.755394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:23:17.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.754759+0000","last_change":"2024-03-16T10:15:03.024684+0000","last_active":"2024-03-16T10:16:28.754759+0000","last_peered":"2024-03-16T10:16:28.754759+0000","last_clean":"2024-03-16T10:16:28.754759+0000","last_became_active":"2024-03-16T10:15:03.024282+0000","last_became_peered":"2024-03-16T10:15:03.024282+0000","last_unstale":"2024-03-16T10:16:28.754759+0000","last_undegraded":"2024-03-16T10:16:28.754759+0000","last_fullsized":"2024-03-16T10:16:28.754759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:29.595162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.758529+0000","last_change":"2024-03-16T10:15:03.025337+0000","last_active":"2024-03-16T10:16:28.758529+0000","last_peered":"2024-03-16T10:16:28.758529+0000","last_clean":"2024-03-16T10:16:28.758529+0000","last_became_active":"2024-03-16T10:15:03.024938+0000","last_became_peered":"2024-03-16T10:15:03.024938+0000","last_unstale":"2024-03-16T10:16:28.758529+0000","last_undegraded":"2024-03-16T10:16:28.758529+0000","last_fullsized":"2024-03-16T10:16:28.758529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:13:15.892354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.765584+0000","last_change":"2024-03-16T10:15:03.015468+0000","last_active":"2024-03-16T10:16:28.765584+0000","last_peered":"2024-03-16T10:16:28.765584+0000","last_clean":"2024-03-16T10:16:28.765584+0000","last_became_active":"2024-03-16T10:15:03.015097+0000","last_became_peered":"2024-03-16T10:15:03.015097+0000","last_unstale":"2024-03-16T10:16:28.765584+0000","last_undegraded":"2024-03-16T10:16:28.765584+0000","last_fullsized":"2024-03-16T10:16:28.765584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:46.114346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.761104+0000","last_change":"2024-03-16T10:15:03.020690+0000","last_active":"2024-03-16T10:16:28.761104+0000","last_peered":"2024-03-16T10:16:28.761104+0000","last_clean":"2024-03-16T10:16:28.761104+0000","last_became_active":"2024-03-16T10:15:03.020365+0000","last_became_peered":"2024-03-16T10:15:03.020365+0000","last_unstale":"2024-03-16T10:16:28.761104+0000","last_undegraded":"2024-03-16T10:16:28.761104+0000","last_fullsized":"2024-03-16T10:16:28.761104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:25.671533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280280,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67925,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1197,"num_read_kb":675,"num_write":253,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":209456,"data_compressed":5583,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":457189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":1108266,"data_compressed":37511,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":456587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":456965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":1049092,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":456304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":518105,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":458175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":518549,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":456129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":1049161,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68795,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63084,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":62316,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T10:17:07.589 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T10:17:07.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.728+0000 7fed9a577640 1 -- 172.21.15.77:0/2185812238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed941022c0 msgr2=0x7fed941046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:07.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.728+0000 7fed9a577640 1 --2- 172.21.15.77:0/2185812238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed941022c0 0x7fed941046b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fed84009860 tx=0x7fed8402bb70 comp rx=0 tx=0).stop 2024-03-16T10:17:07.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.728+0000 7fed9a577640 1 -- 172.21.15.77:0/2185812238 shutdown_connections 2024-03-16T10:17:07.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.728+0000 7fed9a577640 1 --2- 172.21.15.77:0/2185812238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed94107520 0x7fed94109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.728+0000 7fed9a577640 1 --2- 172.21.15.77:0/2185812238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed94104bf0 0x7fed94106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.728+0000 7fed9a577640 1 --2- 172.21.15.77:0/2185812238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed941022c0 0x7fed941046b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.728+0000 7fed9a577640 1 -- 172.21.15.77:0/2185812238 >> 172.21.15.77:0/2185812238 conn(0x7fed940fbc70 msgr2=0x7fed940fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:07.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.728+0000 7fed9a577640 1 -- 172.21.15.77:0/2185812238 shutdown_connections 2024-03-16T10:17:07.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.728+0000 7fed9a577640 1 -- 172.21.15.77:0/2185812238 wait complete. 2024-03-16T10:17:07.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed9a577640 1 Processor -- start 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed9a577640 1 -- start start 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed9a577640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed941022c0 0x7fed94197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed9a577640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed94104bf0 0x7fed94198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed9a577640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed94107520 0x7fed9419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed9a577640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fed9410d4a0 con 0x7fed941022c0 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed9a577640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fed9410d320 con 0x7fed94104bf0 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed9a577640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fed9410d620 con 0x7fed94107520 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed93fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed941022c0 0x7fed94197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed93fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed941022c0 0x7fed94197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56202/0 (socket says 172.21.15.77:56202) 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed93fff640 1 -- 172.21.15.77:0/3904564906 learned_addr learned my addr 172.21.15.77:0/3904564906 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed98b14640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed94107520 0x7fed9419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed937fe640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed94104bf0 0x7fed94198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:07.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed93fff640 1 -- 172.21.15.77:0/3904564906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed94107520 msgr2=0x7fed9419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:07.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed93fff640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed94107520 0x7fed9419f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed93fff640 1 -- 172.21.15.77:0/3904564906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed94104bf0 msgr2=0x7fed94198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:07.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.729+0000 7fed93fff640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed94104bf0 0x7fed94198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.730+0000 7fed93fff640 1 -- 172.21.15.77:0/3904564906 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed9419f770 con 0x7fed941022c0 2024-03-16T10:17:07.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.730+0000 7fed98b14640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed94107520 0x7fed9419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:07.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.730+0000 7fed93fff640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed941022c0 0x7fed94197b50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fed84005940 tx=0x7fed84032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:07.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.730+0000 7fed917fa640 1 -- 172.21.15.77:0/3904564906 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed84041020 con 0x7fed941022c0 2024-03-16T10:17:07.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.730+0000 7fed9a577640 1 -- 172.21.15.77:0/3904564906 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed9419fa00 con 0x7fed941022c0 2024-03-16T10:17:07.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.730+0000 7fed917fa640 1 -- 172.21.15.77:0/3904564906 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed8402c9f0 con 0x7fed941022c0 2024-03-16T10:17:07.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.730+0000 7fed917fa640 1 -- 172.21.15.77:0/3904564906 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed84039040 con 0x7fed941022c0 2024-03-16T10:17:07.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.730+0000 7fed9a577640 1 -- 172.21.15.77:0/3904564906 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed9419ff40 con 0x7fed941022c0 2024-03-16T10:17:07.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.731+0000 7fed9a577640 1 -- 172.21.15.77:0/3904564906 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed940fd760 con 0x7fed941022c0 2024-03-16T10:17:07.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.732+0000 7fed917fa640 1 -- 172.21.15.77:0/3904564906 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fed84044020 con 0x7fed941022c0 2024-03-16T10:17:07.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.732+0000 7fed917fa640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fed7c0a2890 0x7fed7c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:07.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.732+0000 7fed937fe640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fed7c0a2890 0x7fed7c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:07.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.733+0000 7fed917fa640 1 -- 172.21.15.77:0/3904564906 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fed840f03b0 con 0x7fed941022c0 2024-03-16T10:17:07.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.733+0000 7fed937fe640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fed7c0a2890 0x7fed7c0a4d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fed94199060 tx=0x7fed88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:07.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.741+0000 7fed917fa640 1 -- 172.21.15.77:0/3904564906 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fed8404b020 con 0x7fed941022c0 2024-03-16T10:17:07.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.942+0000 7fed9a577640 1 -- 172.21.15.77:0/3904564906 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fed94061c20 con 0x7fed7c0a2890 2024-03-16T10:17:07.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.951+0000 7fed917fa640 1 -- 172.21.15.77:0/3904564906 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259101 (secure 0 0 0) 0x7fed94061c20 con 0x7fed7c0a2890 2024-03-16T10:17:07.954 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:17:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T10:17:07.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.955+0000 7fed9a577640 1 -- 172.21.15.77:0/3904564906 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fed7c0a2890 msgr2=0x7fed7c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:07.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.955+0000 7fed9a577640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fed7c0a2890 0x7fed7c0a4d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fed94199060 tx=0x7fed88005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:07.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.955+0000 7fed9a577640 1 -- 172.21.15.77:0/3904564906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed941022c0 msgr2=0x7fed94197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:07.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.955+0000 7fed9a577640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed941022c0 0x7fed94197b50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fed84005940 tx=0x7fed84032470 comp rx=0 tx=0).stop 2024-03-16T10:17:07.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.955+0000 7fed9a577640 1 -- 172.21.15.77:0/3904564906 shutdown_connections 2024-03-16T10:17:07.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.955+0000 7fed9a577640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fed7c0a2890 0x7fed7c0a4d50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.955+0000 7fed9a577640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed94107520 0x7fed9419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.955+0000 7fed9a577640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed94104bf0 0x7fed94198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.955+0000 7fed9a577640 1 --2- 172.21.15.77:0/3904564906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed941022c0 0x7fed94197b50 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.955+0000 7fed9a577640 1 -- 172.21.15.77:0/3904564906 >> 172.21.15.77:0/3904564906 conn(0x7fed940fbc70 msgr2=0x7fed94102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.955+0000 7fed9a577640 1 -- 172.21.15.77:0/3904564906 shutdown_connections 2024-03-16T10:17:07.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:07.955+0000 7fed9a577640 1 -- 172.21.15.77:0/3904564906 wait complete. 2024-03-16T10:17:07.967 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-03-16T10:17:06.743397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658527,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1323,"num_read_kb":784,"num_write":436,"num_write_kb":3084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":858,"ondisk_log_size":858,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360852,"kb_used_data":11700,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580972,"statfs":{"total":1151940427776,"available":1151570915328,"internally_reserved":0,"allocated":11980800,"data_stored":7440687,"data_compressed":112533,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":699,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":52,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002662"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.757174+0000","last_change":"2024-03-16T10:15:03.016880+0000","last_active":"2024-03-16T10:16:28.757174+0000","last_peered":"2024-03-16T10:16:28.757174+0000","last_clean":"2024-03-16T10:16:28.757174+0000","last_became_active":"2024-03-16T10:15:03.016513+0000","last_became_peered":"2024-03-16T10:15:03.016513+0000","last_unstale":"2024-03-16T10:16:28.757174+0000","last_undegraded":"2024-03-16T10:16:28.757174+0000","last_fullsized":"2024-03-16T10:16:28.757174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:39.834015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.753595+0000","last_change":"2024-03-16T10:15:03.015467+0000","last_active":"2024-03-16T10:16:28.753595+0000","last_peered":"2024-03-16T10:16:28.753595+0000","last_clean":"2024-03-16T10:16:28.753595+0000","last_became_active":"2024-03-16T10:15:03.015152+0000","last_became_peered":"2024-03-16T10:15:03.015152+0000","last_unstale":"2024-03-16T10:16:28.753595+0000","last_undegraded":"2024-03-16T10:16:28.753595+0000","last_fullsized":"2024-03-16T10:16:28.753595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:17:29.988569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.753318+0000","last_change":"2024-03-16T10:15:03.023836+0000","last_active":"2024-03-16T10:16:28.753318+0000","last_peered":"2024-03-16T10:16:28.753318+0000","last_clean":"2024-03-16T10:16:28.753318+0000","last_became_active":"2024-03-16T10:15:03.022774+0000","last_became_peered":"2024-03-16T10:15:03.022774+0000","last_unstale":"2024-03-16T10:16:28.753318+0000","last_undegraded":"2024-03-16T10:16:28.753318+0000","last_fullsized":"2024-03-16T10:16:28.753318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:49.837157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.759581+0000","last_change":"2024-03-16T10:15:03.030891+0000","last_active":"2024-03-16T10:16:28.759581+0000","last_peered":"2024-03-16T10:16:28.759581+0000","last_clean":"2024-03-16T10:16:28.759581+0000","last_became_active":"2024-03-16T10:15:03.030278+0000","last_became_peered":"2024-03-16T10:15:03.030278+0000","last_unstale":"2024-03-16T10:16:28.759581+0000","last_undegraded":"2024-03-16T10:16:28.759581+0000","last_fullsized":"2024-03-16T10:16:28.759581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:16.166885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.742976+0000","last_change":"2024-03-16T10:15:03.017610+0000","last_active":"2024-03-16T10:16:28.742976+0000","last_peered":"2024-03-16T10:16:28.742976+0000","last_clean":"2024-03-16T10:16:28.742976+0000","last_became_active":"2024-03-16T10:15:03.017250+0000","last_became_peered":"2024-03-16T10:15:03.017250+0000","last_unstale":"2024-03-16T10:16:28.742976+0000","last_undegraded":"2024-03-16T10:16:28.742976+0000","last_fullsized":"2024-03-16T10:16:28.742976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:07.899425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.753472+0000","last_change":"2024-03-16T10:15:59.856216+0000","last_active":"2024-03-16T10:16:28.753472+0000","last_peered":"2024-03-16T10:16:28.753472+0000","last_clean":"2024-03-16T10:16:28.753472+0000","last_became_active":"2024-03-16T10:15:59.855882+0000","last_became_peered":"2024-03-16T10:15:59.855882+0000","last_unstale":"2024-03-16T10:16:28.753472+0000","last_undegraded":"2024-03-16T10:16:28.753472+0000","last_fullsized":"2024-03-16T10:16:28.753472+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:27:42.682580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762588+0000","last_change":"2024-03-16T10:15:03.017686+0000","last_active":"2024-03-16T10:16:28.762588+0000","last_peered":"2024-03-16T10:16:28.762588+0000","last_clean":"2024-03-16T10:16:28.762588+0000","last_became_active":"2024-03-16T10:15:03.017262+0000","last_became_peered":"2024-03-16T10:15:03.017262+0000","last_unstale":"2024-03-16T10:16:28.762588+0000","last_undegraded":"2024-03-16T10:16:28.762588+0000","last_fullsized":"2024-03-16T10:16:28.762588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:36:22.270651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.759952+0000","last_change":"2024-03-16T10:15:03.020130+0000","last_active":"2024-03-16T10:16:28.759952+0000","last_peered":"2024-03-16T10:16:28.759952+0000","last_clean":"2024-03-16T10:16:28.759952+0000","last_became_active":"2024-03-16T10:15:03.019742+0000","last_became_peered":"2024-03-16T10:15:03.019742+0000","last_unstale":"2024-03-16T10:16:28.759952+0000","last_undegraded":"2024-03-16T10:16:28.759952+0000","last_fullsized":"2024-03-16T10:16:28.759952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:52.450792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.757769+0000","last_change":"2024-03-16T10:15:03.027589+0000","last_active":"2024-03-16T10:16:28.757769+0000","last_peered":"2024-03-16T10:16:28.757769+0000","last_clean":"2024-03-16T10:16:28.757769+0000","last_became_active":"2024-03-16T10:15:03.027226+0000","last_became_peered":"2024-03-16T10:15:03.027226+0000","last_unstale":"2024-03-16T10:16:28.757769+0000","last_undegraded":"2024-03-16T10:16:28.757769+0000","last_fullsized":"2024-03-16T10:16:28.757769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:14:35.354629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.761345+0000","last_change":"2024-03-16T10:15:03.021163+0000","last_active":"2024-03-16T10:16:28.761345+0000","last_peered":"2024-03-16T10:16:28.761345+0000","last_clean":"2024-03-16T10:16:28.761345+0000","last_became_active":"2024-03-16T10:15:03.020483+0000","last_became_peered":"2024-03-16T10:15:03.020483+0000","last_unstale":"2024-03-16T10:16:28.761345+0000","last_undegraded":"2024-03-16T10:16:28.761345+0000","last_fullsized":"2024-03-16T10:16:28.761345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:01:00.873471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.765112+0000","last_change":"2024-03-16T10:15:03.026111+0000","last_active":"2024-03-16T10:16:28.765112+0000","last_peered":"2024-03-16T10:16:28.765112+0000","last_clean":"2024-03-16T10:16:28.765112+0000","last_became_active":"2024-03-16T10:15:03.025296+0000","last_became_peered":"2024-03-16T10:15:03.025296+0000","last_unstale":"2024-03-16T10:16:28.765112+0000","last_undegraded":"2024-03-16T10:16:28.765112+0000","last_fullsized":"2024-03-16T10:16:28.765112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:33:29.606876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.746441+0000","last_change":"2024-03-16T10:15:03.023013+0000","last_active":"2024-03-16T10:16:28.746441+0000","last_peered":"2024-03-16T10:16:28.746441+0000","last_clean":"2024-03-16T10:16:28.746441+0000","last_became_active":"2024-03-16T10:15:03.022251+0000","last_became_peered":"2024-03-16T10:15:03.022251+0000","last_unstale":"2024-03-16T10:16:28.746441+0000","last_undegraded":"2024-03-16T10:16:28.746441+0000","last_fullsized":"2024-03-16T10:16:28.746441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:33.544709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.750180+0000","last_change":"2024-03-16T10:15:03.009548+0000","last_active":"2024-03-16T10:16:28.750180+0000","last_peered":"2024-03-16T10:16:28.750180+0000","last_clean":"2024-03-16T10:16:28.750180+0000","last_became_active":"2024-03-16T10:15:03.007675+0000","last_became_peered":"2024-03-16T10:15:03.007675+0000","last_unstale":"2024-03-16T10:16:28.750180+0000","last_undegraded":"2024-03-16T10:16:28.750180+0000","last_fullsized":"2024-03-16T10:16:28.750180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:08:02.561602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756541+0000","last_change":"2024-03-16T10:15:03.020146+0000","last_active":"2024-03-16T10:16:28.756541+0000","last_peered":"2024-03-16T10:16:28.756541+0000","last_clean":"2024-03-16T10:16:28.756541+0000","last_became_active":"2024-03-16T10:15:03.019823+0000","last_became_peered":"2024-03-16T10:15:03.019823+0000","last_unstale":"2024-03-16T10:16:28.756541+0000","last_undegraded":"2024-03-16T10:16:28.756541+0000","last_fullsized":"2024-03-16T10:16:28.756541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:52.276506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.760668+0000","last_change":"2024-03-16T10:15:03.019150+0000","last_active":"2024-03-16T10:16:28.760668+0000","last_peered":"2024-03-16T10:16:28.760668+0000","last_clean":"2024-03-16T10:16:28.760668+0000","last_became_active":"2024-03-16T10:15:03.018830+0000","last_became_peered":"2024-03-16T10:15:03.018830+0000","last_unstale":"2024-03-16T10:16:28.760668+0000","last_undegraded":"2024-03-16T10:16:28.760668+0000","last_fullsized":"2024-03-16T10:16:28.760668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:46.207187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762615+0000","last_change":"2024-03-16T10:15:03.033321+0000","last_active":"2024-03-16T10:16:28.762615+0000","last_peered":"2024-03-16T10:16:28.762615+0000","last_clean":"2024-03-16T10:16:28.762615+0000","last_became_active":"2024-03-16T10:15:03.032944+0000","last_became_peered":"2024-03-16T10:15:03.032944+0000","last_unstale":"2024-03-16T10:16:28.762615+0000","last_undegraded":"2024-03-16T10:16:28.762615+0000","last_fullsized":"2024-03-16T10:16:28.762615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:14.345488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.757536+0000","last_change":"2024-03-16T10:15:03.019897+0000","last_active":"2024-03-16T10:16:28.757536+0000","last_peered":"2024-03-16T10:16:28.757536+0000","last_clean":"2024-03-16T10:16:28.757536+0000","last_became_active":"2024-03-16T10:15:03.019555+0000","last_became_peered":"2024-03-16T10:15:03.019555+0000","last_unstale":"2024-03-16T10:16:28.757536+0000","last_undegraded":"2024-03-16T10:16:28.757536+0000","last_fullsized":"2024-03-16T10:16:28.757536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:58.307285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.758416+0000","last_change":"2024-03-16T10:15:03.017482+0000","last_active":"2024-03-16T10:16:28.758416+0000","last_peered":"2024-03-16T10:16:28.758416+0000","last_clean":"2024-03-16T10:16:28.758416+0000","last_became_active":"2024-03-16T10:15:03.016521+0000","last_became_peered":"2024-03-16T10:15:03.016521+0000","last_unstale":"2024-03-16T10:16:28.758416+0000","last_undegraded":"2024-03-16T10:16:28.758416+0000","last_fullsized":"2024-03-16T10:16:28.758416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:11.889111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.747191+0000","last_change":"2024-03-16T10:15:03.018625+0000","last_active":"2024-03-16T10:16:28.747191+0000","last_peered":"2024-03-16T10:16:28.747191+0000","last_clean":"2024-03-16T10:16:28.747191+0000","last_became_active":"2024-03-16T10:15:03.018313+0000","last_became_peered":"2024-03-16T10:15:03.018313+0000","last_unstale":"2024-03-16T10:16:28.747191+0000","last_undegraded":"2024-03-16T10:16:28.747191+0000","last_fullsized":"2024-03-16T10:16:28.747191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:35:58.533559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'14","reported_seq":149,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:02.357489+0000","last_change":"2024-03-16T10:15:02.016545+0000","last_active":"2024-03-16T10:17:02.357489+0000","last_peered":"2024-03-16T10:17:02.357489+0000","last_clean":"2024-03-16T10:17:02.357489+0000","last_became_active":"2024-03-16T10:15:02.012088+0000","last_became_peered":"2024-03-16T10:15:02.012088+0000","last_unstale":"2024-03-16T10:17:02.357489+0000","last_undegraded":"2024-03-16T10:17:02.357489+0000","last_fullsized":"2024-03-16T10:17:02.357489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:50.633539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.740676+0000","last_change":"2024-03-16T10:15:03.026565+0000","last_active":"2024-03-16T10:16:28.740676+0000","last_peered":"2024-03-16T10:16:28.740676+0000","last_clean":"2024-03-16T10:16:28.740676+0000","last_became_active":"2024-03-16T10:15:03.025386+0000","last_became_peered":"2024-03-16T10:15:03.025386+0000","last_unstale":"2024-03-16T10:16:28.740676+0000","last_undegraded":"2024-03-16T10:16:28.740676+0000","last_fullsized":"2024-03-16T10:16:28.740676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:28.831764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838352+0000","last_change":"2024-03-16T10:15:02.018097+0000","last_active":"2024-03-16T10:15:59.838352+0000","last_peered":"2024-03-16T10:15:59.838352+0000","last_clean":"2024-03-16T10:15:59.838352+0000","last_became_active":"2024-03-16T10:15:02.017777+0000","last_became_peered":"2024-03-16T10:15:02.017777+0000","last_unstale":"2024-03-16T10:15:59.838352+0000","last_undegraded":"2024-03-16T10:15:59.838352+0000","last_fullsized":"2024-03-16T10:15:59.838352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:04:58.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.753086+0000","last_change":"2024-03-16T10:15:03.023384+0000","last_active":"2024-03-16T10:16:28.753086+0000","last_peered":"2024-03-16T10:16:28.753086+0000","last_clean":"2024-03-16T10:16:28.753086+0000","last_became_active":"2024-03-16T10:15:03.022996+0000","last_became_peered":"2024-03-16T10:15:03.022996+0000","last_unstale":"2024-03-16T10:16:28.753086+0000","last_undegraded":"2024-03-16T10:16:28.753086+0000","last_fullsized":"2024-03-16T10:16:28.753086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:29:20.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834807+0000","last_change":"2024-03-16T10:15:02.014409+0000","last_active":"2024-03-16T10:15:59.834807+0000","last_peered":"2024-03-16T10:15:59.834807+0000","last_clean":"2024-03-16T10:15:59.834807+0000","last_became_active":"2024-03-16T10:15:02.014015+0000","last_became_peered":"2024-03-16T10:15:02.014015+0000","last_unstale":"2024-03-16T10:15:59.834807+0000","last_undegraded":"2024-03-16T10:15:59.834807+0000","last_fullsized":"2024-03-16T10:15:59.834807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:25:53.133466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.759810+0000","last_change":"2024-03-16T10:15:03.031328+0000","last_active":"2024-03-16T10:16:28.759810+0000","last_peered":"2024-03-16T10:16:28.759810+0000","last_clean":"2024-03-16T10:16:28.759810+0000","last_became_active":"2024-03-16T10:15:03.029787+0000","last_became_peered":"2024-03-16T10:15:03.029787+0000","last_unstale":"2024-03-16T10:16:28.759810+0000","last_undegraded":"2024-03-16T10:16:28.759810+0000","last_fullsized":"2024-03-16T10:16:28.759810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:34.321883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834594+0000","last_change":"2024-03-16T10:15:02.018032+0000","last_active":"2024-03-16T10:15:59.834594+0000","last_peered":"2024-03-16T10:15:59.834594+0000","last_clean":"2024-03-16T10:15:59.834594+0000","last_became_active":"2024-03-16T10:15:02.017677+0000","last_became_peered":"2024-03-16T10:15:02.017677+0000","last_unstale":"2024-03-16T10:15:59.834594+0000","last_undegraded":"2024-03-16T10:15:59.834594+0000","last_fullsized":"2024-03-16T10:15:59.834594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:58:07.179129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762172+0000","last_change":"2024-03-16T10:15:03.024958+0000","last_active":"2024-03-16T10:16:28.762172+0000","last_peered":"2024-03-16T10:16:28.762172+0000","last_clean":"2024-03-16T10:16:28.762172+0000","last_became_active":"2024-03-16T10:15:03.023964+0000","last_became_peered":"2024-03-16T10:15:03.023964+0000","last_unstale":"2024-03-16T10:16:28.762172+0000","last_undegraded":"2024-03-16T10:16:28.762172+0000","last_fullsized":"2024-03-16T10:16:28.762172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:55:00.953258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":146,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:04.348190+0000","last_change":"2024-03-16T10:15:02.015062+0000","last_active":"2024-03-16T10:17:04.348190+0000","last_peered":"2024-03-16T10:17:04.348190+0000","last_clean":"2024-03-16T10:17:04.348190+0000","last_became_active":"2024-03-16T10:15:02.014714+0000","last_became_peered":"2024-03-16T10:15:02.014714+0000","last_unstale":"2024-03-16T10:17:04.348190+0000","last_undegraded":"2024-03-16T10:17:04.348190+0000","last_fullsized":"2024-03-16T10:17:04.348190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:23:55.313009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.743988+0000","last_change":"2024-03-16T10:15:03.013956+0000","last_active":"2024-03-16T10:16:28.743988+0000","last_peered":"2024-03-16T10:16:28.743988+0000","last_clean":"2024-03-16T10:16:28.743988+0000","last_became_active":"2024-03-16T10:15:03.013607+0000","last_became_peered":"2024-03-16T10:15:03.013607+0000","last_unstale":"2024-03-16T10:16:28.743988+0000","last_undegraded":"2024-03-16T10:16:28.743988+0000","last_fullsized":"2024-03-16T10:16:28.743988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:40:55.445395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.841646+0000","last_change":"2024-03-16T10:15:02.015651+0000","last_active":"2024-03-16T10:15:59.841646+0000","last_peered":"2024-03-16T10:15:59.841646+0000","last_clean":"2024-03-16T10:15:59.841646+0000","last_became_active":"2024-03-16T10:15:02.015254+0000","last_became_peered":"2024-03-16T10:15:02.015254+0000","last_unstale":"2024-03-16T10:15:59.841646+0000","last_undegraded":"2024-03-16T10:15:59.841646+0000","last_fullsized":"2024-03-16T10:15:59.841646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:41:59.013606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756364+0000","last_change":"2024-03-16T10:15:03.023074+0000","last_active":"2024-03-16T10:16:28.756364+0000","last_peered":"2024-03-16T10:16:28.756364+0000","last_clean":"2024-03-16T10:16:28.756364+0000","last_became_active":"2024-03-16T10:15:03.022700+0000","last_became_peered":"2024-03-16T10:15:03.022700+0000","last_unstale":"2024-03-16T10:16:28.756364+0000","last_undegraded":"2024-03-16T10:16:28.756364+0000","last_fullsized":"2024-03-16T10:16:28.756364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:22.047929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835321+0000","last_change":"2024-03-16T10:15:02.014793+0000","last_active":"2024-03-16T10:15:59.835321+0000","last_peered":"2024-03-16T10:15:59.835321+0000","last_clean":"2024-03-16T10:15:59.835321+0000","last_became_active":"2024-03-16T10:15:02.014459+0000","last_became_peered":"2024-03-16T10:15:02.014459+0000","last_unstale":"2024-03-16T10:15:59.835321+0000","last_undegraded":"2024-03-16T10:15:59.835321+0000","last_fullsized":"2024-03-16T10:15:59.835321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:59:45.035045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.755311+0000","last_change":"2024-03-16T10:15:03.012713+0000","last_active":"2024-03-16T10:16:28.755311+0000","last_peered":"2024-03-16T10:16:28.755311+0000","last_clean":"2024-03-16T10:16:28.755311+0000","last_became_active":"2024-03-16T10:15:03.012407+0000","last_became_peered":"2024-03-16T10:15:03.012407+0000","last_unstale":"2024-03-16T10:16:28.755311+0000","last_undegraded":"2024-03-16T10:16:28.755311+0000","last_fullsized":"2024-03-16T10:16:28.755311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:23:14.592810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898544+0000","last_change":"2024-03-16T10:15:02.015582+0000","last_active":"2024-03-16T10:15:59.898544+0000","last_peered":"2024-03-16T10:15:59.898544+0000","last_clean":"2024-03-16T10:15:59.898544+0000","last_became_active":"2024-03-16T10:15:02.015073+0000","last_became_peered":"2024-03-16T10:15:02.015073+0000","last_unstale":"2024-03-16T10:15:59.898544+0000","last_undegraded":"2024-03-16T10:15:59.898544+0000","last_fullsized":"2024-03-16T10:15:59.898544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:22:48.212448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.750647+0000","last_change":"2024-03-16T10:15:03.016839+0000","last_active":"2024-03-16T10:16:28.750647+0000","last_peered":"2024-03-16T10:16:28.750647+0000","last_clean":"2024-03-16T10:16:28.750647+0000","last_became_active":"2024-03-16T10:15:03.016463+0000","last_became_peered":"2024-03-16T10:15:03.016463+0000","last_unstale":"2024-03-16T10:16:28.750647+0000","last_undegraded":"2024-03-16T10:16:28.750647+0000","last_fullsized":"2024-03-16T10:16:28.750647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:43:34.259049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'5","reported_seq":256,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:03.346162+0000","last_change":"2024-03-16T10:15:02.017884+0000","last_active":"2024-03-16T10:17:03.346162+0000","last_peered":"2024-03-16T10:17:03.346162+0000","last_clean":"2024-03-16T10:17:03.346162+0000","last_became_active":"2024-03-16T10:15:02.017511+0000","last_became_peered":"2024-03-16T10:15:02.017511+0000","last_unstale":"2024-03-16T10:17:03.346162+0000","last_undegraded":"2024-03-16T10:17:03.346162+0000","last_fullsized":"2024-03-16T10:17:03.346162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:31.281631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":232,"num_read_kb":232,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.754815+0000","last_change":"2024-03-16T10:15:03.032259+0000","last_active":"2024-03-16T10:16:28.754815+0000","last_peered":"2024-03-16T10:16:28.754815+0000","last_clean":"2024-03-16T10:16:28.754815+0000","last_became_active":"2024-03-16T10:15:03.031884+0000","last_became_peered":"2024-03-16T10:15:03.031884+0000","last_unstale":"2024-03-16T10:16:28.754815+0000","last_undegraded":"2024-03-16T10:16:28.754815+0000","last_fullsized":"2024-03-16T10:16:28.754815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:37:48.464051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839326+0000","last_change":"2024-03-16T10:15:02.015372+0000","last_active":"2024-03-16T10:15:59.839326+0000","last_peered":"2024-03-16T10:15:59.839326+0000","last_clean":"2024-03-16T10:15:59.839326+0000","last_became_active":"2024-03-16T10:15:02.015022+0000","last_became_peered":"2024-03-16T10:15:02.015022+0000","last_unstale":"2024-03-16T10:15:59.839326+0000","last_undegraded":"2024-03-16T10:15:59.839326+0000","last_fullsized":"2024-03-16T10:15:59.839326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:42.651903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.760728+0000","last_change":"2024-03-16T10:15:03.018661+0000","last_active":"2024-03-16T10:16:28.760728+0000","last_peered":"2024-03-16T10:16:28.760728+0000","last_clean":"2024-03-16T10:16:28.760728+0000","last_became_active":"2024-03-16T10:15:03.018351+0000","last_became_peered":"2024-03-16T10:15:03.018351+0000","last_unstale":"2024-03-16T10:16:28.760728+0000","last_undegraded":"2024-03-16T10:16:28.760728+0000","last_fullsized":"2024-03-16T10:16:28.760728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:21:33.939811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.899049+0000","last_change":"2024-03-16T10:15:02.014343+0000","last_active":"2024-03-16T10:15:59.899049+0000","last_peered":"2024-03-16T10:15:59.899049+0000","last_clean":"2024-03-16T10:15:59.899049+0000","last_became_active":"2024-03-16T10:15:02.013983+0000","last_became_peered":"2024-03-16T10:15:02.013983+0000","last_unstale":"2024-03-16T10:15:59.899049+0000","last_undegraded":"2024-03-16T10:15:59.899049+0000","last_fullsized":"2024-03-16T10:15:59.899049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:35:04.768079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.761372+0000","last_change":"2024-03-16T10:15:03.031055+0000","last_active":"2024-03-16T10:16:28.761372+0000","last_peered":"2024-03-16T10:16:28.761372+0000","last_clean":"2024-03-16T10:16:28.761372+0000","last_became_active":"2024-03-16T10:15:03.029656+0000","last_became_peered":"2024-03-16T10:15:03.029656+0000","last_unstale":"2024-03-16T10:16:28.761372+0000","last_undegraded":"2024-03-16T10:16:28.761372+0000","last_fullsized":"2024-03-16T10:16:28.761372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:27:35.684774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'37","reported_seq":299,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:03.346714+0000","last_change":"2024-03-16T10:15:02.014752+0000","last_active":"2024-03-16T10:17:03.346714+0000","last_peered":"2024-03-16T10:17:03.346714+0000","last_clean":"2024-03-16T10:17:03.346714+0000","last_became_active":"2024-03-16T10:15:02.013763+0000","last_became_peered":"2024-03-16T10:15:02.013763+0000","last_unstale":"2024-03-16T10:17:03.346714+0000","last_undegraded":"2024-03-16T10:17:03.346714+0000","last_fullsized":"2024-03-16T10:17:03.346714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:12:12.759643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60580,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":243,"num_read_kb":182,"num_write":39,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762495+0000","last_change":"2024-03-16T10:15:03.018838+0000","last_active":"2024-03-16T10:16:28.762495+0000","last_peered":"2024-03-16T10:16:28.762495+0000","last_clean":"2024-03-16T10:16:28.762495+0000","last_became_active":"2024-03-16T10:15:03.017713+0000","last_became_peered":"2024-03-16T10:15:03.017713+0000","last_unstale":"2024-03-16T10:16:28.762495+0000","last_undegraded":"2024-03-16T10:16:28.762495+0000","last_fullsized":"2024-03-16T10:16:28.762495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:03:56.936729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835759+0000","last_change":"2024-03-16T10:15:02.016996+0000","last_active":"2024-03-16T10:15:59.835759+0000","last_peered":"2024-03-16T10:15:59.835759+0000","last_clean":"2024-03-16T10:15:59.835759+0000","last_became_active":"2024-03-16T10:15:02.016421+0000","last_became_peered":"2024-03-16T10:15:02.016421+0000","last_unstale":"2024-03-16T10:15:59.835759+0000","last_undegraded":"2024-03-16T10:15:59.835759+0000","last_fullsized":"2024-03-16T10:15:59.835759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:25:09.866730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.750159+0000","last_change":"2024-03-16T10:15:03.017842+0000","last_active":"2024-03-16T10:16:28.750159+0000","last_peered":"2024-03-16T10:16:28.750159+0000","last_clean":"2024-03-16T10:16:28.750159+0000","last_became_active":"2024-03-16T10:15:03.015119+0000","last_became_peered":"2024-03-16T10:15:03.015119+0000","last_unstale":"2024-03-16T10:16:28.750159+0000","last_undegraded":"2024-03-16T10:16:28.750159+0000","last_fullsized":"2024-03-16T10:16:28.750159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:14:34.474519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898795+0000","last_change":"2024-03-16T10:15:02.021407+0000","last_active":"2024-03-16T10:15:59.898795+0000","last_peered":"2024-03-16T10:15:59.898795+0000","last_clean":"2024-03-16T10:15:59.898795+0000","last_became_active":"2024-03-16T10:15:02.020884+0000","last_became_peered":"2024-03-16T10:15:02.020884+0000","last_unstale":"2024-03-16T10:15:59.898795+0000","last_undegraded":"2024-03-16T10:15:59.898795+0000","last_fullsized":"2024-03-16T10:15:59.898795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:24.858281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.766085+0000","last_change":"2024-03-16T10:15:03.028607+0000","last_active":"2024-03-16T10:16:28.766085+0000","last_peered":"2024-03-16T10:16:28.766085+0000","last_clean":"2024-03-16T10:16:28.766085+0000","last_became_active":"2024-03-16T10:15:03.028280+0000","last_became_peered":"2024-03-16T10:15:03.028280+0000","last_unstale":"2024-03-16T10:16:28.766085+0000","last_undegraded":"2024-03-16T10:16:28.766085+0000","last_fullsized":"2024-03-16T10:16:28.766085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:39.905921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'5","reported_seq":141,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:01.356857+0000","last_change":"2024-03-16T10:15:02.008558+0000","last_active":"2024-03-16T10:17:01.356857+0000","last_peered":"2024-03-16T10:17:01.356857+0000","last_clean":"2024-03-16T10:17:01.356857+0000","last_became_active":"2024-03-16T10:15:02.008014+0000","last_became_peered":"2024-03-16T10:15:02.008014+0000","last_unstale":"2024-03-16T10:17:01.356857+0000","last_undegraded":"2024-03-16T10:17:01.356857+0000","last_fullsized":"2024-03-16T10:17:01.356857+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:40:09.217991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.763448+0000","last_change":"2024-03-16T10:15:03.020847+0000","last_active":"2024-03-16T10:16:28.763448+0000","last_peered":"2024-03-16T10:16:28.763448+0000","last_clean":"2024-03-16T10:16:28.763448+0000","last_became_active":"2024-03-16T10:15:03.020512+0000","last_became_peered":"2024-03-16T10:15:03.020512+0000","last_unstale":"2024-03-16T10:16:28.763448+0000","last_undegraded":"2024-03-16T10:16:28.763448+0000","last_fullsized":"2024-03-16T10:16:28.763448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:04.436195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834281+0000","last_change":"2024-03-16T10:15:02.010690+0000","last_active":"2024-03-16T10:15:59.834281+0000","last_peered":"2024-03-16T10:15:59.834281+0000","last_clean":"2024-03-16T10:15:59.834281+0000","last_became_active":"2024-03-16T10:15:02.010228+0000","last_became_peered":"2024-03-16T10:15:02.010228+0000","last_unstale":"2024-03-16T10:15:59.834281+0000","last_undegraded":"2024-03-16T10:15:59.834281+0000","last_fullsized":"2024-03-16T10:15:59.834281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:30:07.513225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756488+0000","last_change":"2024-03-16T10:15:03.025257+0000","last_active":"2024-03-16T10:16:28.756488+0000","last_peered":"2024-03-16T10:16:28.756488+0000","last_clean":"2024-03-16T10:16:28.756488+0000","last_became_active":"2024-03-16T10:15:03.024498+0000","last_became_peered":"2024-03-16T10:15:03.024498+0000","last_unstale":"2024-03-16T10:16:28.756488+0000","last_undegraded":"2024-03-16T10:16:28.756488+0000","last_fullsized":"2024-03-16T10:16:28.756488+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:27:55.894728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834712+0000","last_change":"2024-03-16T10:15:02.014136+0000","last_active":"2024-03-16T10:15:59.834712+0000","last_peered":"2024-03-16T10:15:59.834712+0000","last_clean":"2024-03-16T10:15:59.834712+0000","last_became_active":"2024-03-16T10:15:02.013721+0000","last_became_peered":"2024-03-16T10:15:02.013721+0000","last_unstale":"2024-03-16T10:15:59.834712+0000","last_undegraded":"2024-03-16T10:15:59.834712+0000","last_fullsized":"2024-03-16T10:15:59.834712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:28:12.301450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.744259+0000","last_change":"2024-03-16T10:15:59.856566+0000","last_active":"2024-03-16T10:16:28.744259+0000","last_peered":"2024-03-16T10:16:28.744259+0000","last_clean":"2024-03-16T10:16:28.744259+0000","last_became_active":"2024-03-16T10:15:59.856152+0000","last_became_peered":"2024-03-16T10:15:59.856152+0000","last_unstale":"2024-03-16T10:16:28.744259+0000","last_undegraded":"2024-03-16T10:16:28.744259+0000","last_fullsized":"2024-03-16T10:16:28.744259+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:42:39.104928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.855367+0000","last_change":"2024-03-16T10:15:02.005754+0000","last_active":"2024-03-16T10:15:59.855367+0000","last_peered":"2024-03-16T10:15:59.855367+0000","last_clean":"2024-03-16T10:15:59.855367+0000","last_became_active":"2024-03-16T10:15:02.005344+0000","last_became_peered":"2024-03-16T10:15:02.005344+0000","last_unstale":"2024-03-16T10:15:59.855367+0000","last_undegraded":"2024-03-16T10:15:59.855367+0000","last_fullsized":"2024-03-16T10:15:59.855367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:37:37.918861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.752059+0000","last_change":"2024-03-16T10:15:03.012887+0000","last_active":"2024-03-16T10:16:28.752059+0000","last_peered":"2024-03-16T10:16:28.752059+0000","last_clean":"2024-03-16T10:16:28.752059+0000","last_became_active":"2024-03-16T10:15:03.012496+0000","last_became_peered":"2024-03-16T10:15:03.012496+0000","last_unstale":"2024-03-16T10:16:28.752059+0000","last_undegraded":"2024-03-16T10:16:28.752059+0000","last_fullsized":"2024-03-16T10:16:28.752059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:11.111049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":138,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:00.355736+0000","last_change":"2024-03-16T10:15:02.014334+0000","last_active":"2024-03-16T10:17:00.355736+0000","last_peered":"2024-03-16T10:17:00.355736+0000","last_clean":"2024-03-16T10:17:00.355736+0000","last_became_active":"2024-03-16T10:15:02.013832+0000","last_became_peered":"2024-03-16T10:15:02.013832+0000","last_unstale":"2024-03-16T10:17:00.355736+0000","last_undegraded":"2024-03-16T10:17:00.355736+0000","last_fullsized":"2024-03-16T10:17:00.355736+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:20.391594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.750566+0000","last_change":"2024-03-16T10:15:03.021276+0000","last_active":"2024-03-16T10:16:28.750566+0000","last_peered":"2024-03-16T10:16:28.750566+0000","last_clean":"2024-03-16T10:16:28.750566+0000","last_became_active":"2024-03-16T10:15:03.020927+0000","last_became_peered":"2024-03-16T10:15:03.020927+0000","last_unstale":"2024-03-16T10:16:28.750566+0000","last_undegraded":"2024-03-16T10:16:28.750566+0000","last_fullsized":"2024-03-16T10:16:28.750566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:53:54.127506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.746363+0000","last_change":"2024-03-16T10:15:03.022083+0000","last_active":"2024-03-16T10:16:28.746363+0000","last_peered":"2024-03-16T10:16:28.746363+0000","last_clean":"2024-03-16T10:16:28.746363+0000","last_became_active":"2024-03-16T10:15:03.021609+0000","last_became_peered":"2024-03-16T10:15:03.021609+0000","last_unstale":"2024-03-16T10:16:28.746363+0000","last_undegraded":"2024-03-16T10:16:28.746363+0000","last_fullsized":"2024-03-16T10:16:28.746363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:53.864402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":139,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:00.355523+0000","last_change":"2024-03-16T10:15:02.015497+0000","last_active":"2024-03-16T10:17:00.355523+0000","last_peered":"2024-03-16T10:17:00.355523+0000","last_clean":"2024-03-16T10:17:00.355523+0000","last_became_active":"2024-03-16T10:15:02.015183+0000","last_became_peered":"2024-03-16T10:15:02.015183+0000","last_unstale":"2024-03-16T10:17:00.355523+0000","last_undegraded":"2024-03-16T10:17:00.355523+0000","last_fullsized":"2024-03-16T10:17:00.355523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:44:05.913030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.755036+0000","last_change":"2024-03-16T10:15:03.020046+0000","last_active":"2024-03-16T10:16:28.755036+0000","last_peered":"2024-03-16T10:16:28.755036+0000","last_clean":"2024-03-16T10:16:28.755036+0000","last_became_active":"2024-03-16T10:15:03.019647+0000","last_became_peered":"2024-03-16T10:15:03.019647+0000","last_unstale":"2024-03-16T10:16:28.755036+0000","last_undegraded":"2024-03-16T10:16:28.755036+0000","last_fullsized":"2024-03-16T10:16:28.755036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:18.343495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134445+0000","last_change":"2024-03-16T10:15:02.008560+0000","last_active":"2024-03-16T10:16:00.134445+0000","last_peered":"2024-03-16T10:16:00.134445+0000","last_clean":"2024-03-16T10:16:00.134445+0000","last_became_active":"2024-03-16T10:15:02.008094+0000","last_became_peered":"2024-03-16T10:15:02.008094+0000","last_unstale":"2024-03-16T10:16:00.134445+0000","last_undegraded":"2024-03-16T10:16:00.134445+0000","last_fullsized":"2024-03-16T10:16:00.134445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:21:43.555120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":342,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838687+0000","last_change":"2024-03-16T10:13:47.211815+0000","last_active":"2024-03-16T10:15:59.838687+0000","last_peered":"2024-03-16T10:15:59.838687+0000","last_clean":"2024-03-16T10:15:59.838687+0000","last_became_active":"2024-03-16T10:13:47.211353+0000","last_became_peered":"2024-03-16T10:13:47.211353+0000","last_unstale":"2024-03-16T10:15:59.838687+0000","last_undegraded":"2024-03-16T10:15:59.838687+0000","last_fullsized":"2024-03-16T10:15:59.838687+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_clean_scrub_stamp":"2024-03-16T10:10:22.965421+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:11:20.607686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.751691+0000","last_change":"2024-03-16T10:15:03.020181+0000","last_active":"2024-03-16T10:16:28.751691+0000","last_peered":"2024-03-16T10:16:28.751691+0000","last_clean":"2024-03-16T10:16:28.751691+0000","last_became_active":"2024-03-16T10:15:03.019781+0000","last_became_peered":"2024-03-16T10:15:03.019781+0000","last_unstale":"2024-03-16T10:16:28.751691+0000","last_undegraded":"2024-03-16T10:16:28.751691+0000","last_fullsized":"2024-03-16T10:16:28.751691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:04:49.369484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838833+0000","last_change":"2024-03-16T10:15:02.014461+0000","last_active":"2024-03-16T10:15:59.838833+0000","last_peered":"2024-03-16T10:15:59.838833+0000","last_clean":"2024-03-16T10:15:59.838833+0000","last_became_active":"2024-03-16T10:15:02.014137+0000","last_became_peered":"2024-03-16T10:15:02.014137+0000","last_unstale":"2024-03-16T10:15:59.838833+0000","last_undegraded":"2024-03-16T10:15:59.838833+0000","last_fullsized":"2024-03-16T10:15:59.838833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:36:41.830893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834705+0000","last_change":"2024-03-16T10:15:02.019441+0000","last_active":"2024-03-16T10:15:59.834705+0000","last_peered":"2024-03-16T10:15:59.834705+0000","last_clean":"2024-03-16T10:15:59.834705+0000","last_became_active":"2024-03-16T10:15:02.019091+0000","last_became_peered":"2024-03-16T10:15:02.019091+0000","last_unstale":"2024-03-16T10:15:59.834705+0000","last_undegraded":"2024-03-16T10:15:59.834705+0000","last_fullsized":"2024-03-16T10:15:59.834705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:04:51.758722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.745121+0000","last_change":"2024-03-16T10:15:03.012141+0000","last_active":"2024-03-16T10:16:28.745121+0000","last_peered":"2024-03-16T10:16:28.745121+0000","last_clean":"2024-03-16T10:16:28.745121+0000","last_became_active":"2024-03-16T10:15:03.011771+0000","last_became_peered":"2024-03-16T10:15:03.011771+0000","last_unstale":"2024-03-16T10:16:28.745121+0000","last_undegraded":"2024-03-16T10:16:28.745121+0000","last_fullsized":"2024-03-16T10:16:28.745121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:02:28.522530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'15","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:29.030048+0000","last_change":"2024-03-16T10:15:02.011898+0000","last_active":"2024-03-16T10:16:29.030048+0000","last_peered":"2024-03-16T10:16:29.030048+0000","last_clean":"2024-03-16T10:16:29.030048+0000","last_became_active":"2024-03-16T10:15:02.011578+0000","last_became_peered":"2024-03-16T10:15:02.011578+0000","last_unstale":"2024-03-16T10:16:29.030048+0000","last_undegraded":"2024-03-16T10:16:29.030048+0000","last_fullsized":"2024-03-16T10:16:29.030048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:27:35.756031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762502+0000","last_change":"2024-03-16T10:15:03.024923+0000","last_active":"2024-03-16T10:16:28.762502+0000","last_peered":"2024-03-16T10:16:28.762502+0000","last_clean":"2024-03-16T10:16:28.762502+0000","last_became_active":"2024-03-16T10:15:03.024577+0000","last_became_peered":"2024-03-16T10:15:03.024577+0000","last_unstale":"2024-03-16T10:16:28.762502+0000","last_undegraded":"2024-03-16T10:16:28.762502+0000","last_fullsized":"2024-03-16T10:16:28.762502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:04.650499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134719+0000","last_change":"2024-03-16T10:15:02.006842+0000","last_active":"2024-03-16T10:16:00.134719+0000","last_peered":"2024-03-16T10:16:00.134719+0000","last_clean":"2024-03-16T10:16:00.134719+0000","last_became_active":"2024-03-16T10:15:02.006483+0000","last_became_peered":"2024-03-16T10:15:02.006483+0000","last_unstale":"2024-03-16T10:16:00.134719+0000","last_undegraded":"2024-03-16T10:16:00.134719+0000","last_fullsized":"2024-03-16T10:16:00.134719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:30:55.856007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.760743+0000","last_change":"2024-03-16T10:15:03.018691+0000","last_active":"2024-03-16T10:16:28.760743+0000","last_peered":"2024-03-16T10:16:28.760743+0000","last_clean":"2024-03-16T10:16:28.760743+0000","last_became_active":"2024-03-16T10:15:03.018304+0000","last_became_peered":"2024-03-16T10:15:03.018304+0000","last_unstale":"2024-03-16T10:16:28.760743+0000","last_undegraded":"2024-03-16T10:16:28.760743+0000","last_fullsized":"2024-03-16T10:16:28.760743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:15.446759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.841306+0000","last_change":"2024-03-16T10:15:59.841306+0000","last_active":"2024-03-16T10:15:59.841306+0000","last_peered":"2024-03-16T10:15:59.841306+0000","last_clean":"2024-03-16T10:15:59.841306+0000","last_became_active":"2024-03-16T10:15:59.838434+0000","last_became_peered":"2024-03-16T10:15:59.838434+0000","last_unstale":"2024-03-16T10:15:59.841306+0000","last_undegraded":"2024-03-16T10:15:59.841306+0000","last_fullsized":"2024-03-16T10:15:59.841306+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:14:21.702809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.719194+0000","last_change":"2024-03-16T10:15:03.017262+0000","last_active":"2024-03-16T10:16:28.719194+0000","last_peered":"2024-03-16T10:16:28.719194+0000","last_clean":"2024-03-16T10:16:28.719194+0000","last_became_active":"2024-03-16T10:15:03.016909+0000","last_became_peered":"2024-03-16T10:15:03.016909+0000","last_unstale":"2024-03-16T10:16:28.719194+0000","last_undegraded":"2024-03-16T10:16:28.719194+0000","last_fullsized":"2024-03-16T10:16:28.719194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:53:11.385236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837856+0000","last_change":"2024-03-16T10:15:02.011325+0000","last_active":"2024-03-16T10:15:59.837856+0000","last_peered":"2024-03-16T10:15:59.837856+0000","last_clean":"2024-03-16T10:15:59.837856+0000","last_became_active":"2024-03-16T10:15:02.010999+0000","last_became_peered":"2024-03-16T10:15:02.010999+0000","last_unstale":"2024-03-16T10:15:59.837856+0000","last_undegraded":"2024-03-16T10:15:59.837856+0000","last_fullsized":"2024-03-16T10:15:59.837856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:30:08.137693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.757020+0000","last_change":"2024-03-16T10:15:03.012146+0000","last_active":"2024-03-16T10:16:28.757020+0000","last_peered":"2024-03-16T10:16:28.757020+0000","last_clean":"2024-03-16T10:16:28.757020+0000","last_became_active":"2024-03-16T10:15:03.011790+0000","last_became_peered":"2024-03-16T10:15:03.011790+0000","last_unstale":"2024-03-16T10:16:28.757020+0000","last_undegraded":"2024-03-16T10:16:28.757020+0000","last_fullsized":"2024-03-16T10:16:28.757020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:45:22.642925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.842829+0000","last_change":"2024-03-16T10:15:59.842829+0000","last_active":"2024-03-16T10:15:59.842829+0000","last_peered":"2024-03-16T10:15:59.842829+0000","last_clean":"2024-03-16T10:15:59.842829+0000","last_became_active":"2024-03-16T10:15:59.838787+0000","last_became_peered":"2024-03-16T10:15:59.838787+0000","last_unstale":"2024-03-16T10:15:59.842829+0000","last_undegraded":"2024-03-16T10:15:59.842829+0000","last_fullsized":"2024-03-16T10:15:59.842829+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:47:27.573657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.759857+0000","last_change":"2024-03-16T10:15:03.021094+0000","last_active":"2024-03-16T10:16:28.759857+0000","last_peered":"2024-03-16T10:16:28.759857+0000","last_clean":"2024-03-16T10:16:28.759857+0000","last_became_active":"2024-03-16T10:15:03.017640+0000","last_became_peered":"2024-03-16T10:15:03.017640+0000","last_unstale":"2024-03-16T10:16:28.759857+0000","last_undegraded":"2024-03-16T10:16:28.759857+0000","last_fullsized":"2024-03-16T10:16:28.759857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:02.030112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834498+0000","last_change":"2024-03-16T10:15:02.015652+0000","last_active":"2024-03-16T10:15:59.834498+0000","last_peered":"2024-03-16T10:15:59.834498+0000","last_clean":"2024-03-16T10:15:59.834498+0000","last_became_active":"2024-03-16T10:15:02.014046+0000","last_became_peered":"2024-03-16T10:15:02.014046+0000","last_unstale":"2024-03-16T10:15:59.834498+0000","last_undegraded":"2024-03-16T10:15:59.834498+0000","last_fullsized":"2024-03-16T10:15:59.834498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.765255+0000","last_change":"2024-03-16T10:15:03.023233+0000","last_active":"2024-03-16T10:16:28.765255+0000","last_peered":"2024-03-16T10:16:28.765255+0000","last_clean":"2024-03-16T10:16:28.765255+0000","last_became_active":"2024-03-16T10:15:03.022906+0000","last_became_peered":"2024-03-16T10:15:03.022906+0000","last_unstale":"2024-03-16T10:16:28.765255+0000","last_undegraded":"2024-03-16T10:16:28.765255+0000","last_fullsized":"2024-03-16T10:16:28.765255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:22.943087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'7","reported_seq":144,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:01.344825+0000","last_change":"2024-03-16T10:15:02.010755+0000","last_active":"2024-03-16T10:17:01.344825+0000","last_peered":"2024-03-16T10:17:01.344825+0000","last_clean":"2024-03-16T10:17:01.344825+0000","last_became_active":"2024-03-16T10:15:02.010422+0000","last_became_peered":"2024-03-16T10:15:02.010422+0000","last_unstale":"2024-03-16T10:17:01.344825+0000","last_undegraded":"2024-03-16T10:17:01.344825+0000","last_fullsized":"2024-03-16T10:17:01.344825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:17.606346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.765240+0000","last_change":"2024-03-16T10:15:03.025789+0000","last_active":"2024-03-16T10:16:28.765240+0000","last_peered":"2024-03-16T10:16:28.765240+0000","last_clean":"2024-03-16T10:16:28.765240+0000","last_became_active":"2024-03-16T10:15:03.024812+0000","last_became_peered":"2024-03-16T10:15:03.024812+0000","last_unstale":"2024-03-16T10:16:28.765240+0000","last_undegraded":"2024-03-16T10:16:28.765240+0000","last_fullsized":"2024-03-16T10:16:28.765240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:53.041234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898933+0000","last_change":"2024-03-16T10:15:02.020599+0000","last_active":"2024-03-16T10:15:59.898933+0000","last_peered":"2024-03-16T10:15:59.898933+0000","last_clean":"2024-03-16T10:15:59.898933+0000","last_became_active":"2024-03-16T10:15:02.020203+0000","last_became_peered":"2024-03-16T10:15:02.020203+0000","last_unstale":"2024-03-16T10:15:59.898933+0000","last_undegraded":"2024-03-16T10:15:59.898933+0000","last_fullsized":"2024-03-16T10:15:59.898933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:46:33.032310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.764526+0000","last_change":"2024-03-16T10:15:03.016223+0000","last_active":"2024-03-16T10:16:28.764526+0000","last_peered":"2024-03-16T10:16:28.764526+0000","last_clean":"2024-03-16T10:16:28.764526+0000","last_became_active":"2024-03-16T10:15:03.015857+0000","last_became_peered":"2024-03-16T10:15:03.015857+0000","last_unstale":"2024-03-16T10:16:28.764526+0000","last_undegraded":"2024-03-16T10:16:28.764526+0000","last_fullsized":"2024-03-16T10:16:28.764526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:40:10.329722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835797+0000","last_change":"2024-03-16T10:15:02.015690+0000","last_active":"2024-03-16T10:15:59.835797+0000","last_peered":"2024-03-16T10:15:59.835797+0000","last_clean":"2024-03-16T10:15:59.835797+0000","last_became_active":"2024-03-16T10:15:02.015375+0000","last_became_peered":"2024-03-16T10:15:02.015375+0000","last_unstale":"2024-03-16T10:15:59.835797+0000","last_undegraded":"2024-03-16T10:15:59.835797+0000","last_fullsized":"2024-03-16T10:15:59.835797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:39.118343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762466+0000","last_change":"2024-03-16T10:15:03.012702+0000","last_active":"2024-03-16T10:16:28.762466+0000","last_peered":"2024-03-16T10:16:28.762466+0000","last_clean":"2024-03-16T10:16:28.762466+0000","last_became_active":"2024-03-16T10:15:03.012387+0000","last_became_peered":"2024-03-16T10:15:03.012387+0000","last_unstale":"2024-03-16T10:16:28.762466+0000","last_undegraded":"2024-03-16T10:16:28.762466+0000","last_fullsized":"2024-03-16T10:16:28.762466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:16.938372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756885+0000","last_change":"2024-03-16T10:15:03.015290+0000","last_active":"2024-03-16T10:16:28.756885+0000","last_peered":"2024-03-16T10:16:28.756885+0000","last_clean":"2024-03-16T10:16:28.756885+0000","last_became_active":"2024-03-16T10:15:03.013846+0000","last_became_peered":"2024-03-16T10:15:03.013846+0000","last_unstale":"2024-03-16T10:16:28.756885+0000","last_undegraded":"2024-03-16T10:16:28.756885+0000","last_fullsized":"2024-03-16T10:16:28.756885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:33.377793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.744383+0000","last_change":"2024-03-16T10:15:03.023733+0000","last_active":"2024-03-16T10:16:28.744383+0000","last_peered":"2024-03-16T10:16:28.744383+0000","last_clean":"2024-03-16T10:16:28.744383+0000","last_became_active":"2024-03-16T10:15:03.023409+0000","last_became_peered":"2024-03-16T10:15:03.023409+0000","last_unstale":"2024-03-16T10:16:28.744383+0000","last_undegraded":"2024-03-16T10:16:28.744383+0000","last_fullsized":"2024-03-16T10:16:28.744383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:44.748152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756641+0000","last_change":"2024-03-16T10:15:59.856644+0000","last_active":"2024-03-16T10:16:28.756641+0000","last_peered":"2024-03-16T10:16:28.756641+0000","last_clean":"2024-03-16T10:16:28.756641+0000","last_became_active":"2024-03-16T10:15:59.856243+0000","last_became_peered":"2024-03-16T10:15:59.856243+0000","last_unstale":"2024-03-16T10:16:28.756641+0000","last_undegraded":"2024-03-16T10:16:28.756641+0000","last_fullsized":"2024-03-16T10:16:28.756641+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:22.831073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762122+0000","last_change":"2024-03-16T10:15:03.029277+0000","last_active":"2024-03-16T10:16:28.762122+0000","last_peered":"2024-03-16T10:16:28.762122+0000","last_clean":"2024-03-16T10:16:28.762122+0000","last_became_active":"2024-03-16T10:15:03.028960+0000","last_became_peered":"2024-03-16T10:15:03.028960+0000","last_unstale":"2024-03-16T10:16:28.762122+0000","last_undegraded":"2024-03-16T10:16:28.762122+0000","last_fullsized":"2024-03-16T10:16:28.762122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:50:48.362914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756920+0000","last_change":"2024-03-16T10:15:59.856671+0000","last_active":"2024-03-16T10:16:28.756920+0000","last_peered":"2024-03-16T10:16:28.756920+0000","last_clean":"2024-03-16T10:16:28.756920+0000","last_became_active":"2024-03-16T10:15:59.856287+0000","last_became_peered":"2024-03-16T10:15:59.856287+0000","last_unstale":"2024-03-16T10:16:28.756920+0000","last_undegraded":"2024-03-16T10:16:28.756920+0000","last_fullsized":"2024-03-16T10:16:28.756920+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:15:58.409554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756231+0000","last_change":"2024-03-16T10:15:03.017499+0000","last_active":"2024-03-16T10:16:28.756231+0000","last_peered":"2024-03-16T10:16:28.756231+0000","last_clean":"2024-03-16T10:16:28.756231+0000","last_became_active":"2024-03-16T10:15:03.017085+0000","last_became_peered":"2024-03-16T10:15:03.017085+0000","last_unstale":"2024-03-16T10:16:28.756231+0000","last_undegraded":"2024-03-16T10:16:28.756231+0000","last_fullsized":"2024-03-16T10:16:28.756231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:57:09.592420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.752712+0000","last_change":"2024-03-16T10:15:03.018943+0000","last_active":"2024-03-16T10:16:28.752712+0000","last_peered":"2024-03-16T10:16:28.752712+0000","last_clean":"2024-03-16T10:16:28.752712+0000","last_became_active":"2024-03-16T10:15:03.017519+0000","last_became_peered":"2024-03-16T10:15:03.017519+0000","last_unstale":"2024-03-16T10:16:28.752712+0000","last_undegraded":"2024-03-16T10:16:28.752712+0000","last_fullsized":"2024-03-16T10:16:28.752712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:29.026988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.754110+0000","last_change":"2024-03-16T10:15:03.024697+0000","last_active":"2024-03-16T10:16:28.754110+0000","last_peered":"2024-03-16T10:16:28.754110+0000","last_clean":"2024-03-16T10:16:28.754110+0000","last_became_active":"2024-03-16T10:15:03.024305+0000","last_became_peered":"2024-03-16T10:15:03.024305+0000","last_unstale":"2024-03-16T10:16:28.754110+0000","last_undegraded":"2024-03-16T10:16:28.754110+0000","last_fullsized":"2024-03-16T10:16:28.754110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:07:35.160746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.755394+0000","last_change":"2024-03-16T10:15:03.017504+0000","last_active":"2024-03-16T10:16:28.755394+0000","last_peered":"2024-03-16T10:16:28.755394+0000","last_clean":"2024-03-16T10:16:28.755394+0000","last_became_active":"2024-03-16T10:15:03.017100+0000","last_became_peered":"2024-03-16T10:15:03.017100+0000","last_unstale":"2024-03-16T10:16:28.755394+0000","last_undegraded":"2024-03-16T10:16:28.755394+0000","last_fullsized":"2024-03-16T10:16:28.755394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:23:17.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.754759+0000","last_change":"2024-03-16T10:15:03.024684+0000","last_active":"2024-03-16T10:16:28.754759+0000","last_peered":"2024-03-16T10:16:28.754759+0000","last_clean":"2024-03-16T10:16:28.754759+0000","last_became_active":"2024-03-16T10:15:03.024282+0000","last_became_peered":"2024-03-16T10:15:03.024282+0000","last_unstale":"2024-03-16T10:16:28.754759+0000","last_undegraded":"2024-03-16T10:16:28.754759+0000","last_fullsized":"2024-03-16T10:16:28.754759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:29.595162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.758529+0000","last_change":"2024-03-16T10:15:03.025337+0000","last_active":"2024-03-16T10:16:28.758529+0000","last_peered":"2024-03-16T10:16:28.758529+0000","last_clean":"2024-03-16T10:16:28.758529+0000","last_became_active":"2024-03-16T10:15:03.024938+0000","last_became_peered":"2024-03-16T10:15:03.024938+0000","last_unstale":"2024-03-16T10:16:28.758529+0000","last_undegraded":"2024-03-16T10:16:28.758529+0000","last_fullsized":"2024-03-16T10:16:28.758529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:13:15.892354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.765584+0000","last_change":"2024-03-16T10:15:03.015468+0000","last_active":"2024-03-16T10:16:28.765584+0000","last_peered":"2024-03-16T10:16:28.765584+0000","last_clean":"2024-03-16T10:16:28.765584+0000","last_became_active":"2024-03-16T10:15:03.015097+0000","last_became_peered":"2024-03-16T10:15:03.015097+0000","last_unstale":"2024-03-16T10:16:28.765584+0000","last_undegraded":"2024-03-16T10:16:28.765584+0000","last_fullsized":"2024-03-16T10:16:28.765584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:46.114346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.761104+0000","last_change":"2024-03-16T10:15:03.020690+0000","last_active":"2024-03-16T10:16:28.761104+0000","last_peered":"2024-03-16T10:16:28.761104+0000","last_clean":"2024-03-16T10:16:28.761104+0000","last_became_active":"2024-03-16T10:15:03.020365+0000","last_became_peered":"2024-03-16T10:15:03.020365+0000","last_unstale":"2024-03-16T10:16:28.761104+0000","last_undegraded":"2024-03-16T10:16:28.761104+0000","last_fullsized":"2024-03-16T10:16:28.761104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:25.671533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280280,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67925,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1197,"num_read_kb":675,"num_write":253,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":209456,"data_compressed":5583,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":457189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":1108266,"data_compressed":37511,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":456587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":456965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":1049092,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":456304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":518105,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":458175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":518549,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":456129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":1049161,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68795,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63084,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":62316,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T10:17:07.971 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-16T10:17:07.971 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T10:17:08.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.112+0000 7f2a83153640 1 -- 172.21.15.77:0/801662885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c1022b0 msgr2=0x7f2a7c108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.112+0000 7f2a83153640 1 --2- 172.21.15.77:0/801662885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c1022b0 0x7f2a7c108b40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009ca0 tx=0x7f2a7402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.112+0000 7f2a83153640 1 -- 172.21.15.77:0/801662885 shutdown_connections 2024-03-16T10:17:08.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.112+0000 7f2a83153640 1 --2- 172.21.15.77:0/801662885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c1022b0 0x7f2a7c108b40 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.112+0000 7f2a83153640 1 --2- 172.21.15.77:0/801662885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a7c1018f0 0x7f2a7c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.112+0000 7f2a83153640 1 --2- 172.21.15.77:0/801662885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a7c100700 0x7f2a7c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.112+0000 7f2a83153640 1 -- 172.21.15.77:0/801662885 >> 172.21.15.77:0/801662885 conn(0x7f2a7c0fbc70 msgr2=0x7f2a7c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.112+0000 7f2a83153640 1 -- 172.21.15.77:0/801662885 shutdown_connections 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.112+0000 7f2a83153640 1 -- 172.21.15.77:0/801662885 wait complete. 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a83153640 1 Processor -- start 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a83153640 1 -- start start 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a83153640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a7c100700 0x7f2a7c1969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a83153640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c1018f0 0x7f2a7c196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a83153640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a7c1022b0 0x7f2a7c19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a83153640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c10c360 con 0x7f2a7c1018f0 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a83153640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c10c1e0 con 0x7f2a7c100700 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a83153640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c10c4e0 con 0x7f2a7c1022b0 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a7bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c1018f0 0x7f2a7c196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:08.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a7bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c1018f0 0x7f2a7c196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56222/0 (socket says 172.21.15.77:56222) 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a7bfff640 1 -- 172.21.15.77:0/2149661682 learned_addr learned my addr 172.21.15.77:0/2149661682 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a80eef640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a7c100700 0x7f2a7c1969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a816f0640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a7c1022b0 0x7f2a7c19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a7bfff640 1 -- 172.21.15.77:0/2149661682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a7c1022b0 msgr2=0x7f2a7c19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a7bfff640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a7c1022b0 0x7f2a7c19df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a7bfff640 1 -- 172.21.15.77:0/2149661682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a7c100700 msgr2=0x7f2a7c1969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a7bfff640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a7c100700 0x7f2a7c1969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.113+0000 7f2a7bfff640 1 -- 172.21.15.77:0/2149661682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7c19e5c0 con 0x7f2a7c1018f0 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.114+0000 7f2a7bfff640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c1018f0 0x7f2a7c196ee0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800e9f0 tx=0x7f2a6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.114+0000 7f2a79ffb640 1 -- 172.21.15.77:0/2149661682 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6800cd80 con 0x7f2a7c1018f0 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.114+0000 7f2a79ffb640 1 -- 172.21.15.77:0/2149661682 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a68010910 con 0x7f2a7c1018f0 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.114+0000 7f2a83153640 1 -- 172.21.15.77:0/2149661682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a7c19e8b0 con 0x7f2a7c1018f0 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.114+0000 7f2a79ffb640 1 -- 172.21.15.77:0/2149661682 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68018990 con 0x7f2a7c1018f0 2024-03-16T10:17:08.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.114+0000 7f2a83153640 1 -- 172.21.15.77:0/2149661682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a7c19ee00 con 0x7f2a7c1018f0 2024-03-16T10:17:08.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.115+0000 7f2a83153640 1 -- 172.21.15.77:0/2149661682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a7c107a20 con 0x7f2a7c1018f0 2024-03-16T10:17:08.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.116+0000 7f2a79ffb640 1 -- 172.21.15.77:0/2149661682 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f2a68010ac0 con 0x7f2a7c1018f0 2024-03-16T10:17:08.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.116+0000 7f2a79ffb640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2a6c0a2890 0x7f2a6c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:08.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.117+0000 7f2a80eef640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2a6c0a2890 0x7f2a6c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:08.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.117+0000 7f2a79ffb640 1 -- 172.21.15.77:0/2149661682 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2a680cee50 con 0x7f2a7c1018f0 2024-03-16T10:17:08.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.117+0000 7f2a80eef640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2a6c0a2890 0x7f2a6c0a4d50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2a700095f0 tx=0x7f2a70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:08.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.121+0000 7f2a79ffb640 1 -- 172.21.15.77:0/2149661682 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a6809b830 con 0x7f2a7c1018f0 2024-03-16T10:17:08.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:07 smithi192 ceph-mon[36354]: pgmap v351: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-16T10:17:08.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:07 smithi192 ceph-mon[36354]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-16T10:17:08.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:07 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2309530245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:17:08.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:07 smithi077 ceph-mon[32400]: pgmap v351: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-16T10:17:08.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:07 smithi077 ceph-mon[32400]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-16T10:17:08.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:07 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2309530245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:17:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:07 smithi193 ceph-mon[36311]: pgmap v351: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-16T10:17:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:07 smithi193 ceph-mon[36311]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-16T10:17:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:07 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2309530245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:17:08.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.299+0000 7f2a83153640 1 -- 172.21.15.77:0/2149661682 --> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2a7c061c20 con 0x7f2a6c0a2890 2024-03-16T10:17:08.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.312+0000 7f2a79ffb640 1 -- 172.21.15.77:0/2149661682 <== mgr.14152 v2:172.21.15.77:6800/1427985582 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259101 (secure 0 0 0) 0x7f2a7c061c20 con 0x7f2a6c0a2890 2024-03-16T10:17:08.315 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:17:08.317 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-03-16T10:17:08.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.315+0000 7f2a83153640 1 -- 172.21.15.77:0/2149661682 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2a6c0a2890 msgr2=0x7f2a6c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.315+0000 7f2a83153640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2a6c0a2890 0x7f2a6c0a4d50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2a700095f0 tx=0x7f2a70005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:08.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.315+0000 7f2a83153640 1 -- 172.21.15.77:0/2149661682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c1018f0 msgr2=0x7f2a7c196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.315+0000 7f2a83153640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c1018f0 0x7f2a7c196ee0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800e9f0 tx=0x7f2a6800eeb0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.315+0000 7f2a83153640 1 -- 172.21.15.77:0/2149661682 shutdown_connections 2024-03-16T10:17:08.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.315+0000 7f2a83153640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2a6c0a2890 0x7f2a6c0a4d50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.315+0000 7f2a83153640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a7c1022b0 0x7f2a7c19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.315+0000 7f2a83153640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a7c1018f0 0x7f2a7c196ee0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.315+0000 7f2a83153640 1 --2- 172.21.15.77:0/2149661682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a7c100700 0x7f2a7c1969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.315+0000 7f2a83153640 1 -- 172.21.15.77:0/2149661682 >> 172.21.15.77:0/2149661682 conn(0x7f2a7c0fbc70 msgr2=0x7f2a7c104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:08.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.315+0000 7f2a83153640 1 -- 172.21.15.77:0/2149661682 shutdown_connections 2024-03-16T10:17:08.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.316+0000 7f2a83153640 1 -- 172.21.15.77:0/2149661682 wait complete. 2024-03-16T10:17:08.327 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-03-16T10:17:06.743397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658527,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1323,"num_read_kb":784,"num_write":436,"num_write_kb":3084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":858,"ondisk_log_size":858,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360852,"kb_used_data":11700,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580972,"statfs":{"total":1151940427776,"available":1151570915328,"internally_reserved":0,"allocated":11980800,"data_stored":7440687,"data_compressed":112533,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":699,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":52,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002662"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.757174+0000","last_change":"2024-03-16T10:15:03.016880+0000","last_active":"2024-03-16T10:16:28.757174+0000","last_peered":"2024-03-16T10:16:28.757174+0000","last_clean":"2024-03-16T10:16:28.757174+0000","last_became_active":"2024-03-16T10:15:03.016513+0000","last_became_peered":"2024-03-16T10:15:03.016513+0000","last_unstale":"2024-03-16T10:16:28.757174+0000","last_undegraded":"2024-03-16T10:16:28.757174+0000","last_fullsized":"2024-03-16T10:16:28.757174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:39.834015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.753595+0000","last_change":"2024-03-16T10:15:03.015467+0000","last_active":"2024-03-16T10:16:28.753595+0000","last_peered":"2024-03-16T10:16:28.753595+0000","last_clean":"2024-03-16T10:16:28.753595+0000","last_became_active":"2024-03-16T10:15:03.015152+0000","last_became_peered":"2024-03-16T10:15:03.015152+0000","last_unstale":"2024-03-16T10:16:28.753595+0000","last_undegraded":"2024-03-16T10:16:28.753595+0000","last_fullsized":"2024-03-16T10:16:28.753595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:17:29.988569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.753318+0000","last_change":"2024-03-16T10:15:03.023836+0000","last_active":"2024-03-16T10:16:28.753318+0000","last_peered":"2024-03-16T10:16:28.753318+0000","last_clean":"2024-03-16T10:16:28.753318+0000","last_became_active":"2024-03-16T10:15:03.022774+0000","last_became_peered":"2024-03-16T10:15:03.022774+0000","last_unstale":"2024-03-16T10:16:28.753318+0000","last_undegraded":"2024-03-16T10:16:28.753318+0000","last_fullsized":"2024-03-16T10:16:28.753318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:49.837157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.759581+0000","last_change":"2024-03-16T10:15:03.030891+0000","last_active":"2024-03-16T10:16:28.759581+0000","last_peered":"2024-03-16T10:16:28.759581+0000","last_clean":"2024-03-16T10:16:28.759581+0000","last_became_active":"2024-03-16T10:15:03.030278+0000","last_became_peered":"2024-03-16T10:15:03.030278+0000","last_unstale":"2024-03-16T10:16:28.759581+0000","last_undegraded":"2024-03-16T10:16:28.759581+0000","last_fullsized":"2024-03-16T10:16:28.759581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:22:16.166885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.742976+0000","last_change":"2024-03-16T10:15:03.017610+0000","last_active":"2024-03-16T10:16:28.742976+0000","last_peered":"2024-03-16T10:16:28.742976+0000","last_clean":"2024-03-16T10:16:28.742976+0000","last_became_active":"2024-03-16T10:15:03.017250+0000","last_became_peered":"2024-03-16T10:15:03.017250+0000","last_unstale":"2024-03-16T10:16:28.742976+0000","last_undegraded":"2024-03-16T10:16:28.742976+0000","last_fullsized":"2024-03-16T10:16:28.742976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:49:07.899425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.753472+0000","last_change":"2024-03-16T10:15:59.856216+0000","last_active":"2024-03-16T10:16:28.753472+0000","last_peered":"2024-03-16T10:16:28.753472+0000","last_clean":"2024-03-16T10:16:28.753472+0000","last_became_active":"2024-03-16T10:15:59.855882+0000","last_became_peered":"2024-03-16T10:15:59.855882+0000","last_unstale":"2024-03-16T10:16:28.753472+0000","last_undegraded":"2024-03-16T10:16:28.753472+0000","last_fullsized":"2024-03-16T10:16:28.753472+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:27:42.682580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762588+0000","last_change":"2024-03-16T10:15:03.017686+0000","last_active":"2024-03-16T10:16:28.762588+0000","last_peered":"2024-03-16T10:16:28.762588+0000","last_clean":"2024-03-16T10:16:28.762588+0000","last_became_active":"2024-03-16T10:15:03.017262+0000","last_became_peered":"2024-03-16T10:15:03.017262+0000","last_unstale":"2024-03-16T10:16:28.762588+0000","last_undegraded":"2024-03-16T10:16:28.762588+0000","last_fullsized":"2024-03-16T10:16:28.762588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:36:22.270651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.759952+0000","last_change":"2024-03-16T10:15:03.020130+0000","last_active":"2024-03-16T10:16:28.759952+0000","last_peered":"2024-03-16T10:16:28.759952+0000","last_clean":"2024-03-16T10:16:28.759952+0000","last_became_active":"2024-03-16T10:15:03.019742+0000","last_became_peered":"2024-03-16T10:15:03.019742+0000","last_unstale":"2024-03-16T10:16:28.759952+0000","last_undegraded":"2024-03-16T10:16:28.759952+0000","last_fullsized":"2024-03-16T10:16:28.759952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:10:52.450792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.757769+0000","last_change":"2024-03-16T10:15:03.027589+0000","last_active":"2024-03-16T10:16:28.757769+0000","last_peered":"2024-03-16T10:16:28.757769+0000","last_clean":"2024-03-16T10:16:28.757769+0000","last_became_active":"2024-03-16T10:15:03.027226+0000","last_became_peered":"2024-03-16T10:15:03.027226+0000","last_unstale":"2024-03-16T10:16:28.757769+0000","last_undegraded":"2024-03-16T10:16:28.757769+0000","last_fullsized":"2024-03-16T10:16:28.757769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:14:35.354629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.761345+0000","last_change":"2024-03-16T10:15:03.021163+0000","last_active":"2024-03-16T10:16:28.761345+0000","last_peered":"2024-03-16T10:16:28.761345+0000","last_clean":"2024-03-16T10:16:28.761345+0000","last_became_active":"2024-03-16T10:15:03.020483+0000","last_became_peered":"2024-03-16T10:15:03.020483+0000","last_unstale":"2024-03-16T10:16:28.761345+0000","last_undegraded":"2024-03-16T10:16:28.761345+0000","last_fullsized":"2024-03-16T10:16:28.761345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:01:00.873471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.765112+0000","last_change":"2024-03-16T10:15:03.026111+0000","last_active":"2024-03-16T10:16:28.765112+0000","last_peered":"2024-03-16T10:16:28.765112+0000","last_clean":"2024-03-16T10:16:28.765112+0000","last_became_active":"2024-03-16T10:15:03.025296+0000","last_became_peered":"2024-03-16T10:15:03.025296+0000","last_unstale":"2024-03-16T10:16:28.765112+0000","last_undegraded":"2024-03-16T10:16:28.765112+0000","last_fullsized":"2024-03-16T10:16:28.765112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:33:29.606876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.746441+0000","last_change":"2024-03-16T10:15:03.023013+0000","last_active":"2024-03-16T10:16:28.746441+0000","last_peered":"2024-03-16T10:16:28.746441+0000","last_clean":"2024-03-16T10:16:28.746441+0000","last_became_active":"2024-03-16T10:15:03.022251+0000","last_became_peered":"2024-03-16T10:15:03.022251+0000","last_unstale":"2024-03-16T10:16:28.746441+0000","last_undegraded":"2024-03-16T10:16:28.746441+0000","last_fullsized":"2024-03-16T10:16:28.746441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:33.544709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.750180+0000","last_change":"2024-03-16T10:15:03.009548+0000","last_active":"2024-03-16T10:16:28.750180+0000","last_peered":"2024-03-16T10:16:28.750180+0000","last_clean":"2024-03-16T10:16:28.750180+0000","last_became_active":"2024-03-16T10:15:03.007675+0000","last_became_peered":"2024-03-16T10:15:03.007675+0000","last_unstale":"2024-03-16T10:16:28.750180+0000","last_undegraded":"2024-03-16T10:16:28.750180+0000","last_fullsized":"2024-03-16T10:16:28.750180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:08:02.561602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756541+0000","last_change":"2024-03-16T10:15:03.020146+0000","last_active":"2024-03-16T10:16:28.756541+0000","last_peered":"2024-03-16T10:16:28.756541+0000","last_clean":"2024-03-16T10:16:28.756541+0000","last_became_active":"2024-03-16T10:15:03.019823+0000","last_became_peered":"2024-03-16T10:15:03.019823+0000","last_unstale":"2024-03-16T10:16:28.756541+0000","last_undegraded":"2024-03-16T10:16:28.756541+0000","last_fullsized":"2024-03-16T10:16:28.756541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:29:52.276506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.760668+0000","last_change":"2024-03-16T10:15:03.019150+0000","last_active":"2024-03-16T10:16:28.760668+0000","last_peered":"2024-03-16T10:16:28.760668+0000","last_clean":"2024-03-16T10:16:28.760668+0000","last_became_active":"2024-03-16T10:15:03.018830+0000","last_became_peered":"2024-03-16T10:15:03.018830+0000","last_unstale":"2024-03-16T10:16:28.760668+0000","last_undegraded":"2024-03-16T10:16:28.760668+0000","last_fullsized":"2024-03-16T10:16:28.760668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:18:46.207187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762615+0000","last_change":"2024-03-16T10:15:03.033321+0000","last_active":"2024-03-16T10:16:28.762615+0000","last_peered":"2024-03-16T10:16:28.762615+0000","last_clean":"2024-03-16T10:16:28.762615+0000","last_became_active":"2024-03-16T10:15:03.032944+0000","last_became_peered":"2024-03-16T10:15:03.032944+0000","last_unstale":"2024-03-16T10:16:28.762615+0000","last_undegraded":"2024-03-16T10:16:28.762615+0000","last_fullsized":"2024-03-16T10:16:28.762615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:55:14.345488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.757536+0000","last_change":"2024-03-16T10:15:03.019897+0000","last_active":"2024-03-16T10:16:28.757536+0000","last_peered":"2024-03-16T10:16:28.757536+0000","last_clean":"2024-03-16T10:16:28.757536+0000","last_became_active":"2024-03-16T10:15:03.019555+0000","last_became_peered":"2024-03-16T10:15:03.019555+0000","last_unstale":"2024-03-16T10:16:28.757536+0000","last_undegraded":"2024-03-16T10:16:28.757536+0000","last_fullsized":"2024-03-16T10:16:28.757536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:58.307285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.758416+0000","last_change":"2024-03-16T10:15:03.017482+0000","last_active":"2024-03-16T10:16:28.758416+0000","last_peered":"2024-03-16T10:16:28.758416+0000","last_clean":"2024-03-16T10:16:28.758416+0000","last_became_active":"2024-03-16T10:15:03.016521+0000","last_became_peered":"2024-03-16T10:15:03.016521+0000","last_unstale":"2024-03-16T10:16:28.758416+0000","last_undegraded":"2024-03-16T10:16:28.758416+0000","last_fullsized":"2024-03-16T10:16:28.758416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:36:11.889111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.747191+0000","last_change":"2024-03-16T10:15:03.018625+0000","last_active":"2024-03-16T10:16:28.747191+0000","last_peered":"2024-03-16T10:16:28.747191+0000","last_clean":"2024-03-16T10:16:28.747191+0000","last_became_active":"2024-03-16T10:15:03.018313+0000","last_became_peered":"2024-03-16T10:15:03.018313+0000","last_unstale":"2024-03-16T10:16:28.747191+0000","last_undegraded":"2024-03-16T10:16:28.747191+0000","last_fullsized":"2024-03-16T10:16:28.747191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:35:58.533559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'14","reported_seq":149,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:02.357489+0000","last_change":"2024-03-16T10:15:02.016545+0000","last_active":"2024-03-16T10:17:02.357489+0000","last_peered":"2024-03-16T10:17:02.357489+0000","last_clean":"2024-03-16T10:17:02.357489+0000","last_became_active":"2024-03-16T10:15:02.012088+0000","last_became_peered":"2024-03-16T10:15:02.012088+0000","last_unstale":"2024-03-16T10:17:02.357489+0000","last_undegraded":"2024-03-16T10:17:02.357489+0000","last_fullsized":"2024-03-16T10:17:02.357489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:10:50.633539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.740676+0000","last_change":"2024-03-16T10:15:03.026565+0000","last_active":"2024-03-16T10:16:28.740676+0000","last_peered":"2024-03-16T10:16:28.740676+0000","last_clean":"2024-03-16T10:16:28.740676+0000","last_became_active":"2024-03-16T10:15:03.025386+0000","last_became_peered":"2024-03-16T10:15:03.025386+0000","last_unstale":"2024-03-16T10:16:28.740676+0000","last_undegraded":"2024-03-16T10:16:28.740676+0000","last_fullsized":"2024-03-16T10:16:28.740676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:28.831764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838352+0000","last_change":"2024-03-16T10:15:02.018097+0000","last_active":"2024-03-16T10:15:59.838352+0000","last_peered":"2024-03-16T10:15:59.838352+0000","last_clean":"2024-03-16T10:15:59.838352+0000","last_became_active":"2024-03-16T10:15:02.017777+0000","last_became_peered":"2024-03-16T10:15:02.017777+0000","last_unstale":"2024-03-16T10:15:59.838352+0000","last_undegraded":"2024-03-16T10:15:59.838352+0000","last_fullsized":"2024-03-16T10:15:59.838352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:04:58.376969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.753086+0000","last_change":"2024-03-16T10:15:03.023384+0000","last_active":"2024-03-16T10:16:28.753086+0000","last_peered":"2024-03-16T10:16:28.753086+0000","last_clean":"2024-03-16T10:16:28.753086+0000","last_became_active":"2024-03-16T10:15:03.022996+0000","last_became_peered":"2024-03-16T10:15:03.022996+0000","last_unstale":"2024-03-16T10:16:28.753086+0000","last_undegraded":"2024-03-16T10:16:28.753086+0000","last_fullsized":"2024-03-16T10:16:28.753086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:29:20.614631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834807+0000","last_change":"2024-03-16T10:15:02.014409+0000","last_active":"2024-03-16T10:15:59.834807+0000","last_peered":"2024-03-16T10:15:59.834807+0000","last_clean":"2024-03-16T10:15:59.834807+0000","last_became_active":"2024-03-16T10:15:02.014015+0000","last_became_peered":"2024-03-16T10:15:02.014015+0000","last_unstale":"2024-03-16T10:15:59.834807+0000","last_undegraded":"2024-03-16T10:15:59.834807+0000","last_fullsized":"2024-03-16T10:15:59.834807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:25:53.133466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.759810+0000","last_change":"2024-03-16T10:15:03.031328+0000","last_active":"2024-03-16T10:16:28.759810+0000","last_peered":"2024-03-16T10:16:28.759810+0000","last_clean":"2024-03-16T10:16:28.759810+0000","last_became_active":"2024-03-16T10:15:03.029787+0000","last_became_peered":"2024-03-16T10:15:03.029787+0000","last_unstale":"2024-03-16T10:16:28.759810+0000","last_undegraded":"2024-03-16T10:16:28.759810+0000","last_fullsized":"2024-03-16T10:16:28.759810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:34.321883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834594+0000","last_change":"2024-03-16T10:15:02.018032+0000","last_active":"2024-03-16T10:15:59.834594+0000","last_peered":"2024-03-16T10:15:59.834594+0000","last_clean":"2024-03-16T10:15:59.834594+0000","last_became_active":"2024-03-16T10:15:02.017677+0000","last_became_peered":"2024-03-16T10:15:02.017677+0000","last_unstale":"2024-03-16T10:15:59.834594+0000","last_undegraded":"2024-03-16T10:15:59.834594+0000","last_fullsized":"2024-03-16T10:15:59.834594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:58:07.179129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762172+0000","last_change":"2024-03-16T10:15:03.024958+0000","last_active":"2024-03-16T10:16:28.762172+0000","last_peered":"2024-03-16T10:16:28.762172+0000","last_clean":"2024-03-16T10:16:28.762172+0000","last_became_active":"2024-03-16T10:15:03.023964+0000","last_became_peered":"2024-03-16T10:15:03.023964+0000","last_unstale":"2024-03-16T10:16:28.762172+0000","last_undegraded":"2024-03-16T10:16:28.762172+0000","last_fullsized":"2024-03-16T10:16:28.762172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:55:00.953258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":146,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:04.348190+0000","last_change":"2024-03-16T10:15:02.015062+0000","last_active":"2024-03-16T10:17:04.348190+0000","last_peered":"2024-03-16T10:17:04.348190+0000","last_clean":"2024-03-16T10:17:04.348190+0000","last_became_active":"2024-03-16T10:15:02.014714+0000","last_became_peered":"2024-03-16T10:15:02.014714+0000","last_unstale":"2024-03-16T10:17:04.348190+0000","last_undegraded":"2024-03-16T10:17:04.348190+0000","last_fullsized":"2024-03-16T10:17:04.348190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:23:55.313009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.743988+0000","last_change":"2024-03-16T10:15:03.013956+0000","last_active":"2024-03-16T10:16:28.743988+0000","last_peered":"2024-03-16T10:16:28.743988+0000","last_clean":"2024-03-16T10:16:28.743988+0000","last_became_active":"2024-03-16T10:15:03.013607+0000","last_became_peered":"2024-03-16T10:15:03.013607+0000","last_unstale":"2024-03-16T10:16:28.743988+0000","last_undegraded":"2024-03-16T10:16:28.743988+0000","last_fullsized":"2024-03-16T10:16:28.743988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:40:55.445395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.841646+0000","last_change":"2024-03-16T10:15:02.015651+0000","last_active":"2024-03-16T10:15:59.841646+0000","last_peered":"2024-03-16T10:15:59.841646+0000","last_clean":"2024-03-16T10:15:59.841646+0000","last_became_active":"2024-03-16T10:15:02.015254+0000","last_became_peered":"2024-03-16T10:15:02.015254+0000","last_unstale":"2024-03-16T10:15:59.841646+0000","last_undegraded":"2024-03-16T10:15:59.841646+0000","last_fullsized":"2024-03-16T10:15:59.841646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:41:59.013606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756364+0000","last_change":"2024-03-16T10:15:03.023074+0000","last_active":"2024-03-16T10:16:28.756364+0000","last_peered":"2024-03-16T10:16:28.756364+0000","last_clean":"2024-03-16T10:16:28.756364+0000","last_became_active":"2024-03-16T10:15:03.022700+0000","last_became_peered":"2024-03-16T10:15:03.022700+0000","last_unstale":"2024-03-16T10:16:28.756364+0000","last_undegraded":"2024-03-16T10:16:28.756364+0000","last_fullsized":"2024-03-16T10:16:28.756364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:16:22.047929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835321+0000","last_change":"2024-03-16T10:15:02.014793+0000","last_active":"2024-03-16T10:15:59.835321+0000","last_peered":"2024-03-16T10:15:59.835321+0000","last_clean":"2024-03-16T10:15:59.835321+0000","last_became_active":"2024-03-16T10:15:02.014459+0000","last_became_peered":"2024-03-16T10:15:02.014459+0000","last_unstale":"2024-03-16T10:15:59.835321+0000","last_undegraded":"2024-03-16T10:15:59.835321+0000","last_fullsized":"2024-03-16T10:15:59.835321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:59:45.035045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.755311+0000","last_change":"2024-03-16T10:15:03.012713+0000","last_active":"2024-03-16T10:16:28.755311+0000","last_peered":"2024-03-16T10:16:28.755311+0000","last_clean":"2024-03-16T10:16:28.755311+0000","last_became_active":"2024-03-16T10:15:03.012407+0000","last_became_peered":"2024-03-16T10:15:03.012407+0000","last_unstale":"2024-03-16T10:16:28.755311+0000","last_undegraded":"2024-03-16T10:16:28.755311+0000","last_fullsized":"2024-03-16T10:16:28.755311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:23:14.592810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898544+0000","last_change":"2024-03-16T10:15:02.015582+0000","last_active":"2024-03-16T10:15:59.898544+0000","last_peered":"2024-03-16T10:15:59.898544+0000","last_clean":"2024-03-16T10:15:59.898544+0000","last_became_active":"2024-03-16T10:15:02.015073+0000","last_became_peered":"2024-03-16T10:15:02.015073+0000","last_unstale":"2024-03-16T10:15:59.898544+0000","last_undegraded":"2024-03-16T10:15:59.898544+0000","last_fullsized":"2024-03-16T10:15:59.898544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:22:48.212448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.750647+0000","last_change":"2024-03-16T10:15:03.016839+0000","last_active":"2024-03-16T10:16:28.750647+0000","last_peered":"2024-03-16T10:16:28.750647+0000","last_clean":"2024-03-16T10:16:28.750647+0000","last_became_active":"2024-03-16T10:15:03.016463+0000","last_became_peered":"2024-03-16T10:15:03.016463+0000","last_unstale":"2024-03-16T10:16:28.750647+0000","last_undegraded":"2024-03-16T10:16:28.750647+0000","last_fullsized":"2024-03-16T10:16:28.750647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:43:34.259049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'5","reported_seq":256,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:03.346162+0000","last_change":"2024-03-16T10:15:02.017884+0000","last_active":"2024-03-16T10:17:03.346162+0000","last_peered":"2024-03-16T10:17:03.346162+0000","last_clean":"2024-03-16T10:17:03.346162+0000","last_became_active":"2024-03-16T10:15:02.017511+0000","last_became_peered":"2024-03-16T10:15:02.017511+0000","last_unstale":"2024-03-16T10:17:03.346162+0000","last_undegraded":"2024-03-16T10:17:03.346162+0000","last_fullsized":"2024-03-16T10:17:03.346162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:31.281631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":232,"num_read_kb":232,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.754815+0000","last_change":"2024-03-16T10:15:03.032259+0000","last_active":"2024-03-16T10:16:28.754815+0000","last_peered":"2024-03-16T10:16:28.754815+0000","last_clean":"2024-03-16T10:16:28.754815+0000","last_became_active":"2024-03-16T10:15:03.031884+0000","last_became_peered":"2024-03-16T10:15:03.031884+0000","last_unstale":"2024-03-16T10:16:28.754815+0000","last_undegraded":"2024-03-16T10:16:28.754815+0000","last_fullsized":"2024-03-16T10:16:28.754815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:37:48.464051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.839326+0000","last_change":"2024-03-16T10:15:02.015372+0000","last_active":"2024-03-16T10:15:59.839326+0000","last_peered":"2024-03-16T10:15:59.839326+0000","last_clean":"2024-03-16T10:15:59.839326+0000","last_became_active":"2024-03-16T10:15:02.015022+0000","last_became_peered":"2024-03-16T10:15:02.015022+0000","last_unstale":"2024-03-16T10:15:59.839326+0000","last_undegraded":"2024-03-16T10:15:59.839326+0000","last_fullsized":"2024-03-16T10:15:59.839326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:24:42.651903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.760728+0000","last_change":"2024-03-16T10:15:03.018661+0000","last_active":"2024-03-16T10:16:28.760728+0000","last_peered":"2024-03-16T10:16:28.760728+0000","last_clean":"2024-03-16T10:16:28.760728+0000","last_became_active":"2024-03-16T10:15:03.018351+0000","last_became_peered":"2024-03-16T10:15:03.018351+0000","last_unstale":"2024-03-16T10:16:28.760728+0000","last_undegraded":"2024-03-16T10:16:28.760728+0000","last_fullsized":"2024-03-16T10:16:28.760728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:21:33.939811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.899049+0000","last_change":"2024-03-16T10:15:02.014343+0000","last_active":"2024-03-16T10:15:59.899049+0000","last_peered":"2024-03-16T10:15:59.899049+0000","last_clean":"2024-03-16T10:15:59.899049+0000","last_became_active":"2024-03-16T10:15:02.013983+0000","last_became_peered":"2024-03-16T10:15:02.013983+0000","last_unstale":"2024-03-16T10:15:59.899049+0000","last_undegraded":"2024-03-16T10:15:59.899049+0000","last_fullsized":"2024-03-16T10:15:59.899049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:35:04.768079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.761372+0000","last_change":"2024-03-16T10:15:03.031055+0000","last_active":"2024-03-16T10:16:28.761372+0000","last_peered":"2024-03-16T10:16:28.761372+0000","last_clean":"2024-03-16T10:16:28.761372+0000","last_became_active":"2024-03-16T10:15:03.029656+0000","last_became_peered":"2024-03-16T10:15:03.029656+0000","last_unstale":"2024-03-16T10:16:28.761372+0000","last_undegraded":"2024-03-16T10:16:28.761372+0000","last_fullsized":"2024-03-16T10:16:28.761372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:27:35.684774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'37","reported_seq":299,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:03.346714+0000","last_change":"2024-03-16T10:15:02.014752+0000","last_active":"2024-03-16T10:17:03.346714+0000","last_peered":"2024-03-16T10:17:03.346714+0000","last_clean":"2024-03-16T10:17:03.346714+0000","last_became_active":"2024-03-16T10:15:02.013763+0000","last_became_peered":"2024-03-16T10:15:02.013763+0000","last_unstale":"2024-03-16T10:17:03.346714+0000","last_undegraded":"2024-03-16T10:17:03.346714+0000","last_fullsized":"2024-03-16T10:17:03.346714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:12:12.759643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60580,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":243,"num_read_kb":182,"num_write":39,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762495+0000","last_change":"2024-03-16T10:15:03.018838+0000","last_active":"2024-03-16T10:16:28.762495+0000","last_peered":"2024-03-16T10:16:28.762495+0000","last_clean":"2024-03-16T10:16:28.762495+0000","last_became_active":"2024-03-16T10:15:03.017713+0000","last_became_peered":"2024-03-16T10:15:03.017713+0000","last_unstale":"2024-03-16T10:16:28.762495+0000","last_undegraded":"2024-03-16T10:16:28.762495+0000","last_fullsized":"2024-03-16T10:16:28.762495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:03:56.936729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835759+0000","last_change":"2024-03-16T10:15:02.016996+0000","last_active":"2024-03-16T10:15:59.835759+0000","last_peered":"2024-03-16T10:15:59.835759+0000","last_clean":"2024-03-16T10:15:59.835759+0000","last_became_active":"2024-03-16T10:15:02.016421+0000","last_became_peered":"2024-03-16T10:15:02.016421+0000","last_unstale":"2024-03-16T10:15:59.835759+0000","last_undegraded":"2024-03-16T10:15:59.835759+0000","last_fullsized":"2024-03-16T10:15:59.835759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:25:09.866730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.750159+0000","last_change":"2024-03-16T10:15:03.017842+0000","last_active":"2024-03-16T10:16:28.750159+0000","last_peered":"2024-03-16T10:16:28.750159+0000","last_clean":"2024-03-16T10:16:28.750159+0000","last_became_active":"2024-03-16T10:15:03.015119+0000","last_became_peered":"2024-03-16T10:15:03.015119+0000","last_unstale":"2024-03-16T10:16:28.750159+0000","last_undegraded":"2024-03-16T10:16:28.750159+0000","last_fullsized":"2024-03-16T10:16:28.750159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:14:34.474519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898795+0000","last_change":"2024-03-16T10:15:02.021407+0000","last_active":"2024-03-16T10:15:59.898795+0000","last_peered":"2024-03-16T10:15:59.898795+0000","last_clean":"2024-03-16T10:15:59.898795+0000","last_became_active":"2024-03-16T10:15:02.020884+0000","last_became_peered":"2024-03-16T10:15:02.020884+0000","last_unstale":"2024-03-16T10:15:59.898795+0000","last_undegraded":"2024-03-16T10:15:59.898795+0000","last_fullsized":"2024-03-16T10:15:59.898795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:30:24.858281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.766085+0000","last_change":"2024-03-16T10:15:03.028607+0000","last_active":"2024-03-16T10:16:28.766085+0000","last_peered":"2024-03-16T10:16:28.766085+0000","last_clean":"2024-03-16T10:16:28.766085+0000","last_became_active":"2024-03-16T10:15:03.028280+0000","last_became_peered":"2024-03-16T10:15:03.028280+0000","last_unstale":"2024-03-16T10:16:28.766085+0000","last_undegraded":"2024-03-16T10:16:28.766085+0000","last_fullsized":"2024-03-16T10:16:28.766085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:39.905921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'5","reported_seq":141,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:01.356857+0000","last_change":"2024-03-16T10:15:02.008558+0000","last_active":"2024-03-16T10:17:01.356857+0000","last_peered":"2024-03-16T10:17:01.356857+0000","last_clean":"2024-03-16T10:17:01.356857+0000","last_became_active":"2024-03-16T10:15:02.008014+0000","last_became_peered":"2024-03-16T10:15:02.008014+0000","last_unstale":"2024-03-16T10:17:01.356857+0000","last_undegraded":"2024-03-16T10:17:01.356857+0000","last_fullsized":"2024-03-16T10:17:01.356857+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:40:09.217991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.763448+0000","last_change":"2024-03-16T10:15:03.020847+0000","last_active":"2024-03-16T10:16:28.763448+0000","last_peered":"2024-03-16T10:16:28.763448+0000","last_clean":"2024-03-16T10:16:28.763448+0000","last_became_active":"2024-03-16T10:15:03.020512+0000","last_became_peered":"2024-03-16T10:15:03.020512+0000","last_unstale":"2024-03-16T10:16:28.763448+0000","last_undegraded":"2024-03-16T10:16:28.763448+0000","last_fullsized":"2024-03-16T10:16:28.763448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:21:04.436195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834281+0000","last_change":"2024-03-16T10:15:02.010690+0000","last_active":"2024-03-16T10:15:59.834281+0000","last_peered":"2024-03-16T10:15:59.834281+0000","last_clean":"2024-03-16T10:15:59.834281+0000","last_became_active":"2024-03-16T10:15:02.010228+0000","last_became_peered":"2024-03-16T10:15:02.010228+0000","last_unstale":"2024-03-16T10:15:59.834281+0000","last_undegraded":"2024-03-16T10:15:59.834281+0000","last_fullsized":"2024-03-16T10:15:59.834281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:30:07.513225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756488+0000","last_change":"2024-03-16T10:15:03.025257+0000","last_active":"2024-03-16T10:16:28.756488+0000","last_peered":"2024-03-16T10:16:28.756488+0000","last_clean":"2024-03-16T10:16:28.756488+0000","last_became_active":"2024-03-16T10:15:03.024498+0000","last_became_peered":"2024-03-16T10:15:03.024498+0000","last_unstale":"2024-03-16T10:16:28.756488+0000","last_undegraded":"2024-03-16T10:16:28.756488+0000","last_fullsized":"2024-03-16T10:16:28.756488+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:27:55.894728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834712+0000","last_change":"2024-03-16T10:15:02.014136+0000","last_active":"2024-03-16T10:15:59.834712+0000","last_peered":"2024-03-16T10:15:59.834712+0000","last_clean":"2024-03-16T10:15:59.834712+0000","last_became_active":"2024-03-16T10:15:02.013721+0000","last_became_peered":"2024-03-16T10:15:02.013721+0000","last_unstale":"2024-03-16T10:15:59.834712+0000","last_undegraded":"2024-03-16T10:15:59.834712+0000","last_fullsized":"2024-03-16T10:15:59.834712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:28:12.301450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.744259+0000","last_change":"2024-03-16T10:15:59.856566+0000","last_active":"2024-03-16T10:16:28.744259+0000","last_peered":"2024-03-16T10:16:28.744259+0000","last_clean":"2024-03-16T10:16:28.744259+0000","last_became_active":"2024-03-16T10:15:59.856152+0000","last_became_peered":"2024-03-16T10:15:59.856152+0000","last_unstale":"2024-03-16T10:16:28.744259+0000","last_undegraded":"2024-03-16T10:16:28.744259+0000","last_fullsized":"2024-03-16T10:16:28.744259+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:42:39.104928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.855367+0000","last_change":"2024-03-16T10:15:02.005754+0000","last_active":"2024-03-16T10:15:59.855367+0000","last_peered":"2024-03-16T10:15:59.855367+0000","last_clean":"2024-03-16T10:15:59.855367+0000","last_became_active":"2024-03-16T10:15:02.005344+0000","last_became_peered":"2024-03-16T10:15:02.005344+0000","last_unstale":"2024-03-16T10:15:59.855367+0000","last_undegraded":"2024-03-16T10:15:59.855367+0000","last_fullsized":"2024-03-16T10:15:59.855367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:37:37.918861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.752059+0000","last_change":"2024-03-16T10:15:03.012887+0000","last_active":"2024-03-16T10:16:28.752059+0000","last_peered":"2024-03-16T10:16:28.752059+0000","last_clean":"2024-03-16T10:16:28.752059+0000","last_became_active":"2024-03-16T10:15:03.012496+0000","last_became_peered":"2024-03-16T10:15:03.012496+0000","last_unstale":"2024-03-16T10:16:28.752059+0000","last_undegraded":"2024-03-16T10:16:28.752059+0000","last_fullsized":"2024-03-16T10:16:28.752059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:09:11.111049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":138,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:00.355736+0000","last_change":"2024-03-16T10:15:02.014334+0000","last_active":"2024-03-16T10:17:00.355736+0000","last_peered":"2024-03-16T10:17:00.355736+0000","last_clean":"2024-03-16T10:17:00.355736+0000","last_became_active":"2024-03-16T10:15:02.013832+0000","last_became_peered":"2024-03-16T10:15:02.013832+0000","last_unstale":"2024-03-16T10:17:00.355736+0000","last_undegraded":"2024-03-16T10:17:00.355736+0000","last_fullsized":"2024-03-16T10:17:00.355736+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:02:20.391594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.750566+0000","last_change":"2024-03-16T10:15:03.021276+0000","last_active":"2024-03-16T10:16:28.750566+0000","last_peered":"2024-03-16T10:16:28.750566+0000","last_clean":"2024-03-16T10:16:28.750566+0000","last_became_active":"2024-03-16T10:15:03.020927+0000","last_became_peered":"2024-03-16T10:15:03.020927+0000","last_unstale":"2024-03-16T10:16:28.750566+0000","last_undegraded":"2024-03-16T10:16:28.750566+0000","last_fullsized":"2024-03-16T10:16:28.750566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:53:54.127506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.746363+0000","last_change":"2024-03-16T10:15:03.022083+0000","last_active":"2024-03-16T10:16:28.746363+0000","last_peered":"2024-03-16T10:16:28.746363+0000","last_clean":"2024-03-16T10:16:28.746363+0000","last_became_active":"2024-03-16T10:15:03.021609+0000","last_became_peered":"2024-03-16T10:15:03.021609+0000","last_unstale":"2024-03-16T10:16:28.746363+0000","last_undegraded":"2024-03-16T10:16:28.746363+0000","last_fullsized":"2024-03-16T10:16:28.746363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:53.864402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":139,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:00.355523+0000","last_change":"2024-03-16T10:15:02.015497+0000","last_active":"2024-03-16T10:17:00.355523+0000","last_peered":"2024-03-16T10:17:00.355523+0000","last_clean":"2024-03-16T10:17:00.355523+0000","last_became_active":"2024-03-16T10:15:02.015183+0000","last_became_peered":"2024-03-16T10:15:02.015183+0000","last_unstale":"2024-03-16T10:17:00.355523+0000","last_undegraded":"2024-03-16T10:17:00.355523+0000","last_fullsized":"2024-03-16T10:17:00.355523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:44:05.913030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.755036+0000","last_change":"2024-03-16T10:15:03.020046+0000","last_active":"2024-03-16T10:16:28.755036+0000","last_peered":"2024-03-16T10:16:28.755036+0000","last_clean":"2024-03-16T10:16:28.755036+0000","last_became_active":"2024-03-16T10:15:03.019647+0000","last_became_peered":"2024-03-16T10:15:03.019647+0000","last_unstale":"2024-03-16T10:16:28.755036+0000","last_undegraded":"2024-03-16T10:16:28.755036+0000","last_fullsized":"2024-03-16T10:16:28.755036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:35:18.343495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134445+0000","last_change":"2024-03-16T10:15:02.008560+0000","last_active":"2024-03-16T10:16:00.134445+0000","last_peered":"2024-03-16T10:16:00.134445+0000","last_clean":"2024-03-16T10:16:00.134445+0000","last_became_active":"2024-03-16T10:15:02.008094+0000","last_became_peered":"2024-03-16T10:15:02.008094+0000","last_unstale":"2024-03-16T10:16:00.134445+0000","last_undegraded":"2024-03-16T10:16:00.134445+0000","last_fullsized":"2024-03-16T10:16:00.134445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:21:43.555120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":342,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838687+0000","last_change":"2024-03-16T10:13:47.211815+0000","last_active":"2024-03-16T10:15:59.838687+0000","last_peered":"2024-03-16T10:15:59.838687+0000","last_clean":"2024-03-16T10:15:59.838687+0000","last_became_active":"2024-03-16T10:13:47.211353+0000","last_became_peered":"2024-03-16T10:13:47.211353+0000","last_unstale":"2024-03-16T10:15:59.838687+0000","last_undegraded":"2024-03-16T10:15:59.838687+0000","last_fullsized":"2024-03-16T10:15:59.838687+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:10:22.965421+0000","last_clean_scrub_stamp":"2024-03-16T10:10:22.965421+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:11:20.607686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.751691+0000","last_change":"2024-03-16T10:15:03.020181+0000","last_active":"2024-03-16T10:16:28.751691+0000","last_peered":"2024-03-16T10:16:28.751691+0000","last_clean":"2024-03-16T10:16:28.751691+0000","last_became_active":"2024-03-16T10:15:03.019781+0000","last_became_peered":"2024-03-16T10:15:03.019781+0000","last_unstale":"2024-03-16T10:16:28.751691+0000","last_undegraded":"2024-03-16T10:16:28.751691+0000","last_fullsized":"2024-03-16T10:16:28.751691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:04:49.369484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.838833+0000","last_change":"2024-03-16T10:15:02.014461+0000","last_active":"2024-03-16T10:15:59.838833+0000","last_peered":"2024-03-16T10:15:59.838833+0000","last_clean":"2024-03-16T10:15:59.838833+0000","last_became_active":"2024-03-16T10:15:02.014137+0000","last_became_peered":"2024-03-16T10:15:02.014137+0000","last_unstale":"2024-03-16T10:15:59.838833+0000","last_undegraded":"2024-03-16T10:15:59.838833+0000","last_fullsized":"2024-03-16T10:15:59.838833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:36:41.830893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834705+0000","last_change":"2024-03-16T10:15:02.019441+0000","last_active":"2024-03-16T10:15:59.834705+0000","last_peered":"2024-03-16T10:15:59.834705+0000","last_clean":"2024-03-16T10:15:59.834705+0000","last_became_active":"2024-03-16T10:15:02.019091+0000","last_became_peered":"2024-03-16T10:15:02.019091+0000","last_unstale":"2024-03-16T10:15:59.834705+0000","last_undegraded":"2024-03-16T10:15:59.834705+0000","last_fullsized":"2024-03-16T10:15:59.834705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:04:51.758722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.745121+0000","last_change":"2024-03-16T10:15:03.012141+0000","last_active":"2024-03-16T10:16:28.745121+0000","last_peered":"2024-03-16T10:16:28.745121+0000","last_clean":"2024-03-16T10:16:28.745121+0000","last_became_active":"2024-03-16T10:15:03.011771+0000","last_became_peered":"2024-03-16T10:15:03.011771+0000","last_unstale":"2024-03-16T10:16:28.745121+0000","last_undegraded":"2024-03-16T10:16:28.745121+0000","last_fullsized":"2024-03-16T10:16:28.745121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:02:28.522530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'15","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:29.030048+0000","last_change":"2024-03-16T10:15:02.011898+0000","last_active":"2024-03-16T10:16:29.030048+0000","last_peered":"2024-03-16T10:16:29.030048+0000","last_clean":"2024-03-16T10:16:29.030048+0000","last_became_active":"2024-03-16T10:15:02.011578+0000","last_became_peered":"2024-03-16T10:15:02.011578+0000","last_unstale":"2024-03-16T10:16:29.030048+0000","last_undegraded":"2024-03-16T10:16:29.030048+0000","last_fullsized":"2024-03-16T10:16:29.030048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:27:35.756031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762502+0000","last_change":"2024-03-16T10:15:03.024923+0000","last_active":"2024-03-16T10:16:28.762502+0000","last_peered":"2024-03-16T10:16:28.762502+0000","last_clean":"2024-03-16T10:16:28.762502+0000","last_became_active":"2024-03-16T10:15:03.024577+0000","last_became_peered":"2024-03-16T10:15:03.024577+0000","last_unstale":"2024-03-16T10:16:28.762502+0000","last_undegraded":"2024-03-16T10:16:28.762502+0000","last_fullsized":"2024-03-16T10:16:28.762502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:29:04.650499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:00.134719+0000","last_change":"2024-03-16T10:15:02.006842+0000","last_active":"2024-03-16T10:16:00.134719+0000","last_peered":"2024-03-16T10:16:00.134719+0000","last_clean":"2024-03-16T10:16:00.134719+0000","last_became_active":"2024-03-16T10:15:02.006483+0000","last_became_peered":"2024-03-16T10:15:02.006483+0000","last_unstale":"2024-03-16T10:16:00.134719+0000","last_undegraded":"2024-03-16T10:16:00.134719+0000","last_fullsized":"2024-03-16T10:16:00.134719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:30:55.856007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.760743+0000","last_change":"2024-03-16T10:15:03.018691+0000","last_active":"2024-03-16T10:16:28.760743+0000","last_peered":"2024-03-16T10:16:28.760743+0000","last_clean":"2024-03-16T10:16:28.760743+0000","last_became_active":"2024-03-16T10:15:03.018304+0000","last_became_peered":"2024-03-16T10:15:03.018304+0000","last_unstale":"2024-03-16T10:16:28.760743+0000","last_undegraded":"2024-03-16T10:16:28.760743+0000","last_fullsized":"2024-03-16T10:16:28.760743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:14:15.446759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.841306+0000","last_change":"2024-03-16T10:15:59.841306+0000","last_active":"2024-03-16T10:15:59.841306+0000","last_peered":"2024-03-16T10:15:59.841306+0000","last_clean":"2024-03-16T10:15:59.841306+0000","last_became_active":"2024-03-16T10:15:59.838434+0000","last_became_peered":"2024-03-16T10:15:59.838434+0000","last_unstale":"2024-03-16T10:15:59.841306+0000","last_undegraded":"2024-03-16T10:15:59.841306+0000","last_fullsized":"2024-03-16T10:15:59.841306+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:14:21.702809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.719194+0000","last_change":"2024-03-16T10:15:03.017262+0000","last_active":"2024-03-16T10:16:28.719194+0000","last_peered":"2024-03-16T10:16:28.719194+0000","last_clean":"2024-03-16T10:16:28.719194+0000","last_became_active":"2024-03-16T10:15:03.016909+0000","last_became_peered":"2024-03-16T10:15:03.016909+0000","last_unstale":"2024-03-16T10:16:28.719194+0000","last_undegraded":"2024-03-16T10:16:28.719194+0000","last_fullsized":"2024-03-16T10:16:28.719194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:53:11.385236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.837856+0000","last_change":"2024-03-16T10:15:02.011325+0000","last_active":"2024-03-16T10:15:59.837856+0000","last_peered":"2024-03-16T10:15:59.837856+0000","last_clean":"2024-03-16T10:15:59.837856+0000","last_became_active":"2024-03-16T10:15:02.010999+0000","last_became_peered":"2024-03-16T10:15:02.010999+0000","last_unstale":"2024-03-16T10:15:59.837856+0000","last_undegraded":"2024-03-16T10:15:59.837856+0000","last_fullsized":"2024-03-16T10:15:59.837856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:30:08.137693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.757020+0000","last_change":"2024-03-16T10:15:03.012146+0000","last_active":"2024-03-16T10:16:28.757020+0000","last_peered":"2024-03-16T10:16:28.757020+0000","last_clean":"2024-03-16T10:16:28.757020+0000","last_became_active":"2024-03-16T10:15:03.011790+0000","last_became_peered":"2024-03-16T10:15:03.011790+0000","last_unstale":"2024-03-16T10:16:28.757020+0000","last_undegraded":"2024-03-16T10:16:28.757020+0000","last_fullsized":"2024-03-16T10:16:28.757020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:45:22.642925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.842829+0000","last_change":"2024-03-16T10:15:59.842829+0000","last_active":"2024-03-16T10:15:59.842829+0000","last_peered":"2024-03-16T10:15:59.842829+0000","last_clean":"2024-03-16T10:15:59.842829+0000","last_became_active":"2024-03-16T10:15:59.838787+0000","last_became_peered":"2024-03-16T10:15:59.838787+0000","last_unstale":"2024-03-16T10:15:59.842829+0000","last_undegraded":"2024-03-16T10:15:59.842829+0000","last_fullsized":"2024-03-16T10:15:59.842829+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T10:47:27.573657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.759857+0000","last_change":"2024-03-16T10:15:03.021094+0000","last_active":"2024-03-16T10:16:28.759857+0000","last_peered":"2024-03-16T10:16:28.759857+0000","last_clean":"2024-03-16T10:16:28.759857+0000","last_became_active":"2024-03-16T10:15:03.017640+0000","last_became_peered":"2024-03-16T10:15:03.017640+0000","last_unstale":"2024-03-16T10:16:28.759857+0000","last_undegraded":"2024-03-16T10:16:28.759857+0000","last_fullsized":"2024-03-16T10:16:28.759857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:27:02.030112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.834498+0000","last_change":"2024-03-16T10:15:02.015652+0000","last_active":"2024-03-16T10:15:59.834498+0000","last_peered":"2024-03-16T10:15:59.834498+0000","last_clean":"2024-03-16T10:15:59.834498+0000","last_became_active":"2024-03-16T10:15:02.014046+0000","last_became_peered":"2024-03-16T10:15:02.014046+0000","last_unstale":"2024-03-16T10:15:59.834498+0000","last_undegraded":"2024-03-16T10:15:59.834498+0000","last_fullsized":"2024-03-16T10:15:59.834498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:23:51.656746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.765255+0000","last_change":"2024-03-16T10:15:03.023233+0000","last_active":"2024-03-16T10:16:28.765255+0000","last_peered":"2024-03-16T10:16:28.765255+0000","last_clean":"2024-03-16T10:16:28.765255+0000","last_became_active":"2024-03-16T10:15:03.022906+0000","last_became_peered":"2024-03-16T10:15:03.022906+0000","last_unstale":"2024-03-16T10:16:28.765255+0000","last_undegraded":"2024-03-16T10:16:28.765255+0000","last_fullsized":"2024-03-16T10:16:28.765255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:31:22.943087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'7","reported_seq":144,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:17:01.344825+0000","last_change":"2024-03-16T10:15:02.010755+0000","last_active":"2024-03-16T10:17:01.344825+0000","last_peered":"2024-03-16T10:17:01.344825+0000","last_clean":"2024-03-16T10:17:01.344825+0000","last_became_active":"2024-03-16T10:15:02.010422+0000","last_became_peered":"2024-03-16T10:15:02.010422+0000","last_unstale":"2024-03-16T10:17:01.344825+0000","last_undegraded":"2024-03-16T10:17:01.344825+0000","last_fullsized":"2024-03-16T10:17:01.344825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:17.606346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.765240+0000","last_change":"2024-03-16T10:15:03.025789+0000","last_active":"2024-03-16T10:16:28.765240+0000","last_peered":"2024-03-16T10:16:28.765240+0000","last_clean":"2024-03-16T10:16:28.765240+0000","last_became_active":"2024-03-16T10:15:03.024812+0000","last_became_peered":"2024-03-16T10:15:03.024812+0000","last_unstale":"2024-03-16T10:16:28.765240+0000","last_undegraded":"2024-03-16T10:16:28.765240+0000","last_fullsized":"2024-03-16T10:16:28.765240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:46:53.041234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.898933+0000","last_change":"2024-03-16T10:15:02.020599+0000","last_active":"2024-03-16T10:15:59.898933+0000","last_peered":"2024-03-16T10:15:59.898933+0000","last_clean":"2024-03-16T10:15:59.898933+0000","last_became_active":"2024-03-16T10:15:02.020203+0000","last_became_peered":"2024-03-16T10:15:02.020203+0000","last_unstale":"2024-03-16T10:15:59.898933+0000","last_undegraded":"2024-03-16T10:15:59.898933+0000","last_fullsized":"2024-03-16T10:15:59.898933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:46:33.032310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.764526+0000","last_change":"2024-03-16T10:15:03.016223+0000","last_active":"2024-03-16T10:16:28.764526+0000","last_peered":"2024-03-16T10:16:28.764526+0000","last_clean":"2024-03-16T10:16:28.764526+0000","last_became_active":"2024-03-16T10:15:03.015857+0000","last_became_peered":"2024-03-16T10:15:03.015857+0000","last_unstale":"2024-03-16T10:16:28.764526+0000","last_undegraded":"2024-03-16T10:16:28.764526+0000","last_fullsized":"2024-03-16T10:16:28.764526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:40:10.329722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:15:59.835797+0000","last_change":"2024-03-16T10:15:02.015690+0000","last_active":"2024-03-16T10:15:59.835797+0000","last_peered":"2024-03-16T10:15:59.835797+0000","last_clean":"2024-03-16T10:15:59.835797+0000","last_became_active":"2024-03-16T10:15:02.015375+0000","last_became_peered":"2024-03-16T10:15:02.015375+0000","last_unstale":"2024-03-16T10:15:59.835797+0000","last_undegraded":"2024-03-16T10:15:59.835797+0000","last_fullsized":"2024-03-16T10:15:59.835797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:00.977818+0000","last_clean_scrub_stamp":"2024-03-16T10:15:00.977818+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:44:39.118343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762466+0000","last_change":"2024-03-16T10:15:03.012702+0000","last_active":"2024-03-16T10:16:28.762466+0000","last_peered":"2024-03-16T10:16:28.762466+0000","last_clean":"2024-03-16T10:16:28.762466+0000","last_became_active":"2024-03-16T10:15:03.012387+0000","last_became_peered":"2024-03-16T10:15:03.012387+0000","last_unstale":"2024-03-16T10:16:28.762466+0000","last_undegraded":"2024-03-16T10:16:28.762466+0000","last_fullsized":"2024-03-16T10:16:28.762466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:16.938372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756885+0000","last_change":"2024-03-16T10:15:03.015290+0000","last_active":"2024-03-16T10:16:28.756885+0000","last_peered":"2024-03-16T10:16:28.756885+0000","last_clean":"2024-03-16T10:16:28.756885+0000","last_became_active":"2024-03-16T10:15:03.013846+0000","last_became_peered":"2024-03-16T10:15:03.013846+0000","last_unstale":"2024-03-16T10:16:28.756885+0000","last_undegraded":"2024-03-16T10:16:28.756885+0000","last_fullsized":"2024-03-16T10:16:28.756885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:32:33.377793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.744383+0000","last_change":"2024-03-16T10:15:03.023733+0000","last_active":"2024-03-16T10:16:28.744383+0000","last_peered":"2024-03-16T10:16:28.744383+0000","last_clean":"2024-03-16T10:16:28.744383+0000","last_became_active":"2024-03-16T10:15:03.023409+0000","last_became_peered":"2024-03-16T10:15:03.023409+0000","last_unstale":"2024-03-16T10:16:28.744383+0000","last_undegraded":"2024-03-16T10:16:28.744383+0000","last_fullsized":"2024-03-16T10:16:28.744383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:44.748152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756641+0000","last_change":"2024-03-16T10:15:59.856644+0000","last_active":"2024-03-16T10:16:28.756641+0000","last_peered":"2024-03-16T10:16:28.756641+0000","last_clean":"2024-03-16T10:16:28.756641+0000","last_became_active":"2024-03-16T10:15:59.856243+0000","last_became_peered":"2024-03-16T10:15:59.856243+0000","last_unstale":"2024-03-16T10:16:28.756641+0000","last_undegraded":"2024-03-16T10:16:28.756641+0000","last_fullsized":"2024-03-16T10:16:28.756641+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:03:22.831073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.762122+0000","last_change":"2024-03-16T10:15:03.029277+0000","last_active":"2024-03-16T10:16:28.762122+0000","last_peered":"2024-03-16T10:16:28.762122+0000","last_clean":"2024-03-16T10:16:28.762122+0000","last_became_active":"2024-03-16T10:15:03.028960+0000","last_became_peered":"2024-03-16T10:15:03.028960+0000","last_unstale":"2024-03-16T10:16:28.762122+0000","last_undegraded":"2024-03-16T10:16:28.762122+0000","last_fullsized":"2024-03-16T10:16:28.762122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:50:48.362914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756920+0000","last_change":"2024-03-16T10:15:59.856671+0000","last_active":"2024-03-16T10:16:28.756920+0000","last_peered":"2024-03-16T10:16:28.756920+0000","last_clean":"2024-03-16T10:16:28.756920+0000","last_became_active":"2024-03-16T10:15:59.856287+0000","last_became_peered":"2024-03-16T10:15:59.856287+0000","last_unstale":"2024-03-16T10:16:28.756920+0000","last_undegraded":"2024-03-16T10:16:28.756920+0000","last_fullsized":"2024-03-16T10:16:28.756920+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:15:58.409554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.756231+0000","last_change":"2024-03-16T10:15:03.017499+0000","last_active":"2024-03-16T10:16:28.756231+0000","last_peered":"2024-03-16T10:16:28.756231+0000","last_clean":"2024-03-16T10:16:28.756231+0000","last_became_active":"2024-03-16T10:15:03.017085+0000","last_became_peered":"2024-03-16T10:15:03.017085+0000","last_unstale":"2024-03-16T10:16:28.756231+0000","last_undegraded":"2024-03-16T10:16:28.756231+0000","last_fullsized":"2024-03-16T10:16:28.756231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:57:09.592420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.752712+0000","last_change":"2024-03-16T10:15:03.018943+0000","last_active":"2024-03-16T10:16:28.752712+0000","last_peered":"2024-03-16T10:16:28.752712+0000","last_clean":"2024-03-16T10:16:28.752712+0000","last_became_active":"2024-03-16T10:15:03.017519+0000","last_became_peered":"2024-03-16T10:15:03.017519+0000","last_unstale":"2024-03-16T10:16:28.752712+0000","last_undegraded":"2024-03-16T10:16:28.752712+0000","last_fullsized":"2024-03-16T10:16:28.752712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:29.026988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.754110+0000","last_change":"2024-03-16T10:15:03.024697+0000","last_active":"2024-03-16T10:16:28.754110+0000","last_peered":"2024-03-16T10:16:28.754110+0000","last_clean":"2024-03-16T10:16:28.754110+0000","last_became_active":"2024-03-16T10:15:03.024305+0000","last_became_peered":"2024-03-16T10:15:03.024305+0000","last_unstale":"2024-03-16T10:16:28.754110+0000","last_undegraded":"2024-03-16T10:16:28.754110+0000","last_fullsized":"2024-03-16T10:16:28.754110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:07:35.160746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.755394+0000","last_change":"2024-03-16T10:15:03.017504+0000","last_active":"2024-03-16T10:16:28.755394+0000","last_peered":"2024-03-16T10:16:28.755394+0000","last_clean":"2024-03-16T10:16:28.755394+0000","last_became_active":"2024-03-16T10:15:03.017100+0000","last_became_peered":"2024-03-16T10:15:03.017100+0000","last_unstale":"2024-03-16T10:16:28.755394+0000","last_undegraded":"2024-03-16T10:16:28.755394+0000","last_fullsized":"2024-03-16T10:16:28.755394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:23:17.074905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.754759+0000","last_change":"2024-03-16T10:15:03.024684+0000","last_active":"2024-03-16T10:16:28.754759+0000","last_peered":"2024-03-16T10:16:28.754759+0000","last_clean":"2024-03-16T10:16:28.754759+0000","last_became_active":"2024-03-16T10:15:03.024282+0000","last_became_peered":"2024-03-16T10:15:03.024282+0000","last_unstale":"2024-03-16T10:16:28.754759+0000","last_undegraded":"2024-03-16T10:16:28.754759+0000","last_fullsized":"2024-03-16T10:16:28.754759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:18:29.595162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.758529+0000","last_change":"2024-03-16T10:15:03.025337+0000","last_active":"2024-03-16T10:16:28.758529+0000","last_peered":"2024-03-16T10:16:28.758529+0000","last_clean":"2024-03-16T10:16:28.758529+0000","last_became_active":"2024-03-16T10:15:03.024938+0000","last_became_peered":"2024-03-16T10:15:03.024938+0000","last_unstale":"2024-03-16T10:16:28.758529+0000","last_undegraded":"2024-03-16T10:16:28.758529+0000","last_fullsized":"2024-03-16T10:16:28.758529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:13:15.892354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.765584+0000","last_change":"2024-03-16T10:15:03.015468+0000","last_active":"2024-03-16T10:16:28.765584+0000","last_peered":"2024-03-16T10:16:28.765584+0000","last_clean":"2024-03-16T10:16:28.765584+0000","last_became_active":"2024-03-16T10:15:03.015097+0000","last_became_peered":"2024-03-16T10:15:03.015097+0000","last_unstale":"2024-03-16T10:16:28.765584+0000","last_undegraded":"2024-03-16T10:16:28.765584+0000","last_fullsized":"2024-03-16T10:16:28.765584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:18:46.114346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T10:16:28.761104+0000","last_change":"2024-03-16T10:15:03.020690+0000","last_active":"2024-03-16T10:16:28.761104+0000","last_peered":"2024-03-16T10:16:28.761104+0000","last_clean":"2024-03-16T10:16:28.761104+0000","last_became_active":"2024-03-16T10:15:03.020365+0000","last_became_peered":"2024-03-16T10:15:03.020365+0000","last_unstale":"2024-03-16T10:16:28.761104+0000","last_undegraded":"2024-03-16T10:16:28.761104+0000","last_fullsized":"2024-03-16T10:16:28.761104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T10:15:01.986178+0000","last_clean_scrub_stamp":"2024-03-16T10:15:01.986178+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:25.671533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280280,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67925,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1197,"num_read_kb":675,"num_write":253,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":209456,"data_compressed":5583,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":457189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":1108266,"data_compressed":37511,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":456587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":456965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":1049092,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":456304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":518105,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":458175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":518549,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":456129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":1049161,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34010,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35650,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":68795,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63084,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":62316,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T10:17:08.331 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:17:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.470+0000 7f4628387640 1 -- 172.21.15.77:0/2034027841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4620100700 msgr2=0x7f4620100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.470+0000 7f4628387640 1 --2- 172.21.15.77:0/2034027841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4620100700 0x7f4620100b20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f461c009900 tx=0x7f461c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.470+0000 7f4628387640 1 -- 172.21.15.77:0/2034027841 shutdown_connections 2024-03-16T10:17:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.470+0000 7f4628387640 1 --2- 172.21.15.77:0/2034027841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46201022b0 0x7f4620108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.470+0000 7f4628387640 1 --2- 172.21.15.77:0/2034027841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46201018f0 0x7f4620101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.470+0000 7f4628387640 1 --2- 172.21.15.77:0/2034027841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4620100700 0x7f4620100b20 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.470+0000 7f4628387640 1 -- 172.21.15.77:0/2034027841 >> 172.21.15.77:0/2034027841 conn(0x7f46200fbc70 msgr2=0x7f46200fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.471+0000 7f4628387640 1 -- 172.21.15.77:0/2034027841 shutdown_connections 2024-03-16T10:17:08.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.471+0000 7f4628387640 1 -- 172.21.15.77:0/2034027841 wait complete. 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.471+0000 7f4628387640 1 Processor -- start 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.471+0000 7f4628387640 1 -- start start 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.471+0000 7f4628387640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4620100700 0x7f46200694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.471+0000 7f4628387640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46201018f0 0x7f462019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.471+0000 7f4628387640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46201022b0 0x7f462019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4628387640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f462010c3a0 con 0x7f46201022b0 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4628387640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f462010c220 con 0x7f46201018f0 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4628387640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f462010c520 con 0x7f4620100700 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4626924640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46201022b0 0x7f462019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4626924640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46201022b0 0x7f462019eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56242/0 (socket says 172.21.15.77:56242) 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4626924640 1 -- 172.21.15.77:0/3443435304 learned_addr learned my addr 172.21.15.77:0/3443435304 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4625922640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46201018f0 0x7f462019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4626123640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4620100700 0x7f46200694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:08.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4626924640 1 -- 172.21.15.77:0/3443435304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4620100700 msgr2=0x7f46200694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4626924640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4620100700 0x7f46200694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4626924640 1 -- 172.21.15.77:0/3443435304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46201018f0 msgr2=0x7f462019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4626924640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46201018f0 0x7f462019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4626924640 1 -- 172.21.15.77:0/3443435304 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462019f7c0 con 0x7f46201022b0 2024-03-16T10:17:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4625922640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46201018f0 0x7f462019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4626924640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46201022b0 0x7f462019eeb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f461800efd0 tx=0x7f461800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f46177fe640 1 -- 172.21.15.77:0/3443435304 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618003b70 con 0x7f46201022b0 2024-03-16T10:17:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f46177fe640 1 -- 172.21.15.77:0/3443435304 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4618003d10 con 0x7f46201022b0 2024-03-16T10:17:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f46177fe640 1 -- 172.21.15.77:0/3443435304 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46180109f0 con 0x7f46201022b0 2024-03-16T10:17:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4628387640 1 -- 172.21.15.77:0/3443435304 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f462019fa50 con 0x7f46201022b0 2024-03-16T10:17:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.472+0000 7f4628387640 1 -- 172.21.15.77:0/3443435304 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f462019ffa0 con 0x7f46201022b0 2024-03-16T10:17:08.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.473+0000 7f4628387640 1 -- 172.21.15.77:0/3443435304 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4620107a20 con 0x7f46201022b0 2024-03-16T10:17:08.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.474+0000 7f46177fe640 1 -- 172.21.15.77:0/3443435304 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f4618010b90 con 0x7f46201022b0 2024-03-16T10:17:08.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.475+0000 7f46177fe640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f460c0a2910 0x7f460c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:08.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.475+0000 7f4626123640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f460c0a2910 0x7f460c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:08.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.475+0000 7f46177fe640 1 -- 172.21.15.77:0/3443435304 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f46180cf2e0 con 0x7f46201022b0 2024-03-16T10:17:08.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.475+0000 7f4626123640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f460c0a2910 0x7f460c0a4dd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f461c02c170 tx=0x7f461c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:08.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.479+0000 7f46177fe640 1 -- 172.21.15.77:0/3443435304 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f461809c2b0 con 0x7f46201022b0 2024-03-16T10:17:08.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.659+0000 7f4628387640 1 -- 172.21.15.77:0/3443435304 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f462002d400 con 0x7f46201022b0 2024-03-16T10:17:08.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.660+0000 7f46177fe640 1 -- 172.21.15.77:0/3443435304 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22496 (secure 0 0 0) 0x7f461809bbb0 con 0x7f46201022b0 2024-03-16T10:17:08.661 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:17:08.661 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":87,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:59.826259+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:17:08.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.662+0000 7f4628387640 1 -- 172.21.15.77:0/3443435304 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f460c0a2910 msgr2=0x7f460c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.662+0000 7f4628387640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f460c0a2910 0x7f460c0a4dd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f461c02c170 tx=0x7f461c02df90 comp rx=0 tx=0).stop 2024-03-16T10:17:08.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.662+0000 7f4628387640 1 -- 172.21.15.77:0/3443435304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46201022b0 msgr2=0x7f462019eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.662+0000 7f4628387640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46201022b0 0x7f462019eeb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f461800efd0 tx=0x7f461800c6a0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.662+0000 7f4628387640 1 -- 172.21.15.77:0/3443435304 shutdown_connections 2024-03-16T10:17:08.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.662+0000 7f4628387640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46201022b0 0x7f462019eeb0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.662+0000 7f4628387640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46201018f0 0x7f462019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.662+0000 7f4628387640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f460c0a2910 0x7f460c0a4dd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.662+0000 7f4628387640 1 --2- 172.21.15.77:0/3443435304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4620100700 0x7f46200694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.662+0000 7f4628387640 1 -- 172.21.15.77:0/3443435304 >> 172.21.15.77:0/3443435304 conn(0x7f46200fbc70 msgr2=0x7f4620104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:08.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.662+0000 7f4628387640 1 -- 172.21.15.77:0/3443435304 shutdown_connections 2024-03-16T10:17:08.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.662+0000 7f4628387640 1 -- 172.21.15.77:0/3443435304 wait complete. 2024-03-16T10:17:08.674 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:10:22.965004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T10:15:00.264039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T10:15:01.536067+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T10:17:08.675 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T10:17:08.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.780+0000 7f747acdc640 1 -- 172.21.15.77:0/2909344313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74741022e0 msgr2=0x7f74741046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.780+0000 7f747acdc640 1 --2- 172.21.15.77:0/2909344313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74741022e0 0x7f74741046d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7468009860 tx=0x7f746802bb70 comp rx=0 tx=0).stop 2024-03-16T10:17:08.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.780+0000 7f747acdc640 1 -- 172.21.15.77:0/2909344313 shutdown_connections 2024-03-16T10:17:08.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.780+0000 7f747acdc640 1 --2- 172.21.15.77:0/2909344313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7474107540 0x7f7474109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.780+0000 7f747acdc640 1 --2- 172.21.15.77:0/2909344313 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7474104c10 0x7f7474107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.780+0000 7f747acdc640 1 --2- 172.21.15.77:0/2909344313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74741022e0 0x7f74741046d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.780+0000 7f747acdc640 1 -- 172.21.15.77:0/2909344313 >> 172.21.15.77:0/2909344313 conn(0x7f74740fbc70 msgr2=0x7f74740fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:08.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.780+0000 7f747acdc640 1 -- 172.21.15.77:0/2909344313 shutdown_connections 2024-03-16T10:17:08.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.780+0000 7f747acdc640 1 -- 172.21.15.77:0/2909344313 wait complete. 2024-03-16T10:17:08.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f747acdc640 1 Processor -- start 2024-03-16T10:17:08.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f747acdc640 1 -- start start 2024-03-16T10:17:08.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f747acdc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741022e0 0x7f7474197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:08.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f747acdc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7474104c10 0x7f7474198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:08.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f747acdc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7474107540 0x7f747419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:08.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f747acdc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f747410d480 con 0x7f7474107540 2024-03-16T10:17:08.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f747acdc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f747410d300 con 0x7f74741022e0 2024-03-16T10:17:08.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f747acdc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f747410d600 con 0x7f7474104c10 2024-03-16T10:17:08.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f7479279640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7474107540 0x7f747419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:08.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f7479279640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7474107540 0x7f747419f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56250/0 (socket says 172.21.15.77:56250) 2024-03-16T10:17:08.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f7478a78640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741022e0 0x7f7474197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:08.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f7479279640 1 -- 172.21.15.77:0/3675236468 learned_addr learned my addr 172.21.15.77:0/3675236468 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:08.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f7473fff640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7474104c10 0x7f7474198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:08.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f7479279640 1 -- 172.21.15.77:0/3675236468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7474104c10 msgr2=0x7f7474198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f7479279640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7474104c10 0x7f7474198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f7479279640 1 -- 172.21.15.77:0/3675236468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741022e0 msgr2=0x7f7474197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.781+0000 7f7479279640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741022e0 0x7f7474197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.782+0000 7f7479279640 1 -- 172.21.15.77:0/3675236468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747419f770 con 0x7f7474107540 2024-03-16T10:17:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.782+0000 7f7479279640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7474107540 0x7f747419f100 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f746c009fd0 tx=0x7f746c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.782+0000 7f7471ffb640 1 -- 172.21.15.77:0/3675236468 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c004b20 con 0x7f7474107540 2024-03-16T10:17:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.782+0000 7f7471ffb640 1 -- 172.21.15.77:0/3675236468 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f746c003e50 con 0x7f7474107540 2024-03-16T10:17:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.782+0000 7f747acdc640 1 -- 172.21.15.77:0/3675236468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f747419fa60 con 0x7f7474107540 2024-03-16T10:17:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.782+0000 7f7471ffb640 1 -- 172.21.15.77:0/3675236468 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c00b630 con 0x7f7474107540 2024-03-16T10:17:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.782+0000 7f747acdc640 1 -- 172.21.15.77:0/3675236468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f747419ffb0 con 0x7f7474107540 2024-03-16T10:17:08.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.783+0000 7f747acdc640 1 -- 172.21.15.77:0/3675236468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74740fd760 con 0x7f7474107540 2024-03-16T10:17:08.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.784+0000 7f7471ffb640 1 -- 172.21.15.77:0/3675236468 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f746c01d020 con 0x7f7474107540 2024-03-16T10:17:08.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.784+0000 7f7471ffb640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f74640a2890 0x7f74640a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:08.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.784+0000 7f7478a78640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f74640a2890 0x7f74640a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:08.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.784+0000 7f7471ffb640 1 -- 172.21.15.77:0/3675236468 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f746c0cea70 con 0x7f7474107540 2024-03-16T10:17:08.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.785+0000 7f7478a78640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f74640a2890 0x7f74640a4d50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f746802c180 tx=0x7f7468034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.789+0000 7f7471ffb640 1 -- 172.21.15.77:0/3675236468 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f746c09baa0 con 0x7f7474107540 2024-03-16T10:17:08.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.969+0000 7f747acdc640 1 -- 172.21.15.77:0/3675236468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f747402d3b0 con 0x7f7474107540 2024-03-16T10:17:08.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.969+0000 7f7471ffb640 1 -- 172.21.15.77:0/3675236468 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f746c09b3a0 con 0x7f7474107540 2024-03-16T10:17:08.970 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T10:17:08.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.971+0000 7f747acdc640 1 -- 172.21.15.77:0/3675236468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f74640a2890 msgr2=0x7f74640a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.971+0000 7f747acdc640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f74640a2890 0x7f74640a4d50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f746802c180 tx=0x7f7468034ad0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.971+0000 7f747acdc640 1 -- 172.21.15.77:0/3675236468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7474107540 msgr2=0x7f747419f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:08.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.971+0000 7f747acdc640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7474107540 0x7f747419f100 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f746c009fd0 tx=0x7f746c00c800 comp rx=0 tx=0).stop 2024-03-16T10:17:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.971+0000 7f747acdc640 1 -- 172.21.15.77:0/3675236468 shutdown_connections 2024-03-16T10:17:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.971+0000 7f747acdc640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7474107540 0x7f747419f100 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.971+0000 7f747acdc640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f74640a2890 0x7f74640a4d50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.971+0000 7f747acdc640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7474104c10 0x7f7474198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.971+0000 7f747acdc640 1 --2- 172.21.15.77:0/3675236468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741022e0 0x7f7474197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.971+0000 7f747acdc640 1 -- 172.21.15.77:0/3675236468 >> 172.21.15.77:0/3675236468 conn(0x7f74740fbc70 msgr2=0x7f7474102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.971+0000 7f747acdc640 1 -- 172.21.15.77:0/3675236468 shutdown_connections 2024-03-16T10:17:08.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:08.971+0000 7f747acdc640 1 -- 172.21.15.77:0/3675236468 wait complete. 2024-03-16T10:17:08.982 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:08 smithi077 ceph-mon[32400]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:17:08.982 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:08 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3443435304' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:17:08.983 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T10:17:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.088+0000 7f591472f640 1 -- 172.21.15.77:0/827436710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590c102a00 msgr2=0x7f590c102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.088+0000 7f591472f640 1 --2- 172.21.15.77:0/827436710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590c102a00 0x7f590c102e80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5900009900 tx=0x7f590002bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 -- 172.21.15.77:0/827436710 shutdown_connections 2024-03-16T10:17:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 --2- 172.21.15.77:0/827436710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f590c1033c0 0x7f590c109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 --2- 172.21.15.77:0/827436710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590c102a00 0x7f590c102e80 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 --2- 172.21.15.77:0/827436710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f590c101810 0x7f590c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 -- 172.21.15.77:0/827436710 >> 172.21.15.77:0/827436710 conn(0x7f590c0fcd80 msgr2=0x7f590c0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 -- 172.21.15.77:0/827436710 shutdown_connections 2024-03-16T10:17:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 -- 172.21.15.77:0/827436710 wait complete. 2024-03-16T10:17:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 Processor -- start 2024-03-16T10:17:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 -- start start 2024-03-16T10:17:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f590c101810 0x7f590c197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590c102a00 0x7f590c198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f590c1033c0 0x7f590c19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f590c10d490 con 0x7f590c102a00 2024-03-16T10:17:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f590c10d310 con 0x7f590c101810 2024-03-16T10:17:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f591472f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f590c10d610 con 0x7f590c1033c0 2024-03-16T10:17:09.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f5911cca640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590c102a00 0x7f590c198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f5911cca640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590c102a00 0x7f590c198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56276/0 (socket says 172.21.15.77:56276) 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f5911cca640 1 -- 172.21.15.77:0/1264293005 learned_addr learned my addr 172.21.15.77:0/1264293005 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f5912ccc640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f590c1033c0 0x7f590c19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.089+0000 7f59124cb640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f590c101810 0x7f590c197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.090+0000 7f5911cca640 1 -- 172.21.15.77:0/1264293005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f590c1033c0 msgr2=0x7f590c19f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.090+0000 7f5911cca640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f590c1033c0 0x7f590c19f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.090+0000 7f5911cca640 1 -- 172.21.15.77:0/1264293005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f590c101810 msgr2=0x7f590c197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.090+0000 7f5911cca640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f590c101810 0x7f590c197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.090+0000 7f5911cca640 1 -- 172.21.15.77:0/1264293005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590c19f750 con 0x7f590c102a00 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.090+0000 7f5911cca640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590c102a00 0x7f590c198070 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5900009230 tx=0x7f590002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.090+0000 7f590b7fe640 1 -- 172.21.15.77:0/1264293005 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900041070 con 0x7f590c102a00 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.090+0000 7f590b7fe640 1 -- 172.21.15.77:0/1264293005 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5900004450 con 0x7f590c102a00 2024-03-16T10:17:09.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.090+0000 7f591472f640 1 -- 172.21.15.77:0/1264293005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f590c19f9e0 con 0x7f590c102a00 2024-03-16T10:17:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.090+0000 7f590b7fe640 1 -- 172.21.15.77:0/1264293005 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900003e20 con 0x7f590c102a00 2024-03-16T10:17:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.090+0000 7f591472f640 1 -- 172.21.15.77:0/1264293005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f590c19fed0 con 0x7f590c102a00 2024-03-16T10:17:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.091+0000 7f591472f640 1 -- 172.21.15.77:0/1264293005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f590c108c20 con 0x7f590c102a00 2024-03-16T10:17:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.093+0000 7f590b7fe640 1 -- 172.21.15.77:0/1264293005 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f5900044070 con 0x7f590c102a00 2024-03-16T10:17:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.093+0000 7f590b7fe640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f58f40a2890 0x7f58f40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.094+0000 7f59124cb640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f58f40a2890 0x7f58f40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.094+0000 7f590b7fe640 1 -- 172.21.15.77:0/1264293005 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f59000f08f0 con 0x7f590c102a00 2024-03-16T10:17:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.094+0000 7f59124cb640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f58f40a2890 0x7f58f40a4d50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f590c102860 tx=0x7f5904005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.097+0000 7f590b7fe640 1 -- 172.21.15.77:0/1264293005 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59000bd3e0 con 0x7f590c102a00 2024-03-16T10:17:09.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:08 smithi192 ceph-mon[36354]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:17:09.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:08 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3443435304' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:17:09.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:08 smithi193 ceph-mon[36311]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:17:09.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:08 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3443435304' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:17:09.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.276+0000 7f591472f640 1 -- 172.21.15.77:0/1264293005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f590c02d400 con 0x7f590c102a00 2024-03-16T10:17:09.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.276+0000 7f590b7fe640 1 -- 172.21.15.77:0/1264293005 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f59000c1480 con 0x7f590c102a00 2024-03-16T10:17:09.277 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T10:17:09.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.279+0000 7f591472f640 1 -- 172.21.15.77:0/1264293005 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f58f40a2890 msgr2=0x7f58f40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.279+0000 7f591472f640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f58f40a2890 0x7f58f40a4d50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f590c102860 tx=0x7f5904005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.279+0000 7f591472f640 1 -- 172.21.15.77:0/1264293005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590c102a00 msgr2=0x7f590c198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.279+0000 7f591472f640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590c102a00 0x7f590c198070 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5900009230 tx=0x7f590002c650 comp rx=0 tx=0).stop 2024-03-16T10:17:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.279+0000 7f591472f640 1 -- 172.21.15.77:0/1264293005 shutdown_connections 2024-03-16T10:17:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.279+0000 7f591472f640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f58f40a2890 0x7f58f40a4d50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.279+0000 7f591472f640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f590c1033c0 0x7f590c19f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.279+0000 7f591472f640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f590c102a00 0x7f590c198070 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.279+0000 7f591472f640 1 --2- 172.21.15.77:0/1264293005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f590c101810 0x7f590c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.279+0000 7f591472f640 1 -- 172.21.15.77:0/1264293005 >> 172.21.15.77:0/1264293005 conn(0x7f590c0fcd80 msgr2=0x7f590c105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.279+0000 7f591472f640 1 -- 172.21.15.77:0/1264293005 shutdown_connections 2024-03-16T10:17:09.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.279+0000 7f591472f640 1 -- 172.21.15.77:0/1264293005 wait complete. 2024-03-16T10:17:09.294 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T10:17:09.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.400+0000 7fd6d696f640 1 -- 172.21.15.77:0/2186181427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6d0107580 msgr2=0x7fd6d0109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.400+0000 7fd6d696f640 1 --2- 172.21.15.77:0/2186181427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6d0107580 0x7fd6d0109970 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009ca0 tx=0x7fd6c802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.400+0000 7fd6d696f640 1 -- 172.21.15.77:0/2186181427 shutdown_connections 2024-03-16T10:17:09.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.400+0000 7fd6d696f640 1 --2- 172.21.15.77:0/2186181427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6d0107580 0x7fd6d0109970 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.400+0000 7fd6d696f640 1 --2- 172.21.15.77:0/2186181427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6d0104c50 0x7fd6d0107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.400+0000 7fd6d696f640 1 --2- 172.21.15.77:0/2186181427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6d0102320 0x7fd6d0104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.400+0000 7fd6d696f640 1 -- 172.21.15.77:0/2186181427 >> 172.21.15.77:0/2186181427 conn(0x7fd6d00fbc90 msgr2=0x7fd6d00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.400+0000 7fd6d696f640 1 -- 172.21.15.77:0/2186181427 shutdown_connections 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.400+0000 7fd6d696f640 1 -- 172.21.15.77:0/2186181427 wait complete. 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6d696f640 1 Processor -- start 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6d696f640 1 -- start start 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6d696f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6d0102320 0x7fd6d0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6d696f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6d0104c50 0x7fd6d0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6d696f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6d0107580 0x7fd6d019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6d696f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d010d4a0 con 0x7fd6d0102320 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6d696f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d010d320 con 0x7fd6d0104c50 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6d696f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d010d620 con 0x7fd6d0107580 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6cffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6d0102320 0x7fd6d0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:09.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6cffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6d0102320 0x7fd6d0197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56292/0 (socket says 172.21.15.77:56292) 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6cffff640 1 -- 172.21.15.77:0/2177380634 learned_addr learned my addr 172.21.15.77:0/2177380634 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6d4f0c640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6d0107580 0x7fd6d019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6cf7fe640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6d0104c50 0x7fd6d0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6cffff640 1 -- 172.21.15.77:0/2177380634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6d0107580 msgr2=0x7fd6d019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6cffff640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6d0107580 0x7fd6d019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6cffff640 1 -- 172.21.15.77:0/2177380634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6d0104c50 msgr2=0x7fd6d0198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6cffff640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6d0104c50 0x7fd6d0198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.401+0000 7fd6cffff640 1 -- 172.21.15.77:0/2177380634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d019f710 con 0x7fd6d0102320 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.402+0000 7fd6cffff640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6d0102320 0x7fd6d0197af0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000ea60 tx=0x7fd6c000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.402+0000 7fd6cd7fa640 1 -- 172.21.15.77:0/2177380634 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c000cde0 con 0x7fd6d0102320 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.402+0000 7fd6cd7fa640 1 -- 172.21.15.77:0/2177380634 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c00035f0 con 0x7fd6d0102320 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.402+0000 7fd6d696f640 1 -- 172.21.15.77:0/2177380634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d019fa00 con 0x7fd6d0102320 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.402+0000 7fd6cd7fa640 1 -- 172.21.15.77:0/2177380634 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0010670 con 0x7fd6d0102320 2024-03-16T10:17:09.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.402+0000 7fd6d696f640 1 -- 172.21.15.77:0/2177380634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d019ff50 con 0x7fd6d0102320 2024-03-16T10:17:09.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.403+0000 7fd6d696f640 1 -- 172.21.15.77:0/2177380634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d00fd780 con 0x7fd6d0102320 2024-03-16T10:17:09.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.404+0000 7fd6cd7fa640 1 -- 172.21.15.77:0/2177380634 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fd6c00037a0 con 0x7fd6d0102320 2024-03-16T10:17:09.406 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.404+0000 7fd6cd7fa640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd6b80a2890 0x7fd6b80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:09.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.404+0000 7fd6cf7fe640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd6b80a2890 0x7fd6b80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:09.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.404+0000 7fd6cd7fa640 1 -- 172.21.15.77:0/2177380634 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd6c00cedf0 con 0x7fd6d0102320 2024-03-16T10:17:09.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.405+0000 7fd6cf7fe640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd6b80a2890 0x7fd6b80a4d50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0199000 tx=0x7fd6c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:09.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.409+0000 7fd6cd7fa640 1 -- 172.21.15.77:0/2177380634 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd6c009be20 con 0x7fd6d0102320 2024-03-16T10:17:09.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.584+0000 7fd6d696f640 1 -- 172.21.15.77:0/2177380634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd6d00623e0 con 0x7fd6d0102320 2024-03-16T10:17:09.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.585+0000 7fd6cd7fa640 1 -- 172.21.15.77:0/2177380634 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fd6c009b720 con 0x7fd6d0102320 2024-03-16T10:17:09.586 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T10:17:09.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.587+0000 7fd6d696f640 1 -- 172.21.15.77:0/2177380634 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd6b80a2890 msgr2=0x7fd6b80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.587+0000 7fd6d696f640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd6b80a2890 0x7fd6b80a4d50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0199000 tx=0x7fd6c4005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:09.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.587+0000 7fd6d696f640 1 -- 172.21.15.77:0/2177380634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6d0102320 msgr2=0x7fd6d0197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.587+0000 7fd6d696f640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6d0102320 0x7fd6d0197af0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000ea60 tx=0x7fd6c000ef20 comp rx=0 tx=0).stop 2024-03-16T10:17:09.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.587+0000 7fd6d696f640 1 -- 172.21.15.77:0/2177380634 shutdown_connections 2024-03-16T10:17:09.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.587+0000 7fd6d696f640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd6b80a2890 0x7fd6b80a4d50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.587+0000 7fd6d696f640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6d0107580 0x7fd6d019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.587+0000 7fd6d696f640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6d0104c50 0x7fd6d0198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.587+0000 7fd6d696f640 1 --2- 172.21.15.77:0/2177380634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6d0102320 0x7fd6d0197af0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.587+0000 7fd6d696f640 1 -- 172.21.15.77:0/2177380634 >> 172.21.15.77:0/2177380634 conn(0x7fd6d00fbc90 msgr2=0x7fd6d0102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:09.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.587+0000 7fd6d696f640 1 -- 172.21.15.77:0/2177380634 shutdown_connections 2024-03-16T10:17:09.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.587+0000 7fd6d696f640 1 -- 172.21.15.77:0/2177380634 wait complete. 2024-03-16T10:17:09.599 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:17:09.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.700+0000 7fc84c861640 1 -- 172.21.15.77:0/4003734700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc848101970 msgr2=0x7fc848101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.700+0000 7fc84c861640 1 --2- 172.21.15.77:0/4003734700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc848101970 0x7fc848101df0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc838009900 tx=0x7fc83802bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.701+0000 7fc84c861640 1 -- 172.21.15.77:0/4003734700 shutdown_connections 2024-03-16T10:17:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.701+0000 7fc84c861640 1 --2- 172.21.15.77:0/4003734700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc848102330 0x7fc848108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.701+0000 7fc84c861640 1 --2- 172.21.15.77:0/4003734700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc848101970 0x7fc848101df0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.701+0000 7fc84c861640 1 --2- 172.21.15.77:0/4003734700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc848100780 0x7fc848100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.701+0000 7fc84c861640 1 -- 172.21.15.77:0/4003734700 >> 172.21.15.77:0/4003734700 conn(0x7fc8480fbd10 msgr2=0x7fc8480fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.701+0000 7fc84c861640 1 -- 172.21.15.77:0/4003734700 shutdown_connections 2024-03-16T10:17:09.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.701+0000 7fc84c861640 1 -- 172.21.15.77:0/4003734700 wait complete. 2024-03-16T10:17:09.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.701+0000 7fc84c861640 1 Processor -- start 2024-03-16T10:17:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84c861640 1 -- start start 2024-03-16T10:17:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84c861640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc848100780 0x7fc8481969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84c861640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc848101970 0x7fc848196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84c861640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc848102330 0x7fc84819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84c861640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc84810c370 con 0x7fc848100780 2024-03-16T10:17:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84c861640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc84810c1f0 con 0x7fc848101970 2024-03-16T10:17:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84c861640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc84810c4f0 con 0x7fc848102330 2024-03-16T10:17:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc848100780 0x7fc8481969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc848100780 0x7fc8481969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56306/0 (socket says 172.21.15.77:56306) 2024-03-16T10:17:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84659c640 1 -- 172.21.15.77:0/3528003975 learned_addr learned my addr 172.21.15.77:0/3528003975 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc845d9b640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc848101970 0x7fc848196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:09.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc846d9d640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc848102330 0x7fc84819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84659c640 1 -- 172.21.15.77:0/3528003975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc848102330 msgr2=0x7fc84819df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84659c640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc848102330 0x7fc84819df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84659c640 1 -- 172.21.15.77:0/3528003975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc848101970 msgr2=0x7fc848196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84659c640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc848101970 0x7fc848196f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84659c640 1 -- 172.21.15.77:0/3528003975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84819e600 con 0x7fc848100780 2024-03-16T10:17:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc846d9d640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc848102330 0x7fc84819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:17:09.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc845d9b640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc848101970 0x7fc848196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:17:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84659c640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc848100780 0x7fc8481969e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc84000ebc0 tx=0x7fc84000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc83f7fe640 1 -- 172.21.15.77:0/3528003975 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840003b70 con 0x7fc848100780 2024-03-16T10:17:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc83f7fe640 1 -- 172.21.15.77:0/3528003975 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc840003d10 con 0x7fc848100780 2024-03-16T10:17:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84c861640 1 -- 172.21.15.77:0/3528003975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84819e8f0 con 0x7fc848100780 2024-03-16T10:17:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc83f7fe640 1 -- 172.21.15.77:0/3528003975 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840019980 con 0x7fc848100780 2024-03-16T10:17:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.702+0000 7fc84c861640 1 -- 172.21.15.77:0/3528003975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc84819ee10 con 0x7fc848100780 2024-03-16T10:17:09.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.703+0000 7fc84c861640 1 -- 172.21.15.77:0/3528003975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc848068a20 con 0x7fc848100780 2024-03-16T10:17:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.704+0000 7fc83f7fe640 1 -- 172.21.15.77:0/3528003975 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fc840019b20 con 0x7fc848100780 2024-03-16T10:17:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.705+0000 7fc83f7fe640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc8340a2890 0x7fc8340a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.705+0000 7fc845d9b640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc8340a2890 0x7fc8340a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.706+0000 7fc83f7fe640 1 -- 172.21.15.77:0/3528003975 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc8400cf620 con 0x7fc848100780 2024-03-16T10:17:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.706+0000 7fc845d9b640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc8340a2890 0x7fc8340a4d50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc838006010 tx=0x7fc83802c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:09.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.709+0000 7fc83f7fe640 1 -- 172.21.15.77:0/3528003975 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc84009c620 con 0x7fc848100780 2024-03-16T10:17:09.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.955+0000 7fc84c861640 1 -- 172.21.15.77:0/3528003975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc848107aa0 con 0x7fc848100780 2024-03-16T10:17:09.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.956+0000 7fc83f7fe640 1 -- 172.21.15.77:0/3528003975 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11938 (secure 0 0 0) 0x7fc84009bf20 con 0x7fc848100780 2024-03-16T10:17:09.976 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:17:09.976 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:17:02.900684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24475,24454,24482,14658,24455]},"id":1}]} 2024-03-16T10:17:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 27 2024-03-16T10:17:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.958+0000 7fc84c861640 1 -- 172.21.15.77:0/3528003975 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc8340a2890 msgr2=0x7fc8340a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.958+0000 7fc84c861640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc8340a2890 0x7fc8340a4d50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc838006010 tx=0x7fc83802c8a0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.958+0000 7fc84c861640 1 -- 172.21.15.77:0/3528003975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc848100780 msgr2=0x7fc8481969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.958+0000 7fc84c861640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc848100780 0x7fc8481969e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc84000ebc0 tx=0x7fc84000c6a0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.958+0000 7fc84c861640 1 -- 172.21.15.77:0/3528003975 shutdown_connections 2024-03-16T10:17:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.958+0000 7fc84c861640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc8340a2890 0x7fc8340a4d50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.958+0000 7fc84c861640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc848102330 0x7fc84819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.958+0000 7fc84c861640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc848101970 0x7fc848196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.958+0000 7fc84c861640 1 --2- 172.21.15.77:0/3528003975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc848100780 0x7fc8481969e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:09.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.958+0000 7fc84c861640 1 -- 172.21.15.77:0/3528003975 >> 172.21.15.77:0/3528003975 conn(0x7fc8480fbd10 msgr2=0x7fc848104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:09.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.958+0000 7fc84c861640 1 -- 172.21.15.77:0/3528003975 shutdown_connections 2024-03-16T10:17:09.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:09.958+0000 7fc84c861640 1 -- 172.21.15.77:0/3528003975 wait complete. 2024-03-16T10:17:09.979 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:17:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.108+0000 7f81f539e640 1 -- 172.21.15.77:0/4145867842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f01022b0 msgr2=0x7f81f0108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 --2- 172.21.15.77:0/4145867842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f01022b0 0x7f81f0108b40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009900 tx=0x7f81e002bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 -- 172.21.15.77:0/4145867842 shutdown_connections 2024-03-16T10:17:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 --2- 172.21.15.77:0/4145867842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f01022b0 0x7f81f0108b40 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 --2- 172.21.15.77:0/4145867842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f01018f0 0x7f81f0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 --2- 172.21.15.77:0/4145867842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81f0100700 0x7f81f0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 -- 172.21.15.77:0/4145867842 >> 172.21.15.77:0/4145867842 conn(0x7f81f00fbc70 msgr2=0x7f81f00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 -- 172.21.15.77:0/4145867842 shutdown_connections 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 -- 172.21.15.77:0/4145867842 wait complete. 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 Processor -- start 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 -- start start 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81f0100700 0x7f81f0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f01018f0 0x7f81f0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f01022b0 0x7f81f019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.109+0000 7f81f539e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f81f010c360 con 0x7f81f01022b0 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81f539e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f81f010c1e0 con 0x7f81f0100700 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81f539e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f81f010c4e0 con 0x7f81f01018f0 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81ef7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f01022b0 0x7f81f019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81eeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81f0100700 0x7f81f0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81ef7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f01022b0 0x7f81f019f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56316/0 (socket says 172.21.15.77:56316) 2024-03-16T10:17:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81eeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81f0100700 0x7f81f0197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:48012/0 (socket says 172.21.15.77:48012) 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81ef7fe640 1 -- 172.21.15.77:0/2705807835 learned_addr learned my addr 172.21.15.77:0/2705807835 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81ee7fc640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f01018f0 0x7f81f0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81eeffd640 1 -- 172.21.15.77:0/2705807835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f01018f0 msgr2=0x7f81f0198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81eeffd640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f01018f0 0x7f81f0198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81eeffd640 1 -- 172.21.15.77:0/2705807835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f01022b0 msgr2=0x7f81f019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81eeffd640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f01022b0 0x7f81f019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81eeffd640 1 -- 172.21.15.77:0/2705807835 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f019f6e0 con 0x7f81f0100700 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81ef7fe640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f01022b0 0x7f81f019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81ee7fc640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f01018f0 0x7f81f0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81eeffd640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81f0100700 0x7f81f0197ac0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f81e800e9c0 tx=0x7f81e800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81d7fff640 1 -- 172.21.15.77:0/2705807835 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e800ccc0 con 0x7f81f0100700 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81f539e640 1 -- 172.21.15.77:0/2705807835 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81f019f9d0 con 0x7f81f0100700 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81d7fff640 1 -- 172.21.15.77:0/2705807835 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81e80035f0 con 0x7f81f0100700 2024-03-16T10:17:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81d7fff640 1 -- 172.21.15.77:0/2705807835 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e800ce60 con 0x7f81f0100700 2024-03-16T10:17:10.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.110+0000 7f81f539e640 1 -- 172.21.15.77:0/2705807835 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81f019ff70 con 0x7f81f0100700 2024-03-16T10:17:10.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.111+0000 7f81f539e640 1 -- 172.21.15.77:0/2705807835 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81f00685d0 con 0x7f81f0100700 2024-03-16T10:17:10.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.112+0000 7f81d7fff640 1 -- 172.21.15.77:0/2705807835 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f81e80037a0 con 0x7f81f0100700 2024-03-16T10:17:10.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.113+0000 7f81d7fff640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f81d80a2890 0x7f81d80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.113+0000 7f81ee7fc640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f81d80a2890 0x7f81d80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.113+0000 7f81d7fff640 1 -- 172.21.15.77:0/2705807835 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f81e80cefd0 con 0x7f81f0100700 2024-03-16T10:17:10.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.113+0000 7f81ee7fc640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f81d80a2890 0x7f81d80a4d50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f81f0198fd0 tx=0x7f81e4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.121+0000 7f81d7fff640 1 -- 172.21.15.77:0/2705807835 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f81e809bf00 con 0x7f81f0100700 2024-03-16T10:17:10.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:09 smithi192 ceph-mon[36354]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:17:10.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:09 smithi192 ceph-mon[36354]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:17:10.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:09 smithi192 ceph-mon[36354]: pgmap v352: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-16T10:17:10.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:09 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3675236468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:17:10.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:09 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1264293005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:17:10.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:09 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2177380634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:17:10.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:09 smithi077 ceph-mon[32400]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:17:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:09 smithi077 ceph-mon[32400]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:17:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:09 smithi077 ceph-mon[32400]: pgmap v352: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-16T10:17:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:09 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3675236468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:17:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:09 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1264293005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:17:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:09 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2177380634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:17:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:09 smithi193 ceph-mon[36311]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:17:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:09 smithi193 ceph-mon[36311]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T10:17:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:09 smithi193 ceph-mon[36311]: pgmap v352: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-16T10:17:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:09 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3675236468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:17:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:09 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1264293005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:17:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:09 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2177380634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:17:10.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.309+0000 7f81f539e640 1 -- 172.21.15.77:0/2705807835 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81f01a03e0 con 0x7f81f0100700 2024-03-16T10:17:10.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.311+0000 7f81d7fff640 1 -- 172.21.15.77:0/2705807835 <== mon.2 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22496 (secure 0 0 0) 0x7f81e809b800 con 0x7f81f0100700 2024-03-16T10:17:10.312 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:17:10.312 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":87,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:15:59.826259+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:17:10.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.313+0000 7f81f539e640 1 -- 172.21.15.77:0/2705807835 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f81d80a2890 msgr2=0x7f81d80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.313+0000 7f81f539e640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f81d80a2890 0x7f81d80a4d50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f81f0198fd0 tx=0x7f81e4009500 comp rx=0 tx=0).stop 2024-03-16T10:17:10.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.313+0000 7f81f539e640 1 -- 172.21.15.77:0/2705807835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81f0100700 msgr2=0x7f81f0197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.313+0000 7f81f539e640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81f0100700 0x7f81f0197ac0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f81e800e9c0 tx=0x7f81e800ee80 comp rx=0 tx=0).stop 2024-03-16T10:17:10.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.313+0000 7f81f539e640 1 -- 172.21.15.77:0/2705807835 shutdown_connections 2024-03-16T10:17:10.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.313+0000 7f81f539e640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81f01022b0 0x7f81f019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.313+0000 7f81f539e640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f81d80a2890 0x7f81d80a4d50 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.313+0000 7f81f539e640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f01018f0 0x7f81f0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.313+0000 7f81f539e640 1 --2- 172.21.15.77:0/2705807835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81f0100700 0x7f81f0197ac0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.313+0000 7f81f539e640 1 -- 172.21.15.77:0/2705807835 >> 172.21.15.77:0/2705807835 conn(0x7f81f00fbc70 msgr2=0x7f81f00fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.314+0000 7f81f539e640 1 -- 172.21.15.77:0/2705807835 shutdown_connections 2024-03-16T10:17:10.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.314+0000 7f81f539e640 1 -- 172.21.15.77:0/2705807835 wait complete. 2024-03-16T10:17:10.327 DEBUG:tasks.fwd_scrub:Yielding 2024-03-16T10:17:10.327 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-16T10:17:10.338 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T10:17:10.338 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-16T10:17:10.338 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-16T10:17:10.338 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T10:17:10.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:17:10.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T10:17:10.341 INFO:teuthology.run_tasks:Running task workunit... 2024-03-16T10:17:10.353 INFO:tasks.workunit:Pulling workunits from ref 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:17:10.353 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-16T10:17:10.353 INFO:tasks.workunit:timeout=3h 2024-03-16T10:17:10.354 INFO:tasks.workunit:cleanup=True 2024-03-16T10:17:10.354 DEBUG:teuthology.orchestra.run.smithi077:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-16T10:17:10.369 INFO:teuthology.orchestra.run.smithi077.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-16T10:17:10.369 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-16T10:17:10.369 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-16T10:17:10.369 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T10:17:10.369 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-16T10:17:10.369 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-03-16 10:15:10.305524447 +0000 2024-03-16T10:17:10.369 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-03-16 10:15:10.305524447 +0000 2024-03-16T10:17:10.369 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-03-16 10:17:02.225501567 +0000 2024-03-16T10:17:10.370 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-03-16 10:15:10.305524447 +0000 2024-03-16T10:17:10.370 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-16T10:17:10.370 DEBUG:teuthology.orchestra.run.smithi077:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-16T10:17:10.443 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T10:17:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.448+0000 7fd355ef8640 1 -- 172.21.15.77:0/3958844066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd350102290 msgr2=0x7fd350108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.448+0000 7fd355ef8640 1 --2- 172.21.15.77:0/3958844066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd350102290 0x7fd350108b20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd348009ca0 tx=0x7fd34802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.448+0000 7fd355ef8640 1 -- 172.21.15.77:0/3958844066 shutdown_connections 2024-03-16T10:17:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.448+0000 7fd355ef8640 1 --2- 172.21.15.77:0/3958844066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd350102290 0x7fd350108b20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.448+0000 7fd355ef8640 1 --2- 172.21.15.77:0/3958844066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3501018d0 0x7fd350101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.448+0000 7fd355ef8640 1 --2- 172.21.15.77:0/3958844066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3501006e0 0x7fd350100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.448+0000 7fd355ef8640 1 -- 172.21.15.77:0/3958844066 >> 172.21.15.77:0/3958844066 conn(0x7fd3500fbc70 msgr2=0x7fd3500fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.448+0000 7fd355ef8640 1 -- 172.21.15.77:0/3958844066 shutdown_connections 2024-03-16T10:17:10.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.448+0000 7fd355ef8640 1 -- 172.21.15.77:0/3958844066 wait complete. 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.448+0000 7fd355ef8640 1 Processor -- start 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd355ef8640 1 -- start start 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd355ef8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3501006e0 0x7fd3501969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd355ef8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3501018d0 0x7fd350196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd355ef8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd350102290 0x7fd35019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd355ef8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd35010c340 con 0x7fd3501006e0 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd355ef8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd35010c1c0 con 0x7fd3501018d0 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd355ef8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd35010c4c0 con 0x7fd350102290 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3501006e0 0x7fd3501969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3501006e0 0x7fd3501969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56326/0 (socket says 172.21.15.77:56326) 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34f7fe640 1 -- 172.21.15.77:0/1855648177 learned_addr learned my addr 172.21.15.77:0/1855648177 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34ffff640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd350102290 0x7fd35019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34effd640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3501018d0 0x7fd350196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34f7fe640 1 -- 172.21.15.77:0/1855648177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd350102290 msgr2=0x7fd35019df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd350102290 0x7fd35019df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34f7fe640 1 -- 172.21.15.77:0/1855648177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3501018d0 msgr2=0x7fd350196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3501018d0 0x7fd350196f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34f7fe640 1 -- 172.21.15.77:0/1855648177 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35019e600 con 0x7fd3501006e0 2024-03-16T10:17:10.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34ffff640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd350102290 0x7fd35019df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:10.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34effd640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3501018d0 0x7fd350196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:10.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.449+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3501006e0 0x7fd3501969e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd34000e9c0 tx=0x7fd34000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.450+0000 7fd34cff9640 1 -- 172.21.15.77:0/1855648177 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34000cd80 con 0x7fd3501006e0 2024-03-16T10:17:10.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.450+0000 7fd34cff9640 1 -- 172.21.15.77:0/1855648177 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3400035f0 con 0x7fd3501006e0 2024-03-16T10:17:10.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.450+0000 7fd355ef8640 1 -- 172.21.15.77:0/1855648177 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35019e8f0 con 0x7fd3501006e0 2024-03-16T10:17:10.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.450+0000 7fd34cff9640 1 -- 172.21.15.77:0/1855648177 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340010690 con 0x7fd3501006e0 2024-03-16T10:17:10.453 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.450+0000 7fd355ef8640 1 -- 172.21.15.77:0/1855648177 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd35019ee90 con 0x7fd3501006e0 2024-03-16T10:17:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.452+0000 7fd358bc0740 1 Processor -- start 2024-03-16T10:17:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.452+0000 7fd358bc0740 1 -- start start 2024-03-16T10:17:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.452+0000 7fd34cff9640 1 -- 172.21.15.77:0/1855648177 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fd3400037a0 con 0x7fd3501006e0 2024-03-16T10:17:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.452+0000 7fd358bc0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b64990 0x5649d2b64db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.452+0000 7fd358bc0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b69e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.452+0000 7fd358bc0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b6a4f0 0x5649d2b6c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.452+0000 7fd358bc0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5649d2ac90d0 con 0x5649d2b65380 2024-03-16T10:17:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.452+0000 7fd358bc0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5649d2a71c70 con 0x5649d2b64990 2024-03-16T10:17:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.452+0000 7fd358bc0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5649d2ac5170 con 0x5649d2b6a4f0 2024-03-16T10:17:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b69e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b69e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56330/0 (socket says 172.21.15.77:56330) 2024-03-16T10:17:10.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b64990 0x5649d2b64db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34f7fe640 1 -- 172.21.15.77:0/1755416651 learned_addr learned my addr 172.21.15.77:0/1755416651 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34effd640 1 --2- 172.21.15.77:0/1755416651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b6a4f0 0x5649d2b6c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34f7fe640 1 -- 172.21.15.77:0/1755416651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b6a4f0 msgr2=0x5649d2b6c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/1755416651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b6a4f0 0x5649d2b6c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34f7fe640 1 -- 172.21.15.77:0/1755416651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b64990 msgr2=0x5649d2b64db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/1755416651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b64990 0x5649d2b64db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34f7fe640 1 -- 172.21.15.77:0/1755416651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649d28a57e0 con 0x5649d2b65380 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34ffff640 1 --2- 172.21.15.77:0/1755416651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b64990 0x5649d2b64db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34cff9640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd3380a2910 0x7fd3380a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/1755416651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b69e80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd3400cc810 tx=0x7fd3400d74b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2d9b4e2fa987ce server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34ffff640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd3380a2910 0x7fd3380a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd34cff9640 1 -- 172.21.15.77:0/1855648177 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd3400cee30 con 0x7fd3501006e0 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.453+0000 7fd335ffb640 1 -- 172.21.15.77:0/1755416651 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340097a00 con 0x5649d2b65380 2024-03-16T10:17:10.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.454+0000 7fd335ffb640 1 -- 172.21.15.77:0/1755416651 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd34009be60 con 0x5649d2b65380 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.454+0000 7fd335ffb640 1 -- 172.21.15.77:0/1755416651 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34009b0c0 con 0x5649d2b65380 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.454+0000 7fd358bc0740 1 -- 172.21.15.77:0/1755416651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 msgr2=0x5649d2b69e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.454+0000 7fd358bc0740 1 --2- 172.21.15.77:0/1755416651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b69e80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd3400cc810 tx=0x7fd3400d74b0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.454+0000 7fd34ffff640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd3380a2910 0x7fd3380a4dd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd348005e30 tx=0x7fd348032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.454+0000 7fd358bc0740 1 -- 172.21.15.77:0/1755416651 shutdown_connections 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.454+0000 7fd358bc0740 1 --2- 172.21.15.77:0/1755416651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b6a4f0 0x5649d2b6c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.454+0000 7fd358bc0740 1 --2- 172.21.15.77:0/1755416651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b69e80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.454+0000 7fd358bc0740 1 --2- 172.21.15.77:0/1755416651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b64990 0x5649d2b64db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.454+0000 7fd358bc0740 1 -- 172.21.15.77:0/1755416651 >> 172.21.15.77:0/1755416651 conn(0x5649d2b2cc50 msgr2=0x5649d2b60300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.454+0000 7fd358bc0740 1 -- 172.21.15.77:0/1755416651 shutdown_connections 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.454+0000 7fd358bc0740 1 -- 172.21.15.77:0/1755416651 wait complete. 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd358bc0740 1 -- start start 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd358bc0740 1 Processor -- start 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd358bc0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 0x5649d2b6eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd358bc0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b6f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd358bc0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b6a4f0 0x5649d2b76450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd358bc0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5649d2b6d4a0 con 0x5649d2b65380 2024-03-16T10:17:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd358bc0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5649d2a850d0 con 0x5649d2b6a4f0 2024-03-16T10:17:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd358bc0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5649d2b6d620 con 0x5649d2b64990 2024-03-16T10:17:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd358bc0740 10 client.0 _ll_register_callbacks cb 0x5649d2a4e760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd34effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b6f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd34effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b6f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56346/0 (socket says 172.21.15.77:56346) 2024-03-16T10:17:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd34f7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 0x5649d2b6eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd34effd640 1 -- 172.21.15.77:0/783688296 learned_addr learned my addr 172.21.15.77:0/783688296 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.455+0000 7fd34ffff640 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b6a4f0 0x5649d2b76450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd34f7fe640 1 -- 172.21.15.77:0/783688296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b6a4f0 msgr2=0x5649d2b76450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b6a4f0 0x5649d2b76450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd34f7fe640 1 -- 172.21.15.77:0/783688296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 msgr2=0x5649d2b6f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b6f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd34f7fe640 1 -- 172.21.15.77:0/783688296 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649d2905f70 con 0x5649d2b64990 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd34effd640 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b6f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 0x5649d2b6eed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x5649d2b65df0 tx=0x7fd3400cc840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fac3e3e5ba4265d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd32effd640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd32effd640 1 -- 172.21.15.77:0/783688296 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34009b2a0 con 0x5649d2b64990 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd358bc0740 1 -- 172.21.15.77:0/783688296 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd34009be60 con 0x5649d2b64990 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd32effd640 1 -- 172.21.15.77:0/783688296 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd340003b30 con 0x5649d2b64990 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd32effd640 1 -- 172.21.15.77:0/783688296 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd34009b530 con 0x5649d2b64990 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd358bc0740 10 client.25294 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd358bc0740 1 -- 172.21.15.77:0/783688296 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649d279d170 con 0x5649d2b64990 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.456+0000 7fd32effd640 1 -- 172.21.15.77:0/783688296 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd34009be60 con 0x5649d2b64990 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.457+0000 7fd32effd640 1 -- 172.21.15.77:0/783688296 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd3400d7c80 con 0x5649d2b64990 2024-03-16T10:17:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.457+0000 7fd358bc0740 10 client.25294 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.457+0000 7fd358bc0740 10 client.25294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.457+0000 7fd358bc0740 20 client.25294 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.457+0000 7fd358bc0740 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5649d2b7a5b0 0x5649d2b7c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.457+0000 7fd358bc0740 4 client.25294 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.457+0000 7fd358bc0740 1 -- 172.21.15.77:0/783688296 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649d2ad6610 con 0x5649d2b7a5b0 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.457+0000 7fd34effd640 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5649d2b7a5b0 0x5649d2b7c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.457+0000 7fd32effd640 10 client.25294 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.457+0000 7fd34effd640 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5649d2b7a5b0 0x5649d2b7c9a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f855968594ec0e1 server_cookie=ac18434ea9775de8 in_seq=0 out_seq=0 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd32effd640 1 -- 172.21.15.77:0/783688296 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5649d2ad6610 con 0x5649d2b7a5b0 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd32effd640 10 client.25294 handle_command_reply: tid=0 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd358bc0740 1 client.25294 shutdown 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd358bc0740 1 -- 172.21.15.77:0/783688296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 msgr2=0x5649d2b6eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd358bc0740 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 0x5649d2b6eed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x5649d2b65df0 tx=0x7fd3400cc840 comp rx=0 tx=0).stop 2024-03-16T10:17:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd358bc0740 1 -- 172.21.15.77:0/783688296 shutdown_connections 2024-03-16T10:17:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd358bc0740 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b6a4f0 0x5649d2b76450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd358bc0740 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5649d2b7a5b0 0x5649d2b7c9a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd358bc0740 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b6f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd358bc0740 1 --2- 172.21.15.77:0/783688296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 0x5649d2b6eed0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd358bc0740 1 -- 172.21.15.77:0/783688296 >> 172.21.15.77:0/783688296 conn(0x5649d2b2cc50 msgr2=0x5649d2b5f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd32effd640 0 client.25294 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd358bc0740 1 -- 172.21.15.77:0/783688296 shutdown_connections 2024-03-16T10:17:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd358bc0740 1 -- 172.21.15.77:0/783688296 wait complete. 2024-03-16T10:17:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.458+0000 7fd358bc0740 20 client.25294 trim_cache size 0 max 16384 2024-03-16T10:17:10.469 INFO:tasks.workunit.client.0.smithi077.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-16T10:17:10.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd358bc0740 1 Processor -- start 2024-03-16T10:17:10.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd358bc0740 1 -- start start 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd358bc0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 0x5649d2b6f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd358bc0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b76410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd358bc0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b6a4f0 0x5649d2b5ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd358bc0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501a7ac0 con 0x5649d2b65380 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd358bc0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501a9600 con 0x5649d2b6a4f0 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd358bc0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501a8190 con 0x5649d2b64990 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd34effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b76410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd34f7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 0x5649d2b6f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd34ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b6a4f0 0x5649d2b5ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd34f7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 0x5649d2b6f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:42038/0 (socket says 172.21.15.77:42038) 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd34effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b76410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56362/0 (socket says 172.21.15.77:56362) 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd34f7fe640 1 -- 172.21.15.77:0/3434059725 learned_addr learned my addr 172.21.15.77:0/3434059725 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd34f7fe640 1 -- 172.21.15.77:0/3434059725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b6a4f0 msgr2=0x5649d2b5ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/3434059725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b6a4f0 0x5649d2b5ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd34f7fe640 1 -- 172.21.15.77:0/3434059725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 msgr2=0x5649d2b76410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/3434059725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b76410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd34f7fe640 1 -- 172.21.15.77:0/3434059725 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649d2b608a0 con 0x5649d2b64990 2024-03-16T10:17:10.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.476+0000 7fd34effd640 1 --2- 172.21.15.77:0/3434059725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b76410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:10.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.477+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/3434059725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 0x5649d2b6f1f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd340099a10 tx=0x7fd340099c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=498de2a15dfd793b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.477+0000 7fd3357fa640 1 -- 172.21.15.77:0/3434059725 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3400ba020 con 0x5649d2b64990 2024-03-16T10:17:10.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.477+0000 7fd3357fa640 1 -- 172.21.15.77:0/3434059725 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd34009bb00 con 0x5649d2b64990 2024-03-16T10:17:10.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.477+0000 7fd3357fa640 1 -- 172.21.15.77:0/3434059725 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340097a00 con 0x5649d2b64990 2024-03-16T10:17:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 -- 172.21.15.77:0/3434059725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 msgr2=0x5649d2b6f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 --2- 172.21.15.77:0/3434059725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 0x5649d2b6f1f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd340099a10 tx=0x7fd340099c70 comp rx=0 tx=0).stop 2024-03-16T10:17:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 -- 172.21.15.77:0/3434059725 shutdown_connections 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 --2- 172.21.15.77:0/3434059725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b6a4f0 0x5649d2b5ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 --2- 172.21.15.77:0/3434059725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b65380 0x5649d2b76410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 --2- 172.21.15.77:0/3434059725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b64990 0x5649d2b6f1f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 -- 172.21.15.77:0/3434059725 >> 172.21.15.77:0/3434059725 conn(0x5649d2b2cc50 msgr2=0x5649d2b6eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 -- 172.21.15.77:0/3434059725 shutdown_connections 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 -- 172.21.15.77:0/3434059725 wait complete. 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 -- start start 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 Processor -- start 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b64990 0x5649d2bdc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b65380 0x5649d2bdec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b6a4f0 0x5649d2be11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501a8f30 con 0x5649d2b6a4f0 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501a8860 con 0x5649d2b64990 2024-03-16T10:17:10.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501a5aa0 con 0x5649d2b65380 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.478+0000 7fd358bc0740 10 client.0 _ll_register_callbacks cb 0x7fd340005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b6a4f0 0x5649d2be11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b6a4f0 0x5649d2be11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56374/0 (socket says 172.21.15.77:56374) 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34f7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b64990 0x5649d2bdc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34ffff640 1 -- 172.21.15.77:0/3557038124 learned_addr learned my addr 172.21.15.77:0/3557038124 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34effd640 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b65380 0x5649d2bdec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34ffff640 1 -- 172.21.15.77:0/3557038124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b65380 msgr2=0x5649d2bdec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34ffff640 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b65380 0x5649d2bdec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34ffff640 1 -- 172.21.15.77:0/3557038124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b64990 msgr2=0x5649d2bdc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34ffff640 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b64990 0x5649d2bdc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34ffff640 1 -- 172.21.15.77:0/3557038124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649d2be19f0 con 0x5649d2b6a4f0 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b64990 0x5649d2bdc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34effd640 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b65380 0x5649d2bdec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:10.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd34ffff640 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b6a4f0 0x5649d2be11c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd348009ca0 tx=0x7fd3480095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385f39f78631252a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd32effd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd32effd640 1 -- 172.21.15.77:0/3557038124 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348039040 con 0x5649d2b6a4f0 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd32effd640 1 -- 172.21.15.77:0/3557038124 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd348034a00 con 0x5649d2b6a4f0 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd358bc0740 1 -- 172.21.15.77:0/3557038124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd34009bb00 con 0x5649d2b6a4f0 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.479+0000 7fd32effd640 1 -- 172.21.15.77:0/3557038124 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd348032a20 con 0x5649d2b6a4f0 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd358bc0740 10 client.15507 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd358bc0740 1 -- 172.21.15.77:0/3557038124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649d2be2da0 con 0x5649d2b6a4f0 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd32effd640 1 -- 172.21.15.77:0/3557038124 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd34009bb00 con 0x5649d2b6a4f0 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd32effd640 1 -- 172.21.15.77:0/3557038124 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd34804e430 con 0x5649d2b6a4f0 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd358bc0740 10 client.15507 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd358bc0740 10 client.15507 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd358bc0740 20 client.15507 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:10.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd358bc0740 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5649d2be37b0 0x5649d2be5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd358bc0740 4 client.15507 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd358bc0740 1 -- 172.21.15.77:0/3557038124 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5649d2ad0960 con 0x5649d2be37b0 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5649d2be37b0 0x5649d2be5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd32effd640 10 client.15507 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.480+0000 7fd34f7fe640 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5649d2be37b0 0x5649d2be5bd0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12ef0f93ce1391db server_cookie=3fcf2480718f1365 in_seq=0 out_seq=0 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.481+0000 7fd32effd640 1 -- 172.21.15.77:0/3557038124 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5649d2ad0960 con 0x5649d2be37b0 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.481+0000 7fd32effd640 10 client.15507 handle_command_reply: tid=0 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.481+0000 7fd358bc0740 1 client.15507 shutdown 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd358bc0740 1 -- 172.21.15.77:0/3557038124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b6a4f0 msgr2=0x5649d2be11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd358bc0740 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b6a4f0 0x5649d2be11c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd348009ca0 tx=0x7fd3480095c0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd358bc0740 1 -- 172.21.15.77:0/3557038124 shutdown_connections 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd358bc0740 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5649d2be37b0 0x5649d2be5bd0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd358bc0740 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649d2b6a4f0 0x5649d2be11c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd358bc0740 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649d2b65380 0x5649d2bdec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd32effd640 0 client.15507 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd358bc0740 1 --2- 172.21.15.77:0/3557038124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649d2b64990 0x5649d2bdc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd358bc0740 1 -- 172.21.15.77:0/3557038124 >> 172.21.15.77:0/3557038124 conn(0x5649d2b2cc50 msgr2=0x5649d2b6f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd358bc0740 1 -- 172.21.15.77:0/3557038124 shutdown_connections 2024-03-16T10:17:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd358bc0740 1 -- 172.21.15.77:0/3557038124 wait complete. 2024-03-16T10:17:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd358bc0740 20 client.15507 trim_cache size 0 max 16384 2024-03-16T10:17:10.488 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:17:10.488 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T10:17:10.488 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "5334ea35-26e4-46e7-909c-ed7bb337b4de", 2024-03-16T10:17:10.488 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T10:17:10.488 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:17:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd355ef8640 1 -- 172.21.15.77:0/1855648177 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd3380a2910 msgr2=0x7fd3380a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd355ef8640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd3380a2910 0x7fd3380a4dd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd348005e30 tx=0x7fd348032590 comp rx=0 tx=0).stop 2024-03-16T10:17:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd355ef8640 1 -- 172.21.15.77:0/1855648177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3501006e0 msgr2=0x7fd3501969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.482+0000 7fd355ef8640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3501006e0 0x7fd3501969e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd34000e9c0 tx=0x7fd34000ee80 comp rx=0 tx=0).stop 2024-03-16T10:17:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.483+0000 7fd355ef8640 1 -- 172.21.15.77:0/1855648177 shutdown_connections 2024-03-16T10:17:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.483+0000 7fd355ef8640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd3380a2910 0x7fd3380a4dd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.483+0000 7fd355ef8640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd350102290 0x7fd35019df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.483+0000 7fd355ef8640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3501018d0 0x7fd350196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.483+0000 7fd355ef8640 1 --2- 172.21.15.77:0/1855648177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3501006e0 0x7fd3501969e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.483+0000 7fd355ef8640 1 -- 172.21.15.77:0/1855648177 >> 172.21.15.77:0/1855648177 conn(0x7fd3500fbc70 msgr2=0x7fd350104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.483+0000 7fd355ef8640 1 -- 172.21.15.77:0/1855648177 shutdown_connections 2024-03-16T10:17:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.483+0000 7fd355ef8640 1 -- 172.21.15.77:0/1855648177 wait complete. 2024-03-16T10:17:10.495 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:17:10.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:17:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 -- 172.21.15.77:0/885434725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5468104c30 msgr2=0x7f5468107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 --2- 172.21.15.77:0/885434725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5468104c30 0x7f5468107020 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f545c009900 tx=0x7f545c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 -- 172.21.15.77:0/885434725 shutdown_connections 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 --2- 172.21.15.77:0/885434725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5468107560 0x7f5468109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 --2- 172.21.15.77:0/885434725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5468104c30 0x7f5468107020 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 --2- 172.21.15.77:0/885434725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5468102300 0x7f54681046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 -- 172.21.15.77:0/885434725 >> 172.21.15.77:0/885434725 conn(0x7f54680fbc70 msgr2=0x7f54680fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 -- 172.21.15.77:0/885434725 shutdown_connections 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 -- 172.21.15.77:0/885434725 wait complete. 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 Processor -- start 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 -- start start 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5468102300 0x7f5468101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5468104c30 0x7f54681002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5468107560 0x7f54681007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f546810d430 con 0x7f5468107560 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f546810d2b0 con 0x7f5468104c30 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546f861640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f546810d5b0 con 0x7f5468102300 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.643+0000 7f546ddfe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5468107560 0x7f54681007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f546ddfe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5468107560 0x7f54681007f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56380/0 (socket says 172.21.15.77:56380) 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f546d5fd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5468102300 0x7f5468101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f546ddfe640 1 -- 172.21.15.77:0/2432376795 learned_addr learned my addr 172.21.15.77:0/2432376795 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f546cdfc640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5468104c30 0x7f54681002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f546ddfe640 1 -- 172.21.15.77:0/2432376795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5468102300 msgr2=0x7f5468101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f546ddfe640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5468102300 0x7f5468101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f546ddfe640 1 -- 172.21.15.77:0/2432376795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5468104c30 msgr2=0x7f54681002b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f546ddfe640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5468104c30 0x7f54681002b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f546ddfe640 1 -- 172.21.15.77:0/2432376795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5468101080 con 0x7f5468107560 2024-03-16T10:17:10.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f546ddfe640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5468107560 0x7f54681007f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f546000eb10 tx=0x7f546000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f54667fc640 1 -- 172.21.15.77:0/2432376795 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460004b20 con 0x7f5468107560 2024-03-16T10:17:10.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f54667fc640 1 -- 172.21.15.77:0/2432376795 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5460004cc0 con 0x7f5468107560 2024-03-16T10:17:10.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f546f861640 1 -- 172.21.15.77:0/2432376795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54681a1c60 con 0x7f5468107560 2024-03-16T10:17:10.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f54667fc640 1 -- 172.21.15.77:0/2432376795 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460010760 con 0x7f5468107560 2024-03-16T10:17:10.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.644+0000 7f546f861640 1 -- 172.21.15.77:0/2432376795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54681a2180 con 0x7f5468107560 2024-03-16T10:17:10.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.647+0000 7f5472451740 1 Processor -- start 2024-03-16T10:17:10.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.647+0000 7f54667fc640 1 -- 172.21.15.77:0/2432376795 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f5460005020 con 0x7f5468107560 2024-03-16T10:17:10.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f5472451740 1 -- start start 2024-03-16T10:17:10.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f5472451740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 0x55f5e20df590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f5472451740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20dfb60 0x55f5e20e4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f5472451740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20e4cd0 0x55f5e20e7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f5472451740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f5e2044500 con 0x55f5e20e4cd0 2024-03-16T10:17:10.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f5472451740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55f5e1fecbe0 con 0x55f5e20dfb60 2024-03-16T10:17:10.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f5472451740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f5e20405a0 con 0x55f5e20df170 2024-03-16T10:17:10.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546cdfc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20e4cd0 0x55f5e20e7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546cdfc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20e4cd0 0x55f5e20e7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56396/0 (socket says 172.21.15.77:56396) 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546cdfc640 1 -- 172.21.15.77:0/289950190 learned_addr learned my addr 172.21.15.77:0/289950190 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546d5fd640 1 --2- 172.21.15.77:0/289950190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20dfb60 0x55f5e20e4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546cdfc640 1 --2- 172.21.15.77:0/289950190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 0x55f5e20df590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f54667fc640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f54500a2910 0x7f54500a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546cdfc640 1 -- 172.21.15.77:0/289950190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 msgr2=0x55f5e20df590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546cdfc640 1 --2- 172.21.15.77:0/289950190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 0x55f5e20df590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546cdfc640 1 -- 172.21.15.77:0/289950190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20dfb60 msgr2=0x55f5e20e4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546cdfc640 1 --2- 172.21.15.77:0/289950190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20dfb60 0x55f5e20e4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546cdfc640 1 -- 172.21.15.77:0/289950190 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5e20004b0 con 0x55f5e20e4cd0 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546d5fd640 1 --2- 172.21.15.77:0/289950190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20dfb60 0x55f5e20e4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546ddfe640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f54500a2910 0x7f54500a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f546cdfc640 1 --2- 172.21.15.77:0/289950190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20e4cd0 0x55f5e20e7160 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f545c02cfd0 tx=0x7f545c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5c405371971b78 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.648+0000 7f54667fc640 1 -- 172.21.15.77:0/2432376795 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f54600cf060 con 0x7f5468107560 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.649+0000 7f544f7fe640 1 -- 172.21.15.77:0/289950190 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c004030 con 0x55f5e20e4cd0 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.649+0000 7f544f7fe640 1 -- 172.21.15.77:0/289950190 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f545c0041d0 con 0x55f5e20e4cd0 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.649+0000 7f544f7fe640 1 -- 172.21.15.77:0/289950190 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c03f440 con 0x55f5e20e4cd0 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.649+0000 7f5472451740 1 -- 172.21.15.77:0/289950190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20e4cd0 msgr2=0x55f5e20e7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.649+0000 7f546ddfe640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f54500a2910 0x7f54500a4dd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5460098360 tx=0x7f5460086510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.649+0000 7f5472451740 1 --2- 172.21.15.77:0/289950190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20e4cd0 0x55f5e20e7160 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f545c02cfd0 tx=0x7f545c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 -- 172.21.15.77:0/289950190 shutdown_connections 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 --2- 172.21.15.77:0/289950190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20e4cd0 0x55f5e20e7160 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 --2- 172.21.15.77:0/289950190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20dfb60 0x55f5e20e4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 --2- 172.21.15.77:0/289950190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 0x55f5e20df590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 -- 172.21.15.77:0/289950190 >> 172.21.15.77:0/289950190 conn(0x55f5e20da6e0 msgr2=0x55f5e20daae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 -- 172.21.15.77:0/289950190 shutdown_connections 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 -- 172.21.15.77:0/289950190 wait complete. 2024-03-16T10:17:10.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 -- start start 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 Processor -- start 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 0x55f5e20e9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 0x55f5e20e9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20e4cd0 0x55f5e20f0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f5e20e7c20 con 0x55f5e20dfb60 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55f5e2000160 con 0x55f5e20e4cd0 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f5e20e7da0 con 0x55f5e20df170 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.650+0000 7f5472451740 10 client.0 _ll_register_callbacks cb 0x55f5e1fe4150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f546d5fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 0x55f5e20e9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f546d5fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 0x55f5e20e9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56398/0 (socket says 172.21.15.77:56398) 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f546d5fd640 1 -- 172.21.15.77:0/481574191 learned_addr learned my addr 172.21.15.77:0/481574191 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f546cdfc640 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20e4cd0 0x55f5e20f0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f546cdfc640 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 0x55f5e20e9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f546d5fd640 1 -- 172.21.15.77:0/481574191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 msgr2=0x55f5e20e9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f546d5fd640 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 0x55f5e20e9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f546d5fd640 1 -- 172.21.15.77:0/481574191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20e4cd0 msgr2=0x55f5e20f0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f546d5fd640 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20e4cd0 0x55f5e20f0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f546d5fd640 1 -- 172.21.15.77:0/481574191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5e1e227d0 con 0x55f5e20dfb60 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f546d5fd640 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 0x55f5e20e9c60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f54580126e0 tx=0x7f5458012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7334c9015048a52 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f544cff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f544cff9640 1 -- 172.21.15.77:0/481574191 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458012190 con 0x55f5e20dfb60 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f544cff9640 1 -- 172.21.15.77:0/481574191 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f545800f8c0 con 0x55f5e20dfb60 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f5472451740 1 -- 172.21.15.77:0/481574191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f545c0041d0 con 0x55f5e20dfb60 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f544cff9640 1 -- 172.21.15.77:0/481574191 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f545800fac0 con 0x55f5e20dfb60 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f5472451740 10 client.15519 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.651+0000 7f5472451740 1 -- 172.21.15.77:0/481574191 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f5e1d583f0 con 0x55f5e20dfb60 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.652+0000 7f544cff9640 1 -- 172.21.15.77:0/481574191 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f545c0041d0 con 0x55f5e20dfb60 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.652+0000 7f544cff9640 1 -- 172.21.15.77:0/481574191 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5458025050 con 0x55f5e20dfb60 2024-03-16T10:17:10.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.652+0000 7f5472451740 10 client.15519 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.652+0000 7f5472451740 10 client.15519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.652+0000 7f5472451740 20 client.15519 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.652+0000 7f5472451740 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f5e20f5190 0x55f5e20f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.652+0000 7f5472451740 4 client.15519 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.652+0000 7f5472451740 1 -- 172.21.15.77:0/481574191 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5e2051a40 con 0x55f5e20f5190 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.652+0000 7f546cdfc640 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f5e20f5190 0x55f5e20f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.652+0000 7f546cdfc640 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f5e20f5190 0x55f5e20f75d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89bec723710f1399 server_cookie=9be65f72f371dbb3 in_seq=0 out_seq=0 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.652+0000 7f544cff9640 10 client.15519 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.653+0000 7f544cff9640 1 -- 172.21.15.77:0/481574191 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f5e2051a40 con 0x55f5e20f5190 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.653+0000 7f544cff9640 10 client.15519 handle_command_reply: tid=0 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.653+0000 7f5472451740 1 client.15519 shutdown 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.653+0000 7f5472451740 1 -- 172.21.15.77:0/481574191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 msgr2=0x55f5e20e9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.653+0000 7f5472451740 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 0x55f5e20e9c60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f54580126e0 tx=0x7f5458012ba0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.653+0000 7f5472451740 1 -- 172.21.15.77:0/481574191 shutdown_connections 2024-03-16T10:17:10.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.653+0000 7f5472451740 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20e4cd0 0x55f5e20f0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.653+0000 7f5472451740 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f5e20f5190 0x55f5e20f75d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.653+0000 7f5472451740 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 0x55f5e20e9c60 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.653+0000 7f5472451740 1 --2- 172.21.15.77:0/481574191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 0x55f5e20e9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.653+0000 7f5472451740 1 -- 172.21.15.77:0/481574191 >> 172.21.15.77:0/481574191 conn(0x55f5e20da6e0 msgr2=0x55f5e20d9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.653+0000 7f544cff9640 0 client.15519 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:10.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.654+0000 7f5472451740 1 -- 172.21.15.77:0/481574191 shutdown_connections 2024-03-16T10:17:10.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.654+0000 7f5472451740 1 -- 172.21.15.77:0/481574191 wait complete. 2024-03-16T10:17:10.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.654+0000 7f5472451740 20 client.15519 trim_cache size 0 max 16384 2024-03-16T10:17:10.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f5472451740 1 Processor -- start 2024-03-16T10:17:10.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f5472451740 1 -- start start 2024-03-16T10:17:10.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f5472451740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 0x55f5e20daac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f5472451740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 0x55f5e20f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f5472451740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20e4cd0 0x55f5e20d9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f5472451740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f54681aadb0 con 0x55f5e20dfb60 2024-03-16T10:17:10.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f5472451740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f54681ac8f0 con 0x55f5e20e4cd0 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f5472451740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f54681ab480 con 0x55f5e20df170 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f546d5fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 0x55f5e20f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f546d5fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 0x55f5e20f0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56402/0 (socket says 172.21.15.77:56402) 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f546d5fd640 1 -- 172.21.15.77:0/2663216515 learned_addr learned my addr 172.21.15.77:0/2663216515 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f546cdfc640 1 --2- 172.21.15.77:0/2663216515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 0x55f5e20daac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f546cdfc640 1 --2- 172.21.15.77:0/2663216515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20e4cd0 0x55f5e20d9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f546d5fd640 1 -- 172.21.15.77:0/2663216515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 msgr2=0x55f5e20daac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f546d5fd640 1 --2- 172.21.15.77:0/2663216515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 0x55f5e20daac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f546d5fd640 1 -- 172.21.15.77:0/2663216515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20e4cd0 msgr2=0x55f5e20d9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f546d5fd640 1 --2- 172.21.15.77:0/2663216515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20e4cd0 0x55f5e20d9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f546d5fd640 1 -- 172.21.15.77:0/2663216515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5e1d180f0 con 0x55f5e20dfb60 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f546cdfc640 1 --2- 172.21.15.77:0/2663216515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20e4cd0 0x55f5e20d9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f546d5fd640 1 --2- 172.21.15.77:0/2663216515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 0x55f5e20f0bf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5458005eb0 tx=0x7f54580123a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5e088b4f909be6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f544effd640 1 -- 172.21.15.77:0/2663216515 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458000e60 con 0x55f5e20dfb60 2024-03-16T10:17:10.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f544effd640 1 -- 172.21.15.77:0/2663216515 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f545800f810 con 0x55f5e20dfb60 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.672+0000 7f544effd640 1 -- 172.21.15.77:0/2663216515 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458010850 con 0x55f5e20dfb60 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.673+0000 7f5472451740 1 -- 172.21.15.77:0/2663216515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 msgr2=0x55f5e20f0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.673+0000 7f5472451740 1 --2- 172.21.15.77:0/2663216515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 0x55f5e20f0bf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5458005eb0 tx=0x7f54580123a0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.673+0000 7f5472451740 1 -- 172.21.15.77:0/2663216515 shutdown_connections 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.673+0000 7f5472451740 1 --2- 172.21.15.77:0/2663216515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20e4cd0 0x55f5e20d9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.673+0000 7f5472451740 1 --2- 172.21.15.77:0/2663216515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20dfb60 0x55f5e20f0bf0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.673+0000 7f5472451740 1 --2- 172.21.15.77:0/2663216515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20df170 0x55f5e20daac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.673+0000 7f5472451740 1 -- 172.21.15.77:0/2663216515 >> 172.21.15.77:0/2663216515 conn(0x55f5e20da6e0 msgr2=0x55f5e20e1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.673+0000 7f5472451740 1 -- 172.21.15.77:0/2663216515 shutdown_connections 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.673+0000 7f5472451740 1 -- 172.21.15.77:0/2663216515 wait complete. 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.673+0000 7f5472451740 1 -- start start 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.673+0000 7f5472451740 1 Processor -- start 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f5472451740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20df170 0x55f5e20ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f5472451740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20dfb60 0x55f5e20ee910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f5472451740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20e4cd0 0x55f5e20ef920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f5472451740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f54681ac220 con 0x55f5e20df170 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f5472451740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f54681abb50 con 0x55f5e20dfb60 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f5472451740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f54681a8d90 con 0x55f5e20e4cd0 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f5472451740 10 client.0 _ll_register_callbacks cb 0x7f5458009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f546cdfc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20df170 0x55f5e20ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f546cdfc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20df170 0x55f5e20ee3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56408/0 (socket says 172.21.15.77:56408) 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f546cdfc640 1 -- 172.21.15.77:0/2849207128 learned_addr learned my addr 172.21.15.77:0/2849207128 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f546d5fd640 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20dfb60 0x55f5e20ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f546cdfc640 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20e4cd0 0x55f5e20ef920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f546d5fd640 1 -- 172.21.15.77:0/2849207128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20e4cd0 msgr2=0x55f5e20ef920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f546d5fd640 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20e4cd0 0x55f5e20ef920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f546d5fd640 1 -- 172.21.15.77:0/2849207128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20df170 msgr2=0x55f5e20ee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f546d5fd640 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20df170 0x55f5e20ee3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f546d5fd640 1 -- 172.21.15.77:0/2849207128 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5e200b9d0 con 0x55f5e20dfb60 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f546cdfc640 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20df170 0x55f5e20ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f546d5fd640 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20dfb60 0x55f5e20ee910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55f5e20d6320 tx=0x7f5458010bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a083d91d5e394b38 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.674+0000 7f544cff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f544cff9640 1 -- 172.21.15.77:0/2849207128 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458005040 con 0x55f5e20dfb60 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f544cff9640 1 -- 172.21.15.77:0/2849207128 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f54580051e0 con 0x55f5e20dfb60 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f5472451740 1 -- 172.21.15.77:0/2849207128 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f545800f810 con 0x55f5e20dfb60 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f544cff9640 1 -- 172.21.15.77:0/2849207128 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5458005400 con 0x55f5e20dfb60 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f5472451740 10 client.25331 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f5472451740 1 -- 172.21.15.77:0/2849207128 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f5e215b4d0 con 0x55f5e20dfb60 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f544cff9640 1 -- 172.21.15.77:0/2849207128 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f545800f810 con 0x55f5e20dfb60 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f544cff9640 1 -- 172.21.15.77:0/2849207128 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5458025070 con 0x55f5e20dfb60 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f5472451740 10 client.25331 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f5472451740 10 client.25331 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:10.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f5472451740 20 client.25331 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f5472451740 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f5e215bce0 0x55f5e215e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f5472451740 4 client.25331 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f5472451740 1 -- 172.21.15.77:0/2849207128 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f5e204bd90 con 0x55f5e215bce0 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.675+0000 7f546cdfc640 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f5e215bce0 0x55f5e215e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f544cff9640 10 client.25331 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f546cdfc640 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f5e215bce0 0x55f5e215e100 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f401493730da5cf8 server_cookie=7efa586c4f9edf5b in_seq=0 out_seq=0 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f544cff9640 1 -- 172.21.15.77:0/2849207128 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f5e204bd90 con 0x55f5e215bce0 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f544cff9640 10 client.25331 handle_command_reply: tid=0 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f5472451740 1 client.25331 shutdown 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f5472451740 1 -- 172.21.15.77:0/2849207128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20dfb60 msgr2=0x55f5e20ee910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f5472451740 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20dfb60 0x55f5e20ee910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55f5e20d6320 tx=0x7f5458010bd0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f5472451740 1 -- 172.21.15.77:0/2849207128 shutdown_connections 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f5472451740 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5e20e4cd0 0x55f5e20ef920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f5472451740 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5e20dfb60 0x55f5e20ee910 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f5472451740 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f5e215bce0 0x55f5e215e100 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f5472451740 1 --2- 172.21.15.77:0/2849207128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f5e20df170 0x55f5e20ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f5472451740 1 -- 172.21.15.77:0/2849207128 >> 172.21.15.77:0/2849207128 conn(0x55f5e20da6e0 msgr2=0x55f5e20dd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f544cff9640 0 client.25331 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:10.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f5472451740 1 -- 172.21.15.77:0/2849207128 shutdown_connections 2024-03-16T10:17:10.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f5472451740 1 -- 172.21.15.77:0/2849207128 wait complete. 2024-03-16T10:17:10.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.676+0000 7f5472451740 20 client.25331 trim_cache size 0 max 16384 2024-03-16T10:17:10.684 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:17:10.684 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:17:10.684 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:17:10.684 INFO:teuthology.orchestra.run.smithi077.stdout: "5334ea35-26e4-46e7-909c-ed7bb337b4de": { 2024-03-16T10:17:10.684 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:17:10.685 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "5334ea35-26e4-46e7-909c-ed7bb337b4de", 2024-03-16T10:17:10.685 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:17:10.685 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:17:10.685 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:17:10.685 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:17:10.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.677+0000 7f546f861640 1 -- 172.21.15.77:0/2432376795 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f54500a2910 msgr2=0x7f54500a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.677+0000 7f546f861640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f54500a2910 0x7f54500a4dd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5460098360 tx=0x7f5460086510 comp rx=0 tx=0).stop 2024-03-16T10:17:10.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.677+0000 7f546f861640 1 -- 172.21.15.77:0/2432376795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5468107560 msgr2=0x7f54681007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:10.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.677+0000 7f546f861640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5468107560 0x7f54681007f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f546000eb10 tx=0x7f546000c2d0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.677+0000 7f546f861640 1 -- 172.21.15.77:0/2432376795 shutdown_connections 2024-03-16T10:17:10.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.677+0000 7f546f861640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5468107560 0x7f54681007f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.677+0000 7f546f861640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5468104c30 0x7f54681002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.677+0000 7f546f861640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f54500a2910 0x7f54500a4dd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.677+0000 7f546f861640 1 --2- 172.21.15.77:0/2432376795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5468102300 0x7f5468101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:10.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.677+0000 7f546f861640 1 -- 172.21.15.77:0/2432376795 >> 172.21.15.77:0/2432376795 conn(0x7f54680fbc70 msgr2=0x7f5468102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:10.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.677+0000 7f546f861640 1 -- 172.21.15.77:0/2432376795 shutdown_connections 2024-03-16T10:17:10.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:10.677+0000 7f546f861640 1 -- 172.21.15.77:0/2432376795 wait complete. 2024-03-16T10:17:10.690 INFO:tasks.cephfs.filesystem:scrub status for tag:5334ea35-26e4-46e7-909c-ed7bb337b4de - {'path': '/', 'tag': '5334ea35-26e4-46e7-909c-ed7bb337b4de', 'options': 'recursive,force'} 2024-03-16T10:17:11.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:10 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3528003975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:17:11.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:10 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2705807835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:17:11.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:10 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3528003975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:17:11.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:10 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2705807835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:17:11.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:10 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3528003975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:17:11.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:10 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2705807835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:17:12.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:11 smithi192 ceph-mon[36354]: scrub queued for path: / 2024-03-16T10:17:12.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:11 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:17:12.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:11 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:17:12.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:11 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:17:12.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:11 smithi192 ceph-mon[36354]: pgmap v353: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-16T10:17:12.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:11 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:17:12.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:11 smithi077 ceph-mon[32400]: scrub queued for path: / 2024-03-16T10:17:12.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:11 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:17:12.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:11 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:17:12.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:11 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:17:12.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:11 smithi077 ceph-mon[32400]: pgmap v353: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-16T10:17:12.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:11 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:17:12.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:11 smithi193 ceph-mon[36311]: scrub queued for path: / 2024-03-16T10:17:12.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:11 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:17:12.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:11 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:17:12.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:11 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:17:12.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:11 smithi193 ceph-mon[36311]: pgmap v353: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-16T10:17:12.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:11 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:17:14.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:13 smithi192 ceph-mon[36354]: pgmap v354: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-16T10:17:14.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:13 smithi077 ceph-mon[32400]: pgmap v354: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-16T10:17:14.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:13 smithi193 ceph-mon[36311]: pgmap v354: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-16T10:17:16.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:15 smithi192 ceph-mon[36354]: pgmap v355: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-03-16T10:17:16.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:15 smithi077 ceph-mon[32400]: pgmap v355: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-03-16T10:17:16.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:15 smithi193 ceph-mon[36311]: pgmap v355: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-03-16T10:17:18.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:17 smithi192 ceph-mon[36354]: pgmap v356: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-03-16T10:17:18.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:17 smithi077 ceph-mon[32400]: pgmap v356: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-03-16T10:17:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:17 smithi193 ceph-mon[36311]: pgmap v356: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-03-16T10:17:19.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:18 smithi077 ceph-mon[32400]: pgmap v357: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-03-16T10:17:19.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:18 smithi193 ceph-mon[36311]: pgmap v357: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-03-16T10:17:19.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:18 smithi192 ceph-mon[36354]: pgmap v357: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-03-16T10:17:20.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:19 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:17:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:19 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:17:20.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:19 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:17:21.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:20 smithi077 ceph-mon[32400]: pgmap v358: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-16T10:17:21.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:20 smithi193 ceph-mon[36311]: pgmap v358: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-16T10:17:21.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:20 smithi192 ceph-mon[36354]: pgmap v358: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-16T10:17:22.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:17:22.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:17:22.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:17:22.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:17:22.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:22.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:17:22.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:17:22.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.504+0000 7f75903cd640 1 -- 172.21.15.77:0/795534766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75881022c0 msgr2=0x7f7588108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.504+0000 7f75903cd640 1 --2- 172.21.15.77:0/795534766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75881022c0 0x7f7588108b50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7580009ca0 tx=0x7f758002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.504+0000 7f75903cd640 1 -- 172.21.15.77:0/795534766 shutdown_connections 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.504+0000 7f75903cd640 1 --2- 172.21.15.77:0/795534766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75881022c0 0x7f7588108b50 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.504+0000 7f75903cd640 1 --2- 172.21.15.77:0/795534766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7588101900 0x7f7588101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.504+0000 7f75903cd640 1 --2- 172.21.15.77:0/795534766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7588100710 0x7f7588100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.504+0000 7f75903cd640 1 -- 172.21.15.77:0/795534766 >> 172.21.15.77:0/795534766 conn(0x7f75880fbc80 msgr2=0x7f75880fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.504+0000 7f75903cd640 1 -- 172.21.15.77:0/795534766 shutdown_connections 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.504+0000 7f75903cd640 1 -- 172.21.15.77:0/795534766 wait complete. 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f75903cd640 1 Processor -- start 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f75903cd640 1 -- start start 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f75903cd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7588100710 0x7f758806ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f75903cd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7588101900 0x7f758819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f75903cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75881022c0 0x7f758819edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f75903cd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f758810c390 con 0x7f75881022c0 2024-03-16T10:17:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f75903cd640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f758810c210 con 0x7f7588100710 2024-03-16T10:17:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f75903cd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f758810c510 con 0x7f7588101900 2024-03-16T10:17:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f758e96a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75881022c0 0x7f758819edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f758e96a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75881022c0 0x7f758819edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39398/0 (socket says 172.21.15.77:39398) 2024-03-16T10:17:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f758e96a640 1 -- 172.21.15.77:0/1437890035 learned_addr learned my addr 172.21.15.77:0/1437890035 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f758e169640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7588100710 0x7f758806ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f758d968640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7588101900 0x7f758819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f758e96a640 1 -- 172.21.15.77:0/1437890035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7588101900 msgr2=0x7f758819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f758e96a640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7588101900 0x7f758819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f758e96a640 1 -- 172.21.15.77:0/1437890035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7588100710 msgr2=0x7f758806ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f758e96a640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7588100710 0x7f758806ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f758e96a640 1 -- 172.21.15.77:0/1437890035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f758819f700 con 0x7f75881022c0 2024-03-16T10:17:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.505+0000 7f758e96a640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75881022c0 0x7f758819edf0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7580037040 tx=0x7f7580032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.506+0000 7f757f7fe640 1 -- 172.21.15.77:0/1437890035 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7580041020 con 0x7f75881022c0 2024-03-16T10:17:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.506+0000 7f757f7fe640 1 -- 172.21.15.77:0/1437890035 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7580034cd0 con 0x7f75881022c0 2024-03-16T10:17:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.506+0000 7f757f7fe640 1 -- 172.21.15.77:0/1437890035 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7580039040 con 0x7f75881022c0 2024-03-16T10:17:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.506+0000 7f75903cd640 1 -- 172.21.15.77:0/1437890035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f758819f930 con 0x7f75881022c0 2024-03-16T10:17:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.506+0000 7f75903cd640 1 -- 172.21.15.77:0/1437890035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f758819fe70 con 0x7f75881022c0 2024-03-16T10:17:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f7592fb0740 1 Processor -- start 2024-03-16T10:17:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f757f7fe640 1 -- 172.21.15.77:0/1437890035 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f7580044020 con 0x7f75881022c0 2024-03-16T10:17:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f7592fb0740 1 -- start start 2024-03-16T10:17:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f7592fb0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141de340 0x5649141de760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f7592fb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 0x5649141e3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f7592fb0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141e3ea0 0x5649141e6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f7592fb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564914143a40 con 0x5649141ded30 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f7592fb0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5649140ec920 con 0x5649141de340 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f7592fb0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56491413fae0 con 0x5649141e3ea0 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f758e169640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 0x5649141e3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f758e169640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 0x5649141e3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39414/0 (socket says 172.21.15.77:39414) 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f758e169640 1 -- 172.21.15.77:0/3732445098 learned_addr learned my addr 172.21.15.77:0/3732445098 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f758d968640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141de340 0x5649141de760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.509+0000 7f758d968640 1 --2- 172.21.15.77:0/3732445098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141e3ea0 0x5649141e6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.510+0000 7f757f7fe640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f75740a29c0 0x7f75740a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.510+0000 7f758d968640 1 -- 172.21.15.77:0/3732445098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141e3ea0 msgr2=0x5649141e6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.510+0000 7f758d968640 1 --2- 172.21.15.77:0/3732445098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141e3ea0 0x5649141e6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.510+0000 7f758d968640 1 -- 172.21.15.77:0/3732445098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 msgr2=0x5649141e3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.510+0000 7f758d968640 1 --2- 172.21.15.77:0/3732445098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 0x5649141e3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.510+0000 7f758d968640 1 -- 172.21.15.77:0/3732445098 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564914104460 con 0x5649141de340 2024-03-16T10:17:23.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.510+0000 7f757f7fe640 1 -- 172.21.15.77:0/1437890035 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f75800f07e0 con 0x7f75881022c0 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.510+0000 7f758e96a640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f75740a29c0 0x7f75740a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.510+0000 7f758d968640 1 --2- 172.21.15.77:0/3732445098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141de340 0x5649141de760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f758806a900 tx=0x7f75780096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b207a8cde19af95a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.510+0000 7f757cbfe640 1 -- 172.21.15.77:0/3732445098 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7578004030 con 0x5649141de340 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.510+0000 7f757cbfe640 1 -- 172.21.15.77:0/3732445098 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75780041d0 con 0x5649141de340 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.510+0000 7f757cbfe640 1 -- 172.21.15.77:0/3732445098 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757800f6b0 con 0x5649141de340 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.511+0000 7f7592fb0740 1 -- 172.21.15.77:0/3732445098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141de340 msgr2=0x5649141de760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.511+0000 7f758e96a640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f75740a29c0 0x7f75740a4e80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7580005e30 tx=0x7f7580004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.511+0000 7f7592fb0740 1 --2- 172.21.15.77:0/3732445098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141de340 0x5649141de760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f758806a900 tx=0x7f75780096a0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.511+0000 7f7592fb0740 1 -- 172.21.15.77:0/3732445098 shutdown_connections 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.511+0000 7f7592fb0740 1 --2- 172.21.15.77:0/3732445098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141e3ea0 0x5649141e6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.511+0000 7f7592fb0740 1 --2- 172.21.15.77:0/3732445098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 0x5649141e3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.511+0000 7f7592fb0740 1 --2- 172.21.15.77:0/3732445098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141de340 0x5649141de760 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.511+0000 7f7592fb0740 1 -- 172.21.15.77:0/3732445098 >> 172.21.15.77:0/3732445098 conn(0x5649141d98b0 msgr2=0x5649141d9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.511+0000 7f7592fb0740 1 -- 172.21.15.77:0/3732445098 shutdown_connections 2024-03-16T10:17:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.511+0000 7f7592fb0740 1 -- 172.21.15.77:0/3732445098 wait complete. 2024-03-16T10:17:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f7592fb0740 1 -- start start 2024-03-16T10:17:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f7592fb0740 1 Processor -- start 2024-03-16T10:17:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f7592fb0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141de340 0x5649141e8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f7592fb0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141ded30 0x5649141e8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f7592fb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 0x5649141efc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f7592fb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5649141e6e50 con 0x5649141e3ea0 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f7592fb0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5649140ffd70 con 0x5649141de340 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f7592fb0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5649141e6fd0 con 0x5649141ded30 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f7592fb0740 10 client.0 _ll_register_callbacks cb 0x564913e57a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f758d968640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 0x5649141efc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f758d968640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 0x5649141efc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39426/0 (socket says 172.21.15.77:39426) 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f758e169640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141ded30 0x5649141e8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f758d968640 1 -- 172.21.15.77:0/3350772672 learned_addr learned my addr 172.21.15.77:0/3350772672 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f758d968640 1 --2- 172.21.15.77:0/3350772672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141de340 0x5649141e8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f758d968640 1 -- 172.21.15.77:0/3350772672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141ded30 msgr2=0x5649141e8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.512+0000 7f758d968640 1 --2- 172.21.15.77:0/3350772672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141ded30 0x5649141e8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f758d968640 1 -- 172.21.15.77:0/3350772672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141de340 msgr2=0x5649141e8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f758d968640 1 --2- 172.21.15.77:0/3350772672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141de340 0x5649141e8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f758d968640 1 -- 172.21.15.77:0/3350772672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564913e58210 con 0x5649141e3ea0 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f758d968640 1 --2- 172.21.15.77:0/3350772672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 0x5649141efc10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7578003af0 tx=0x7f7578009a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4451ea41aac2138 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f7571ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f7571ffb640 1 -- 172.21.15.77:0/3350772672 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7578004380 con 0x5649141e3ea0 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f7571ffb640 1 -- 172.21.15.77:0/3350772672 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f757802a040 con 0x5649141e3ea0 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f7592fb0740 1 -- 172.21.15.77:0/3350772672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75780041d0 con 0x5649141e3ea0 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f7571ffb640 1 -- 172.21.15.77:0/3350772672 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f757800f4b0 con 0x5649141e3ea0 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f7592fb0740 10 client.15537 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f7592fb0740 1 -- 172.21.15.77:0/3350772672 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564913f24620 con 0x5649141e3ea0 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f7571ffb640 1 -- 172.21.15.77:0/3350772672 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75780041d0 con 0x5649141e3ea0 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f7571ffb640 1 -- 172.21.15.77:0/3350772672 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f757802f3f0 con 0x5649141e3ea0 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.513+0000 7f7592fb0740 10 client.15537 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.514+0000 7f7592fb0740 10 client.15537 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.514+0000 7f7592fb0740 20 client.15537 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.514+0000 7f7592fb0740 1 --2- 172.21.15.77:0/3350772672 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5649141f42a0 0x5649141f6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.514+0000 7f7592fb0740 4 client.15537 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.514+0000 7f7592fb0740 1 -- 172.21.15.77:0/3350772672 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564914150f80 con 0x5649141f42a0 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.514+0000 7f758e169640 1 --2- 172.21.15.77:0/3350772672 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5649141f42a0 0x5649141f6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.514+0000 7f758e169640 1 --2- 172.21.15.77:0/3350772672 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5649141f42a0 0x5649141f6690 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7907cbc0f9866d2e server_cookie=ccea81c6bc35798f in_seq=0 out_seq=0 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.514+0000 7f7571ffb640 10 client.15537 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7571ffb640 1 -- 172.21.15.77:0/3350772672 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564914150f80 con 0x5649141f42a0 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7571ffb640 10 client.15537 handle_command_reply: tid=0 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7592fb0740 1 client.15537 shutdown 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7592fb0740 1 -- 172.21.15.77:0/3350772672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 msgr2=0x5649141efc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7592fb0740 1 --2- 172.21.15.77:0/3350772672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 0x5649141efc10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7578003af0 tx=0x7f7578009a40 comp rx=0 tx=0).stop 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7592fb0740 1 -- 172.21.15.77:0/3350772672 shutdown_connections 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7592fb0740 1 --2- 172.21.15.77:0/3350772672 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5649141f42a0 0x5649141f6690 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7592fb0740 1 --2- 172.21.15.77:0/3350772672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 0x5649141efc10 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7592fb0740 1 --2- 172.21.15.77:0/3350772672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141ded30 0x5649141e8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7592fb0740 1 --2- 172.21.15.77:0/3350772672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141de340 0x5649141e8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7592fb0740 1 -- 172.21.15.77:0/3350772672 >> 172.21.15.77:0/3350772672 conn(0x5649141d98b0 msgr2=0x5649141d8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7571ffb640 0 client.15537 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7592fb0740 1 -- 172.21.15.77:0/3350772672 shutdown_connections 2024-03-16T10:17:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7592fb0740 1 -- 172.21.15.77:0/3350772672 wait complete. 2024-03-16T10:17:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.515+0000 7f7592fb0740 20 client.15537 trim_cache size 0 max 16384 2024-03-16T10:17:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.543+0000 7f7592fb0740 1 Processor -- start 2024-03-16T10:17:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f7592fb0740 1 -- start start 2024-03-16T10:17:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f7592fb0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141de340 0x5649141d8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f7592fb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 0x5649141efb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f7592fb0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141e3ea0 0x5649141d8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f7592fb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f75881a8b60 con 0x5649141ded30 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f7592fb0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f75881aa6a0 con 0x5649141e3ea0 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f7592fb0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f75881a9230 con 0x5649141de340 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758e169640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 0x5649141efb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758d968640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141de340 0x5649141d8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758d968640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141de340 0x5649141d8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:42782/0 (socket says 172.21.15.77:42782) 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758d968640 1 -- 172.21.15.77:0/299864181 learned_addr learned my addr 172.21.15.77:0/299864181 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758e169640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 0x5649141efb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39430/0 (socket says 172.21.15.77:39430) 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758d968640 1 --2- 172.21.15.77:0/299864181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141e3ea0 0x5649141d8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758d968640 1 -- 172.21.15.77:0/299864181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141e3ea0 msgr2=0x5649141d8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758d968640 1 --2- 172.21.15.77:0/299864181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141e3ea0 0x5649141d8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758d968640 1 -- 172.21.15.77:0/299864181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 msgr2=0x5649141efb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758d968640 1 --2- 172.21.15.77:0/299864181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 0x5649141efb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758d968640 1 -- 172.21.15.77:0/299864181 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649141de720 con 0x5649141de340 2024-03-16T10:17:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758e169640 1 --2- 172.21.15.77:0/299864181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 0x5649141efb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.544+0000 7f758d968640 1 --2- 172.21.15.77:0/299864181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141de340 0x5649141d8320 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7578003c70 tx=0x7f7578005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a09a7f0f7bb3b06 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.545+0000 7f7573fff640 1 -- 172.21.15.77:0/299864181 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7578023040 con 0x5649141de340 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.545+0000 7f7573fff640 1 -- 172.21.15.77:0/299864181 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7578005500 con 0x5649141de340 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.545+0000 7f7573fff640 1 -- 172.21.15.77:0/299864181 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7578003fb0 con 0x5649141de340 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.545+0000 7f7592fb0740 1 -- 172.21.15.77:0/299864181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141de340 msgr2=0x5649141d8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.545+0000 7f7592fb0740 1 --2- 172.21.15.77:0/299864181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141de340 0x5649141d8320 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7578003c70 tx=0x7f7578005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.545+0000 7f7592fb0740 1 -- 172.21.15.77:0/299864181 shutdown_connections 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.545+0000 7f7592fb0740 1 --2- 172.21.15.77:0/299864181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141e3ea0 0x5649141d8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.545+0000 7f7592fb0740 1 --2- 172.21.15.77:0/299864181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141ded30 0x5649141efb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.545+0000 7f7592fb0740 1 --2- 172.21.15.77:0/299864181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141de340 0x5649141d8320 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.545+0000 7f7592fb0740 1 -- 172.21.15.77:0/299864181 >> 172.21.15.77:0/299864181 conn(0x5649141d98b0 msgr2=0x5649141e0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f7592fb0740 1 -- 172.21.15.77:0/299864181 shutdown_connections 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f7592fb0740 1 -- 172.21.15.77:0/299864181 wait complete. 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f7592fb0740 1 -- start start 2024-03-16T10:17:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f7592fb0740 1 Processor -- start 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f7592fb0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141de340 0x5649141ecc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f7592fb0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141ded30 0x5649141ed160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f7592fb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 0x564914258570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f7592fb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f75881a9fd0 con 0x5649141e3ea0 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f7592fb0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f75881a9900 con 0x5649141ded30 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f7592fb0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f75881a6b40 con 0x5649141de340 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f7592fb0740 10 client.0 _ll_register_callbacks cb 0x7f75840097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f758d968640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141de340 0x5649141ecc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f758d968640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 0x564914258570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f758d968640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 0x564914258570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39446/0 (socket says 172.21.15.77:39446) 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f758e169640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141ded30 0x5649141ed160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.546+0000 7f758d968640 1 -- 172.21.15.77:0/641682854 learned_addr learned my addr 172.21.15.77:0/641682854 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f758d968640 1 -- 172.21.15.77:0/641682854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141de340 msgr2=0x5649141ecc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f758d968640 1 --2- 172.21.15.77:0/641682854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141de340 0x5649141ecc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f758d968640 1 -- 172.21.15.77:0/641682854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141ded30 msgr2=0x5649141ed160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f758d968640 1 --2- 172.21.15.77:0/641682854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141ded30 0x5649141ed160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f758d968640 1 -- 172.21.15.77:0/641682854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564914258ae0 con 0x5649141e3ea0 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f758d968640 1 --2- 172.21.15.77:0/641682854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 0x564914258570 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f757802f9a0 tx=0x7f7578003ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b0665a9561a159 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f7571ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f7571ffb640 1 -- 172.21.15.77:0/641682854 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7578023040 con 0x5649141e3ea0 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f7571ffb640 1 -- 172.21.15.77:0/641682854 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7578042040 con 0x5649141e3ea0 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f7592fb0740 1 -- 172.21.15.77:0/641682854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7578005500 con 0x5649141e3ea0 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f7571ffb640 1 -- 172.21.15.77:0/641682854 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7578005070 con 0x5649141e3ea0 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f7592fb0740 10 client.15543 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.547+0000 7f7592fb0740 1 -- 172.21.15.77:0/641682854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564914259df0 con 0x5649141e3ea0 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.548+0000 7f7571ffb640 1 -- 172.21.15.77:0/641682854 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7578005500 con 0x5649141e3ea0 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.548+0000 7f7571ffb640 1 -- 172.21.15.77:0/641682854 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7578048470 con 0x5649141e3ea0 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.548+0000 7f7592fb0740 10 client.15543 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.548+0000 7f7592fb0740 10 client.15543 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.548+0000 7f7592fb0740 20 client.15543 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.548+0000 7f7592fb0740 1 --2- 172.21.15.77:0/641682854 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56491425a770 0x56491425cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.548+0000 7f7592fb0740 4 client.15543 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:17:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.548+0000 7f7592fb0740 1 -- 172.21.15.77:0/641682854 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56491414b2d0 con 0x56491425a770 2024-03-16T10:17:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.548+0000 7f758e169640 1 --2- 172.21.15.77:0/641682854 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56491425a770 0x56491425cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.548+0000 7f758e169640 1 --2- 172.21.15.77:0/641682854 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56491425a770 0x56491425cb90 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a5469b04e7e485e server_cookie=b54b2ccfdfac2b47 in_seq=0 out_seq=0 2024-03-16T10:17:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.548+0000 7f7571ffb640 10 client.15543 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7571ffb640 1 -- 172.21.15.77:0/641682854 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56491414b2d0 con 0x56491425a770 2024-03-16T10:17:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7571ffb640 10 client.15543 handle_command_reply: tid=0 2024-03-16T10:17:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7592fb0740 1 client.15543 shutdown 2024-03-16T10:17:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7592fb0740 1 -- 172.21.15.77:0/641682854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 msgr2=0x564914258570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7592fb0740 1 --2- 172.21.15.77:0/641682854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 0x564914258570 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f757802f9a0 tx=0x7f7578003ca0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7592fb0740 1 -- 172.21.15.77:0/641682854 shutdown_connections 2024-03-16T10:17:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7592fb0740 1 --2- 172.21.15.77:0/641682854 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56491425a770 0x56491425cb90 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7592fb0740 1 --2- 172.21.15.77:0/641682854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5649141e3ea0 0x564914258570 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7592fb0740 1 --2- 172.21.15.77:0/641682854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5649141ded30 0x5649141ed160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7592fb0740 1 --2- 172.21.15.77:0/641682854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649141de340 0x5649141ecc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7592fb0740 1 -- 172.21.15.77:0/641682854 >> 172.21.15.77:0/641682854 conn(0x5649141d98b0 msgr2=0x5649141dcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7571ffb640 0 client.15543 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7592fb0740 1 -- 172.21.15.77:0/641682854 shutdown_connections 2024-03-16T10:17:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.558+0000 7f7592fb0740 1 -- 172.21.15.77:0/641682854 wait complete. 2024-03-16T10:17:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f7592fb0740 20 client.15543 trim_cache size 0 max 16384 2024-03-16T10:17:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:17:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:17:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:17:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:17:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f75903cd640 1 -- 172.21.15.77:0/1437890035 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f75740a29c0 msgr2=0x7f75740a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f75903cd640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f75740a29c0 0x7f75740a4e80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7580005e30 tx=0x7f7580004a40 comp rx=0 tx=0).stop 2024-03-16T10:17:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f75903cd640 1 -- 172.21.15.77:0/1437890035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75881022c0 msgr2=0x7f758819edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f75903cd640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75881022c0 0x7f758819edf0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7580037040 tx=0x7f7580032f40 comp rx=0 tx=0).stop 2024-03-16T10:17:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f75903cd640 1 -- 172.21.15.77:0/1437890035 shutdown_connections 2024-03-16T10:17:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f75903cd640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75881022c0 0x7f758819edf0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f75903cd640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f75740a29c0 0x7f75740a4e80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f75903cd640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7588101900 0x7f758819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f75903cd640 1 --2- 172.21.15.77:0/1437890035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7588100710 0x7f758806ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f75903cd640 1 -- 172.21.15.77:0/1437890035 >> 172.21.15.77:0/1437890035 conn(0x7f75880fbc80 msgr2=0x7f7588104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f75903cd640 1 -- 172.21.15.77:0/1437890035 shutdown_connections 2024-03-16T10:17:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:23.559+0000 7f75903cd640 1 -- 172.21.15.77:0/1437890035 wait complete. 2024-03-16T10:17:23.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:23 smithi192 ceph-mon[36354]: pgmap v359: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-16T10:17:23.744 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:23 smithi077 ceph-mon[32400]: pgmap v359: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-16T10:17:23.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:23 smithi193 ceph-mon[36311]: pgmap v359: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-16T10:17:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:25 smithi192 ceph-mon[36354]: pgmap v360: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-16T10:17:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:25 smithi077 ceph-mon[32400]: pgmap v360: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-16T10:17:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:25 smithi193 ceph-mon[36311]: pgmap v360: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-16T10:17:28.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:27 smithi192 ceph-mon[36354]: pgmap v361: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-16T10:17:28.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:27 smithi077 ceph-mon[32400]: pgmap v361: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-16T10:17:28.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:27 smithi193 ceph-mon[36311]: pgmap v361: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-16T10:17:30.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:29 smithi192 ceph-mon[36354]: pgmap v362: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-03-16T10:17:30.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:29 smithi077 ceph-mon[32400]: pgmap v362: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-03-16T10:17:30.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:29 smithi193 ceph-mon[36311]: pgmap v362: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-03-16T10:17:32.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:31 smithi192 ceph-mon[36354]: pgmap v363: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-16T10:17:32.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:31 smithi077 ceph-mon[32400]: pgmap v363: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-16T10:17:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:31 smithi193 ceph-mon[36311]: pgmap v363: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-16T10:17:34.089 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:33 smithi077 ceph-mon[32400]: pgmap v364: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-16T10:17:34.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:33 smithi192 ceph-mon[36354]: pgmap v364: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-16T10:17:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:33 smithi193 ceph-mon[36311]: pgmap v364: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-16T10:17:36.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:35 smithi192 ceph-mon[36354]: pgmap v365: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-03-16T10:17:36.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:35 smithi077 ceph-mon[32400]: pgmap v365: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-03-16T10:17:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:35 smithi193 ceph-mon[36311]: pgmap v365: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-03-16T10:17:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.645+0000 7f8e80881640 1 -- 172.21.15.77:0/3800592124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e7c1018f0 msgr2=0x7f8e7c101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.645+0000 7f8e80881640 1 --2- 172.21.15.77:0/3800592124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e7c1018f0 0x7f8e7c101d70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8e64009900 tx=0x7f8e6402bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.646+0000 7f8e80881640 1 -- 172.21.15.77:0/3800592124 shutdown_connections 2024-03-16T10:17:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.646+0000 7f8e80881640 1 --2- 172.21.15.77:0/3800592124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e7c1022b0 0x7f8e7c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.646+0000 7f8e80881640 1 --2- 172.21.15.77:0/3800592124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e7c1018f0 0x7f8e7c101d70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.646+0000 7f8e80881640 1 --2- 172.21.15.77:0/3800592124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e7c100700 0x7f8e7c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.646+0000 7f8e80881640 1 -- 172.21.15.77:0/3800592124 >> 172.21.15.77:0/3800592124 conn(0x7f8e7c0fbc70 msgr2=0x7f8e7c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.646+0000 7f8e80881640 1 -- 172.21.15.77:0/3800592124 shutdown_connections 2024-03-16T10:17:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.646+0000 7f8e80881640 1 -- 172.21.15.77:0/3800592124 wait complete. 2024-03-16T10:17:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.646+0000 7f8e80881640 1 Processor -- start 2024-03-16T10:17:36.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.646+0000 7f8e80881640 1 -- start start 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.646+0000 7f8e80881640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e7c100700 0x7f8e7c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.646+0000 7f8e80881640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e7c1018f0 0x7f8e7c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e80881640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e7c1022b0 0x7f8e7c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e80881640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c10d4a0 con 0x7f8e7c100700 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e80881640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c10d320 con 0x7f8e7c1022b0 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e80881640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c10d620 con 0x7f8e7c1018f0 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e7a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e7c100700 0x7f8e7c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e7a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e7c100700 0x7f8e7c197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47724/0 (socket says 172.21.15.77:47724) 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e7a59c640 1 -- 172.21.15.77:0/686468513 learned_addr learned my addr 172.21.15.77:0/686468513 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e79d9b640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e7c1018f0 0x7f8e7c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e7ad9d640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e7c1022b0 0x7f8e7c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e7a59c640 1 -- 172.21.15.77:0/686468513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e7c1018f0 msgr2=0x7f8e7c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:36.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e7a59c640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e7c1018f0 0x7f8e7c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e7a59c640 1 -- 172.21.15.77:0/686468513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e7c1022b0 msgr2=0x7f8e7c19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e7a59c640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e7c1022b0 0x7f8e7c19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e7a59c640 1 -- 172.21.15.77:0/686468513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7c19f670 con 0x7f8e7c100700 2024-03-16T10:17:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e7a59c640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e7c100700 0x7f8e7c197a50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400ebc0 tx=0x7f8e7400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e737fe640 1 -- 172.21.15.77:0/686468513 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74003b70 con 0x7f8e7c100700 2024-03-16T10:17:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e737fe640 1 -- 172.21.15.77:0/686468513 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e74003d10 con 0x7f8e7c100700 2024-03-16T10:17:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e80881640 1 -- 172.21.15.77:0/686468513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7c19f960 con 0x7f8e7c100700 2024-03-16T10:17:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e737fe640 1 -- 172.21.15.77:0/686468513 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74019980 con 0x7f8e7c100700 2024-03-16T10:17:36.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.647+0000 7f8e80881640 1 -- 172.21.15.77:0/686468513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e7c19fed0 con 0x7f8e7c100700 2024-03-16T10:17:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e8346b740 1 Processor -- start 2024-03-16T10:17:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e8346b740 1 -- start start 2024-03-16T10:17:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e8346b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d145340 0x559f7d145760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e8346b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145d30 0x559f7d14a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e8346b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d14aea0 0x559f7d14d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e8346b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559f7d0aaa40 con 0x559f7d145d30 2024-03-16T10:17:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e8346b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559f7d053920 con 0x559f7d145340 2024-03-16T10:17:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e8346b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559f7d0a6ae0 con 0x559f7d14aea0 2024-03-16T10:17:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e737fe640 1 -- 172.21.15.77:0/686468513 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f8e74019b20 con 0x7f8e7c100700 2024-03-16T10:17:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e7a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145d30 0x559f7d14a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e79d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d14aea0 0x559f7d14d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e7ad9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d145340 0x559f7d145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e79d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d14aea0 0x559f7d14d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:37670/0 (socket says 172.21.15.77:37670) 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.650+0000 7f8e79d9b640 1 -- 172.21.15.77:0/1966506346 learned_addr learned my addr 172.21.15.77:0/1966506346 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e79d9b640 1 -- 172.21.15.77:0/1966506346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d145340 msgr2=0x559f7d145760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e79d9b640 1 --2- 172.21.15.77:0/1966506346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d145340 0x559f7d145760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e79d9b640 1 -- 172.21.15.77:0/1966506346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145d30 msgr2=0x559f7d14a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e79d9b640 1 --2- 172.21.15.77:0/1966506346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145d30 0x559f7d14a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e79d9b640 1 -- 172.21.15.77:0/1966506346 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f7d06b460 con 0x559f7d14aea0 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e7ad9d640 1 --2- 172.21.15.77:0/1966506346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d145340 0x559f7d145760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e79d9b640 1 --2- 172.21.15.77:0/1966506346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d14aea0 0x559f7d14d330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8e64009900 tx=0x7f8e64009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c71a5d280b65dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e737fe640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8e680a29c0 0x7f8e680a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e70ff9640 1 -- 172.21.15.77:0/1966506346 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6402cb80 con 0x559f7d14aea0 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e70ff9640 1 -- 172.21.15.77:0/1966506346 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e64005040 con 0x559f7d14aea0 2024-03-16T10:17:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e70ff9640 1 -- 172.21.15.77:0/1966506346 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e64004900 con 0x559f7d14aea0 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e7ad9d640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8e680a29c0 0x7f8e680a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.651+0000 7f8e737fe640 1 -- 172.21.15.77:0/686468513 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8e740cf320 con 0x7f8e7c100700 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.652+0000 7f8e8346b740 1 -- 172.21.15.77:0/1966506346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d14aea0 msgr2=0x559f7d14d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.652+0000 7f8e7ad9d640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8e680a29c0 0x7f8e680a4e80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x559f7d146590 tx=0x7f8e6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.652+0000 7f8e8346b740 1 --2- 172.21.15.77:0/1966506346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d14aea0 0x559f7d14d330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8e64009900 tx=0x7f8e64009560 comp rx=0 tx=0).stop 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.652+0000 7f8e8346b740 1 -- 172.21.15.77:0/1966506346 shutdown_connections 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.652+0000 7f8e8346b740 1 --2- 172.21.15.77:0/1966506346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d14aea0 0x559f7d14d330 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.652+0000 7f8e8346b740 1 --2- 172.21.15.77:0/1966506346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145d30 0x559f7d14a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.652+0000 7f8e8346b740 1 --2- 172.21.15.77:0/1966506346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d145340 0x559f7d145760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.652+0000 7f8e8346b740 1 -- 172.21.15.77:0/1966506346 >> 172.21.15.77:0/1966506346 conn(0x559f7d1408b0 msgr2=0x559f7d140cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.652+0000 7f8e8346b740 1 -- 172.21.15.77:0/1966506346 shutdown_connections 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.652+0000 7f8e8346b740 1 -- 172.21.15.77:0/1966506346 wait complete. 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.652+0000 7f8e8346b740 1 -- start start 2024-03-16T10:17:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.652+0000 7f8e8346b740 1 Processor -- start 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e8346b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d14f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e8346b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d145d30 0x559f7d14fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e8346b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d14aea0 0x559f7d156b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e8346b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559f7d14ddf0 con 0x559f7d145340 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e8346b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559f7d066d70 con 0x559f7d145d30 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e8346b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559f7d14df70 con 0x559f7d14aea0 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e8346b740 10 client.0 _ll_register_callbacks cb 0x559f7cdbea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e7a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d14f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e7a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d14f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47736/0 (socket says 172.21.15.77:47736) 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e7a59c640 1 -- 172.21.15.77:0/1410245740 learned_addr learned my addr 172.21.15.77:0/1410245740 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e7ad9d640 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d14aea0 0x559f7d156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e79d9b640 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d145d30 0x559f7d14fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e7a59c640 1 -- 172.21.15.77:0/1410245740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d14aea0 msgr2=0x559f7d156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e7a59c640 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d14aea0 0x559f7d156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e7a59c640 1 -- 172.21.15.77:0/1410245740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d145d30 msgr2=0x559f7d14fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e7a59c640 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d145d30 0x559f7d14fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e7a59c640 1 -- 172.21.15.77:0/1410245740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f7cdbf210 con 0x559f7d145340 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e7a59c640 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d14f610 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8e7409b890 tx=0x7f8e740096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa64505bb636e72 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e61ffb640 1 -- 172.21.15.77:0/1410245740 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7409b960 con 0x559f7d145340 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e61ffb640 1 -- 172.21.15.77:0/1410245740 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8e74004b60 con 0x559f7d145340 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e8346b740 1 -- 172.21.15.77:0/1410245740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e64005040 con 0x559f7d145340 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e61ffb640 1 -- 172.21.15.77:0/1410245740 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8e7409b4f0 con 0x559f7d145340 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e8346b740 10 client.15555 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.653+0000 7f8e8346b740 1 -- 172.21.15.77:0/1410245740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559f7ce8b620 con 0x559f7d145340 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.654+0000 7f8e61ffb640 1 -- 172.21.15.77:0/1410245740 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e64005040 con 0x559f7d145340 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.654+0000 7f8e61ffb640 1 -- 172.21.15.77:0/1410245740 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8e7409afe0 con 0x559f7d145340 2024-03-16T10:17:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.654+0000 7f8e8346b740 10 client.15555 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.654+0000 7f8e8346b740 10 client.15555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.654+0000 7f8e8346b740 20 client.15555 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.654+0000 7f8e8346b740 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559f7d15b520 0x559f7d15d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.654+0000 7f8e8346b740 4 client.15555 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.654+0000 7f8e8346b740 1 -- 172.21.15.77:0/1410245740 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f7d0b7f80 con 0x559f7d15b520 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.654+0000 7f8e79d9b640 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559f7d15b520 0x559f7d15d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.654+0000 7f8e61ffb640 10 client.15555 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.654+0000 7f8e79d9b640 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559f7d15b520 0x559f7d15d970 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86bd9b58f94fc87b server_cookie=35ff79e1573b2c9 in_seq=0 out_seq=0 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e61ffb640 1 -- 172.21.15.77:0/1410245740 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559f7d0b7f80 con 0x559f7d15b520 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e61ffb640 10 client.15555 handle_command_reply: tid=0 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e8346b740 1 client.15555 shutdown 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e8346b740 1 -- 172.21.15.77:0/1410245740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 msgr2=0x559f7d14f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e8346b740 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d14f610 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8e7409b890 tx=0x7f8e740096c0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e8346b740 1 -- 172.21.15.77:0/1410245740 shutdown_connections 2024-03-16T10:17:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e8346b740 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d14aea0 0x559f7d156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e8346b740 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d145d30 0x559f7d14fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e8346b740 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559f7d15b520 0x559f7d15d970 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e8346b740 1 --2- 172.21.15.77:0/1410245740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d14f610 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e61ffb640 0 client.15555 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e8346b740 1 -- 172.21.15.77:0/1410245740 >> 172.21.15.77:0/1410245740 conn(0x559f7d1408b0 msgr2=0x559f7d13fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e8346b740 1 -- 172.21.15.77:0/1410245740 shutdown_connections 2024-03-16T10:17:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e8346b740 1 -- 172.21.15.77:0/1410245740 wait complete. 2024-03-16T10:17:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.655+0000 7f8e8346b740 20 client.15555 trim_cache size 0 max 16384 2024-03-16T10:17:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e8346b740 1 Processor -- start 2024-03-16T10:17:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e8346b740 1 -- start start 2024-03-16T10:17:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e8346b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d144180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e8346b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d145d30 0x559f7d14fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e8346b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d14aea0 0x559f7d156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e8346b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1a89f0 con 0x559f7d145340 2024-03-16T10:17:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e8346b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1aa530 con 0x559f7d14aea0 2024-03-16T10:17:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e8346b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1a90c0 con 0x559f7d145d30 2024-03-16T10:17:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e7a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e7a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d144180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47752/0 (socket says 172.21.15.77:47752) 2024-03-16T10:17:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e7a59c640 1 -- 172.21.15.77:0/2128731267 learned_addr learned my addr 172.21.15.77:0/2128731267 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e79d9b640 1 --2- 172.21.15.77:0/2128731267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d145d30 0x559f7d14fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.670+0000 7f8e7ad9d640 1 --2- 172.21.15.77:0/2128731267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d14aea0 0x559f7d156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e7a59c640 1 -- 172.21.15.77:0/2128731267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d145d30 msgr2=0x559f7d14fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e7a59c640 1 --2- 172.21.15.77:0/2128731267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d145d30 0x559f7d14fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e7a59c640 1 -- 172.21.15.77:0/2128731267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d14aea0 msgr2=0x559f7d156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e7a59c640 1 --2- 172.21.15.77:0/2128731267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d14aea0 0x559f7d156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e7a59c640 1 -- 172.21.15.77:0/2128731267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f7d14a2e0 con 0x559f7d145340 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e7ad9d640 1 --2- 172.21.15.77:0/2128731267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d14aea0 0x559f7d156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e7a59c640 1 --2- 172.21.15.77:0/2128731267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d144180 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8e740048c0 tx=0x7f8e7400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289e07d41bb06c3e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e63fff640 1 -- 172.21.15.77:0/2128731267 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7409b6d0 con 0x559f7d145340 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e63fff640 1 -- 172.21.15.77:0/2128731267 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e7409c290 con 0x559f7d145340 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e63fff640 1 -- 172.21.15.77:0/2128731267 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7409fce0 con 0x559f7d145340 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e8346b740 1 -- 172.21.15.77:0/2128731267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 msgr2=0x559f7d144180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e8346b740 1 --2- 172.21.15.77:0/2128731267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d144180 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8e740048c0 tx=0x7f8e7400cf90 comp rx=0 tx=0).stop 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.671+0000 7f8e8346b740 1 -- 172.21.15.77:0/2128731267 shutdown_connections 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 --2- 172.21.15.77:0/2128731267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d14aea0 0x559f7d156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 --2- 172.21.15.77:0/2128731267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d145d30 0x559f7d14fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 --2- 172.21.15.77:0/2128731267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d144180 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 -- 172.21.15.77:0/2128731267 >> 172.21.15.77:0/2128731267 conn(0x559f7d1408b0 msgr2=0x559f7d14e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 -- 172.21.15.77:0/2128731267 shutdown_connections 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 -- 172.21.15.77:0/2128731267 wait complete. 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 -- start start 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 Processor -- start 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d154880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d145d30 0x559f7d154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d14aea0 0x559f7d155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1a9e60 con 0x559f7d145340 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1a9790 con 0x559f7d14aea0 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1a69d0 con 0x559f7d145d30 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e8346b740 10 client.0 _ll_register_callbacks cb 0x7f8e74004810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e7a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e7a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d154880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47756/0 (socket says 172.21.15.77:47756) 2024-03-16T10:17:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e7a59c640 1 -- 172.21.15.77:0/3294447090 learned_addr learned my addr 172.21.15.77:0/3294447090 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e79d9b640 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d145d30 0x559f7d154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e7ad9d640 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d14aea0 0x559f7d155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e7a59c640 1 -- 172.21.15.77:0/3294447090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d145d30 msgr2=0x559f7d154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e7a59c640 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d145d30 0x559f7d154dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e7a59c640 1 -- 172.21.15.77:0/3294447090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d14aea0 msgr2=0x559f7d155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e7a59c640 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d14aea0 0x559f7d155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.672+0000 7f8e7a59c640 1 -- 172.21.15.77:0/3294447090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f7d1563c0 con 0x559f7d145340 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e7a59c640 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d154880 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8e740d4040 tx=0x7f8e740ccdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12043ef2e170df60 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e61ffb640 1 -- 172.21.15.77:0/3294447090 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7409b6d0 con 0x559f7d145340 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e61ffb640 1 -- 172.21.15.77:0/3294447090 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8e740978e0 con 0x559f7d145340 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e8346b740 1 -- 172.21.15.77:0/3294447090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e7409c290 con 0x559f7d145340 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e61ffb640 1 -- 172.21.15.77:0/3294447090 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8e7409b960 con 0x559f7d145340 2024-03-16T10:17:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e8346b740 10 client.15561 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e8346b740 1 -- 172.21.15.77:0/3294447090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559f7d1563c0 con 0x559f7d145340 2024-03-16T10:17:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e61ffb640 1 -- 172.21.15.77:0/3294447090 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e7409c290 con 0x559f7d145340 2024-03-16T10:17:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e61ffb640 1 -- 172.21.15.77:0/3294447090 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8e740a0c00 con 0x559f7d145340 2024-03-16T10:17:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e8346b740 10 client.15561 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e8346b740 10 client.15561 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e8346b740 20 client.15561 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e8346b740 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559f7d1c2400 0x559f7d1c47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e8346b740 4 client.15561 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:17:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e8346b740 1 -- 172.21.15.77:0/3294447090 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559f7d0b22d0 con 0x559f7d1c2400 2024-03-16T10:17:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.673+0000 7f8e79d9b640 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559f7d1c2400 0x559f7d1c47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.674+0000 7f8e79d9b640 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559f7d1c2400 0x559f7d1c47f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a7a86cb9a661a41 server_cookie=4b91b4d167a6cd2c in_seq=0 out_seq=0 2024-03-16T10:17:36.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.674+0000 7f8e61ffb640 10 client.15561 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e61ffb640 1 -- 172.21.15.77:0/3294447090 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559f7d0b22d0 con 0x559f7d1c2400 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e61ffb640 10 client.15561 handle_command_reply: tid=0 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e8346b740 1 client.15561 shutdown 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e8346b740 1 -- 172.21.15.77:0/3294447090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 msgr2=0x559f7d154880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e8346b740 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d154880 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8e740d4040 tx=0x7f8e740ccdb0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e8346b740 1 -- 172.21.15.77:0/3294447090 shutdown_connections 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e8346b740 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559f7d14aea0 0x559f7d155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e8346b740 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f7d145d30 0x559f7d154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e8346b740 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559f7d1c2400 0x559f7d1c47f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e8346b740 1 --2- 172.21.15.77:0/3294447090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559f7d145340 0x559f7d154880 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e8346b740 1 -- 172.21.15.77:0/3294447090 >> 172.21.15.77:0/3294447090 conn(0x559f7d1408b0 msgr2=0x559f7d14f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e61ffb640 0 client.15561 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:36.682 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e8346b740 1 -- 172.21.15.77:0/3294447090 shutdown_connections 2024-03-16T10:17:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e8346b740 1 -- 172.21.15.77:0/3294447090 wait complete. 2024-03-16T10:17:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e8346b740 20 client.15561 trim_cache size 0 max 16384 2024-03-16T10:17:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:17:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:17:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:17:36.683 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:17:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e80881640 1 -- 172.21.15.77:0/686468513 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8e680a29c0 msgr2=0x7f8e680a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e80881640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8e680a29c0 0x7f8e680a4e80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x559f7d146590 tx=0x7f8e6c005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e80881640 1 -- 172.21.15.77:0/686468513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e7c100700 msgr2=0x7f8e7c197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e80881640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e7c100700 0x7f8e7c197a50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400ebc0 tx=0x7f8e7400c6a0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e80881640 1 -- 172.21.15.77:0/686468513 shutdown_connections 2024-03-16T10:17:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e80881640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e7c1022b0 0x7f8e7c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e80881640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8e680a29c0 0x7f8e680a4e80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e80881640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e7c1018f0 0x7f8e7c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e80881640 1 --2- 172.21.15.77:0/686468513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e7c100700 0x7f8e7c197a50 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e80881640 1 -- 172.21.15.77:0/686468513 >> 172.21.15.77:0/686468513 conn(0x7f8e7c0fbc70 msgr2=0x7f8e7c104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e80881640 1 -- 172.21.15.77:0/686468513 shutdown_connections 2024-03-16T10:17:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:36.677+0000 7f8e80881640 1 -- 172.21.15.77:0/686468513 wait complete. 2024-03-16T10:17:38.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:37 smithi192 ceph-mon[36354]: pgmap v366: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-16T10:17:38.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:37 smithi077 ceph-mon[32400]: pgmap v366: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-16T10:17:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:37 smithi193 ceph-mon[36311]: pgmap v366: 97 pgs: 97 active+clean; 658 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-16T10:17:40.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:39 smithi192 ceph-mon[36354]: pgmap v367: 97 pgs: 97 active+clean; 659 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-03-16T10:17:40.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:39 smithi077 ceph-mon[32400]: pgmap v367: 97 pgs: 97 active+clean; 659 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-03-16T10:17:40.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:39 smithi193 ceph-mon[36311]: pgmap v367: 97 pgs: 97 active+clean; 659 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-03-16T10:17:40.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:17:40.691 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:17:40.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 -- 172.21.15.77:0/3144637861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1801006e0 msgr2=0x7fa180100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 --2- 172.21.15.77:0/3144637861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1801006e0 0x7fa180100b00 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa174009900 tx=0x7fa17402bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 -- 172.21.15.77:0/3144637861 shutdown_connections 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 --2- 172.21.15.77:0/3144637861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa180102290 0x7fa180108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 --2- 172.21.15.77:0/3144637861 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1801018d0 0x7fa180101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 --2- 172.21.15.77:0/3144637861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1801006e0 0x7fa180100b00 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 -- 172.21.15.77:0/3144637861 >> 172.21.15.77:0/3144637861 conn(0x7fa1800fbc70 msgr2=0x7fa1800fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 -- 172.21.15.77:0/3144637861 shutdown_connections 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 -- 172.21.15.77:0/3144637861 wait complete. 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 Processor -- start 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 -- start start 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1801006e0 0x7fa180196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1801018d0 0x7fa180196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa180102290 0x7fa18019dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa18010c360 con 0x7fa180102290 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa18010c1e0 con 0x7fa1801006e0 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.805+0000 7fa186abf640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa18010c4e0 con 0x7fa1801018d0 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa18505c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa180102290 0x7fa18019dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa18485b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1801006e0 0x7fa180196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa18505c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa180102290 0x7fa18019dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47774/0 (socket says 172.21.15.77:47774) 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa18505c640 1 -- 172.21.15.77:0/4038963945 learned_addr learned my addr 172.21.15.77:0/4038963945 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa17ffff640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1801018d0 0x7fa180196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa18505c640 1 -- 172.21.15.77:0/4038963945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1801018d0 msgr2=0x7fa180196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa18505c640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1801018d0 0x7fa180196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa18505c640 1 -- 172.21.15.77:0/4038963945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1801006e0 msgr2=0x7fa180196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa18505c640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1801006e0 0x7fa180196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa18505c640 1 -- 172.21.15.77:0/4038963945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa18019e640 con 0x7fa180102290 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa18505c640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa180102290 0x7fa18019dfd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa17800eb10 tx=0x7fa17800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa17dffb640 1 -- 172.21.15.77:0/4038963945 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa178004b20 con 0x7fa180102290 2024-03-16T10:17:40.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa17dffb640 1 -- 172.21.15.77:0/4038963945 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa178004cc0 con 0x7fa180102290 2024-03-16T10:17:40.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa17dffb640 1 -- 172.21.15.77:0/4038963945 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa178010760 con 0x7fa180102290 2024-03-16T10:17:40.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.806+0000 7fa186abf640 1 -- 172.21.15.77:0/4038963945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa18019e930 con 0x7fa180102290 2024-03-16T10:17:40.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.807+0000 7fa186abf640 1 -- 172.21.15.77:0/4038963945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa18019eed0 con 0x7fa180102290 2024-03-16T10:17:40.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.809+0000 7fa17dffb640 1 -- 172.21.15.77:0/4038963945 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fa178005020 con 0x7fa180102290 2024-03-16T10:17:40.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.809+0000 7fa189788740 1 Processor -- start 2024-03-16T10:17:40.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.809+0000 7fa189788740 1 -- start start 2024-03-16T10:17:40.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.809+0000 7fa189788740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366170 0x5558c3366590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa189788740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366b60 0x5558c336b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa189788740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c336bcd0 0x5558c336e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa189788740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5558c32cb500 con 0x5558c336bcd0 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa189788740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5558c3273be0 con 0x5558c3366170 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa189788740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5558c32c75a0 con 0x5558c3366b60 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa17ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c336bcd0 0x5558c336e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa18485b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366b60 0x5558c336b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa17ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c336bcd0 0x5558c336e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47776/0 (socket says 172.21.15.77:47776) 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa18485b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366b60 0x5558c336b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:37726/0 (socket says 172.21.15.77:37726) 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa17ffff640 1 -- 172.21.15.77:0/3770782891 learned_addr learned my addr 172.21.15.77:0/3770782891 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa17ffff640 1 --2- 172.21.15.77:0/3770782891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366170 0x5558c3366590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa17dffb640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa1700a2910 0x7fa1700a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa17ffff640 1 -- 172.21.15.77:0/3770782891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366b60 msgr2=0x5558c336b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa17ffff640 1 --2- 172.21.15.77:0/3770782891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366b60 0x5558c336b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa17ffff640 1 -- 172.21.15.77:0/3770782891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366170 msgr2=0x5558c3366590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa17ffff640 1 --2- 172.21.15.77:0/3770782891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366170 0x5558c3366590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa17ffff640 1 -- 172.21.15.77:0/3770782891 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558c32874b0 con 0x5558c336bcd0 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa18505c640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa1700a2910 0x7fa1700a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa18485b640 1 --2- 172.21.15.77:0/3770782891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366b60 0x5558c336b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa17dffb640 1 -- 172.21.15.77:0/4038963945 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa1780cf150 con 0x7fa180102290 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa17ffff640 1 --2- 172.21.15.77:0/3770782891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c336bcd0 0x5558c336e160 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009fd0 tx=0x7fa16c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=507ddc872c3c2d2f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa16affd640 1 -- 172.21.15.77:0/3770782891 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c004030 con 0x5558c336bcd0 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa16affd640 1 -- 172.21.15.77:0/3770782891 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa16c00fc70 con 0x5558c336bcd0 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.810+0000 7fa16affd640 1 -- 172.21.15.77:0/3770782891 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c0039e0 con 0x5558c336bcd0 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.811+0000 7fa189788740 1 -- 172.21.15.77:0/3770782891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c336bcd0 msgr2=0x5558c336e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.811+0000 7fa18505c640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa1700a2910 0x7fa1700a4dd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa1780a3cf0 tx=0x7fa178097680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.811+0000 7fa189788740 1 --2- 172.21.15.77:0/3770782891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c336bcd0 0x5558c336e160 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009fd0 tx=0x7fa16c005850 comp rx=0 tx=0).stop 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.811+0000 7fa189788740 1 -- 172.21.15.77:0/3770782891 shutdown_connections 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.811+0000 7fa189788740 1 --2- 172.21.15.77:0/3770782891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c336bcd0 0x5558c336e160 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.811+0000 7fa189788740 1 --2- 172.21.15.77:0/3770782891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366b60 0x5558c336b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.811+0000 7fa189788740 1 --2- 172.21.15.77:0/3770782891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366170 0x5558c3366590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.811+0000 7fa189788740 1 -- 172.21.15.77:0/3770782891 >> 172.21.15.77:0/3770782891 conn(0x5558c33616e0 msgr2=0x5558c3361ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.811+0000 7fa189788740 1 -- 172.21.15.77:0/3770782891 shutdown_connections 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.811+0000 7fa189788740 1 -- 172.21.15.77:0/3770782891 wait complete. 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa189788740 1 -- start start 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa189788740 1 Processor -- start 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa189788740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366170 0x5558c3370630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa189788740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366b60 0x5558c3370b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa189788740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c336bcd0 0x5558c3377bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa189788740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5558c336ec80 con 0x5558c3366170 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa189788740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5558c3287160 con 0x5558c3366b60 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa189788740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5558c336ee00 con 0x5558c336bcd0 2024-03-16T10:17:40.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa189788740 10 client.0 _ll_register_callbacks cb 0x5558c326b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa17ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366170 0x5558c3370630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa17ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366170 0x5558c3370630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47784/0 (socket says 172.21.15.77:47784) 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa17ffff640 1 -- 172.21.15.77:0/2131026629 learned_addr learned my addr 172.21.15.77:0/2131026629 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa18485b640 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366b60 0x5558c3370b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.812+0000 7fa17ffff640 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c336bcd0 0x5558c3377bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa17ffff640 1 -- 172.21.15.77:0/2131026629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c336bcd0 msgr2=0x5558c3377bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa17ffff640 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c336bcd0 0x5558c3377bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa17ffff640 1 -- 172.21.15.77:0/2131026629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366b60 msgr2=0x5558c3370b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa17ffff640 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366b60 0x5558c3370b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa17ffff640 1 -- 172.21.15.77:0/2131026629 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558c30a97d0 con 0x5558c3366170 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa17ffff640 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366170 0x5558c3370630 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa16c005970 tx=0x7fa16c009c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a55b0fd3669111 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa15ffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:40.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa15ffff640 1 -- 172.21.15.77:0/2131026629 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c023020 con 0x5558c3366170 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa15ffff640 1 -- 172.21.15.77:0/2131026629 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa16c005050 con 0x5558c3366170 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa189788740 1 -- 172.21.15.77:0/2131026629 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa16c00fc70 con 0x5558c3366170 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa15ffff640 1 -- 172.21.15.77:0/2131026629 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa16c005250 con 0x5558c3366170 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa189788740 10 client.15573 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa189788740 1 -- 172.21.15.77:0/2131026629 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5558c2fdf3f0 con 0x5558c3366170 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.813+0000 7fa15ffff640 1 -- 172.21.15.77:0/2131026629 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa16c00fc70 con 0x5558c3366170 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.814+0000 7fa15ffff640 1 -- 172.21.15.77:0/2131026629 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa16c01f3f0 con 0x5558c3366170 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.814+0000 7fa189788740 10 client.15573 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.814+0000 7fa189788740 10 client.15573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.814+0000 7fa189788740 20 client.15573 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.814+0000 7fa189788740 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5558c337beb0 0x5558c337e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.814+0000 7fa189788740 4 client.15573 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.814+0000 7fa189788740 1 -- 172.21.15.77:0/2131026629 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558c32d8a40 con 0x5558c337beb0 2024-03-16T10:17:40.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.814+0000 7fa18485b640 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5558c337beb0 0x5558c337e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.814+0000 7fa18485b640 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5558c337beb0 0x5558c337e2c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a96c0bf8c35e5a4e server_cookie=404a9940c1fd96b0 in_seq=0 out_seq=0 2024-03-16T10:17:40.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.814+0000 7fa15ffff640 10 client.15573 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:40.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.815+0000 7fa15ffff640 1 -- 172.21.15.77:0/2131026629 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5558c32d8a40 con 0x5558c337beb0 2024-03-16T10:17:40.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.815+0000 7fa15ffff640 10 client.15573 handle_command_reply: tid=0 2024-03-16T10:17:40.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.815+0000 7fa189788740 1 client.15573 shutdown 2024-03-16T10:17:40.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.815+0000 7fa189788740 1 -- 172.21.15.77:0/2131026629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366170 msgr2=0x5558c3370630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:40.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.815+0000 7fa189788740 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366170 0x5558c3370630 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa16c005970 tx=0x7fa16c009c70 comp rx=0 tx=0).stop 2024-03-16T10:17:40.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.816+0000 7fa189788740 1 -- 172.21.15.77:0/2131026629 shutdown_connections 2024-03-16T10:17:40.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.816+0000 7fa189788740 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c336bcd0 0x5558c3377bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.816+0000 7fa189788740 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366b60 0x5558c3370b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.816+0000 7fa189788740 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5558c337beb0 0x5558c337e2c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.816+0000 7fa189788740 1 --2- 172.21.15.77:0/2131026629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366170 0x5558c3370630 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.816+0000 7fa15ffff640 0 client.15573 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:40.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.816+0000 7fa189788740 1 -- 172.21.15.77:0/2131026629 >> 172.21.15.77:0/2131026629 conn(0x5558c33616e0 msgr2=0x5558c3360900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:40.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.816+0000 7fa189788740 1 -- 172.21.15.77:0/2131026629 shutdown_connections 2024-03-16T10:17:40.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.816+0000 7fa189788740 1 -- 172.21.15.77:0/2131026629 wait complete. 2024-03-16T10:17:40.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.816+0000 7fa189788740 20 client.15573 trim_cache size 0 max 16384 2024-03-16T10:17:40.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa189788740 1 Processor -- start 2024-03-16T10:17:40.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa189788740 1 -- start start 2024-03-16T10:17:40.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa189788740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366170 0x5558c3360070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa189788740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366b60 0x5558c3377b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa189788740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c336bcd0 0x5558c33605b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa189788740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801a69d0 con 0x5558c3366b60 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa189788740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801a8510 con 0x5558c3366170 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa189788740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801a70a0 con 0x5558c336bcd0 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa18485b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366b60 0x5558c3377b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa18485b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366b60 0x5558c3377b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47792/0 (socket says 172.21.15.77:47792) 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa18485b640 1 -- 172.21.15.77:0/2363369953 learned_addr learned my addr 172.21.15.77:0/2363369953 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa17ffff640 1 --2- 172.21.15.77:0/2363369953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c336bcd0 0x5558c33605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa17ffff640 1 --2- 172.21.15.77:0/2363369953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366170 0x5558c3360070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.833+0000 7fa18485b640 1 -- 172.21.15.77:0/2363369953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c336bcd0 msgr2=0x5558c33605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.834+0000 7fa18485b640 1 --2- 172.21.15.77:0/2363369953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c336bcd0 0x5558c33605b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.834+0000 7fa18485b640 1 -- 172.21.15.77:0/2363369953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366170 msgr2=0x5558c3360070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.834+0000 7fa18485b640 1 --2- 172.21.15.77:0/2363369953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366170 0x5558c3360070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.834+0000 7fa18485b640 1 -- 172.21.15.77:0/2363369953 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558c2f9f0f0 con 0x5558c3366b60 2024-03-16T10:17:40.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.834+0000 7fa18485b640 1 --2- 172.21.15.77:0/2363369953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366b60 0x5558c3377b70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa174005940 tx=0x7fa17403e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648205999674da36 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.834+0000 7fa16a7fc640 1 -- 172.21.15.77:0/2363369953 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17404b020 con 0x5558c3366b60 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.834+0000 7fa16a7fc640 1 -- 172.21.15.77:0/2363369953 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa17402cc80 con 0x5558c3366b60 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.834+0000 7fa16a7fc640 1 -- 172.21.15.77:0/2363369953 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa174004920 con 0x5558c3366b60 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.834+0000 7fa189788740 1 -- 172.21.15.77:0/2363369953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366b60 msgr2=0x5558c3377b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 --2- 172.21.15.77:0/2363369953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366b60 0x5558c3377b70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa174005940 tx=0x7fa17403e040 comp rx=0 tx=0).stop 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 -- 172.21.15.77:0/2363369953 shutdown_connections 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 --2- 172.21.15.77:0/2363369953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c336bcd0 0x5558c33605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 --2- 172.21.15.77:0/2363369953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c3366b60 0x5558c3377b70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 --2- 172.21.15.77:0/2363369953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366170 0x5558c3360070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 -- 172.21.15.77:0/2363369953 >> 172.21.15.77:0/2363369953 conn(0x5558c33616e0 msgr2=0x5558c3370240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 -- 172.21.15.77:0/2363369953 shutdown_connections 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 -- 172.21.15.77:0/2363369953 wait complete. 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 -- start start 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 Processor -- start 2024-03-16T10:17:40.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366170 0x5558c3375220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366b60 0x5558c3375760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c336bcd0 0x5558c3376690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801a7e40 con 0x5558c336bcd0 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801a7770 con 0x5558c3366b60 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801a49d0 con 0x5558c3366170 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa189788740 10 client.0 _ll_register_callbacks cb 0x7fa16c009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa17ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c336bcd0 0x5558c3376690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa17ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366170 0x5558c3375220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa18485b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366b60 0x5558c3375760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.835+0000 7fa17ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366170 0x5558c3375220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:37758/0 (socket says 172.21.15.77:37758) 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa17ffff640 1 -- 172.21.15.77:0/2773292366 learned_addr learned my addr 172.21.15.77:0/2773292366 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa17ffff640 1 -- 172.21.15.77:0/2773292366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366b60 msgr2=0x5558c3375760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa17ffff640 1 --2- 172.21.15.77:0/2773292366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366b60 0x5558c3375760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa17ffff640 1 -- 172.21.15.77:0/2773292366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c336bcd0 msgr2=0x5558c3376690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa17ffff640 1 --2- 172.21.15.77:0/2773292366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c336bcd0 0x5558c3376690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa17ffff640 1 -- 172.21.15.77:0/2773292366 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558c32929d0 con 0x5558c3366170 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa17ffff640 1 --2- 172.21.15.77:0/2773292366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366170 0x5558c3375220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0117f0 tx=0x7fa16c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0903a46d786f66b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa15ffff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa15ffff640 1 -- 172.21.15.77:0/2773292366 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c037040 con 0x5558c3366170 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa189788740 1 -- 172.21.15.77:0/2773292366 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa17402cc80 con 0x5558c3366170 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa15ffff640 1 -- 172.21.15.77:0/2773292366 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa16c01f5f0 con 0x5558c3366170 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa15ffff640 1 -- 172.21.15.77:0/2773292366 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa16c023040 con 0x5558c3366170 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa189788740 10 client.25372 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.836+0000 7fa189788740 1 -- 172.21.15.77:0/2773292366 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5558c3377e30 con 0x5558c3366170 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa15ffff640 1 -- 172.21.15.77:0/2773292366 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa17402cc80 con 0x5558c3366170 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa15ffff640 1 -- 172.21.15.77:0/2773292366 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa16c04a050 con 0x5558c3366170 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa189788740 10 client.25372 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa189788740 10 client.25372 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:40.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa189788740 20 client.25372 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa189788740 1 --2- 172.21.15.77:0/2773292366 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5558c33e2920 0x5558c33e4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa189788740 4 client.25372 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa189788740 1 -- 172.21.15.77:0/2773292366 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5558c32d2d90 con 0x5558c33e2920 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa18485b640 1 --2- 172.21.15.77:0/2773292366 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5558c33e2920 0x5558c33e4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa15ffff640 10 client.25372 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa18485b640 1 --2- 172.21.15.77:0/2773292366 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5558c33e2920 0x5558c33e4d40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd13cb13c8a7c583 server_cookie=5b6b77686b2c6214 in_seq=0 out_seq=0 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa15ffff640 1 -- 172.21.15.77:0/2773292366 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5558c32d2d90 con 0x5558c33e2920 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa15ffff640 10 client.25372 handle_command_reply: tid=0 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.837+0000 7fa189788740 1 client.25372 shutdown 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa189788740 1 -- 172.21.15.77:0/2773292366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366170 msgr2=0x5558c3375220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa189788740 1 --2- 172.21.15.77:0/2773292366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366170 0x5558c3375220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0117f0 tx=0x7fa16c000bc0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa189788740 1 -- 172.21.15.77:0/2773292366 shutdown_connections 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa189788740 1 --2- 172.21.15.77:0/2773292366 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5558c33e2920 0x5558c33e4d40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa189788740 1 --2- 172.21.15.77:0/2773292366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5558c336bcd0 0x5558c3376690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa189788740 1 --2- 172.21.15.77:0/2773292366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5558c3366b60 0x5558c3375760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa15ffff640 0 client.25372 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:40.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa189788740 1 --2- 172.21.15.77:0/2773292366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558c3366170 0x5558c3375220 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa189788740 1 -- 172.21.15.77:0/2773292366 >> 172.21.15.77:0/2773292366 conn(0x5558c33616e0 msgr2=0x5558c336f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:40.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa189788740 1 -- 172.21.15.77:0/2773292366 shutdown_connections 2024-03-16T10:17:40.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa189788740 1 -- 172.21.15.77:0/2773292366 wait complete. 2024-03-16T10:17:40.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa189788740 20 client.25372 trim_cache size 0 max 16384 2024-03-16T10:17:40.845 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:17:40.845 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T10:17:40.845 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T10:17:40.846 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:17:40.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa186abf640 1 -- 172.21.15.77:0/4038963945 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa1700a2910 msgr2=0x7fa1700a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:40.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa186abf640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa1700a2910 0x7fa1700a4dd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa1780a3cf0 tx=0x7fa178097680 comp rx=0 tx=0).stop 2024-03-16T10:17:40.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa186abf640 1 -- 172.21.15.77:0/4038963945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa180102290 msgr2=0x7fa18019dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:40.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa186abf640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa180102290 0x7fa18019dfd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa17800eb10 tx=0x7fa17800c2d0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa186abf640 1 -- 172.21.15.77:0/4038963945 shutdown_connections 2024-03-16T10:17:40.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa186abf640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa180102290 0x7fa18019dfd0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa186abf640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa1700a2910 0x7fa1700a4dd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa186abf640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1801018d0 0x7fa180196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa186abf640 1 --2- 172.21.15.77:0/4038963945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1801006e0 0x7fa180196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa186abf640 1 -- 172.21.15.77:0/4038963945 >> 172.21.15.77:0/4038963945 conn(0x7fa1800fbc70 msgr2=0x7fa180104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:40.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.838+0000 7fa186abf640 1 -- 172.21.15.77:0/4038963945 shutdown_connections 2024-03-16T10:17:40.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.839+0000 7fa186abf640 1 -- 172.21.15.77:0/4038963945 wait complete. 2024-03-16T10:17:40.850 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T10:17:40.850 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:17:40.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.975+0000 7fd933812640 1 -- 172.21.15.77:0/3996346629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd92c0ffaf0 msgr2=0x7fd92c0fff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:40.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.975+0000 7fd933812640 1 --2- 172.21.15.77:0/3996346629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd92c0ffaf0 0x7fd92c0fff10 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd924009930 tx=0x7fd92402bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:40.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.975+0000 7fd933812640 1 -- 172.21.15.77:0/3996346629 shutdown_connections 2024-03-16T10:17:40.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.975+0000 7fd933812640 1 --2- 172.21.15.77:0/3996346629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd92c103520 0x7fd92c109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.975+0000 7fd933812640 1 --2- 172.21.15.77:0/3996346629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd92c102b60 0x7fd92c102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.975+0000 7fd933812640 1 --2- 172.21.15.77:0/3996346629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd92c0ffaf0 0x7fd92c0fff10 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.975+0000 7fd933812640 1 -- 172.21.15.77:0/3996346629 >> 172.21.15.77:0/3996346629 conn(0x7fd92c0fb400 msgr2=0x7fd92c0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:40.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.975+0000 7fd933812640 1 -- 172.21.15.77:0/3996346629 shutdown_connections 2024-03-16T10:17:40.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.975+0000 7fd933812640 1 -- 172.21.15.77:0/3996346629 wait complete. 2024-03-16T10:17:40.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd933812640 1 Processor -- start 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd933812640 1 -- start start 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd933812640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd92c0ffaf0 0x7fd92c197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd933812640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd92c102b60 0x7fd92c1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd933812640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd92c103520 0x7fd92c19f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd933812640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c10d4a0 con 0x7fd92c103520 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd933812640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c10d320 con 0x7fd92c102b60 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd933812640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c10d620 con 0x7fd92c0ffaf0 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd931daf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd92c103520 0x7fd92c19f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd931daf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd92c103520 0x7fd92c19f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47824/0 (socket says 172.21.15.77:47824) 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd9315ae640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd92c0ffaf0 0x7fd92c197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd931daf640 1 -- 172.21.15.77:0/3548774071 learned_addr learned my addr 172.21.15.77:0/3548774071 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd930dad640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd92c102b60 0x7fd92c1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd931daf640 1 -- 172.21.15.77:0/3548774071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd92c0ffaf0 msgr2=0x7fd92c197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd931daf640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd92c0ffaf0 0x7fd92c197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd931daf640 1 -- 172.21.15.77:0/3548774071 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd92c102b60 msgr2=0x7fd92c1980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd931daf640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd92c102b60 0x7fd92c1980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.976+0000 7fd931daf640 1 -- 172.21.15.77:0/3548774071 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd92c19f7c0 con 0x7fd92c103520 2024-03-16T10:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.977+0000 7fd931daf640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd92c103520 0x7fd92c19f150 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd91c00eb10 tx=0x7fd91c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.977+0000 7fd92a7fc640 1 -- 172.21.15.77:0/3548774071 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c004b20 con 0x7fd92c103520 2024-03-16T10:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.977+0000 7fd933812640 1 -- 172.21.15.77:0/3548774071 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd92c19fab0 con 0x7fd92c103520 2024-03-16T10:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.977+0000 7fd92a7fc640 1 -- 172.21.15.77:0/3548774071 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd91c004cc0 con 0x7fd92c103520 2024-03-16T10:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.977+0000 7fd92a7fc640 1 -- 172.21.15.77:0/3548774071 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c010760 con 0x7fd92c103520 2024-03-16T10:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.977+0000 7fd933812640 1 -- 172.21.15.77:0/3548774071 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd92c19ffb0 con 0x7fd92c103520 2024-03-16T10:17:40.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.978+0000 7fd933812640 1 -- 172.21.15.77:0/3548774071 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd92c108db0 con 0x7fd92c103520 2024-03-16T10:17:40.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.979+0000 7fd92a7fc640 1 -- 172.21.15.77:0/3548774071 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fd91c010900 con 0x7fd92c103520 2024-03-16T10:17:40.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.979+0000 7fd92a7fc640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd9140a2890 0x7fd9140a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:40.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.980+0000 7fd9315ae640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd9140a2890 0x7fd9140a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:40.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.980+0000 7fd92a7fc640 1 -- 172.21.15.77:0/3548774071 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd91c0cf2a0 con 0x7fd92c103520 2024-03-16T10:17:40.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.980+0000 7fd9315ae640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd9140a2890 0x7fd9140a4d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd92402c7b0 tx=0x7fd924005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:40.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:40.983+0000 7fd92a7fc640 1 -- 172.21.15.77:0/3548774071 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd91c053740 con 0x7fd92c103520 2024-03-16T10:17:41.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.202+0000 7fd933812640 1 -- 172.21.15.77:0/3548774071 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd92c02d1b0 con 0x7fd92c103520 2024-03-16T10:17:41.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.203+0000 7fd92a7fc640 1 -- 172.21.15.77:0/3548774071 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11938 (secure 0 0 0) 0x7fd91c09c2d0 con 0x7fd92c103520 2024-03-16T10:17:41.203 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:17:41.204 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:17:02.900684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24475,24454,24482,14658,24455]},"id":1}]} 2024-03-16T10:17:41.204 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 27 2024-03-16T10:17:41.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.205+0000 7fd933812640 1 -- 172.21.15.77:0/3548774071 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd9140a2890 msgr2=0x7fd9140a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.205+0000 7fd933812640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd9140a2890 0x7fd9140a4d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd92402c7b0 tx=0x7fd924005fd0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.205+0000 7fd933812640 1 -- 172.21.15.77:0/3548774071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd92c103520 msgr2=0x7fd92c19f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.205+0000 7fd933812640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd92c103520 0x7fd92c19f150 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd91c00eb10 tx=0x7fd91c00c2d0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.205+0000 7fd933812640 1 -- 172.21.15.77:0/3548774071 shutdown_connections 2024-03-16T10:17:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.205+0000 7fd933812640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd92c103520 0x7fd92c19f150 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.205+0000 7fd933812640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd92c102b60 0x7fd92c1980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.205+0000 7fd933812640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd9140a2890 0x7fd9140a4d50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.205+0000 7fd933812640 1 --2- 172.21.15.77:0/3548774071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd92c0ffaf0 0x7fd92c197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.205+0000 7fd933812640 1 -- 172.21.15.77:0/3548774071 >> 172.21.15.77:0/3548774071 conn(0x7fd92c0fb400 msgr2=0x7fd92c105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.205+0000 7fd933812640 1 -- 172.21.15.77:0/3548774071 shutdown_connections 2024-03-16T10:17:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.205+0000 7fd933812640 1 -- 172.21.15.77:0/3548774071 wait complete. 2024-03-16T10:17:41.217 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-16T10:17:41.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 -- 172.21.15.77:0/2959141786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91981006e0 msgr2=0x7f9198100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 --2- 172.21.15.77:0/2959141786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91981006e0 0x7f9198100b00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9190009ca0 tx=0x7f919002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 -- 172.21.15.77:0/2959141786 shutdown_connections 2024-03-16T10:17:41.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 --2- 172.21.15.77:0/2959141786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9198102290 0x7f9198108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 --2- 172.21.15.77:0/2959141786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91981018d0 0x7f9198101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 --2- 172.21.15.77:0/2959141786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91981006e0 0x7f9198100b00 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 -- 172.21.15.77:0/2959141786 >> 172.21.15.77:0/2959141786 conn(0x7f91980fbc70 msgr2=0x7f91980fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 -- 172.21.15.77:0/2959141786 shutdown_connections 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 -- 172.21.15.77:0/2959141786 wait complete. 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 Processor -- start 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 -- start start 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91981006e0 0x7f9198197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91981018d0 0x7f91981980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9198102290 0x7f919819f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f919810c350 con 0x7f91981006e0 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f919810c1d0 con 0x7f91981018d0 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919ed7e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f919810c4d0 con 0x7f9198102290 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.321+0000 7f919cb1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91981006e0 0x7f9198197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f919cb1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91981006e0 0x7f9198197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47838/0 (socket says 172.21.15.77:47838) 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f919cb1a640 1 -- 172.21.15.77:0/1259442231 learned_addr learned my addr 172.21.15.77:0/1259442231 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f919d31b640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9198102290 0x7f919819f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f9197fff640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91981018d0 0x7f91981980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f919cb1a640 1 -- 172.21.15.77:0/1259442231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9198102290 msgr2=0x7f919819f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f919cb1a640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9198102290 0x7f919819f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f919cb1a640 1 -- 172.21.15.77:0/1259442231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91981018d0 msgr2=0x7f91981980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f919cb1a640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91981018d0 0x7f91981980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f919cb1a640 1 -- 172.21.15.77:0/1259442231 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919819f7a0 con 0x7f91981006e0 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f919d31b640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9198102290 0x7f919819f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f919cb1a640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91981006e0 0x7f9198197b80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f91900324c0 tx=0x7f9190032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f9195ffb640 1 -- 172.21.15.77:0/1259442231 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190041020 con 0x7f91981006e0 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f919ed7e640 1 -- 172.21.15.77:0/1259442231 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f919819fa30 con 0x7f91981006e0 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f9195ffb640 1 -- 172.21.15.77:0/1259442231 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9190034cd0 con 0x7f91981006e0 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f919ed7e640 1 -- 172.21.15.77:0/1259442231 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f919819ff70 con 0x7f91981006e0 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.322+0000 7f9195ffb640 1 -- 172.21.15.77:0/1259442231 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190039040 con 0x7f91981006e0 2024-03-16T10:17:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f9195ffb640 1 -- 172.21.15.77:0/1259442231 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f9190044020 con 0x7f91981006e0 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f91a196c740 1 Processor -- start 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f91a196c740 1 -- start start 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f91a196c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 0x55cc29eea080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f91a196c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29eef150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f91a196c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29eef7c0 0x55cc29ef1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f91a196c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cc29e4a490 con 0x55cc29ee9c60 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f91a196c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55cc29df2c90 con 0x55cc29eea650 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f91a196c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cc29e46530 con 0x55cc29eef7c0 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f919d31b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 0x55cc29eea080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f919d31b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 0x55cc29eea080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47844/0 (socket says 172.21.15.77:47844) 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f919d31b640 1 -- 172.21.15.77:0/216038481 learned_addr learned my addr 172.21.15.77:0/216038481 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f919cb1a640 1 --2- 172.21.15.77:0/216038481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29eef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f9197fff640 1 --2- 172.21.15.77:0/216038481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29eef7c0 0x55cc29ef1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.325+0000 7f9195ffb640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f91840a2890 0x7f91840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f9197fff640 1 -- 172.21.15.77:0/216038481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 msgr2=0x55cc29eef150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f9197fff640 1 --2- 172.21.15.77:0/216038481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29eef150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f9197fff640 1 -- 172.21.15.77:0/216038481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 msgr2=0x55cc29eea080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f9197fff640 1 --2- 172.21.15.77:0/216038481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 0x55cc29eea080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f9197fff640 1 -- 172.21.15.77:0/216038481 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc29ead500 con 0x55cc29eef7c0 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f919d31b640 1 --2- 172.21.15.77:0/216038481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 0x55cc29eea080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f919cb1a640 1 --2- 172.21.15.77:0/216038481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29eef150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f919d31b640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f91840a2890 0x7f91840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f9195ffb640 1 -- 172.21.15.77:0/1259442231 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f91900f09a0 con 0x7f91981006e0 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f9197fff640 1 --2- 172.21.15.77:0/216038481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29eef7c0 0x55cc29ef1c50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9188009bd0 tx=0x7f9188011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d319b85d2eef385 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f918effd640 1 -- 172.21.15.77:0/216038481 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188004230 con 0x55cc29eef7c0 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f919d31b640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f91840a2890 0x7f91840a4d50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55cc29eeaeb0 tx=0x7f91800046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f918effd640 1 -- 172.21.15.77:0/216038481 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9188009e40 con 0x55cc29eef7c0 2024-03-16T10:17:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.326+0000 7f918effd640 1 -- 172.21.15.77:0/216038481 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188004d70 con 0x55cc29eef7c0 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.327+0000 7f91a196c740 1 -- 172.21.15.77:0/216038481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29eef7c0 msgr2=0x55cc29ef1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.327+0000 7f91a196c740 1 --2- 172.21.15.77:0/216038481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29eef7c0 0x55cc29ef1c50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9188009bd0 tx=0x7f9188011810 comp rx=0 tx=0).stop 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.327+0000 7f91a196c740 1 -- 172.21.15.77:0/216038481 shutdown_connections 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.327+0000 7f91a196c740 1 --2- 172.21.15.77:0/216038481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29eef7c0 0x55cc29ef1c50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.327+0000 7f91a196c740 1 --2- 172.21.15.77:0/216038481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29eef150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.327+0000 7f91a196c740 1 --2- 172.21.15.77:0/216038481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 0x55cc29eea080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.327+0000 7f91a196c740 1 -- 172.21.15.77:0/216038481 >> 172.21.15.77:0/216038481 conn(0x55cc29ee0dc0 msgr2=0x55cc29ee11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.327+0000 7f91a196c740 1 -- 172.21.15.77:0/216038481 shutdown_connections 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.327+0000 7f91a196c740 1 -- 172.21.15.77:0/216038481 wait complete. 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.327+0000 7f91a196c740 1 -- start start 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f91a196c740 1 Processor -- start 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f91a196c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee9c60 0x55cc29ef4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f91a196c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29ef4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f91a196c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eef7c0 0x55cc29efb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f91a196c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cc29ef2770 con 0x55cc29eef7c0 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f91a196c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55cc29e06230 con 0x55cc29eea650 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f91a196c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cc29ef28f0 con 0x55cc29ee9c60 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f91a196c740 10 client.0 _ll_register_callbacks cb 0x55cc29e42010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f919cb1a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee9c60 0x55cc29ef4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f919d31b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eef7c0 0x55cc29efb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f919cb1a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee9c60 0x55cc29ef4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:37804/0 (socket says 172.21.15.77:37804) 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f919cb1a640 1 -- 172.21.15.77:0/568017924 learned_addr learned my addr 172.21.15.77:0/568017924 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f9197fff640 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29ef4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f919d31b640 1 -- 172.21.15.77:0/568017924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee9c60 msgr2=0x55cc29ef4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f919d31b640 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee9c60 0x55cc29ef4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f919d31b640 1 -- 172.21.15.77:0/568017924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 msgr2=0x55cc29ef4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f919d31b640 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29ef4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f919d31b640 1 -- 172.21.15.77:0/568017924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc29e15100 con 0x55cc29eef7c0 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.328+0000 7f919cb1a640 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee9c60 0x55cc29ef4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f919d31b640 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eef7c0 0x55cc29efb6d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9180012d10 tx=0x7f9180004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95519b1d6bf953bc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f9177fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f9177fff640 1 -- 172.21.15.77:0/568017924 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918001d9d0 con 0x55cc29eef7c0 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f9177fff640 1 -- 172.21.15.77:0/568017924 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f918001db70 con 0x55cc29eef7c0 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f91a196c740 1 -- 172.21.15.77:0/568017924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cc29e579d0 con 0x55cc29eef7c0 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f9177fff640 1 -- 172.21.15.77:0/568017924 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f918001dd70 con 0x55cc29eef7c0 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f91a196c740 10 client.15597 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f91a196c740 1 -- 172.21.15.77:0/568017924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cc29b5e3f0 con 0x55cc29eef7c0 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f9177fff640 1 -- 172.21.15.77:0/568017924 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55cc29e579d0 con 0x55cc29eef7c0 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f9177fff640 1 -- 172.21.15.77:0/568017924 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9180025020 con 0x55cc29eef7c0 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f91a196c740 10 client.15597 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f91a196c740 10 client.15597 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:17:41.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f91a196c740 20 client.15597 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:41.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f91a196c740 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55cc29f00090 0x55cc29f02550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f91a196c740 4 client.15597 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:41.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f91a196c740 1 -- 172.21.15.77:0/568017924 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc29e51d20 con 0x55cc29f00090 2024-03-16T10:17:41.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.329+0000 7f919cb1a640 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55cc29f00090 0x55cc29f02550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.330+0000 7f9177fff640 10 client.15597 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:17:41.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.330+0000 7f919cb1a640 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55cc29f00090 0x55cc29f02550 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6165d0fac160440f server_cookie=62764adefe836eb4 in_seq=0 out_seq=0 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.330+0000 7f9177fff640 1 -- 172.21.15.77:0/568017924 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cc29e51d20 con 0x55cc29f00090 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.330+0000 7f9177fff640 10 client.15597 handle_command_reply: tid=0 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.330+0000 7f91a196c740 1 client.15597 shutdown 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.330+0000 7f91a196c740 1 -- 172.21.15.77:0/568017924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eef7c0 msgr2=0x55cc29efb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.330+0000 7f91a196c740 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eef7c0 0x55cc29efb6d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9180012d10 tx=0x7f9180004730 comp rx=0 tx=0).stop 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.330+0000 7f91a196c740 1 -- 172.21.15.77:0/568017924 shutdown_connections 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.331+0000 7f91a196c740 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eef7c0 0x55cc29efb6d0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.331+0000 7f91a196c740 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29ef4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.331+0000 7f91a196c740 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55cc29f00090 0x55cc29f02550 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.331+0000 7f91a196c740 1 --2- 172.21.15.77:0/568017924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee9c60 0x55cc29ef4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.331+0000 7f91a196c740 1 -- 172.21.15.77:0/568017924 >> 172.21.15.77:0/568017924 conn(0x55cc29ee0dc0 msgr2=0x55cc29edffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.331+0000 7f9177fff640 0 client.15597 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.331+0000 7f91a196c740 1 -- 172.21.15.77:0/568017924 shutdown_connections 2024-03-16T10:17:41.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.331+0000 7f91a196c740 1 -- 172.21.15.77:0/568017924 wait complete. 2024-03-16T10:17:41.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.331+0000 7f91a196c740 20 client.15597 trim_cache size 0 max 16384 2024-03-16T10:17:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f91a196c740 1 Processor -- start 2024-03-16T10:17:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f91a196c740 1 -- start start 2024-03-16T10:17:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f91a196c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee0dc0 0x55cc29ef2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f91a196c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 0x55cc29efb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f91a196c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29edf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f91a196c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f91981a69f0 con 0x55cc29ee9c60 2024-03-16T10:17:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f91a196c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f91981a8530 con 0x55cc29eea650 2024-03-16T10:17:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f91a196c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91981a70c0 con 0x55cc29ee0dc0 2024-03-16T10:17:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f9197fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 0x55cc29efb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f9197fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 0x55cc29efb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47862/0 (socket says 172.21.15.77:47862) 2024-03-16T10:17:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f9197fff640 1 -- 172.21.15.77:0/1976803111 learned_addr learned my addr 172.21.15.77:0/1976803111 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f919cb1a640 1 --2- 172.21.15.77:0/1976803111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee0dc0 0x55cc29ef2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f919d31b640 1 --2- 172.21.15.77:0/1976803111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29edf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f9197fff640 1 -- 172.21.15.77:0/1976803111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee0dc0 msgr2=0x55cc29ef2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f9197fff640 1 --2- 172.21.15.77:0/1976803111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee0dc0 0x55cc29ef2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f9197fff640 1 -- 172.21.15.77:0/1976803111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 msgr2=0x55cc29edf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f9197fff640 1 --2- 172.21.15.77:0/1976803111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29edf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f9197fff640 1 -- 172.21.15.77:0/1976803111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc29c287d0 con 0x55cc29ee9c60 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.344+0000 7f9197fff640 1 --2- 172.21.15.77:0/1976803111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 0x55cc29efb690 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9188009bd0 tx=0x7f9188009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f567bb1404bb1da4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f918e7fc640 1 -- 172.21.15.77:0/1976803111 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918801e040 con 0x55cc29ee9c60 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f918e7fc640 1 -- 172.21.15.77:0/1976803111 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9188005390 con 0x55cc29ee9c60 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f918e7fc640 1 -- 172.21.15.77:0/1976803111 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188024760 con 0x55cc29ee9c60 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f91a196c740 1 -- 172.21.15.77:0/1976803111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 msgr2=0x55cc29efb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f91a196c740 1 --2- 172.21.15.77:0/1976803111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 0x55cc29efb690 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9188009bd0 tx=0x7f9188009d80 comp rx=0 tx=0).stop 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f91a196c740 1 -- 172.21.15.77:0/1976803111 shutdown_connections 2024-03-16T10:17:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f91a196c740 1 --2- 172.21.15.77:0/1976803111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29eea650 0x55cc29edf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f91a196c740 1 --2- 172.21.15.77:0/1976803111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29ee9c60 0x55cc29efb690 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f91a196c740 1 --2- 172.21.15.77:0/1976803111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee0dc0 0x55cc29ef2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f91a196c740 1 -- 172.21.15.77:0/1976803111 >> 172.21.15.77:0/1976803111 conn(0x55cc29eeb430 msgr2=0x55cc29eebb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f91a196c740 1 -- 172.21.15.77:0/1976803111 shutdown_connections 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f91a196c740 1 -- 172.21.15.77:0/1976803111 wait complete. 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.345+0000 7f91a196c740 1 -- start start 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f91a196c740 1 Processor -- start 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f91a196c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29ee0dc0 0x55cc29ef8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f91a196c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee9c60 0x55cc29ef8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f91a196c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eea650 0x55cc29ef9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f91a196c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f91981a7e60 con 0x55cc29eea650 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f91a196c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f91981a7790 con 0x55cc29ee0dc0 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f91a196c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91981a49d0 con 0x55cc29ee9c60 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f91a196c740 10 client.0 _ll_register_callbacks cb 0x7f918000bce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f919d31b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eea650 0x55cc29ef9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f919d31b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eea650 0x55cc29ef9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47874/0 (socket says 172.21.15.77:47874) 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f919d31b640 1 -- 172.21.15.77:0/2725627482 learned_addr learned my addr 172.21.15.77:0/2725627482 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f9197fff640 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee9c60 0x55cc29ef8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f919cb1a640 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29ee0dc0 0x55cc29ef8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f919d31b640 1 -- 172.21.15.77:0/2725627482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee9c60 msgr2=0x55cc29ef8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f919d31b640 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee9c60 0x55cc29ef8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f919d31b640 1 -- 172.21.15.77:0/2725627482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29ee0dc0 msgr2=0x55cc29ef8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f919d31b640 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29ee0dc0 0x55cc29ef8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f919d31b640 1 -- 172.21.15.77:0/2725627482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc29efa3a0 con 0x55cc29eea650 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f919d31b640 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eea650 0x55cc29ef9db0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f918000b620 tx=0x7f918001da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55324bc95d86ee2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.346+0000 7f9177fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f9177fff640 1 -- 172.21.15.77:0/2725627482 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91800043f0 con 0x55cc29eea650 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f9177fff640 1 -- 172.21.15.77:0/2725627482 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f918001b890 con 0x55cc29eea650 2024-03-16T10:17:41.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f91a196c740 1 -- 172.21.15.77:0/2725627482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9188005390 con 0x55cc29eea650 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f9177fff640 1 -- 172.21.15.77:0/2725627482 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f918000fe50 con 0x55cc29eea650 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f91a196c740 10 client.15603 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f91a196c740 1 -- 172.21.15.77:0/2725627482 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cc29efb6a0 con 0x55cc29eea650 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f9177fff640 1 -- 172.21.15.77:0/2725627482 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9188005390 con 0x55cc29eea650 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f9177fff640 1 -- 172.21.15.77:0/2725627482 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9180025040 con 0x55cc29eea650 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f91a196c740 10 client.15603 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f91a196c740 10 client.15603 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f91a196c740 20 client.15603 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f91a196c740 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55cc29eef7c0 0x55cc29f68960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f91a196c740 4 client.15603 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f91a196c740 1 -- 172.21.15.77:0/2725627482 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cc29dcf6a0 con 0x55cc29eef7c0 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.347+0000 7f919cb1a640 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55cc29eef7c0 0x55cc29f68960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f9177fff640 10 client.15603 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:17:41.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f919cb1a640 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55cc29eef7c0 0x55cc29f68960 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cf5498dc38b028c0 server_cookie=ff37cf35552e1de4 in_seq=0 out_seq=0 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f9177fff640 1 -- 172.21.15.77:0/2725627482 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cc29dcf6a0 con 0x55cc29eef7c0 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f9177fff640 10 client.15603 handle_command_reply: tid=0 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f91a196c740 1 client.15603 shutdown 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f91a196c740 1 -- 172.21.15.77:0/2725627482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eea650 msgr2=0x55cc29ef9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f91a196c740 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eea650 0x55cc29ef9db0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f918000b620 tx=0x7f918001da80 comp rx=0 tx=0).stop 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f91a196c740 1 -- 172.21.15.77:0/2725627482 shutdown_connections 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f91a196c740 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc29eea650 0x55cc29ef9db0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f91a196c740 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55cc29eef7c0 0x55cc29f68960 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f9177fff640 0 client.15603 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f91a196c740 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc29ee9c60 0x55cc29ef8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f91a196c740 1 --2- 172.21.15.77:0/2725627482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc29ee0dc0 0x55cc29ef8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f91a196c740 1 -- 172.21.15.77:0/2725627482 >> 172.21.15.77:0/2725627482 conn(0x55cc29eeb430 msgr2=0x55cc29ee4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f91a196c740 1 -- 172.21.15.77:0/2725627482 shutdown_connections 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f91a196c740 1 -- 172.21.15.77:0/2725627482 wait complete. 2024-03-16T10:17:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.348+0000 7f91a196c740 20 client.15603 trim_cache size 0 max 16384 2024-03-16T10:17:41.357 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:17:41.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.349+0000 7f919ed7e640 1 -- 172.21.15.77:0/1259442231 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f91840a2890 msgr2=0x7f91840a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.349+0000 7f919ed7e640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f91840a2890 0x7f91840a4d50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55cc29eeaeb0 tx=0x7f91800046c0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.349+0000 7f919ed7e640 1 -- 172.21.15.77:0/1259442231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91981006e0 msgr2=0x7f9198197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.349+0000 7f919ed7e640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91981006e0 0x7f9198197b80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f91900324c0 tx=0x7f9190032f40 comp rx=0 tx=0).stop 2024-03-16T10:17:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.349+0000 7f919ed7e640 1 -- 172.21.15.77:0/1259442231 shutdown_connections 2024-03-16T10:17:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.349+0000 7f919ed7e640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f91840a2890 0x7f91840a4d50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.349+0000 7f919ed7e640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9198102290 0x7f919819f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.349+0000 7f919ed7e640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91981018d0 0x7f91981980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.349+0000 7f919ed7e640 1 --2- 172.21.15.77:0/1259442231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91981006e0 0x7f9198197b80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.349+0000 7f919ed7e640 1 -- 172.21.15.77:0/1259442231 >> 172.21.15.77:0/1259442231 conn(0x7f91980fbc70 msgr2=0x7f9198068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.349+0000 7f919ed7e640 1 -- 172.21.15.77:0/1259442231 shutdown_connections 2024-03-16T10:17:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.349+0000 7f919ed7e640 1 -- 172.21.15.77:0/1259442231 wait complete. 2024-03-16T10:17:41.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-16T10:17:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.465+0000 7ff88b84b640 1 -- 172.21.15.77:0/3118131976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff884102290 msgr2=0x7ff884108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.465+0000 7ff88b84b640 1 --2- 172.21.15.77:0/3118131976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff884102290 0x7ff884108b20 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff87c009ca0 tx=0x7ff87c02bde0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.465+0000 7ff88b84b640 1 -- 172.21.15.77:0/3118131976 shutdown_connections 2024-03-16T10:17:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.465+0000 7ff88b84b640 1 --2- 172.21.15.77:0/3118131976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff884102290 0x7ff884108b20 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.465+0000 7ff88b84b640 1 --2- 172.21.15.77:0/3118131976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8841018d0 0x7ff884101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.465+0000 7ff88b84b640 1 --2- 172.21.15.77:0/3118131976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8841006e0 0x7ff884100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.465+0000 7ff88b84b640 1 -- 172.21.15.77:0/3118131976 >> 172.21.15.77:0/3118131976 conn(0x7ff8840fbc70 msgr2=0x7ff8840fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.465+0000 7ff88b84b640 1 -- 172.21.15.77:0/3118131976 shutdown_connections 2024-03-16T10:17:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.465+0000 7ff88b84b640 1 -- 172.21.15.77:0/3118131976 wait complete. 2024-03-16T10:17:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.465+0000 7ff88b84b640 1 Processor -- start 2024-03-16T10:17:41.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.465+0000 7ff88b84b640 1 -- start start 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff88b84b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8841006e0 0x7ff8841958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff88b84b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8841018d0 0x7ff884195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff88b84b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff884102290 0x7ff88419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff88b84b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff88410b220 con 0x7ff8841006e0 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff88b84b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff88410b0a0 con 0x7ff8841018d0 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff88b84b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff88410b3a0 con 0x7ff884102290 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff8895e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8841006e0 0x7ff8841958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff888de6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8841018d0 0x7ff884195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff8895e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8841006e0 0x7ff8841958c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47890/0 (socket says 172.21.15.77:47890) 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff8895e7640 1 -- 172.21.15.77:0/1704331301 learned_addr learned my addr 172.21.15.77:0/1704331301 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff889de8640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff884102290 0x7ff88419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff8895e7640 1 -- 172.21.15.77:0/1704331301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff884102290 msgr2=0x7ff88419ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff8895e7640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff884102290 0x7ff88419ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff8895e7640 1 -- 172.21.15.77:0/1704331301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8841018d0 msgr2=0x7ff884195e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff8895e7640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8841018d0 0x7ff884195e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff8895e7640 1 -- 172.21.15.77:0/1704331301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88419d4e0 con 0x7ff8841006e0 2024-03-16T10:17:41.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff888de6640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8841018d0 0x7ff884195e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:41.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff8895e7640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8841006e0 0x7ff8841958c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff880009be0 tx=0x7ff880007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff87a7fc640 1 -- 172.21.15.77:0/1704331301 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff880007c40 con 0x7ff8841006e0 2024-03-16T10:17:41.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff87a7fc640 1 -- 172.21.15.77:0/1704331301 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff880007de0 con 0x7ff8841006e0 2024-03-16T10:17:41.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.466+0000 7ff87a7fc640 1 -- 172.21.15.77:0/1704331301 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff880019650 con 0x7ff8841006e0 2024-03-16T10:17:41.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.467+0000 7ff88b84b640 1 -- 172.21.15.77:0/1704331301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff88419d7d0 con 0x7ff8841006e0 2024-03-16T10:17:41.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.467+0000 7ff88b84b640 1 -- 172.21.15.77:0/1704331301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff88419dcd0 con 0x7ff8841006e0 2024-03-16T10:17:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.469+0000 7ff88e433740 1 Processor -- start 2024-03-16T10:17:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.469+0000 7ff88e433740 1 -- start start 2024-03-16T10:17:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.469+0000 7ff88e433740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 0x5653a2dc4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.469+0000 7ff88e433740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc5240 0x5653a2dc9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.469+0000 7ff88e433740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dca3b0 0x5653a2dcc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.469+0000 7ff88e433740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5653a2d29490 con 0x5653a2dc4850 2024-03-16T10:17:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.469+0000 7ff88e433740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5653a2cd1c90 con 0x5653a2dca3b0 2024-03-16T10:17:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.469+0000 7ff88e433740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5653a2d25530 con 0x5653a2dc5240 2024-03-16T10:17:41.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.469+0000 7ff889de8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 0x5653a2dc4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.469+0000 7ff889de8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 0x5653a2dc4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47900/0 (socket says 172.21.15.77:47900) 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.469+0000 7ff889de8640 1 -- 172.21.15.77:0/3197721168 learned_addr learned my addr 172.21.15.77:0/3197721168 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff888de6640 1 --2- 172.21.15.77:0/3197721168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dca3b0 0x5653a2dcc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff87a7fc640 1 -- 172.21.15.77:0/1704331301 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7ff880020030 con 0x7ff8841006e0 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff889de8640 1 -- 172.21.15.77:0/3197721168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc5240 msgr2=0x5653a2dc9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff889de8640 1 --2- 172.21.15.77:0/3197721168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc5240 0x5653a2dc9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff889de8640 1 -- 172.21.15.77:0/3197721168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dca3b0 msgr2=0x5653a2dcc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff889de8640 1 --2- 172.21.15.77:0/3197721168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dca3b0 0x5653a2dcc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff889de8640 1 -- 172.21.15.77:0/3197721168 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653a2d8c500 con 0x5653a2dc4850 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff889de8640 1 --2- 172.21.15.77:0/3197721168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 0x5653a2dc4c70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff87c02c3f0 tx=0x7ff87c035f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6bf31a009e561c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff86f7fe640 1 -- 172.21.15.77:0/3197721168 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c044070 con 0x5653a2dc4850 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff86f7fe640 1 -- 172.21.15.77:0/3197721168 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff87c02cb90 con 0x5653a2dc4850 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff86f7fe640 1 -- 172.21.15.77:0/3197721168 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c039040 con 0x5653a2dc4850 2024-03-16T10:17:41.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff87a7fc640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff8700a2910 0x7ff8700a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.470+0000 7ff8895e7640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff8700a2910 0x7ff8700a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff87a7fc640 1 -- 172.21.15.77:0/1704331301 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ff8800d1fe0 con 0x7ff8841006e0 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff88e433740 1 -- 172.21.15.77:0/3197721168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 msgr2=0x5653a2dc4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff88e433740 1 --2- 172.21.15.77:0/3197721168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 0x5653a2dc4c70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff87c02c3f0 tx=0x7ff87c035f40 comp rx=0 tx=0).stop 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff8895e7640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff8700a2910 0x7ff8700a4dd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x5653a2dc5cb0 tx=0x7ff880000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff88e433740 1 -- 172.21.15.77:0/3197721168 shutdown_connections 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff88e433740 1 --2- 172.21.15.77:0/3197721168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dca3b0 0x5653a2dcc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff88e433740 1 --2- 172.21.15.77:0/3197721168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc5240 0x5653a2dc9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff88e433740 1 --2- 172.21.15.77:0/3197721168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 0x5653a2dc4c70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff88e433740 1 -- 172.21.15.77:0/3197721168 >> 172.21.15.77:0/3197721168 conn(0x5653a2dbfdc0 msgr2=0x5653a2dc01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff88e433740 1 -- 172.21.15.77:0/3197721168 shutdown_connections 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff88e433740 1 -- 172.21.15.77:0/3197721168 wait complete. 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff88e433740 1 -- start start 2024-03-16T10:17:41.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.471+0000 7ff88e433740 1 Processor -- start 2024-03-16T10:17:41.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff88e433740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 0x5653a2dcece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff88e433740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc5240 0x5653a2dcf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff88e433740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dca3b0 0x5653a2dd6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff88e433740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5653a2dcd300 con 0x5653a2dc4850 2024-03-16T10:17:41.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff88e433740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5653a2ce5230 con 0x5653a2dca3b0 2024-03-16T10:17:41.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff88e433740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5653a2dcd480 con 0x5653a2dc5240 2024-03-16T10:17:41.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff88e433740 10 client.0 _ll_register_callbacks cb 0x5653a2d21010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:41.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff889de8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 0x5653a2dcece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff888de6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc5240 0x5653a2dcf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff889de8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 0x5653a2dcece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47912/0 (socket says 172.21.15.77:47912) 2024-03-16T10:17:41.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff889de8640 1 -- 172.21.15.77:0/268133600 learned_addr learned my addr 172.21.15.77:0/268133600 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff889de8640 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dca3b0 0x5653a2dd6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff889de8640 1 -- 172.21.15.77:0/268133600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc5240 msgr2=0x5653a2dcf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff889de8640 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc5240 0x5653a2dcf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff889de8640 1 -- 172.21.15.77:0/268133600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dca3b0 msgr2=0x5653a2dd6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff889de8640 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dca3b0 0x5653a2dd6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff889de8640 1 -- 172.21.15.77:0/268133600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653a2cf4100 con 0x5653a2dc4850 2024-03-16T10:17:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff888de6640 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc5240 0x5653a2dcf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff889de8640 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 0x5653a2dcece0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff87c037040 tx=0x7ff87c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20f5caad4e014dc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.472+0000 7ff86cff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff86cff9640 1 -- 172.21.15.77:0/268133600 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c04f070 con 0x5653a2dc4850 2024-03-16T10:17:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff86cff9640 1 -- 172.21.15.77:0/268133600 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ff87c003b70 con 0x5653a2dc4850 2024-03-16T10:17:41.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff88e433740 1 -- 172.21.15.77:0/268133600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff87c02cb90 con 0x5653a2dc4850 2024-03-16T10:17:41.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff86cff9640 1 -- 172.21.15.77:0/268133600 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff87c039040 con 0x5653a2dc4850 2024-03-16T10:17:41.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff88e433740 10 client.15615 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:41.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff88e433740 1 -- 172.21.15.77:0/268133600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5653a2a3d3f0 con 0x5653a2dc4850 2024-03-16T10:17:41.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff86cff9640 1 -- 172.21.15.77:0/268133600 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff87c02cb90 con 0x5653a2dc4850 2024-03-16T10:17:41.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff86cff9640 1 -- 172.21.15.77:0/268133600 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff87c0041a0 con 0x5653a2dc4850 2024-03-16T10:17:41.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff88e433740 10 client.15615 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:41.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff88e433740 10 client.15615 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:17:41.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff88e433740 20 client.15615 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:41.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff88e433740 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5653a2dda740 0x5653a2ddcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff88e433740 4 client.15615 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:41.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff88e433740 1 -- 172.21.15.77:0/268133600 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653a2d369d0 con 0x5653a2dda740 2024-03-16T10:17:41.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.473+0000 7ff888de6640 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5653a2dda740 0x5653a2ddcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.474+0000 7ff888de6640 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5653a2dda740 0x5653a2ddcb30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9ec9e5dd36ff38f3 server_cookie=d82cdf21732a126a in_seq=0 out_seq=0 2024-03-16T10:17:41.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.474+0000 7ff86cff9640 10 client.15615 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:17:41.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.474+0000 7ff86cff9640 1 -- 172.21.15.77:0/268133600 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5653a2d369d0 con 0x5653a2dda740 2024-03-16T10:17:41.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.474+0000 7ff86cff9640 10 client.15615 handle_command_reply: tid=0 2024-03-16T10:17:41.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.474+0000 7ff88e433740 1 client.15615 shutdown 2024-03-16T10:17:41.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.475+0000 7ff88e433740 1 -- 172.21.15.77:0/268133600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 msgr2=0x5653a2dcece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.475+0000 7ff88e433740 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 0x5653a2dcece0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff87c037040 tx=0x7ff87c0095c0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.475+0000 7ff88e433740 1 -- 172.21.15.77:0/268133600 shutdown_connections 2024-03-16T10:17:41.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.475+0000 7ff88e433740 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5653a2dda740 0x5653a2ddcb30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.475+0000 7ff88e433740 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dca3b0 0x5653a2dd6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.475+0000 7ff88e433740 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc5240 0x5653a2dcf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.475+0000 7ff86cff9640 0 client.15615 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:17:41.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.475+0000 7ff88e433740 1 --2- 172.21.15.77:0/268133600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc4850 0x5653a2dcece0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.475+0000 7ff88e433740 1 -- 172.21.15.77:0/268133600 >> 172.21.15.77:0/268133600 conn(0x5653a2dbfdc0 msgr2=0x5653a2dbefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.475+0000 7ff88e433740 1 -- 172.21.15.77:0/268133600 shutdown_connections 2024-03-16T10:17:41.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.475+0000 7ff88e433740 1 -- 172.21.15.77:0/268133600 wait complete. 2024-03-16T10:17:41.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.475+0000 7ff88e433740 20 client.15615 trim_cache size 0 max 16384 2024-03-16T10:17:41.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff88e433740 1 Processor -- start 2024-03-16T10:17:41.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff88e433740 1 -- start start 2024-03-16T10:17:41.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff88e433740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dc4850 0x5653a2dcf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff88e433740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc5240 0x5653a2dd6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff88e433740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dca3b0 0x5653a2dbe750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff88e433740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff8841a6a10 con 0x5653a2dc5240 2024-03-16T10:17:41.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff88e433740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff8841a8550 con 0x5653a2dc4850 2024-03-16T10:17:41.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff88e433740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff8841a70e0 con 0x5653a2dca3b0 2024-03-16T10:17:41.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff888de6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc5240 0x5653a2dd6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff888de6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc5240 0x5653a2dd6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47920/0 (socket says 172.21.15.77:47920) 2024-03-16T10:17:41.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff888de6640 1 -- 172.21.15.77:0/3523676324 learned_addr learned my addr 172.21.15.77:0/3523676324 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff889de8640 1 --2- 172.21.15.77:0/3523676324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dc4850 0x5653a2dcf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff889de8640 1 --2- 172.21.15.77:0/3523676324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dca3b0 0x5653a2dbe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff888de6640 1 -- 172.21.15.77:0/3523676324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dca3b0 msgr2=0x5653a2dbe750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff888de6640 1 --2- 172.21.15.77:0/3523676324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dca3b0 0x5653a2dbe750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff888de6640 1 -- 172.21.15.77:0/3523676324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dc4850 msgr2=0x5653a2dcf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff888de6640 1 --2- 172.21.15.77:0/3523676324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dc4850 0x5653a2dcf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff888de6640 1 -- 172.21.15.77:0/3523676324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653a2b077d0 con 0x5653a2dc5240 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.488+0000 7ff889de8640 1 --2- 172.21.15.77:0/3523676324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dca3b0 0x5653a2dbe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff888de6640 1 --2- 172.21.15.77:0/3523676324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc5240 0x5653a2dd6210 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff874005d00 tx=0x7ff874005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d87dced0f54cbb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff86effd640 1 -- 172.21.15.77:0/3523676324 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff874012110 con 0x5653a2dc5240 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff86effd640 1 -- 172.21.15.77:0/3523676324 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff874022170 con 0x5653a2dc5240 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff86effd640 1 -- 172.21.15.77:0/3523676324 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8740122b0 con 0x5653a2dc5240 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff88e433740 1 -- 172.21.15.77:0/3523676324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc5240 msgr2=0x5653a2dd6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff88e433740 1 --2- 172.21.15.77:0/3523676324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc5240 0x5653a2dd6210 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff874005d00 tx=0x7ff874005f90 comp rx=0 tx=0).stop 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff88e433740 1 -- 172.21.15.77:0/3523676324 shutdown_connections 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff88e433740 1 --2- 172.21.15.77:0/3523676324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dca3b0 0x5653a2dbe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff88e433740 1 --2- 172.21.15.77:0/3523676324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dc5240 0x5653a2dd6210 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff88e433740 1 --2- 172.21.15.77:0/3523676324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dc4850 0x5653a2dcf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff88e433740 1 -- 172.21.15.77:0/3523676324 >> 172.21.15.77:0/3523676324 conn(0x5653a2dbfdc0 msgr2=0x5653a2dce8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff88e433740 1 -- 172.21.15.77:0/3523676324 shutdown_connections 2024-03-16T10:17:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.489+0000 7ff88e433740 1 -- 172.21.15.77:0/3523676324 wait complete. 2024-03-16T10:17:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff88e433740 1 -- start start 2024-03-16T10:17:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff88e433740 1 Processor -- start 2024-03-16T10:17:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff88e433740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc4850 0x5653a2dd3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff88e433740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dc5240 0x5653a2dd4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff88e433740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dca3b0 0x5653a2dd5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff88e433740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff8841a7e80 con 0x5653a2dca3b0 2024-03-16T10:17:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff88e433740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff8841a77b0 con 0x5653a2dc5240 2024-03-16T10:17:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff88e433740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff8841a49f0 con 0x5653a2dc4850 2024-03-16T10:17:41.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff88e433740 10 client.0 _ll_register_callbacks cb 0x7ff874009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff889de8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dca3b0 0x5653a2dd5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff889de8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dca3b0 0x5653a2dd5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47932/0 (socket says 172.21.15.77:47932) 2024-03-16T10:17:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff889de8640 1 -- 172.21.15.77:0/1756197178 learned_addr learned my addr 172.21.15.77:0/1756197178 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff889de8640 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc4850 0x5653a2dd3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff888de6640 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dc5240 0x5653a2dd4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff889de8640 1 -- 172.21.15.77:0/1756197178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc4850 msgr2=0x5653a2dd3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff889de8640 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc4850 0x5653a2dd3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff889de8640 1 -- 172.21.15.77:0/1756197178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dc5240 msgr2=0x5653a2dd4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff889de8640 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dc5240 0x5653a2dd4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.490+0000 7ff889de8640 1 -- 172.21.15.77:0/1756197178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653a2dd5670 con 0x5653a2dca3b0 2024-03-16T10:17:41.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff889de8640 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dca3b0 0x5653a2dd5080 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff87c032410 tx=0x7ff87c032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3b401b66eef702 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff86cff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:41.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff86cff9640 1 -- 172.21.15.77:0/1756197178 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c04f040 con 0x5653a2dca3b0 2024-03-16T10:17:41.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff86cff9640 1 -- 172.21.15.77:0/1756197178 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ff87c034b90 con 0x5653a2dca3b0 2024-03-16T10:17:41.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff88e433740 1 -- 172.21.15.77:0/1756197178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff874022170 con 0x5653a2dca3b0 2024-03-16T10:17:41.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff86cff9640 1 -- 172.21.15.77:0/1756197178 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff87c044040 con 0x5653a2dca3b0 2024-03-16T10:17:41.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff88e433740 10 client.15621 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:41.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff88e433740 1 -- 172.21.15.77:0/1756197178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5653a2e40a40 con 0x5653a2dca3b0 2024-03-16T10:17:41.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff86cff9640 1 -- 172.21.15.77:0/1756197178 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff874022170 con 0x5653a2dca3b0 2024-03-16T10:17:41.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff86cff9640 1 -- 172.21.15.77:0/1756197178 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff87c05fb10 con 0x5653a2dca3b0 2024-03-16T10:17:41.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff88e433740 10 client.15621 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:41.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff88e433740 10 client.15621 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:17:41.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff88e433740 20 client.15621 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff88e433740 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5653a2e41380 0x5653a2e437a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff88e433740 4 client.15621 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:17:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff88e433740 1 -- 172.21.15.77:0/1756197178 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5653a2d30d20 con 0x5653a2e41380 2024-03-16T10:17:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.491+0000 7ff888de6640 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5653a2e41380 0x5653a2e437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff86cff9640 10 client.15621 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:17:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff888de6640 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5653a2e41380 0x5653a2e437a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=76ab5aa4fcec64aa server_cookie=e41ad37ca9fd939a in_seq=0 out_seq=0 2024-03-16T10:17:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff86cff9640 1 -- 172.21.15.77:0/1756197178 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5653a2d30d20 con 0x5653a2e41380 2024-03-16T10:17:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff86cff9640 10 client.15621 handle_command_reply: tid=0 2024-03-16T10:17:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff88e433740 1 client.15621 shutdown 2024-03-16T10:17:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff88e433740 1 -- 172.21.15.77:0/1756197178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dca3b0 msgr2=0x5653a2dd5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff88e433740 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dca3b0 0x5653a2dd5080 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff87c032410 tx=0x7ff87c032e90 comp rx=0 tx=0).stop 2024-03-16T10:17:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff88e433740 1 -- 172.21.15.77:0/1756197178 shutdown_connections 2024-03-16T10:17:41.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff88e433740 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5653a2e41380 0x5653a2e437a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff88e433740 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5653a2dca3b0 0x5653a2dd5080 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff88e433740 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653a2dc5240 0x5653a2dd4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff88e433740 1 --2- 172.21.15.77:0/1756197178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5653a2dc4850 0x5653a2dd3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff88e433740 1 -- 172.21.15.77:0/1756197178 >> 172.21.15.77:0/1756197178 conn(0x5653a2dbfdc0 msgr2=0x5653a2dc3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff86cff9640 0 client.15621 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:17:41.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff88e433740 1 -- 172.21.15.77:0/1756197178 shutdown_connections 2024-03-16T10:17:41.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff88e433740 1 -- 172.21.15.77:0/1756197178 wait complete. 2024-03-16T10:17:41.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.492+0000 7ff88e433740 20 client.15621 trim_cache size 0 max 16384 2024-03-16T10:17:41.503 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:17:41.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.493+0000 7ff88b84b640 1 -- 172.21.15.77:0/1704331301 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff8700a2910 msgr2=0x7ff8700a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.493+0000 7ff88b84b640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff8700a2910 0x7ff8700a4dd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x5653a2dc5cb0 tx=0x7ff880000bc0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.493+0000 7ff88b84b640 1 -- 172.21.15.77:0/1704331301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8841006e0 msgr2=0x7ff8841958c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.493+0000 7ff88b84b640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8841006e0 0x7ff8841958c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff880009be0 tx=0x7ff880007590 comp rx=0 tx=0).stop 2024-03-16T10:17:41.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.493+0000 7ff88b84b640 1 -- 172.21.15.77:0/1704331301 shutdown_connections 2024-03-16T10:17:41.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.493+0000 7ff88b84b640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff8700a2910 0x7ff8700a4dd0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.493+0000 7ff88b84b640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff884102290 0x7ff88419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.493+0000 7ff88b84b640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8841018d0 0x7ff884195e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.493+0000 7ff88b84b640 1 --2- 172.21.15.77:0/1704331301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8841006e0 0x7ff8841958c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.493+0000 7ff88b84b640 1 -- 172.21.15.77:0/1704331301 >> 172.21.15.77:0/1704331301 conn(0x7ff8840fbc70 msgr2=0x7ff884104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.493+0000 7ff88b84b640 1 -- 172.21.15.77:0/1704331301 shutdown_connections 2024-03-16T10:17:41.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.493+0000 7ff88b84b640 1 -- 172.21.15.77:0/1704331301 wait complete. 2024-03-16T10:17:41.505 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T10:17:41.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.609+0000 7f2588576640 1 -- 172.21.15.77:0/1885712546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2580102290 msgr2=0x7f2580108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.609+0000 7f2588576640 1 --2- 172.21.15.77:0/1885712546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2580102290 0x7f2580108b20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2578009ca0 tx=0x7f257802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 -- 172.21.15.77:0/1885712546 shutdown_connections 2024-03-16T10:17:41.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 --2- 172.21.15.77:0/1885712546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2580102290 0x7f2580108b20 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 --2- 172.21.15.77:0/1885712546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25801018d0 0x7f2580101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 --2- 172.21.15.77:0/1885712546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25801006e0 0x7f2580100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 -- 172.21.15.77:0/1885712546 >> 172.21.15.77:0/1885712546 conn(0x7f25800fbc70 msgr2=0x7f25800fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 -- 172.21.15.77:0/1885712546 shutdown_connections 2024-03-16T10:17:41.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 -- 172.21.15.77:0/1885712546 wait complete. 2024-03-16T10:17:41.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 Processor -- start 2024-03-16T10:17:41.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 -- start start 2024-03-16T10:17:41.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25801006e0 0x7f2580197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25801018d0 0x7f2580198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2580102290 0x7f258019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f258010c360 con 0x7f25801006e0 2024-03-16T10:17:41.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f258010c1e0 con 0x7f25801018d0 2024-03-16T10:17:41.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2588576640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f258010c4e0 con 0x7f2580102290 2024-03-16T10:17:41.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2586312640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25801006e0 0x7f2580197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2586312640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25801006e0 0x7f2580197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52342/0 (socket says 172.21.15.77:52342) 2024-03-16T10:17:41.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.610+0000 7f2586312640 1 -- 172.21.15.77:0/2811210514 learned_addr learned my addr 172.21.15.77:0/2811210514 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f2585b11640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25801018d0 0x7f2580198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f2586b13640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2580102290 0x7f258019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f2586312640 1 -- 172.21.15.77:0/2811210514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2580102290 msgr2=0x7f258019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f2586312640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2580102290 0x7f258019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f2586312640 1 -- 172.21.15.77:0/2811210514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25801018d0 msgr2=0x7f2580198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f2586312640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25801018d0 0x7f2580198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f2586312640 1 -- 172.21.15.77:0/2811210514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258019f720 con 0x7f25801006e0 2024-03-16T10:17:41.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f2586312640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25801006e0 0x7f2580197b00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f257000e9c0 tx=0x7f257000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f25777fe640 1 -- 172.21.15.77:0/2811210514 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257000cd80 con 0x7f25801006e0 2024-03-16T10:17:41.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f25777fe640 1 -- 172.21.15.77:0/2811210514 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25700035f0 con 0x7f25801006e0 2024-03-16T10:17:41.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f2588576640 1 -- 172.21.15.77:0/2811210514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f258019fa10 con 0x7f25801006e0 2024-03-16T10:17:41.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f25777fe640 1 -- 172.21.15.77:0/2811210514 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2570010690 con 0x7f25801006e0 2024-03-16T10:17:41.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.611+0000 7f2588576640 1 -- 172.21.15.77:0/2811210514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f258019ff60 con 0x7f25801006e0 2024-03-16T10:17:41.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.613+0000 7f25777fe640 1 -- 172.21.15.77:0/2811210514 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f25700037a0 con 0x7f25801006e0 2024-03-16T10:17:41.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f258b235740 1 Processor -- start 2024-03-16T10:17:41.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f258b235740 1 -- start start 2024-03-16T10:17:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f258b235740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 0x5615abd55c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f258b235740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd5ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f258b235740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd5d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f258b235740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5615abcba490 con 0x5615abd56240 2024-03-16T10:17:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f258b235740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5615abc62c90 con 0x5615abd55850 2024-03-16T10:17:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f258b235740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5615abcb6530 con 0x5615abd5b3b0 2024-03-16T10:17:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f2586312640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd5ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f25777fe640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f256c0a2910 0x7f256c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f2586b13640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 0x5615abd55c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f2586312640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd5ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52352/0 (socket says 172.21.15.77:52352) 2024-03-16T10:17:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f2586312640 1 -- 172.21.15.77:0/3597582889 learned_addr learned my addr 172.21.15.77:0/3597582889 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.614+0000 7f2585b11640 1 --2- 172.21.15.77:0/3597582889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd5d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f2586b13640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f256c0a2910 0x7f256c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f2585b11640 1 -- 172.21.15.77:0/3597582889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 msgr2=0x5615abd55c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f2585b11640 1 --2- 172.21.15.77:0/3597582889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 0x5615abd55c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f25777fe640 1 -- 172.21.15.77:0/2811210514 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f25700cedd0 con 0x7f25801006e0 2024-03-16T10:17:41.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f2585b11640 1 -- 172.21.15.77:0/3597582889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 msgr2=0x5615abd5ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f2585b11640 1 --2- 172.21.15.77:0/3597582889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd5ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f2585b11640 1 -- 172.21.15.77:0/3597582889 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615abd1d500 con 0x5615abd5b3b0 2024-03-16T10:17:41.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f2586312640 1 --2- 172.21.15.77:0/3597582889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd5ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:41.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f2585b11640 1 --2- 172.21.15.77:0/3597582889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd5d840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f257c009bf0 tx=0x7f257c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ac38ddf0e21e5a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f2586b13640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f256c0a2910 0x7f256c0a4dd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2578005e30 tx=0x7f257802c0f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f2574bfe640 1 -- 172.21.15.77:0/3597582889 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c01e070 con 0x5615abd5b3b0 2024-03-16T10:17:41.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f2574bfe640 1 -- 172.21.15.77:0/3597582889 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f257c009da0 con 0x5615abd5b3b0 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.615+0000 7f2574bfe640 1 -- 172.21.15.77:0/3597582889 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c018840 con 0x5615abd5b3b0 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.616+0000 7f258b235740 1 -- 172.21.15.77:0/3597582889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 msgr2=0x5615abd5d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.616+0000 7f258b235740 1 --2- 172.21.15.77:0/3597582889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd5d840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f257c009bf0 tx=0x7f257c011810 comp rx=0 tx=0).stop 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.616+0000 7f258b235740 1 -- 172.21.15.77:0/3597582889 shutdown_connections 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.616+0000 7f258b235740 1 --2- 172.21.15.77:0/3597582889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd5d840 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.616+0000 7f258b235740 1 --2- 172.21.15.77:0/3597582889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd5ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.616+0000 7f258b235740 1 --2- 172.21.15.77:0/3597582889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 0x5615abd55c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.616+0000 7f258b235740 1 -- 172.21.15.77:0/3597582889 >> 172.21.15.77:0/3597582889 conn(0x5615abd50dc0 msgr2=0x5615abd511c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.616+0000 7f258b235740 1 -- 172.21.15.77:0/3597582889 shutdown_connections 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.616+0000 7f258b235740 1 -- 172.21.15.77:0/3597582889 wait complete. 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.616+0000 7f258b235740 1 -- start start 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.616+0000 7f258b235740 1 Processor -- start 2024-03-16T10:17:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.616+0000 7f258b235740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 0x5615abd5fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f258b235740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd602c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f258b235740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd672c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f258b235740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5615abd5e360 con 0x5615abd56240 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f258b235740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5615abc76230 con 0x5615abd55850 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f258b235740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5615abd5e4e0 con 0x5615abd5b3b0 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2585b11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f258b235740 10 client.0 _ll_register_callbacks cb 0x5615abcb2010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2585b11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd602c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52360/0 (socket says 172.21.15.77:52360) 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2585b11640 1 -- 172.21.15.77:0/1453730913 learned_addr learned my addr 172.21.15.77:0/1453730913 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2586312640 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 0x5615abd5fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2586b13640 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd672c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2585b11640 1 -- 172.21.15.77:0/1453730913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 msgr2=0x5615abd672c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2585b11640 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd672c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2585b11640 1 -- 172.21.15.77:0/1453730913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 msgr2=0x5615abd5fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2585b11640 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 0x5615abd5fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2585b11640 1 -- 172.21.15.77:0/1453730913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615abc85100 con 0x5615abd56240 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2585b11640 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd602c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f257c01c040 tx=0x7f257c019eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c3392ea4b4b056 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2569ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2569ffb640 1 -- 172.21.15.77:0/1453730913 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c026020 con 0x5615abd56240 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2569ffb640 1 -- 172.21.15.77:0/1453730913 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f257c005050 con 0x5615abd56240 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f258b235740 1 -- 172.21.15.77:0/1453730913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f257c009da0 con 0x5615abd56240 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f2569ffb640 1 -- 172.21.15.77:0/1453730913 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f257c031050 con 0x5615abd56240 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f258b235740 10 client.15633 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.617+0000 7f258b235740 1 -- 172.21.15.77:0/1453730913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5615ab9ce3f0 con 0x5615abd56240 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.618+0000 7f2569ffb640 1 -- 172.21.15.77:0/1453730913 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f257c009da0 con 0x5615abd56240 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.618+0000 7f2569ffb640 1 -- 172.21.15.77:0/1453730913 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f257c021030 con 0x5615abd56240 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.618+0000 7f258b235740 10 client.15633 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.618+0000 7f258b235740 10 client.15633 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:17:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.618+0000 7f258b235740 20 client.15633 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.618+0000 7f258b235740 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5615abd6b830 0x5615abd6dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.618+0000 7f258b235740 4 client.15633 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.618+0000 7f258b235740 1 -- 172.21.15.77:0/1453730913 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615abcc79d0 con 0x5615abd6b830 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.618+0000 7f2586312640 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5615abd6b830 0x5615abd6dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.618+0000 7f2586312640 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5615abd6b830 0x5615abd6dc80 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d895cf94c48aaeb9 server_cookie=e9825820b2c9dc8d in_seq=0 out_seq=0 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.618+0000 7f2569ffb640 10 client.15633 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f2569ffb640 1 -- 172.21.15.77:0/1453730913 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5615abcc79d0 con 0x5615abd6b830 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f2569ffb640 10 client.15633 handle_command_reply: tid=0 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f258b235740 1 client.15633 shutdown 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f258b235740 1 -- 172.21.15.77:0/1453730913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 msgr2=0x5615abd602c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f258b235740 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd602c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f257c01c040 tx=0x7f257c019eb0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f258b235740 1 -- 172.21.15.77:0/1453730913 shutdown_connections 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f258b235740 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd672c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f258b235740 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd602c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f258b235740 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5615abd6b830 0x5615abd6dc80 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f258b235740 1 --2- 172.21.15.77:0/1453730913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 0x5615abd5fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f2569ffb640 0 client.15633 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:17:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f258b235740 1 -- 172.21.15.77:0/1453730913 >> 172.21.15.77:0/1453730913 conn(0x5615abd50dc0 msgr2=0x5615abd4ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f258b235740 1 -- 172.21.15.77:0/1453730913 shutdown_connections 2024-03-16T10:17:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f258b235740 1 -- 172.21.15.77:0/1453730913 wait complete. 2024-03-16T10:17:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.619+0000 7f258b235740 20 client.15633 trim_cache size 0 max 16384 2024-03-16T10:17:41.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f258b235740 1 Processor -- start 2024-03-16T10:17:41.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f258b235740 1 -- start start 2024-03-16T10:17:41.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f258b235740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd55850 0x5615abd4f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f258b235740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd56240 0x5615abd67280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f258b235740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd5b3b0 0x5615abd4fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f258b235740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f25801a8c20 con 0x5615abd5b3b0 2024-03-16T10:17:41.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f258b235740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f25801aa760 con 0x5615abd56240 2024-03-16T10:17:41.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f258b235740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25801a92f0 con 0x5615abd55850 2024-03-16T10:17:41.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f2586b13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd5b3b0 0x5615abd4fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f2586b13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd5b3b0 0x5615abd4fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52376/0 (socket says 172.21.15.77:52376) 2024-03-16T10:17:41.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f2586b13640 1 -- 172.21.15.77:0/3415758027 learned_addr learned my addr 172.21.15.77:0/3415758027 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f2586312640 1 --2- 172.21.15.77:0/3415758027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd55850 0x5615abd4f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.635+0000 7f2585b11640 1 --2- 172.21.15.77:0/3415758027 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd56240 0x5615abd67280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.636+0000 7f2586b13640 1 -- 172.21.15.77:0/3415758027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd55850 msgr2=0x5615abd4f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.636+0000 7f2586b13640 1 --2- 172.21.15.77:0/3415758027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd55850 0x5615abd4f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.636+0000 7f2586b13640 1 -- 172.21.15.77:0/3415758027 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd56240 msgr2=0x5615abd67280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.636+0000 7f2586b13640 1 --2- 172.21.15.77:0/3415758027 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd56240 0x5615abd67280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.636+0000 7f2586b13640 1 -- 172.21.15.77:0/3415758027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615aba987d0 con 0x5615abd5b3b0 2024-03-16T10:17:41.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.636+0000 7f2586b13640 1 --2- 172.21.15.77:0/3415758027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd5b3b0 0x5615abd4fc90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2578036000 tx=0x7f2578009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b66e893f4c55151 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.636+0000 7f256bfff640 1 -- 172.21.15.77:0/3415758027 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2578039040 con 0x5615abd5b3b0 2024-03-16T10:17:41.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.636+0000 7f256bfff640 1 -- 172.21.15.77:0/3415758027 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2578054070 con 0x5615abd5b3b0 2024-03-16T10:17:41.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.636+0000 7f256bfff640 1 -- 172.21.15.77:0/3415758027 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257804f690 con 0x5615abd5b3b0 2024-03-16T10:17:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.637+0000 7f258b235740 1 -- 172.21.15.77:0/3415758027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd5b3b0 msgr2=0x5615abd4fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.637+0000 7f258b235740 1 --2- 172.21.15.77:0/3415758027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd5b3b0 0x5615abd4fc90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2578036000 tx=0x7f2578009360 comp rx=0 tx=0).stop 2024-03-16T10:17:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.637+0000 7f258b235740 1 -- 172.21.15.77:0/3415758027 shutdown_connections 2024-03-16T10:17:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.637+0000 7f258b235740 1 --2- 172.21.15.77:0/3415758027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd5b3b0 0x5615abd4fc90 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.637+0000 7f258b235740 1 --2- 172.21.15.77:0/3415758027 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd56240 0x5615abd67280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.637+0000 7f258b235740 1 --2- 172.21.15.77:0/3415758027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd55850 0x5615abd4f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.637+0000 7f258b235740 1 -- 172.21.15.77:0/3415758027 >> 172.21.15.77:0/3415758027 conn(0x5615abd50dc0 msgr2=0x5615abd57740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.637+0000 7f258b235740 1 -- 172.21.15.77:0/3415758027 shutdown_connections 2024-03-16T10:17:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.637+0000 7f258b235740 1 -- 172.21.15.77:0/3415758027 wait complete. 2024-03-16T10:17:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.637+0000 7f258b235740 1 -- start start 2024-03-16T10:17:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.637+0000 7f258b235740 1 Processor -- start 2024-03-16T10:17:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f258b235740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 0x5615abd64620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f258b235740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd64b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f258b235740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd65b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f258b235740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f25801aa090 con 0x5615abd56240 2024-03-16T10:17:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f258b235740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f25801a99c0 con 0x5615abd55850 2024-03-16T10:17:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f258b235740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25801a6c00 con 0x5615abd5b3b0 2024-03-16T10:17:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f258b235740 10 client.0 _ll_register_callbacks cb 0x7f257c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2585b11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd64b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2585b11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd64b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52382/0 (socket says 172.21.15.77:52382) 2024-03-16T10:17:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2585b11640 1 -- 172.21.15.77:0/3096501457 learned_addr learned my addr 172.21.15.77:0/3096501457 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2586312640 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 0x5615abd64620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2586b13640 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd65b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2585b11640 1 -- 172.21.15.77:0/3096501457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 msgr2=0x5615abd65b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2585b11640 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd65b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2585b11640 1 -- 172.21.15.77:0/3096501457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 msgr2=0x5615abd64620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2585b11640 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 0x5615abd64620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2585b11640 1 -- 172.21.15.77:0/3096501457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615abd66160 con 0x5615abd56240 2024-03-16T10:17:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2585b11640 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd64b60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x5615abd4c910 tx=0x7f257c0095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402d9dc8e4ee7de2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2569ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2569ffb640 1 -- 172.21.15.77:0/3096501457 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c026040 con 0x5615abd56240 2024-03-16T10:17:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2569ffb640 1 -- 172.21.15.77:0/3096501457 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f257c019ca0 con 0x5615abd56240 2024-03-16T10:17:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f258b235740 1 -- 172.21.15.77:0/3096501457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2578054070 con 0x5615abd56240 2024-03-16T10:17:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f2569ffb640 1 -- 172.21.15.77:0/3096501457 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f257c03d050 con 0x5615abd56240 2024-03-16T10:17:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f258b235740 10 client.15639 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.638+0000 7f258b235740 1 -- 172.21.15.77:0/3096501457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5615abd67460 con 0x5615abd56240 2024-03-16T10:17:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f2569ffb640 1 -- 172.21.15.77:0/3096501457 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2578054070 con 0x5615abd56240 2024-03-16T10:17:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f2569ffb640 1 -- 172.21.15.77:0/3096501457 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f257c021070 con 0x5615abd56240 2024-03-16T10:17:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f258b235740 10 client.15639 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f258b235740 10 client.15639 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:17:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f258b235740 20 client.15639 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f258b235740 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5615abdd21c0 0x5615abdd45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f258b235740 4 client.15639 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:17:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f258b235740 1 -- 172.21.15.77:0/3096501457 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615abcc1d20 con 0x5615abdd21c0 2024-03-16T10:17:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f2586312640 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5615abdd21c0 0x5615abdd45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f2569ffb640 10 client.15639 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:17:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f2586312640 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5615abdd21c0 0x5615abdd45e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cea0bf247c4a527d server_cookie=365115ff31b2d55f in_seq=0 out_seq=0 2024-03-16T10:17:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f2569ffb640 1 -- 172.21.15.77:0/3096501457 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615abcc1d20 con 0x5615abdd21c0 2024-03-16T10:17:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f2569ffb640 10 client.15639 handle_command_reply: tid=0 2024-03-16T10:17:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.639+0000 7f258b235740 1 client.15639 shutdown 2024-03-16T10:17:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f258b235740 1 -- 172.21.15.77:0/3096501457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 msgr2=0x5615abd64b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f258b235740 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd64b60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x5615abd4c910 tx=0x7f257c0095b0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f258b235740 1 -- 172.21.15.77:0/3096501457 shutdown_connections 2024-03-16T10:17:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f258b235740 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615abd5b3b0 0x5615abd65b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f258b235740 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615abd56240 0x5615abd64b60 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f258b235740 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5615abdd21c0 0x5615abdd45e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f258b235740 1 --2- 172.21.15.77:0/3096501457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5615abd55850 0x5615abd64620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2569ffb640 0 client.15639 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:17:41.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f258b235740 1 -- 172.21.15.77:0/3096501457 >> 172.21.15.77:0/3096501457 conn(0x5615abd50dc0 msgr2=0x5615abd54020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f258b235740 1 -- 172.21.15.77:0/3096501457 shutdown_connections 2024-03-16T10:17:41.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f258b235740 1 -- 172.21.15.77:0/3096501457 wait complete. 2024-03-16T10:17:41.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f258b235740 20 client.15639 trim_cache size 0 max 16384 2024-03-16T10:17:41.648 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:17:41.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2588576640 1 -- 172.21.15.77:0/2811210514 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f256c0a2910 msgr2=0x7f256c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2588576640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f256c0a2910 0x7f256c0a4dd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2578005e30 tx=0x7f257802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2588576640 1 -- 172.21.15.77:0/2811210514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25801006e0 msgr2=0x7f2580197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2588576640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25801006e0 0x7f2580197b00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f257000e9c0 tx=0x7f257000ee80 comp rx=0 tx=0).stop 2024-03-16T10:17:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2588576640 1 -- 172.21.15.77:0/2811210514 shutdown_connections 2024-03-16T10:17:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2588576640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f256c0a2910 0x7f256c0a4dd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2588576640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2580102290 0x7f258019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2588576640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25801018d0 0x7f2580198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2588576640 1 --2- 172.21.15.77:0/2811210514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25801006e0 0x7f2580197b00 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2588576640 1 -- 172.21.15.77:0/2811210514 >> 172.21.15.77:0/2811210514 conn(0x7f25800fbc70 msgr2=0x7f2580068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2588576640 1 -- 172.21.15.77:0/2811210514 shutdown_connections 2024-03-16T10:17:41.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.640+0000 7f2588576640 1 -- 172.21.15.77:0/2811210514 wait complete. 2024-03-16T10:17:41.652 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T10:17:41.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.756+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3246275567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9dc1006e0 msgr2=0x7fa9dc100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.756+0000 7fa9e1aae640 1 --2- 172.21.15.77:0/3246275567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9dc1006e0 0x7fa9dc100b00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009900 tx=0x7fa9d402bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:41.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.756+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3246275567 shutdown_connections 2024-03-16T10:17:41.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.756+0000 7fa9e1aae640 1 --2- 172.21.15.77:0/3246275567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9dc102290 0x7fa9dc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.756+0000 7fa9e1aae640 1 --2- 172.21.15.77:0/3246275567 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9dc1018d0 0x7fa9dc101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.756+0000 7fa9e1aae640 1 --2- 172.21.15.77:0/3246275567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9dc1006e0 0x7fa9dc100b00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.756+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3246275567 >> 172.21.15.77:0/3246275567 conn(0x7fa9dc0fbc70 msgr2=0x7fa9dc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.756+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3246275567 shutdown_connections 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.756+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3246275567 wait complete. 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.757+0000 7fa9e1aae640 1 Processor -- start 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.757+0000 7fa9e1aae640 1 -- start start 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.757+0000 7fa9e1aae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9dc1006e0 0x7fa9dc10b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.757+0000 7fa9e1aae640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9dc1018d0 0x7fa9dc1096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.757+0000 7fa9e1aae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9dc102290 0x7fa9dc109be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.757+0000 7fa9e1aae640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc10d480 con 0x7fa9dc1006e0 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.757+0000 7fa9e1aae640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc10d300 con 0x7fa9dc102290 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.757+0000 7fa9e1aae640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc10d600 con 0x7fa9dc1018d0 2024-03-16T10:17:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.757+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9dc1006e0 0x7fa9dc10b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.757+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9dc1006e0 0x7fa9dc10b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52412/0 (socket says 172.21.15.77:52412) 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.757+0000 7fa9db7fe640 1 -- 172.21.15.77:0/3156979651 learned_addr learned my addr 172.21.15.77:0/3156979651 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.757+0000 7fa9daffd640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9dc1018d0 0x7fa9dc1096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.758+0000 7fa9dbfff640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9dc102290 0x7fa9dc109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.758+0000 7fa9db7fe640 1 -- 172.21.15.77:0/3156979651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9dc1018d0 msgr2=0x7fa9dc1096a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.758+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9dc1018d0 0x7fa9dc1096a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.758+0000 7fa9db7fe640 1 -- 172.21.15.77:0/3156979651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9dc102290 msgr2=0x7fa9dc109be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.758+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9dc102290 0x7fa9dc109be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.758+0000 7fa9db7fe640 1 -- 172.21.15.77:0/3156979651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9dc10a470 con 0x7fa9dc1006e0 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.758+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9dc1006e0 0x7fa9dc10b060 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4000c00 tx=0x7fa9d402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.758+0000 7fa9d8ff9640 1 -- 172.21.15.77:0/3156979651 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4041070 con 0x7fa9dc1006e0 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.758+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3156979651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9dc1a1c00 con 0x7fa9dc1006e0 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.758+0000 7fa9d8ff9640 1 -- 172.21.15.77:0/3156979651 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d4004450 con 0x7fa9dc1006e0 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.758+0000 7fa9d8ff9640 1 -- 172.21.15.77:0/3156979651 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4034440 con 0x7fa9dc1006e0 2024-03-16T10:17:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.758+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3156979651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9dc1a2150 con 0x7fa9dc1006e0 2024-03-16T10:17:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.760+0000 7fa9d8ff9640 1 -- 172.21.15.77:0/3156979651 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fa9d4044070 con 0x7fa9dc1006e0 2024-03-16T10:17:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9e4770740 1 Processor -- start 2024-03-16T10:17:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9e4770740 1 -- start start 2024-03-16T10:17:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9d8ff9640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa9c80a2890 0x7fa9c80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9e4770740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 0x5637d4daac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9e4770740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4dab240 0x5637d4dafd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9e4770740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4db2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9e4770740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5637d4d0f490 con 0x5637d4daa850 2024-03-16T10:17:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9e4770740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5637d4cb7c90 con 0x5637d4db03b0 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9e4770740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5637d4d0b530 con 0x5637d4dab240 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9dbfff640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa9c80a2890 0x7fa9c80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9d8ff9640 1 -- 172.21.15.77:0/3156979651 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa9d40f04b0 con 0x7fa9dc1006e0 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9daffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4dab240 0x5637d4dafd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 0x5637d4daac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9daffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4dab240 0x5637d4dafd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58238/0 (socket says 172.21.15.77:58238) 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9daffd640 1 -- 172.21.15.77:0/4032310405 learned_addr learned my addr 172.21.15.77:0/4032310405 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9dbfff640 1 --2- 172.21.15.77:0/4032310405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4db2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9db7fe640 1 -- 172.21.15.77:0/4032310405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4dab240 msgr2=0x5637d4dafd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/4032310405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4dab240 0x5637d4dafd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9db7fe640 1 -- 172.21.15.77:0/4032310405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 msgr2=0x5637d4db2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/4032310405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4db2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.761+0000 7fa9db7fe640 1 -- 172.21.15.77:0/4032310405 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637d4d72500 con 0x5637d4daa850 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.762+0000 7fa9dbfff640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa9c80a2890 0x7fa9c80a4d50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc009fd0 tx=0x7fa9cc009b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.762+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/4032310405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 0x5637d4daac70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40ee060 tx=0x7fa9d40b8c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2e81d2cfe093ab6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.762+0000 7fa9d1ffb640 1 -- 172.21.15.77:0/4032310405 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d404b070 con 0x5637d4daa850 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.762+0000 7fa9d1ffb640 1 -- 172.21.15.77:0/4032310405 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d4004c80 con 0x5637d4daa850 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.762+0000 7fa9d1ffb640 1 -- 172.21.15.77:0/4032310405 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40c10d0 con 0x5637d4daa850 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.762+0000 7fa9e4770740 1 -- 172.21.15.77:0/4032310405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 msgr2=0x5637d4daac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.762+0000 7fa9e4770740 1 --2- 172.21.15.77:0/4032310405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 0x5637d4daac70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40ee060 tx=0x7fa9d40b8c50 comp rx=0 tx=0).stop 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.762+0000 7fa9e4770740 1 -- 172.21.15.77:0/4032310405 shutdown_connections 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.762+0000 7fa9e4770740 1 --2- 172.21.15.77:0/4032310405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4db2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.762+0000 7fa9e4770740 1 --2- 172.21.15.77:0/4032310405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4dab240 0x5637d4dafd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.762+0000 7fa9e4770740 1 --2- 172.21.15.77:0/4032310405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 0x5637d4daac70 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9e4770740 1 -- 172.21.15.77:0/4032310405 >> 172.21.15.77:0/4032310405 conn(0x5637d4da5dc0 msgr2=0x5637d4da61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9e4770740 1 -- 172.21.15.77:0/4032310405 shutdown_connections 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9e4770740 1 -- 172.21.15.77:0/4032310405 wait complete. 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9e4770740 1 -- start start 2024-03-16T10:17:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9e4770740 1 Processor -- start 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9e4770740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4daa850 0x5637d4db49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9e4770740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 0x5637d4db4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9e4770740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4dbbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9e4770740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5637d4db3120 con 0x5637d4dab240 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9e4770740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5637d4ccb230 con 0x5637d4db03b0 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9e4770740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5637d4db32a0 con 0x5637d4daa850 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9e4770740 10 client.0 _ll_register_callbacks cb 0x5637d4d07010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9daffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 0x5637d4db4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9daffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 0x5637d4db4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52418/0 (socket says 172.21.15.77:52418) 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4daa850 0x5637d4db49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9daffd640 1 -- 172.21.15.77:0/1536669172 learned_addr learned my addr 172.21.15.77:0/1536669172 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.763+0000 7fa9dbfff640 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4dbbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9daffd640 1 -- 172.21.15.77:0/1536669172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4daa850 msgr2=0x5637d4db49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9daffd640 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4daa850 0x5637d4db49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9daffd640 1 -- 172.21.15.77:0/1536669172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 msgr2=0x5637d4dbbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9daffd640 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4dbbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9daffd640 1 -- 172.21.15.77:0/1536669172 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637d4cda100 con 0x5637d4dab240 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4daa850 0x5637d4db49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9dbfff640 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4dbbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9daffd640 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 0x5637d4db4f30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4011d10 tx=0x7fa9c400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449025da950b0d96 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9baffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9baffd640 1 -- 172.21.15.77:0/1536669172 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4003970 con 0x5637d4dab240 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9baffd640 1 -- 172.21.15.77:0/1536669172 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa9c4003b10 con 0x5637d4dab240 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9e4770740 1 -- 172.21.15.77:0/1536669172 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9d4004c80 con 0x5637d4dab240 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9baffd640 1 -- 172.21.15.77:0/1536669172 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa9c4003d10 con 0x5637d4dab240 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9e4770740 10 client.15651 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.764+0000 7fa9e4770740 1 -- 172.21.15.77:0/1536669172 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5637d4a233f0 con 0x5637d4dab240 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.765+0000 7fa9baffd640 1 -- 172.21.15.77:0/1536669172 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d4004c80 con 0x5637d4dab240 2024-03-16T10:17:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.765+0000 7fa9baffd640 1 -- 172.21.15.77:0/1536669172 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa9c4019070 con 0x5637d4dab240 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.765+0000 7fa9e4770740 10 client.15651 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.765+0000 7fa9e4770740 10 client.15651 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.765+0000 7fa9e4770740 20 client.15651 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.765+0000 7fa9e4770740 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x5637d4dc08a0 0x5637d4dc2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.765+0000 7fa9e4770740 4 client.15651 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.765+0000 7fa9e4770740 1 -- 172.21.15.77:0/1536669172 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637d4d1c9d0 con 0x5637d4dc08a0 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.765+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x5637d4dc08a0 0x5637d4dc2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.765+0000 7fa9baffd640 10 client.15651 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.765+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x5637d4dc08a0 0x5637d4dc2d40 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2084e35666bf5d83 server_cookie=9a159733969f78fb in_seq=0 out_seq=0 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9baffd640 1 -- 172.21.15.77:0/1536669172 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5637d4d1c9d0 con 0x5637d4dc08a0 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9baffd640 10 client.15651 handle_command_reply: tid=0 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9e4770740 1 client.15651 shutdown 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9e4770740 1 -- 172.21.15.77:0/1536669172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 msgr2=0x5637d4db4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9e4770740 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 0x5637d4db4f30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4011d10 tx=0x7fa9c400f790 comp rx=0 tx=0).stop 2024-03-16T10:17:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9e4770740 1 -- 172.21.15.77:0/1536669172 shutdown_connections 2024-03-16T10:17:41.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9e4770740 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x5637d4dc08a0 0x5637d4dc2d40 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9e4770740 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4dbbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9e4770740 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 0x5637d4db4f30 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9e4770740 1 --2- 172.21.15.77:0/1536669172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4daa850 0x5637d4db49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9baffd640 0 client.15651 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:17:41.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9e4770740 1 -- 172.21.15.77:0/1536669172 >> 172.21.15.77:0/1536669172 conn(0x5637d4da5dc0 msgr2=0x5637d4da4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9e4770740 1 -- 172.21.15.77:0/1536669172 shutdown_connections 2024-03-16T10:17:41.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9e4770740 1 -- 172.21.15.77:0/1536669172 wait complete. 2024-03-16T10:17:41.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.766+0000 7fa9e4770740 20 client.15651 trim_cache size 0 max 16384 2024-03-16T10:17:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9e4770740 1 Processor -- start 2024-03-16T10:17:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9e4770740 1 -- start start 2024-03-16T10:17:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9e4770740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 0x5637d4da61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9e4770740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4dab240 0x5637d4dbbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9e4770740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4da4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9e4770740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc1aade0 con 0x5637d4daa850 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9e4770740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc1ac920 con 0x5637d4db03b0 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9e4770740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc1ab4b0 con 0x5637d4dab240 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 0x5637d4da61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 0x5637d4da61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52428/0 (socket says 172.21.15.77:52428) 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9db7fe640 1 -- 172.21.15.77:0/3327351901 learned_addr learned my addr 172.21.15.77:0/3327351901 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9daffd640 1 --2- 172.21.15.77:0/3327351901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4dab240 0x5637d4dbbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9dbfff640 1 --2- 172.21.15.77:0/3327351901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4da4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9db7fe640 1 -- 172.21.15.77:0/3327351901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4dab240 msgr2=0x5637d4dbbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/3327351901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4dab240 0x5637d4dbbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9db7fe640 1 -- 172.21.15.77:0/3327351901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 msgr2=0x5637d4da4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/3327351901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4da4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9db7fe640 1 -- 172.21.15.77:0/3327351901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637d4aed7d0 con 0x5637d4daa850 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/3327351901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 0x5637d4da61a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa9d402cfd0 tx=0x7fa9d4003990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbe18541568b240 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9d17fa640 1 -- 172.21.15.77:0/3327351901 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d404b040 con 0x5637d4daa850 2024-03-16T10:17:41.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9d17fa640 1 -- 172.21.15.77:0/3327351901 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d4004c80 con 0x5637d4daa850 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.780+0000 7fa9d17fa640 1 -- 172.21.15.77:0/3327351901 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40bb330 con 0x5637d4daa850 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.781+0000 7fa9e4770740 1 -- 172.21.15.77:0/3327351901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 msgr2=0x5637d4da61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.781+0000 7fa9e4770740 1 --2- 172.21.15.77:0/3327351901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 0x5637d4da61a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa9d402cfd0 tx=0x7fa9d4003990 comp rx=0 tx=0).stop 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.781+0000 7fa9e4770740 1 -- 172.21.15.77:0/3327351901 shutdown_connections 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.781+0000 7fa9e4770740 1 --2- 172.21.15.77:0/3327351901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4da4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.781+0000 7fa9e4770740 1 --2- 172.21.15.77:0/3327351901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4dab240 0x5637d4dbbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.781+0000 7fa9e4770740 1 --2- 172.21.15.77:0/3327351901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4daa850 0x5637d4da61a0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.781+0000 7fa9e4770740 1 -- 172.21.15.77:0/3327351901 >> 172.21.15.77:0/3327351901 conn(0x5637d4da5dc0 msgr2=0x5637d4da10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.781+0000 7fa9e4770740 1 -- 172.21.15.77:0/3327351901 shutdown_connections 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.781+0000 7fa9e4770740 1 -- 172.21.15.77:0/3327351901 wait complete. 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.781+0000 7fa9e4770740 1 -- start start 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9e4770740 1 Processor -- start 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9e4770740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4daa850 0x5637d4db8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9e4770740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 0x5637d4db92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9e4770740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4e24c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9e4770740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc1ac250 con 0x5637d4dab240 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9e4770740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc1abb80 con 0x5637d4db03b0 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9e4770740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc1a8dc0 con 0x5637d4daa850 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9e4770740 10 client.0 _ll_register_callbacks cb 0x7fa9c4011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9daffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 0x5637d4db92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9daffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 0x5637d4db92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52438/0 (socket says 172.21.15.77:52438) 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9daffd640 1 -- 172.21.15.77:0/2863583472 learned_addr learned my addr 172.21.15.77:0/2863583472 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4daa850 0x5637d4db8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9dbfff640 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4e24c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9daffd640 1 -- 172.21.15.77:0/2863583472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4daa850 msgr2=0x5637d4db8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9daffd640 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4daa850 0x5637d4db8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9daffd640 1 -- 172.21.15.77:0/2863583472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 msgr2=0x5637d4e24c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9daffd640 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4e24c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9daffd640 1 -- 172.21.15.77:0/2863583472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637d4e25170 con 0x5637d4dab240 2024-03-16T10:17:41.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9daffd640 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 0x5637d4db92e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4006010 tx=0x7fa9c40118a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bcfafe8d10f7fb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9baffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.782+0000 7fa9baffd640 1 -- 172.21.15.77:0/2863583472 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4029040 con 0x5637d4dab240 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9baffd640 1 -- 172.21.15.77:0/2863583472 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa9c401a8d0 con 0x5637d4dab240 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9e4770740 1 -- 172.21.15.77:0/2863583472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9d4004c80 con 0x5637d4dab240 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9baffd640 1 -- 172.21.15.77:0/2863583472 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa9c401aaf0 con 0x5637d4dab240 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9e4770740 10 client.15657 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9e4770740 1 -- 172.21.15.77:0/2863583472 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5637d4e26370 con 0x5637d4dab240 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9baffd640 1 -- 172.21.15.77:0/2863583472 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d4004c80 con 0x5637d4dab240 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9baffd640 1 -- 172.21.15.77:0/2863583472 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa9c402c070 con 0x5637d4dab240 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9e4770740 10 client.15657 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9e4770740 10 client.15657 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9e4770740 20 client.15657 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9e4770740 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x5637d4e26da0 0x5637d4e291c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9e4770740 4 client.15657 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9e4770740 1 -- 172.21.15.77:0/2863583472 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5637d4d16d20 con 0x5637d4e26da0 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.783+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x5637d4e26da0 0x5637d4e291c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9db7fe640 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x5637d4e26da0 0x5637d4e291c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=699feee1df7a67eb server_cookie=8aa46df537ce8193 in_seq=0 out_seq=0 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9baffd640 10 client.15657 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9baffd640 1 -- 172.21.15.77:0/2863583472 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5637d4d16d20 con 0x5637d4e26da0 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9baffd640 10 client.15657 handle_command_reply: tid=0 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9e4770740 1 client.15657 shutdown 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9e4770740 1 -- 172.21.15.77:0/2863583472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 msgr2=0x5637d4db92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9e4770740 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 0x5637d4db92e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4006010 tx=0x7fa9c40118a0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9e4770740 1 -- 172.21.15.77:0/2863583472 shutdown_connections 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9e4770740 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x5637d4e26da0 0x5637d4e291c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9e4770740 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5637d4db03b0 0x5637d4e24c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9e4770740 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5637d4dab240 0x5637d4db92e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9baffd640 0 client.15657 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:17:41.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9e4770740 1 --2- 172.21.15.77:0/2863583472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5637d4daa850 0x5637d4db8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9e4770740 1 -- 172.21.15.77:0/2863583472 >> 172.21.15.77:0/2863583472 conn(0x5637d4da5dc0 msgr2=0x5637d4da61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9e4770740 1 -- 172.21.15.77:0/2863583472 shutdown_connections 2024-03-16T10:17:41.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9e4770740 1 -- 172.21.15.77:0/2863583472 wait complete. 2024-03-16T10:17:41.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.784+0000 7fa9e4770740 20 client.15657 trim_cache size 0 max 16384 2024-03-16T10:17:41.792 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:17:41.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.785+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3156979651 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa9c80a2890 msgr2=0x7fa9c80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.785+0000 7fa9e1aae640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa9c80a2890 0x7fa9c80a4d50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc009fd0 tx=0x7fa9cc009b80 comp rx=0 tx=0).stop 2024-03-16T10:17:41.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.785+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3156979651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9dc1006e0 msgr2=0x7fa9dc10b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.785+0000 7fa9e1aae640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9dc1006e0 0x7fa9dc10b060 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4000c00 tx=0x7fa9d402c640 comp rx=0 tx=0).stop 2024-03-16T10:17:41.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.785+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3156979651 shutdown_connections 2024-03-16T10:17:41.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.785+0000 7fa9e1aae640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9dc102290 0x7fa9dc109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.785+0000 7fa9e1aae640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa9c80a2890 0x7fa9c80a4d50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.785+0000 7fa9e1aae640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9dc1018d0 0x7fa9dc1096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.785+0000 7fa9e1aae640 1 --2- 172.21.15.77:0/3156979651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9dc1006e0 0x7fa9dc10b060 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.785+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3156979651 >> 172.21.15.77:0/3156979651 conn(0x7fa9dc0fbc70 msgr2=0x7fa9dc104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.785+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3156979651 shutdown_connections 2024-03-16T10:17:41.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.785+0000 7fa9e1aae640 1 -- 172.21.15.77:0/3156979651 wait complete. 2024-03-16T10:17:41.797 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T10:17:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.904+0000 7f4929a9a640 1 -- 172.21.15.77:0/2894472014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49241033b0 msgr2=0x7f4924109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.904+0000 7f4929a9a640 1 --2- 172.21.15.77:0/2894472014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49241033b0 0x7f4924109c40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f491c009be0 tx=0x7f491c02bef0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.904+0000 7f4929a9a640 1 -- 172.21.15.77:0/2894472014 shutdown_connections 2024-03-16T10:17:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.904+0000 7f4929a9a640 1 --2- 172.21.15.77:0/2894472014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49241033b0 0x7f4924109c40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.904+0000 7f4929a9a640 1 --2- 172.21.15.77:0/2894472014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49241029f0 0x7f4924102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.904+0000 7f4929a9a640 1 --2- 172.21.15.77:0/2894472014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4924101800 0x7f4924101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.904+0000 7f4929a9a640 1 -- 172.21.15.77:0/2894472014 >> 172.21.15.77:0/2894472014 conn(0x7f49240fcd70 msgr2=0x7f49240ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.904+0000 7f4929a9a640 1 -- 172.21.15.77:0/2894472014 shutdown_connections 2024-03-16T10:17:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.904+0000 7f4929a9a640 1 -- 172.21.15.77:0/2894472014 wait complete. 2024-03-16T10:17:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4929a9a640 1 Processor -- start 2024-03-16T10:17:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4929a9a640 1 -- start start 2024-03-16T10:17:41.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4929a9a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4924101800 0x7f4924197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4929a9a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49241029f0 0x7f4924197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4929a9a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49241033b0 0x7f492419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4929a9a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f492410d460 con 0x7f49241033b0 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4929a9a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f492410d2e0 con 0x7f4924101800 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4929a9a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f492410d5e0 con 0x7f49241029f0 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4923fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49241033b0 0x7f492419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4923fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49241033b0 0x7f492419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52462/0 (socket says 172.21.15.77:52462) 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4923fff640 1 -- 172.21.15.77:0/3603765657 learned_addr learned my addr 172.21.15.77:0/3603765657 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f491bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49241029f0 0x7f4924197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f49237fe640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4924101800 0x7f4924197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4923fff640 1 -- 172.21.15.77:0/3603765657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49241029f0 msgr2=0x7f4924197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4923fff640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49241029f0 0x7f4924197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4923fff640 1 -- 172.21.15.77:0/3603765657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4924101800 msgr2=0x7f4924197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4923fff640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4924101800 0x7f4924197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.905+0000 7f4923fff640 1 -- 172.21.15.77:0/3603765657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492419f660 con 0x7f49241033b0 2024-03-16T10:17:41.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.906+0000 7f4923fff640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49241033b0 0x7f492419eff0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f491c036040 tx=0x7f491c0337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.906+0000 7f49217fa640 1 -- 172.21.15.77:0/3603765657 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c043020 con 0x7f49241033b0 2024-03-16T10:17:41.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.906+0000 7f4929a9a640 1 -- 172.21.15.77:0/3603765657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f492419f8f0 con 0x7f49241033b0 2024-03-16T10:17:41.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.906+0000 7f49217fa640 1 -- 172.21.15.77:0/3603765657 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f491c02ca60 con 0x7f49241033b0 2024-03-16T10:17:41.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.906+0000 7f4929a9a640 1 -- 172.21.15.77:0/3603765657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f492419fe40 con 0x7f49241033b0 2024-03-16T10:17:41.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.906+0000 7f49217fa640 1 -- 172.21.15.77:0/3603765657 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c034630 con 0x7f49241033b0 2024-03-16T10:17:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.908+0000 7f492c764740 1 Processor -- start 2024-03-16T10:17:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.908+0000 7f49217fa640 1 -- 172.21.15.77:0/3603765657 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f491c046020 con 0x7f49241033b0 2024-03-16T10:17:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f492c764740 1 -- start start 2024-03-16T10:17:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f492c764740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb0539850 0x55dcb0539c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f492c764740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb053a240 0x55dcb053ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f492c764740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053f3b0 0x55dcb0541840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f492c764740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55dcb049e490 con 0x55dcb0539850 2024-03-16T10:17:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f492c764740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55dcb0446c90 con 0x55dcb053f3b0 2024-03-16T10:17:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f492c764740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dcb049a530 con 0x55dcb053a240 2024-03-16T10:17:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f491bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb0539850 0x55dcb0539c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f49237fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb053a240 0x55dcb053ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f491bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb0539850 0x55dcb0539c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52474/0 (socket says 172.21.15.77:52474) 2024-03-16T10:17:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f491bfff640 1 -- 172.21.15.77:0/1637549457 learned_addr learned my addr 172.21.15.77:0/1637549457 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f491bfff640 1 --2- 172.21.15.77:0/1637549457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053f3b0 0x55dcb0541840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f49217fa640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f49100a2890 0x7f49100a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f49237fe640 1 -- 172.21.15.77:0/1637549457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053f3b0 msgr2=0x55dcb0541840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f49237fe640 1 --2- 172.21.15.77:0/1637549457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053f3b0 0x55dcb0541840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f49237fe640 1 -- 172.21.15.77:0/1637549457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb0539850 msgr2=0x55dcb0539c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f49237fe640 1 --2- 172.21.15.77:0/1637549457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb0539850 0x55dcb0539c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f49237fe640 1 -- 172.21.15.77:0/1637549457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcb0501500 con 0x55dcb053a240 2024-03-16T10:17:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f491bfff640 1 --2- 172.21.15.77:0/1637549457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb0539850 0x55dcb0539c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f4923fff640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f49100a2890 0x7f49100a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f49217fa640 1 -- 172.21.15.77:0/3603765657 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f491c0f0fa0 con 0x7f49241033b0 2024-03-16T10:17:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.909+0000 7f49237fe640 1 --2- 172.21.15.77:0/1637549457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb053a240 0x55dcb053ed40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4914005fd0 tx=0x7f4914011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8825543d74e82979 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f4919ffb640 1 -- 172.21.15.77:0/1637549457 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491401e070 con 0x55dcb053a240 2024-03-16T10:17:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f4919ffb640 1 -- 172.21.15.77:0/1637549457 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4914009db0 con 0x55dcb053a240 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f4919ffb640 1 -- 172.21.15.77:0/1637549457 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4914019450 con 0x55dcb053a240 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f492c764740 1 -- 172.21.15.77:0/1637549457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb053a240 msgr2=0x55dcb053ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f492c764740 1 --2- 172.21.15.77:0/1637549457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb053a240 0x55dcb053ed40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4914005fd0 tx=0x7f4914011810 comp rx=0 tx=0).stop 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f4923fff640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f49100a2890 0x7f49100a4d50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f491c0c5c50 tx=0x7f491c004620 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f492c764740 1 -- 172.21.15.77:0/1637549457 shutdown_connections 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f492c764740 1 --2- 172.21.15.77:0/1637549457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053f3b0 0x55dcb0541840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f492c764740 1 --2- 172.21.15.77:0/1637549457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb053a240 0x55dcb053ed40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f492c764740 1 --2- 172.21.15.77:0/1637549457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb0539850 0x55dcb0539c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f492c764740 1 -- 172.21.15.77:0/1637549457 >> 172.21.15.77:0/1637549457 conn(0x55dcb0534dc0 msgr2=0x55dcb05351c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f492c764740 1 -- 172.21.15.77:0/1637549457 shutdown_connections 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.910+0000 7f492c764740 1 -- 172.21.15.77:0/1637549457 wait complete. 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f492c764740 1 -- start start 2024-03-16T10:17:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f492c764740 1 Processor -- start 2024-03-16T10:17:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f492c764740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 0x55dcb0543da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f492c764740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 0x55dcb05442e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f492c764740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053f3b0 0x55dcb054b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f492c764740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55dcb0542360 con 0x55dcb053a240 2024-03-16T10:17:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f492c764740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55dcb045a230 con 0x55dcb053f3b0 2024-03-16T10:17:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f492c764740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dcb05424e0 con 0x55dcb0539850 2024-03-16T10:17:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f492c764740 10 client.0 _ll_register_callbacks cb 0x55dcb0496010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f491bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 0x55dcb0543da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f49237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 0x55dcb05442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f491bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 0x55dcb0543da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58300/0 (socket says 172.21.15.77:58300) 2024-03-16T10:17:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f491bfff640 1 -- 172.21.15.77:0/3628733127 learned_addr learned my addr 172.21.15.77:0/3628733127 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f491bfff640 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053f3b0 0x55dcb054b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f491bfff640 1 -- 172.21.15.77:0/3628733127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053f3b0 msgr2=0x55dcb054b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f491bfff640 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053f3b0 0x55dcb054b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f491bfff640 1 -- 172.21.15.77:0/3628733127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 msgr2=0x55dcb05442e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f491bfff640 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 0x55dcb05442e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f491bfff640 1 -- 172.21.15.77:0/3628733127 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcb0469100 con 0x55dcb0539850 2024-03-16T10:17:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.911+0000 7f49237fe640 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 0x55dcb05442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f491bfff640 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 0x55dcb0543da0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f490c010ee0 tx=0x7f490c009950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2357ef1fb6c8e528 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f4902ffd640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:17:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f4902ffd640 1 -- 172.21.15.77:0/3628733127 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490c01c040 con 0x55dcb0539850 2024-03-16T10:17:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f4902ffd640 1 -- 172.21.15.77:0/3628733127 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f490c004230 con 0x55dcb0539850 2024-03-16T10:17:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f492c764740 1 -- 172.21.15.77:0/3628733127 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4914009db0 con 0x55dcb0539850 2024-03-16T10:17:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f4902ffd640 1 -- 172.21.15.77:0/3628733127 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f490c004430 con 0x55dcb0539850 2024-03-16T10:17:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f492c764740 10 client.25459 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f492c764740 1 -- 172.21.15.77:0/3628733127 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dcb01b23f0 con 0x55dcb0539850 2024-03-16T10:17:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f4902ffd640 1 -- 172.21.15.77:0/3628733127 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4914009db0 con 0x55dcb0539850 2024-03-16T10:17:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f4902ffd640 1 -- 172.21.15.77:0/3628733127 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f490c023550 con 0x55dcb0539850 2024-03-16T10:17:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f492c764740 10 client.25459 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f492c764740 10 client.25459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f492c764740 20 client.25459 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f492c764740 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dcb054fbd0 0x55dcb0552040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f492c764740 4 client.25459 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.912+0000 7f492c764740 1 -- 172.21.15.77:0/3628733127 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcb04ab9d0 con 0x55dcb054fbd0 2024-03-16T10:17:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.913+0000 7f49237fe640 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dcb054fbd0 0x55dcb0552040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.913+0000 7f49237fe640 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dcb054fbd0 0x55dcb0552040 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=359193827a912578 server_cookie=c4890fe872c5eccb in_seq=0 out_seq=0 2024-03-16T10:17:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.913+0000 7f4902ffd640 10 client.25459 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f4902ffd640 1 -- 172.21.15.77:0/3628733127 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dcb04ab9d0 con 0x55dcb054fbd0 2024-03-16T10:17:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f4902ffd640 10 client.25459 handle_command_reply: tid=0 2024-03-16T10:17:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f492c764740 1 client.25459 shutdown 2024-03-16T10:17:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f492c764740 1 -- 172.21.15.77:0/3628733127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 msgr2=0x55dcb0543da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f492c764740 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 0x55dcb0543da0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f490c010ee0 tx=0x7f490c009950 comp rx=0 tx=0).stop 2024-03-16T10:17:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f492c764740 1 -- 172.21.15.77:0/3628733127 shutdown_connections 2024-03-16T10:17:41.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f492c764740 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053f3b0 0x55dcb054b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f492c764740 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dcb054fbd0 0x55dcb0552040 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f492c764740 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 0x55dcb05442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f492c764740 1 --2- 172.21.15.77:0/3628733127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 0x55dcb0543da0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f4902ffd640 0 client.25459 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f492c764740 1 -- 172.21.15.77:0/3628733127 >> 172.21.15.77:0/3628733127 conn(0x55dcb0534dc0 msgr2=0x55dcb0533fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f492c764740 1 -- 172.21.15.77:0/3628733127 shutdown_connections 2024-03-16T10:17:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f492c764740 1 -- 172.21.15.77:0/3628733127 wait complete. 2024-03-16T10:17:41.924 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.914+0000 7f492c764740 20 client.25459 trim_cache size 0 max 16384 2024-03-16T10:17:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f492c764740 1 Processor -- start 2024-03-16T10:17:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f492c764740 1 -- start start 2024-03-16T10:17:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f492c764740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 0x55dcb0544240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f492c764740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053a240 0x55dcb054b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f492c764740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053f3b0 0x55dcb0538730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f492c764740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49241a8ad0 con 0x55dcb053f3b0 2024-03-16T10:17:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f492c764740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f49241aa610 con 0x55dcb053a240 2024-03-16T10:17:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f492c764740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f49241a91a0 con 0x55dcb0539850 2024-03-16T10:17:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f491bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053f3b0 0x55dcb0538730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f491bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 0x55dcb0544240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f491bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 0x55dcb0544240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58302/0 (socket says 172.21.15.77:58302) 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f491bfff640 1 -- 172.21.15.77:0/590486896 learned_addr learned my addr 172.21.15.77:0/590486896 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.934+0000 7f49237fe640 1 --2- 172.21.15.77:0/590486896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053a240 0x55dcb054b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.935+0000 7f491bfff640 1 -- 172.21.15.77:0/590486896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053a240 msgr2=0x55dcb054b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.935+0000 7f491bfff640 1 --2- 172.21.15.77:0/590486896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053a240 0x55dcb054b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.935+0000 7f491bfff640 1 -- 172.21.15.77:0/590486896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053f3b0 msgr2=0x55dcb0538730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.935+0000 7f491bfff640 1 --2- 172.21.15.77:0/590486896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053f3b0 0x55dcb0538730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.935+0000 7f491bfff640 1 -- 172.21.15.77:0/590486896 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcb027c7d0 con 0x55dcb0539850 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.935+0000 7f491bfff640 1 --2- 172.21.15.77:0/590486896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 0x55dcb0544240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f490c009830 tx=0x7f490c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63c647a54dfbd348 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.935+0000 7f49197fa640 1 -- 172.21.15.77:0/590486896 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490c023d90 con 0x55dcb0539850 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.935+0000 7f49197fa640 1 -- 172.21.15.77:0/590486896 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f490c023430 con 0x55dcb0539850 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.935+0000 7f49197fa640 1 -- 172.21.15.77:0/590486896 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490c02d410 con 0x55dcb0539850 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 -- 172.21.15.77:0/590486896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 msgr2=0x55dcb0544240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 --2- 172.21.15.77:0/590486896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 0x55dcb0544240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f490c009830 tx=0x7f490c000bc0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 -- 172.21.15.77:0/590486896 shutdown_connections 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 --2- 172.21.15.77:0/590486896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053f3b0 0x55dcb0538730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 --2- 172.21.15.77:0/590486896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb053a240 0x55dcb054b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 --2- 172.21.15.77:0/590486896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb0539850 0x55dcb0544240 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 -- 172.21.15.77:0/590486896 >> 172.21.15.77:0/590486896 conn(0x55dcb0534dc0 msgr2=0x55dcb0538020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 -- 172.21.15.77:0/590486896 shutdown_connections 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 -- 172.21.15.77:0/590486896 wait complete. 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 -- start start 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 Processor -- start 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb0539850 0x55dcb0548460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 0x55dcb05489a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb053f3b0 0x55dcb05499b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49241a9f40 con 0x55dcb053a240 2024-03-16T10:17:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f49241a9870 con 0x55dcb0539850 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.936+0000 7f492c764740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f49241a6ab0 con 0x55dcb053f3b0 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f492c764740 10 client.0 _ll_register_callbacks cb 0x7f490c010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f49237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 0x55dcb05489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f49237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 0x55dcb05489a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52510/0 (socket says 172.21.15.77:52510) 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f491bfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb0539850 0x55dcb0548460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f49237fe640 1 -- 172.21.15.77:0/1033274412 learned_addr learned my addr 172.21.15.77:0/1033274412 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f491bfff640 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb053f3b0 0x55dcb05499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f49237fe640 1 -- 172.21.15.77:0/1033274412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb053f3b0 msgr2=0x55dcb05499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f49237fe640 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb053f3b0 0x55dcb05499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f49237fe640 1 -- 172.21.15.77:0/1033274412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb0539850 msgr2=0x55dcb0548460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f49237fe640 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb0539850 0x55dcb0548460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f49237fe640 1 -- 172.21.15.77:0/1033274412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcb0549fa0 con 0x55dcb053a240 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f491bfff640 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb0539850 0x55dcb0548460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f49237fe640 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 0x55dcb05489a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f491401c040 tx=0x7f49140097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be44ec95c4fff5b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f4902ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f4902ffd640 1 -- 172.21.15.77:0/1033274412 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491401e040 con 0x55dcb053a240 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f4902ffd640 1 -- 172.21.15.77:0/1033274412 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f4914016040 con 0x55dcb053a240 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f492c764740 1 -- 172.21.15.77:0/1033274412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f490c023430 con 0x55dcb053a240 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f4902ffd640 1 -- 172.21.15.77:0/1033274412 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4914005350 con 0x55dcb053a240 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f492c764740 10 client.15675 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.937+0000 7f492c764740 1 -- 172.21.15.77:0/1033274412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dcb054b310 con 0x55dcb053a240 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f4902ffd640 1 -- 172.21.15.77:0/1033274412 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f490c023430 con 0x55dcb053a240 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f4902ffd640 1 -- 172.21.15.77:0/1033274412 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4914003930 con 0x55dcb053a240 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f492c764740 10 client.15675 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f492c764740 10 client.15675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f492c764740 20 client.15675 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f492c764740 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dcb05b6150 0x55dcb05b8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f492c764740 4 client.15675 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f492c764740 1 -- 172.21.15.77:0/1033274412 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dcb04a5d20 con 0x55dcb05b6150 2024-03-16T10:17:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f491bfff640 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dcb05b6150 0x55dcb05b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f4902ffd640 10 client.15675 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f491bfff640 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dcb05b6150 0x55dcb05b8540 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50d1c8bc365d93d7 server_cookie=7629639b55f11ee6 in_seq=0 out_seq=0 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f4902ffd640 1 -- 172.21.15.77:0/1033274412 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dcb04a5d20 con 0x55dcb05b6150 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.938+0000 7f4902ffd640 10 client.15675 handle_command_reply: tid=0 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f492c764740 1 client.15675 shutdown 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f492c764740 1 -- 172.21.15.77:0/1033274412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 msgr2=0x55dcb05489a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f492c764740 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 0x55dcb05489a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f491401c040 tx=0x7f49140097b0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f492c764740 1 -- 172.21.15.77:0/1033274412 shutdown_connections 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f492c764740 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dcb053f3b0 0x55dcb05499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f492c764740 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dcb05b6150 0x55dcb05b8540 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f492c764740 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dcb053a240 0x55dcb05489a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f492c764740 1 --2- 172.21.15.77:0/1033274412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dcb0539850 0x55dcb0548460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f492c764740 1 -- 172.21.15.77:0/1033274412 >> 172.21.15.77:0/1033274412 conn(0x55dcb0534dc0 msgr2=0x55dcb05388b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f4902ffd640 0 client.15675 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f492c764740 1 -- 172.21.15.77:0/1033274412 shutdown_connections 2024-03-16T10:17:41.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f492c764740 1 -- 172.21.15.77:0/1033274412 wait complete. 2024-03-16T10:17:41.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f492c764740 20 client.15675 trim_cache size 0 max 16384 2024-03-16T10:17:41.947 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:17:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f4929a9a640 1 -- 172.21.15.77:0/3603765657 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f49100a2890 msgr2=0x7f49100a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f4929a9a640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f49100a2890 0x7f49100a4d50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f491c0c5c50 tx=0x7f491c004620 comp rx=0 tx=0).stop 2024-03-16T10:17:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f4929a9a640 1 -- 172.21.15.77:0/3603765657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49241033b0 msgr2=0x7f492419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.939+0000 7f4929a9a640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49241033b0 0x7f492419eff0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f491c036040 tx=0x7f491c0337b0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.940+0000 7f4929a9a640 1 -- 172.21.15.77:0/3603765657 shutdown_connections 2024-03-16T10:17:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.940+0000 7f4929a9a640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49241033b0 0x7f492419eff0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.940+0000 7f4929a9a640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f49100a2890 0x7f49100a4d50 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.940+0000 7f4929a9a640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49241029f0 0x7f4924197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.940+0000 7f4929a9a640 1 --2- 172.21.15.77:0/3603765657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4924101800 0x7f4924197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.940+0000 7f4929a9a640 1 -- 172.21.15.77:0/3603765657 >> 172.21.15.77:0/3603765657 conn(0x7f49240fcd70 msgr2=0x7f4924105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.940+0000 7f4929a9a640 1 -- 172.21.15.77:0/3603765657 shutdown_connections 2024-03-16T10:17:41.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:41.940+0000 7f4929a9a640 1 -- 172.21.15.77:0/3603765657 wait complete. 2024-03-16T10:17:42.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:41 smithi192 ceph-mon[36354]: pgmap v368: 97 pgs: 97 active+clean; 659 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T10:17:42.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:41 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3548774071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:17:42.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:41 smithi077 ceph-mon[32400]: pgmap v368: 97 pgs: 97 active+clean; 659 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T10:17:42.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:41 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3548774071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:17:42.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:41 smithi193 ceph-mon[36311]: pgmap v368: 97 pgs: 97 active+clean; 659 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T10:17:42.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:41 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3548774071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:17:42.954 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T10:17:42.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:17:42.955 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T10:17:43.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.069+0000 7fdb852c0640 1 -- 172.21.15.77:0/4022953020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb801006e0 msgr2=0x7fdb80100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.069+0000 7fdb852c0640 1 --2- 172.21.15.77:0/4022953020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb801006e0 0x7fdb80100b00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009900 tx=0x7fdb7002bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.069+0000 7fdb852c0640 1 -- 172.21.15.77:0/4022953020 shutdown_connections 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.069+0000 7fdb852c0640 1 --2- 172.21.15.77:0/4022953020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb80102290 0x7fdb80108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.069+0000 7fdb852c0640 1 --2- 172.21.15.77:0/4022953020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb801018d0 0x7fdb80101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.069+0000 7fdb852c0640 1 --2- 172.21.15.77:0/4022953020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb801006e0 0x7fdb80100b00 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.069+0000 7fdb852c0640 1 -- 172.21.15.77:0/4022953020 >> 172.21.15.77:0/4022953020 conn(0x7fdb800fbc70 msgr2=0x7fdb800fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.069+0000 7fdb852c0640 1 -- 172.21.15.77:0/4022953020 shutdown_connections 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.069+0000 7fdb852c0640 1 -- 172.21.15.77:0/4022953020 wait complete. 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb852c0640 1 Processor -- start 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb852c0640 1 -- start start 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb852c0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb801006e0 0x7fdb800694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb852c0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb801018d0 0x7fdb8019ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb852c0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb80102290 0x7fdb801a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb852c0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8010c360 con 0x7fdb80102290 2024-03-16T10:17:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb852c0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8010c1e0 con 0x7fdb801006e0 2024-03-16T10:17:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb852c0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8010c4e0 con 0x7fdb801018d0 2024-03-16T10:17:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb7f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb80102290 0x7fdb801a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb767fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb801018d0 0x7fdb8019ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb7effd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb801006e0 0x7fdb800694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb7f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb80102290 0x7fdb801a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52534/0 (socket says 172.21.15.77:52534) 2024-03-16T10:17:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb7f7fe640 1 -- 172.21.15.77:0/1734607105 learned_addr learned my addr 172.21.15.77:0/1734607105 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.070+0000 7fdb767fc640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb801018d0 0x7fdb8019ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58334/0 (socket says 172.21.15.77:58334) 2024-03-16T10:17:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.071+0000 7fdb7f7fe640 1 -- 172.21.15.77:0/1734607105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb801018d0 msgr2=0x7fdb8019ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.071+0000 7fdb7f7fe640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb801018d0 0x7fdb8019ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.071+0000 7fdb7f7fe640 1 -- 172.21.15.77:0/1734607105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb801006e0 msgr2=0x7fdb800694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.071+0000 7fdb7f7fe640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb801006e0 0x7fdb800694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.071+0000 7fdb7f7fe640 1 -- 172.21.15.77:0/1734607105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb801a19b0 con 0x7fdb80102290 2024-03-16T10:17:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.071+0000 7fdb7f7fe640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb80102290 0x7fdb801a11c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800eb10 tx=0x7fdb7800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.071+0000 7fdb7cff9640 1 -- 172.21.15.77:0/1734607105 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb78004b20 con 0x7fdb80102290 2024-03-16T10:17:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.071+0000 7fdb852c0640 1 -- 172.21.15.77:0/1734607105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb801a1c40 con 0x7fdb80102290 2024-03-16T10:17:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.071+0000 7fdb7cff9640 1 -- 172.21.15.77:0/1734607105 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb78004cc0 con 0x7fdb80102290 2024-03-16T10:17:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.071+0000 7fdb7cff9640 1 -- 172.21.15.77:0/1734607105 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb78010760 con 0x7fdb80102290 2024-03-16T10:17:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.071+0000 7fdb852c0640 1 -- 172.21.15.77:0/1734607105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb801a2190 con 0x7fdb80102290 2024-03-16T10:17:43.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.073+0000 7fdb7cff9640 1 -- 172.21.15.77:0/1734607105 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fdb78004e70 con 0x7fdb80102290 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb87f85740 1 Processor -- start 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb87f85740 1 -- start start 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb87f85740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c19db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb87f85740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1a380 0x563a46c1ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb87f85740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1f4f0 0x563a46c21980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb87f85740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563a46b7e0d0 con 0x563a46c1a380 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb87f85740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563a46b26c70 con 0x563a46c19990 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb87f85740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563a46b7a170 con 0x563a46c1f4f0 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb7cff9640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdb6c0a28c0 0x7fdb6c0a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb7effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1a380 0x563a46c1ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb7effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1a380 0x563a46c1ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52546/0 (socket says 172.21.15.77:52546) 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb7effd640 1 -- 172.21.15.77:0/30378958 learned_addr learned my addr 172.21.15.77:0/30378958 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:43.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb767fc640 1 --2- 172.21.15.77:0/30378958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c19db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb7f7fe640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdb6c0a28c0 0x7fdb6c0a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb767fc640 1 --2- 172.21.15.77:0/30378958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1f4f0 0x563a46c21980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.074+0000 7fdb7cff9640 1 -- 172.21.15.77:0/1734607105 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fdb780cf280 con 0x7fdb80102290 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.075+0000 7fdb767fc640 1 -- 172.21.15.77:0/30378958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1f4f0 msgr2=0x563a46c21980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.075+0000 7fdb767fc640 1 --2- 172.21.15.77:0/30378958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1f4f0 0x563a46c21980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.075+0000 7fdb767fc640 1 -- 172.21.15.77:0/30378958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1a380 msgr2=0x563a46c1ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.075+0000 7fdb767fc640 1 --2- 172.21.15.77:0/30378958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1a380 0x563a46c1ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.075+0000 7fdb767fc640 1 -- 172.21.15.77:0/30378958 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a4695a7e0 con 0x563a46c19990 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.075+0000 7fdb7effd640 1 --2- 172.21.15.77:0/30378958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1a380 0x563a46c1ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.075+0000 7fdb767fc640 1 --2- 172.21.15.77:0/30378958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c19db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009730 tx=0x7fdb680096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d640b3e00ac30b02 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.075+0000 7fdb7f7fe640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdb6c0a28c0 0x7fdb6c0a4d80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdb7809ba40 tx=0x7fdb78086040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.075+0000 7fdb757fa640 1 -- 172.21.15.77:0/30378958 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68004050 con 0x563a46c19990 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.075+0000 7fdb757fa640 1 -- 172.21.15.77:0/30378958 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb68005500 con 0x563a46c19990 2024-03-16T10:17:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.075+0000 7fdb757fa640 1 -- 172.21.15.77:0/30378958 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68004d30 con 0x563a46c19990 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.076+0000 7fdb87f85740 1 -- 172.21.15.77:0/30378958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 msgr2=0x563a46c19db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.076+0000 7fdb87f85740 1 --2- 172.21.15.77:0/30378958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c19db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009730 tx=0x7fdb680096c0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.076+0000 7fdb87f85740 1 -- 172.21.15.77:0/30378958 shutdown_connections 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.076+0000 7fdb87f85740 1 --2- 172.21.15.77:0/30378958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1f4f0 0x563a46c21980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.076+0000 7fdb87f85740 1 --2- 172.21.15.77:0/30378958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1a380 0x563a46c1ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.076+0000 7fdb87f85740 1 --2- 172.21.15.77:0/30378958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c19db0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.076+0000 7fdb87f85740 1 -- 172.21.15.77:0/30378958 >> 172.21.15.77:0/30378958 conn(0x563a46be1c50 msgr2=0x563a46c15300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.076+0000 7fdb87f85740 1 -- 172.21.15.77:0/30378958 shutdown_connections 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.076+0000 7fdb87f85740 1 -- 172.21.15.77:0/30378958 wait complete. 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.076+0000 7fdb87f85740 1 -- start start 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.076+0000 7fdb87f85740 1 Processor -- start 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb87f85740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c23ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb87f85740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1a380 0x563a46c24420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb87f85740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 0x563a46c2b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb87f85740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563a46c224a0 con 0x563a46c1f4f0 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb87f85740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563a46b3a0d0 con 0x563a46c19990 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb87f85740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563a46c22620 con 0x563a46c1a380 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb87f85740 10 client.0 _ll_register_callbacks cb 0x563a46b03760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb767fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 0x563a46c2b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb767fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 0x563a46c2b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52554/0 (socket says 172.21.15.77:52554) 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb767fc640 1 -- 172.21.15.77:0/3943485942 learned_addr learned my addr 172.21.15.77:0/3943485942 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb7effd640 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1a380 0x563a46c24420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb767fc640 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c23ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb767fc640 1 -- 172.21.15.77:0/3943485942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1a380 msgr2=0x563a46c24420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb767fc640 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1a380 0x563a46c24420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb767fc640 1 -- 172.21.15.77:0/3943485942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 msgr2=0x563a46c23ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb767fc640 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c23ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb767fc640 1 -- 172.21.15.77:0/3943485942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a469baf70 con 0x563a46c1f4f0 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb767fc640 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 0x563a46c2b460 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fdb680097c0 tx=0x7fdb68009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35dfa6be04fa277b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb667fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb667fc640 1 -- 172.21.15.77:0/3943485942 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb680056b0 con 0x563a46c1f4f0 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb667fc640 1 -- 172.21.15.77:0/3943485942 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fdb68005160 con 0x563a46c1f4f0 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb87f85740 1 -- 172.21.15.77:0/3943485942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb68005500 con 0x563a46c1f4f0 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb667fc640 1 -- 172.21.15.77:0/3943485942 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdb6800f790 con 0x563a46c1f4f0 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb87f85740 10 client.15687 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.077+0000 7fdb87f85740 1 -- 172.21.15.77:0/3943485942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563a46852170 con 0x563a46c1f4f0 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.078+0000 7fdb667fc640 1 -- 172.21.15.77:0/3943485942 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb68005500 con 0x563a46c1f4f0 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.078+0000 7fdb667fc640 1 -- 172.21.15.77:0/3943485942 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdb6801f9a0 con 0x563a46c1f4f0 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.078+0000 7fdb87f85740 10 client.15687 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.078+0000 7fdb87f85740 10 client.15687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.078+0000 7fdb87f85740 20 client.15687 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.078+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563a46c2f660 0x563a46c31a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.078+0000 7fdb87f85740 4 client.15687 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.078+0000 7fdb87f85740 1 -- 172.21.15.77:0/3943485942 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a46b8b610 con 0x563a46c2f660 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.078+0000 7fdb7effd640 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563a46c2f660 0x563a46c31a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.078+0000 7fdb7effd640 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563a46c2f660 0x563a46c31a90 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d74e4b054d1d2fb server_cookie=473e325eebc27c23 in_seq=0 out_seq=0 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.078+0000 7fdb667fc640 10 client.15687 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb667fc640 1 -- 172.21.15.77:0/3943485942 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563a46b8b610 con 0x563a46c2f660 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb667fc640 10 client.15687 handle_command_reply: tid=0 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb87f85740 1 client.15687 shutdown 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb87f85740 1 -- 172.21.15.77:0/3943485942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 msgr2=0x563a46c2b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 0x563a46c2b460 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fdb680097c0 tx=0x7fdb68009ac0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb87f85740 1 -- 172.21.15.77:0/3943485942 shutdown_connections 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563a46c2f660 0x563a46c31a90 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 0x563a46c2b460 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1a380 0x563a46c24420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb667fc640 0 client.15687 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3943485942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c23ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb87f85740 1 -- 172.21.15.77:0/3943485942 >> 172.21.15.77:0/3943485942 conn(0x563a46be1c50 msgr2=0x563a46c144e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:43.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb87f85740 1 -- 172.21.15.77:0/3943485942 shutdown_connections 2024-03-16T10:17:43.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb87f85740 1 -- 172.21.15.77:0/3943485942 wait complete. 2024-03-16T10:17:43.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.079+0000 7fdb87f85740 20 client.15687 trim_cache size 0 max 16384 2024-03-16T10:17:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.096+0000 7fdb87f85740 1 Processor -- start 2024-03-16T10:17:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb87f85740 1 -- start start 2024-03-16T10:17:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb87f85740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c13c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb87f85740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1a380 0x563a46c2b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb87f85740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 0x563a46c14190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb87f85740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdb801aaee0 con 0x563a46c1f4f0 2024-03-16T10:17:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb87f85740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fdb801aca20 con 0x563a46c19990 2024-03-16T10:17:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb87f85740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdb801ab5b0 con 0x563a46c1a380 2024-03-16T10:17:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb7effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1a380 0x563a46c2b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb767fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 0x563a46c14190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb7effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1a380 0x563a46c2b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58352/0 (socket says 172.21.15.77:58352) 2024-03-16T10:17:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb767fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 0x563a46c14190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52556/0 (socket says 172.21.15.77:52556) 2024-03-16T10:17:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb7effd640 1 -- 172.21.15.77:0/3193859790 learned_addr learned my addr 172.21.15.77:0/3193859790 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb767fc640 1 --2- 172.21.15.77:0/3193859790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c13c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb767fc640 1 -- 172.21.15.77:0/3193859790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1a380 msgr2=0x563a46c2b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb767fc640 1 --2- 172.21.15.77:0/3193859790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1a380 0x563a46c2b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb767fc640 1 -- 172.21.15.77:0/3193859790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 msgr2=0x563a46c13c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb767fc640 1 --2- 172.21.15.77:0/3193859790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c13c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb767fc640 1 -- 172.21.15.77:0/3193859790 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a46c158a0 con 0x563a46c1f4f0 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb7effd640 1 --2- 172.21.15.77:0/3193859790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1a380 0x563a46c2b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.097+0000 7fdb767fc640 1 --2- 172.21.15.77:0/3193859790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 0x563a46c14190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdb68006010 tx=0x7fdb68004f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fdf89e5da2f534d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.098+0000 7fdb74ff9640 1 -- 172.21.15.77:0/3193859790 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68023070 con 0x563a46c1f4f0 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.098+0000 7fdb74ff9640 1 -- 172.21.15.77:0/3193859790 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb6800fe10 con 0x563a46c1f4f0 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.098+0000 7fdb74ff9640 1 -- 172.21.15.77:0/3193859790 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb680048e0 con 0x563a46c1f4f0 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.098+0000 7fdb87f85740 1 -- 172.21.15.77:0/3193859790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 msgr2=0x563a46c14190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.098+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3193859790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 0x563a46c14190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdb68006010 tx=0x7fdb68004f10 comp rx=0 tx=0).stop 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.098+0000 7fdb87f85740 1 -- 172.21.15.77:0/3193859790 shutdown_connections 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.098+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3193859790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c1f4f0 0x563a46c14190 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.098+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3193859790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1a380 0x563a46c2b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.098+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3193859790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c19990 0x563a46c13c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.098+0000 7fdb87f85740 1 -- 172.21.15.77:0/3193859790 >> 172.21.15.77:0/3193859790 conn(0x563a46be1c50 msgr2=0x563a46c36350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.098+0000 7fdb87f85740 1 -- 172.21.15.77:0/3193859790 shutdown_connections 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.098+0000 7fdb87f85740 1 -- 172.21.15.77:0/3193859790 wait complete. 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb87f85740 1 -- start start 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb87f85740 1 Processor -- start 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb87f85740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c19990 0x563a46c29ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb87f85740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c1a380 0x563a46c29ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb87f85740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1f4f0 0x563a46c2a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb87f85740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdb801ac350 con 0x563a46c19990 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb87f85740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fdb801abc80 con 0x563a46c1a380 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb87f85740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdb801a8ec0 con 0x563a46c1f4f0 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb87f85740 10 client.0 _ll_register_callbacks cb 0x7fdb780093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb767fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c19990 0x563a46c29ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb767fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c19990 0x563a46c29ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52572/0 (socket says 172.21.15.77:52572) 2024-03-16T10:17:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb767fc640 1 -- 172.21.15.77:0/3487231233 learned_addr learned my addr 172.21.15.77:0/3487231233 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb7effd640 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c1a380 0x563a46c29ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb767fc640 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1f4f0 0x563a46c2a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb767fc640 1 -- 172.21.15.77:0/3487231233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1f4f0 msgr2=0x563a46c2a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb767fc640 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1f4f0 0x563a46c2a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb767fc640 1 -- 172.21.15.77:0/3487231233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c1a380 msgr2=0x563a46c29ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb767fc640 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c1a380 0x563a46c29ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb767fc640 1 -- 172.21.15.77:0/3487231233 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a46c21e40 con 0x563a46c19990 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.099+0000 7fdb7effd640 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c1a380 0x563a46c29ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb767fc640 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c19990 0x563a46c29ab0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fdb6800a010 tx=0x7fdb68000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f78c444c55a69a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb667fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb667fc640 1 -- 172.21.15.77:0/3487231233 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68023070 con 0x563a46c19990 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb667fc640 1 -- 172.21.15.77:0/3487231233 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fdb6800f930 con 0x563a46c19990 2024-03-16T10:17:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb87f85740 1 -- 172.21.15.77:0/3487231233 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb6800fe10 con 0x563a46c19990 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb667fc640 1 -- 172.21.15.77:0/3487231233 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdb6800fb30 con 0x563a46c19990 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb87f85740 10 client.15693 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb87f85740 1 -- 172.21.15.77:0/3487231233 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563a46c969b0 con 0x563a46c19990 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb667fc640 1 -- 172.21.15.77:0/3487231233 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb6800fe10 con 0x563a46c19990 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb667fc640 1 -- 172.21.15.77:0/3487231233 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdb6801f4d0 con 0x563a46c19990 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb87f85740 10 client.15693 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb87f85740 10 client.15693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb87f85740 20 client.15693 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563a46c972a0 0x563a46c996c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb87f85740 4 client.15693 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.100+0000 7fdb87f85740 1 -- 172.21.15.77:0/3487231233 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563a46b85960 con 0x563a46c972a0 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.101+0000 7fdb7effd640 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563a46c972a0 0x563a46c996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.101+0000 7fdb7effd640 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563a46c972a0 0x563a46c996c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b780384627df214c server_cookie=86bdc11e0cae826d in_seq=0 out_seq=0 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.101+0000 7fdb667fc640 10 client.15693 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb667fc640 1 -- 172.21.15.77:0/3487231233 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563a46b85960 con 0x563a46c972a0 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb667fc640 10 client.15693 handle_command_reply: tid=0 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb87f85740 1 client.15693 shutdown 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb87f85740 1 -- 172.21.15.77:0/3487231233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c19990 msgr2=0x563a46c29ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c19990 0x563a46c29ab0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fdb6800a010 tx=0x7fdb68000bc0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb87f85740 1 -- 172.21.15.77:0/3487231233 shutdown_connections 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a46c1f4f0 0x563a46c2a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563a46c1a380 0x563a46c29ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563a46c972a0 0x563a46c996c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb87f85740 1 --2- 172.21.15.77:0/3487231233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563a46c19990 0x563a46c29ab0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb87f85740 1 -- 172.21.15.77:0/3487231233 >> 172.21.15.77:0/3487231233 conn(0x563a46be1c50 msgr2=0x563a46c36be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb667fc640 0 client.15693 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb87f85740 1 -- 172.21.15.77:0/3487231233 shutdown_connections 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb87f85740 1 -- 172.21.15.77:0/3487231233 wait complete. 2024-03-16T10:17:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.102+0000 7fdb87f85740 20 client.15693 trim_cache size 0 max 16384 2024-03-16T10:17:43.111 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:17:43.111 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T10:17:43.111 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "547aa8c2-0338-4404-9e02-32352b122e60", 2024-03-16T10:17:43.111 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T10:17:43.111 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:17:43.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.103+0000 7fdb852c0640 1 -- 172.21.15.77:0/1734607105 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdb6c0a28c0 msgr2=0x7fdb6c0a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.103+0000 7fdb852c0640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdb6c0a28c0 0x7fdb6c0a4d80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdb7809ba40 tx=0x7fdb78086040 comp rx=0 tx=0).stop 2024-03-16T10:17:43.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.103+0000 7fdb852c0640 1 -- 172.21.15.77:0/1734607105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb80102290 msgr2=0x7fdb801a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.103+0000 7fdb852c0640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb80102290 0x7fdb801a11c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800eb10 tx=0x7fdb7800c2d0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.103+0000 7fdb852c0640 1 -- 172.21.15.77:0/1734607105 shutdown_connections 2024-03-16T10:17:43.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.103+0000 7fdb852c0640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb80102290 0x7fdb801a11c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.103+0000 7fdb852c0640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdb6c0a28c0 0x7fdb6c0a4d80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.103+0000 7fdb852c0640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb801018d0 0x7fdb8019ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.103+0000 7fdb852c0640 1 --2- 172.21.15.77:0/1734607105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb801006e0 0x7fdb800694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.103+0000 7fdb852c0640 1 -- 172.21.15.77:0/1734607105 >> 172.21.15.77:0/1734607105 conn(0x7fdb800fbc70 msgr2=0x7fdb80104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:43.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.103+0000 7fdb852c0640 1 -- 172.21.15.77:0/1734607105 shutdown_connections 2024-03-16T10:17:43.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.103+0000 7fdb852c0640 1 -- 172.21.15.77:0/1734607105 wait complete. 2024-03-16T10:17:43.116 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:17:43.116 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:17:43.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.220+0000 7f48ed8b2640 1 -- 172.21.15.77:0/691704853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e81006e0 msgr2=0x7f48e8100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.220+0000 7f48ed8b2640 1 --2- 172.21.15.77:0/691704853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e81006e0 0x7f48e8100b00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f48e0009900 tx=0x7f48e002bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:43.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.220+0000 7f48ed8b2640 1 -- 172.21.15.77:0/691704853 shutdown_connections 2024-03-16T10:17:43.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.220+0000 7f48ed8b2640 1 --2- 172.21.15.77:0/691704853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e8102290 0x7f48e8108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.220+0000 7f48ed8b2640 1 --2- 172.21.15.77:0/691704853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e81018d0 0x7f48e8101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.220+0000 7f48ed8b2640 1 --2- 172.21.15.77:0/691704853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e81006e0 0x7f48e8100b00 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.220+0000 7f48ed8b2640 1 -- 172.21.15.77:0/691704853 >> 172.21.15.77:0/691704853 conn(0x7f48e80fbc70 msgr2=0x7f48e80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:43.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.220+0000 7f48ed8b2640 1 -- 172.21.15.77:0/691704853 shutdown_connections 2024-03-16T10:17:43.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.220+0000 7f48ed8b2640 1 -- 172.21.15.77:0/691704853 wait complete. 2024-03-16T10:17:43.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48ed8b2640 1 Processor -- start 2024-03-16T10:17:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48ed8b2640 1 -- start start 2024-03-16T10:17:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48ed8b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e81006e0 0x7f48e80694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48ed8b2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e81018d0 0x7f48e819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48ed8b2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e8102290 0x7f48e819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48ed8b2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f48e810c360 con 0x7f48e81006e0 2024-03-16T10:17:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48ed8b2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f48e810c1e0 con 0x7f48e81018d0 2024-03-16T10:17:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48ed8b2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f48e810c4e0 con 0x7f48e8102290 2024-03-16T10:17:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e81006e0 0x7f48e80694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48e67fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e81018d0 0x7f48e819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e81006e0 0x7f48e80694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52594/0 (socket says 172.21.15.77:52594) 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48e6ffd640 1 -- 172.21.15.77:0/1629682121 learned_addr learned my addr 172.21.15.77:0/1629682121 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.221+0000 7f48e77fe640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e8102290 0x7f48e819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48e67fc640 1 -- 172.21.15.77:0/1629682121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e8102290 msgr2=0x7f48e819ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48e67fc640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e8102290 0x7f48e819ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48e67fc640 1 -- 172.21.15.77:0/1629682121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e81006e0 msgr2=0x7f48e80694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48e67fc640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e81006e0 0x7f48e80694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48e67fc640 1 -- 172.21.15.77:0/1629682121 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e819f7a0 con 0x7f48e81018d0 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48e6ffd640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e81006e0 0x7f48e80694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48e77fe640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e8102290 0x7f48e819ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48e67fc640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e81018d0 0x7f48e819c870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f48dc00eb00 tx=0x7f48dc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48ec8b0640 1 -- 172.21.15.77:0/1629682121 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc010430 con 0x7f48e81018d0 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48ec8b0640 1 -- 172.21.15.77:0/1629682121 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48dc010ab0 con 0x7f48e81018d0 2024-03-16T10:17:43.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48ed8b2640 1 -- 172.21.15.77:0/1629682121 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48e819fa30 con 0x7f48e81018d0 2024-03-16T10:17:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48ec8b0640 1 -- 172.21.15.77:0/1629682121 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc017bb0 con 0x7f48e81018d0 2024-03-16T10:17:43.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.222+0000 7f48ed8b2640 1 -- 172.21.15.77:0/1629682121 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48e819ff80 con 0x7f48e81018d0 2024-03-16T10:17:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.224+0000 7f48f057b740 1 Processor -- start 2024-03-16T10:17:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.224+0000 7f48ec8b0640 1 -- 172.21.15.77:0/1629682121 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f48dc010c60 con 0x7f48e81018d0 2024-03-16T10:17:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.224+0000 7f48f057b740 1 -- start start 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48f057b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b980590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48f057b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 0x55c51b985660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48f057b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b985cd0 0x55c51b988160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48f057b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c51b8e5500 con 0x55c51b980b60 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48f057b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c51b88dbe0 con 0x55c51b980170 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48f057b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c51b8e15a0 con 0x55c51b985cd0 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 0x55c51b985660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e77fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b980590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 0x55c51b985660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52604/0 (socket says 172.21.15.77:52604) 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e67fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b985cd0 0x55c51b988160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e6ffd640 1 -- 172.21.15.77:0/428122751 learned_addr learned my addr 172.21.15.77:0/428122751 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48ec8b0640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f48d00a2890 0x7f48d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e6ffd640 1 -- 172.21.15.77:0/428122751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b985cd0 msgr2=0x55c51b988160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e6ffd640 1 --2- 172.21.15.77:0/428122751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b985cd0 0x55c51b988160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e6ffd640 1 -- 172.21.15.77:0/428122751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 msgr2=0x55c51b980590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e6ffd640 1 --2- 172.21.15.77:0/428122751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b980590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e6ffd640 1 -- 172.21.15.77:0/428122751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c51b8a14b0 con 0x55c51b980b60 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e67fc640 1 --2- 172.21.15.77:0/428122751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b985cd0 0x55c51b988160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e77fe640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f48d00a2890 0x7f48d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48ec8b0640 1 -- 172.21.15.77:0/1629682121 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f48dc0cf5a0 con 0x7f48e81018d0 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.225+0000 7f48e6ffd640 1 --2- 172.21.15.77:0/428122751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 0x55c51b985660 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f48e0000c00 tx=0x7f48e002cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=775af65c0c5e0d44 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48cdffb640 1 -- 172.21.15.77:0/428122751 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e0042070 con 0x55c51b980b60 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48cdffb640 1 -- 172.21.15.77:0/428122751 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48e0004ba0 con 0x55c51b980b60 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48cdffb640 1 -- 172.21.15.77:0/428122751 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e00052d0 con 0x55c51b980b60 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48f057b740 1 -- 172.21.15.77:0/428122751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 msgr2=0x55c51b985660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48e77fe640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f48d00a2890 0x7f48d00a4d50 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55c51b9813c0 tx=0x7f48d80046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48f057b740 1 --2- 172.21.15.77:0/428122751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 0x55c51b985660 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f48e0000c00 tx=0x7f48e002cd30 comp rx=0 tx=0).stop 2024-03-16T10:17:43.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48f057b740 1 -- 172.21.15.77:0/428122751 shutdown_connections 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48f057b740 1 --2- 172.21.15.77:0/428122751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b985cd0 0x55c51b988160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48f057b740 1 --2- 172.21.15.77:0/428122751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 0x55c51b985660 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48f057b740 1 --2- 172.21.15.77:0/428122751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b980590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48f057b740 1 -- 172.21.15.77:0/428122751 >> 172.21.15.77:0/428122751 conn(0x55c51b97b6e0 msgr2=0x55c51b97bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48f057b740 1 -- 172.21.15.77:0/428122751 shutdown_connections 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.226+0000 7f48f057b740 1 -- 172.21.15.77:0/428122751 wait complete. 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48f057b740 1 -- start start 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48f057b740 1 Processor -- start 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48f057b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b98a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48f057b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b980b60 0x55c51b98ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48f057b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 0x55c51b991be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48f057b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c51b988c20 con 0x55c51b985cd0 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48f057b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c51b8a1160 con 0x55c51b980170 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48f057b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c51b988da0 con 0x55c51b980b60 2024-03-16T10:17:43.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48f057b740 10 client.0 _ll_register_callbacks cb 0x55c51b885150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48e77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 0x55c51b991be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48e6ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b98a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48e67fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b980b60 0x55c51b98ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48e77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 0x55c51b991be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52610/0 (socket says 172.21.15.77:52610) 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48e77fe640 1 -- 172.21.15.77:0/1400858909 learned_addr learned my addr 172.21.15.77:0/1400858909 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48e77fe640 1 -- 172.21.15.77:0/1400858909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b980b60 msgr2=0x55c51b98ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48e77fe640 1 --2- 172.21.15.77:0/1400858909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b980b60 0x55c51b98ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48e77fe640 1 -- 172.21.15.77:0/1400858909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 msgr2=0x55c51b98a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48e77fe640 1 --2- 172.21.15.77:0/1400858909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b98a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.227+0000 7f48e77fe640 1 -- 172.21.15.77:0/1400858909 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c51b6c37d0 con 0x55c51b985cd0 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48e77fe640 1 --2- 172.21.15.77:0/1400858909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 0x55c51b991be0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f48d8010b80 tx=0x7f48d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f254d64ff634add5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:43.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48c6ffd640 1 -- 172.21.15.77:0/1400858909 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d802e060 con 0x55c51b985cd0 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48c6ffd640 1 -- 172.21.15.77:0/1400858909 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f48d801dd00 con 0x55c51b985cd0 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48f057b740 1 -- 172.21.15.77:0/1400858909 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48e0004ba0 con 0x55c51b985cd0 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48c6ffd640 1 -- 172.21.15.77:0/1400858909 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f48d80343f0 con 0x55c51b985cd0 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48f057b740 10 client.15705 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48f057b740 1 -- 172.21.15.77:0/1400858909 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c51b5f93f0 con 0x55c51b985cd0 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48c6ffd640 1 -- 172.21.15.77:0/1400858909 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48e0004ba0 con 0x55c51b985cd0 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48c6ffd640 1 -- 172.21.15.77:0/1400858909 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f48d801b4f0 con 0x55c51b985cd0 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48f057b740 10 client.15705 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48f057b740 10 client.15705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48f057b740 20 client.15705 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48f057b740 1 --2- 172.21.15.77:0/1400858909 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c51b996210 0x55c51b9986b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48f057b740 4 client.15705 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.228+0000 7f48f057b740 1 -- 172.21.15.77:0/1400858909 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c51b8f2a40 con 0x55c51b996210 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.229+0000 7f48e6ffd640 1 --2- 172.21.15.77:0/1400858909 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c51b996210 0x55c51b9986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.229+0000 7f48e6ffd640 1 --2- 172.21.15.77:0/1400858909 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c51b996210 0x55c51b9986b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab363cf8bee38624 server_cookie=f95d61755de11e63 in_seq=0 out_seq=0 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.229+0000 7f48c6ffd640 10 client.15705 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48c6ffd640 1 -- 172.21.15.77:0/1400858909 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c51b8f2a40 con 0x55c51b996210 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48c6ffd640 10 client.15705 handle_command_reply: tid=0 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48f057b740 1 client.15705 shutdown 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48f057b740 1 -- 172.21.15.77:0/1400858909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 msgr2=0x55c51b991be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48f057b740 1 --2- 172.21.15.77:0/1400858909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 0x55c51b991be0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f48d8010b80 tx=0x7f48d8004730 comp rx=0 tx=0).stop 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48f057b740 1 -- 172.21.15.77:0/1400858909 shutdown_connections 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48f057b740 1 --2- 172.21.15.77:0/1400858909 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c51b996210 0x55c51b9986b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48f057b740 1 --2- 172.21.15.77:0/1400858909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 0x55c51b991be0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48f057b740 1 --2- 172.21.15.77:0/1400858909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b980b60 0x55c51b98ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48c6ffd640 0 client.15705 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48f057b740 1 --2- 172.21.15.77:0/1400858909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b98a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48f057b740 1 -- 172.21.15.77:0/1400858909 >> 172.21.15.77:0/1400858909 conn(0x55c51b97b6e0 msgr2=0x55c51b97a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48f057b740 1 -- 172.21.15.77:0/1400858909 shutdown_connections 2024-03-16T10:17:43.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48f057b740 1 -- 172.21.15.77:0/1400858909 wait complete. 2024-03-16T10:17:43.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.230+0000 7f48f057b740 20 client.15705 trim_cache size 0 max 16384 2024-03-16T10:17:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48f057b740 1 Processor -- start 2024-03-16T10:17:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48f057b740 1 -- start start 2024-03-16T10:17:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48f057b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b992270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48f057b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b980b60 0x55c51b991ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48f057b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 0x55c51b9927b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48f057b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f48e81a8ba0 con 0x55c51b985cd0 2024-03-16T10:17:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48f057b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f48e81aa6e0 con 0x55c51b980170 2024-03-16T10:17:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48f057b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f48e81a9270 con 0x55c51b980b60 2024-03-16T10:17:43.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48e77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 0x55c51b9927b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48e77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 0x55c51b9927b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52614/0 (socket says 172.21.15.77:52614) 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48e77fe640 1 -- 172.21.15.77:0/38465236 learned_addr learned my addr 172.21.15.77:0/38465236 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48e67fc640 1 --2- 172.21.15.77:0/38465236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b980b60 0x55c51b991ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48e6ffd640 1 --2- 172.21.15.77:0/38465236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b992270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48e77fe640 1 -- 172.21.15.77:0/38465236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b980b60 msgr2=0x55c51b991ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48e77fe640 1 --2- 172.21.15.77:0/38465236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b980b60 0x55c51b991ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48e77fe640 1 -- 172.21.15.77:0/38465236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 msgr2=0x55c51b992270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48e77fe640 1 --2- 172.21.15.77:0/38465236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b992270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48e77fe640 1 -- 172.21.15.77:0/38465236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c51b5b90f0 con 0x55c51b985cd0 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48e67fc640 1 --2- 172.21.15.77:0/38465236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b980b60 0x55c51b991ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48e77fe640 1 --2- 172.21.15.77:0/38465236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 0x55c51b9927b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f48d8005ff0 tx=0x7f48d8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3c90991d83eb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48cd7fa640 1 -- 172.21.15.77:0/38465236 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d801bdd0 con 0x55c51b985cd0 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48cd7fa640 1 -- 172.21.15.77:0/38465236 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48d801b3d0 con 0x55c51b985cd0 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.248+0000 7f48cd7fa640 1 -- 172.21.15.77:0/38465236 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d8034b40 con 0x55c51b985cd0 2024-03-16T10:17:43.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.249+0000 7f48f057b740 1 -- 172.21.15.77:0/38465236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 msgr2=0x55c51b9927b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.249+0000 7f48f057b740 1 --2- 172.21.15.77:0/38465236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 0x55c51b9927b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f48d8005ff0 tx=0x7f48d8009f80 comp rx=0 tx=0).stop 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.249+0000 7f48f057b740 1 -- 172.21.15.77:0/38465236 shutdown_connections 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.249+0000 7f48f057b740 1 --2- 172.21.15.77:0/38465236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b985cd0 0x55c51b9927b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.249+0000 7f48f057b740 1 --2- 172.21.15.77:0/38465236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b980b60 0x55c51b991ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.249+0000 7f48f057b740 1 --2- 172.21.15.77:0/38465236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b992270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.249+0000 7f48f057b740 1 -- 172.21.15.77:0/38465236 >> 172.21.15.77:0/38465236 conn(0x55c51b97b6e0 msgr2=0x55c51b989170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.249+0000 7f48f057b740 1 -- 172.21.15.77:0/38465236 shutdown_connections 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.249+0000 7f48f057b740 1 -- 172.21.15.77:0/38465236 wait complete. 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.249+0000 7f48f057b740 1 -- start start 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48f057b740 1 Processor -- start 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48f057b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b98ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48f057b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 0x55c51b98f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48f057b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b985cd0 0x55c51b9902e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48f057b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f48e81aa010 con 0x55c51b980b60 2024-03-16T10:17:43.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48f057b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f48e81a9940 con 0x55c51b980170 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48f057b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f48e81a6ba0 con 0x55c51b985cd0 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48f057b740 10 client.0 _ll_register_callbacks cb 0x7f48d8009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48e67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 0x55c51b98f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48e67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 0x55c51b98f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52626/0 (socket says 172.21.15.77:52626) 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48e67fc640 1 -- 172.21.15.77:0/468621317 learned_addr learned my addr 172.21.15.77:0/468621317 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48e77fe640 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b985cd0 0x55c51b9902e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48e6ffd640 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b98ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48e67fc640 1 -- 172.21.15.77:0/468621317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b985cd0 msgr2=0x55c51b9902e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48e67fc640 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b985cd0 0x55c51b9902e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48e67fc640 1 -- 172.21.15.77:0/468621317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 msgr2=0x55c51b98ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48e67fc640 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b98ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.250+0000 7f48e67fc640 1 -- 172.21.15.77:0/468621317 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c51b8ac9d0 con 0x55c51b980b60 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48e67fc640 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 0x55c51b98f320 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f48dc00c7d0 tx=0x7f48dc09bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe2a6dd22e7a4d32 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48c6ffd640 1 -- 172.21.15.77:0/468621317 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc09c4d0 con 0x55c51b980b60 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48f057b740 1 -- 172.21.15.77:0/468621317 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48d801b3d0 con 0x55c51b980b60 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48c6ffd640 1 -- 172.21.15.77:0/468621317 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f48dc09b220 con 0x55c51b980b60 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48c6ffd640 1 -- 172.21.15.77:0/468621317 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f48dc09bdd0 con 0x55c51b980b60 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48f057b740 10 client.15711 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48f057b740 1 -- 172.21.15.77:0/468621317 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c51b991a80 con 0x55c51b980b60 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48c6ffd640 1 -- 172.21.15.77:0/468621317 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48d801b3d0 con 0x55c51b980b60 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48c6ffd640 1 -- 172.21.15.77:0/468621317 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f48dc097b20 con 0x55c51b980b60 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48f057b740 10 client.15711 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48f057b740 10 client.15711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48f057b740 20 client.15711 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48f057b740 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c51b9fc970 0x55c51b9fed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48f057b740 4 client.15711 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:17:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48f057b740 1 -- 172.21.15.77:0/468621317 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c51b8ecd90 con 0x55c51b9fc970 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.251+0000 7f48e6ffd640 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c51b9fc970 0x55c51b9fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48c6ffd640 10 client.15711 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48e6ffd640 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c51b9fc970 0x55c51b9fed90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af5f8aba159e08d8 server_cookie=3b7fe47ea4e5ca0b in_seq=0 out_seq=0 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48c6ffd640 1 -- 172.21.15.77:0/468621317 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c51b8ecd90 con 0x55c51b9fc970 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48c6ffd640 10 client.15711 handle_command_reply: tid=0 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48f057b740 1 client.15711 shutdown 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48f057b740 1 -- 172.21.15.77:0/468621317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 msgr2=0x55c51b98f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48f057b740 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 0x55c51b98f320 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f48dc00c7d0 tx=0x7f48dc09bcd0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48f057b740 1 -- 172.21.15.77:0/468621317 shutdown_connections 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48f057b740 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c51b985cd0 0x55c51b9902e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48f057b740 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c51b9fc970 0x55c51b9fed90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48f057b740 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c51b980b60 0x55c51b98f320 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48f057b740 1 --2- 172.21.15.77:0/468621317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c51b980170 0x55c51b98ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48c6ffd640 0 client.15711 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:43.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48f057b740 1 -- 172.21.15.77:0/468621317 >> 172.21.15.77:0/468621317 conn(0x55c51b97b6e0 msgr2=0x55c51b992270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48f057b740 1 -- 172.21.15.77:0/468621317 shutdown_connections 2024-03-16T10:17:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48f057b740 1 -- 172.21.15.77:0/468621317 wait complete. 2024-03-16T10:17:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.252+0000 7f48f057b740 20 client.15711 trim_cache size 0 max 16384 2024-03-16T10:17:43.260 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:17:43.260 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:17:43.260 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:17:43.260 INFO:teuthology.orchestra.run.smithi077.stdout: "547aa8c2-0338-4404-9e02-32352b122e60": { 2024-03-16T10:17:43.260 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:17:43.260 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "547aa8c2-0338-4404-9e02-32352b122e60", 2024-03-16T10:17:43.260 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:17:43.261 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:17:43.261 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:17:43.261 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:17:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.253+0000 7f48ed8b2640 1 -- 172.21.15.77:0/1629682121 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f48d00a2890 msgr2=0x7f48d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.253+0000 7f48ed8b2640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f48d00a2890 0x7f48d00a4d50 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55c51b9813c0 tx=0x7f48d80046c0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.253+0000 7f48ed8b2640 1 -- 172.21.15.77:0/1629682121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e81018d0 msgr2=0x7f48e819c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.253+0000 7f48ed8b2640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e81018d0 0x7f48e819c870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f48dc00eb00 tx=0x7f48dc00c2d0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.253+0000 7f48ed8b2640 1 -- 172.21.15.77:0/1629682121 shutdown_connections 2024-03-16T10:17:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.253+0000 7f48ed8b2640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f48d00a2890 0x7f48d00a4d50 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.253+0000 7f48ed8b2640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e8102290 0x7f48e819ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.253+0000 7f48ed8b2640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e81018d0 0x7f48e819c870 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.253+0000 7f48ed8b2640 1 --2- 172.21.15.77:0/1629682121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e81006e0 0x7f48e80694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.253+0000 7f48ed8b2640 1 -- 172.21.15.77:0/1629682121 >> 172.21.15.77:0/1629682121 conn(0x7f48e80fbc70 msgr2=0x7f48e8104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.253+0000 7f48ed8b2640 1 -- 172.21.15.77:0/1629682121 shutdown_connections 2024-03-16T10:17:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:17:43.253+0000 7f48ed8b2640 1 -- 172.21.15.77:0/1629682121 wait complete. 2024-03-16T10:17:43.265 INFO:tasks.cephfs.filesystem:scrub status for tag:547aa8c2-0338-4404-9e02-32352b122e60 - {'path': '/', 'tag': '547aa8c2-0338-4404-9e02-32352b122e60', 'options': 'recursive,force'} 2024-03-16T10:17:44.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:43 smithi192 ceph-mon[36354]: pgmap v369: 97 pgs: 97 active+clean; 659 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-16T10:17:44.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:43 smithi077 ceph-mon[32400]: pgmap v369: 97 pgs: 97 active+clean; 659 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-16T10:17:44.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:43 smithi193 ceph-mon[36311]: pgmap v369: 97 pgs: 97 active+clean; 659 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-16T10:17:45.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:44 smithi192 ceph-mon[36354]: scrub queued for path: / 2024-03-16T10:17:45.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:44 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:17:45.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:44 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:17:45.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:44 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:17:45.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:44 smithi077 ceph-mon[32400]: scrub queued for path: / 2024-03-16T10:17:45.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:44 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:17:45.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:44 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:17:45.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:44 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:17:45.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:44 smithi193 ceph-mon[36311]: scrub queued for path: / 2024-03-16T10:17:45.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:44 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:17:45.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:44 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:17:45.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:44 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:17:46.153 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:45 smithi193 ceph-mon[36311]: pgmap v370: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-16T10:17:46.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:45 smithi192 ceph-mon[36354]: pgmap v370: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-16T10:17:46.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:45 smithi077 ceph-mon[32400]: pgmap v370: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-16T10:17:47.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:46 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:17:47.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:46 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:17:47.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:46 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:17:48.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:47 smithi192 ceph-mon[36354]: pgmap v371: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-16T10:17:48.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:47 smithi077 ceph-mon[32400]: pgmap v371: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-16T10:17:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:47 smithi193 ceph-mon[36311]: pgmap v371: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-16T10:17:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.767+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2970406787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40ff920 msgr2=0x7fa8f40ffd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.767+0000 7fa8f97ff640 1 --2- 172.21.15.77:0/2970406787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40ff920 0x7fa8f40ffd40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009930 tx=0x7fa8e402bc30 comp rx=0 tx=0).stop 2024-03-16T10:17:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.767+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2970406787 shutdown_connections 2024-03-16T10:17:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.767+0000 7fa8f97ff640 1 --2- 172.21.15.77:0/2970406787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8f41032b0 0x7fa8f4109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.767+0000 7fa8f97ff640 1 --2- 172.21.15.77:0/2970406787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8f41028f0 0x7fa8f4102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.767+0000 7fa8f97ff640 1 --2- 172.21.15.77:0/2970406787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40ff920 0x7fa8f40ffd40 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.767+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2970406787 >> 172.21.15.77:0/2970406787 conn(0x7fa8f40fb250 msgr2=0x7fa8f40fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.768+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2970406787 shutdown_connections 2024-03-16T10:17:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.768+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2970406787 wait complete. 2024-03-16T10:17:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.768+0000 7fa8f97ff640 1 Processor -- start 2024-03-16T10:17:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.768+0000 7fa8f97ff640 1 -- start start 2024-03-16T10:17:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.768+0000 7fa8f97ff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8f40ff920 0x7fa8f41979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.768+0000 7fa8f97ff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8f41028f0 0x7fa8f4197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.768+0000 7fa8f97ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f41032b0 0x7fa8f419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.768+0000 7fa8f97ff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f410d2e0 con 0x7fa8f41032b0 2024-03-16T10:17:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.768+0000 7fa8f97ff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f410d160 con 0x7fa8f40ff920 2024-03-16T10:17:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.768+0000 7fa8f97ff640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f410d460 con 0x7fa8f41028f0 2024-03-16T10:17:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f41032b0 0x7fa8f419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f2ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8f40ff920 0x7fa8f41979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f41032b0 0x7fa8f419ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52644/0 (socket says 172.21.15.77:52644) 2024-03-16T10:17:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f37fe640 1 -- 172.21.15.77:0/2784439892 learned_addr learned my addr 172.21.15.77:0/2784439892 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f27fc640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8f41028f0 0x7fa8f4197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f37fe640 1 -- 172.21.15.77:0/2784439892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8f41028f0 msgr2=0x7fa8f4197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f37fe640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8f41028f0 0x7fa8f4197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f37fe640 1 -- 172.21.15.77:0/2784439892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8f40ff920 msgr2=0x7fa8f41979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f37fe640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8f40ff920 0x7fa8f41979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f37fe640 1 -- 172.21.15.77:0/2784439892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f419f5d0 con 0x7fa8f41032b0 2024-03-16T10:17:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8f40ff920 0x7fa8f41979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f27fc640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8f41028f0 0x7fa8f4197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f37fe640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f41032b0 0x7fa8f419ef60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec00eb10 tx=0x7fa8ec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8dbfff640 1 -- 172.21.15.77:0/2784439892 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec004b20 con 0x7fa8f41032b0 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2784439892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f419f8c0 con 0x7fa8f41032b0 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2784439892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f419fe10 con 0x7fa8f41032b0 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8dbfff640 1 -- 172.21.15.77:0/2784439892 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ec004cc0 con 0x7fa8f41032b0 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.769+0000 7fa8dbfff640 1 -- 172.21.15.77:0/2784439892 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec010760 con 0x7fa8f41032b0 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.772+0000 7fa8dbfff640 1 -- 172.21.15.77:0/2784439892 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fa8ec010900 con 0x7fa8f41032b0 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.772+0000 7fa8dbfff640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa8dc0a2900 0x7fa8dc0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.772+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa8dc0a2900 0x7fa8dc0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.772+0000 7fa8dbfff640 1 -- 172.21.15.77:0/2784439892 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa8ec0cefd0 con 0x7fa8f41032b0 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.772+0000 7fa8fc3e1740 1 Processor -- start 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.772+0000 7fa8fc3e1740 1 -- start start 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8fc3e1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc70024d0 0x561fc70028f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8fc3e1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7002ec0 0x561fc70079c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8fc3e1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7008030 0x561fc700a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8fc3e1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561fc6f67a40 con 0x561fc7002ec0 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8fc3e1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561fc6f10920 con 0x561fc7008030 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8fc3e1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561fc6f63ae0 con 0x561fc70024d0 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8f27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7002ec0 0x561fc70079c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa8dc0a2900 0x7fa8dc0a4dc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa8e402c7b0 tx=0x7fa8e4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8f27fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7002ec0 0x561fc70079c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52658/0 (socket says 172.21.15.77:52658) 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8f27fc640 1 -- 172.21.15.77:0/1040956952 learned_addr learned my addr 172.21.15.77:0/1040956952 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8f37fe640 1 --2- 172.21.15.77:0/1040956952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7008030 0x561fc700a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/1040956952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc70024d0 0x561fc70028f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8f27fc640 1 -- 172.21.15.77:0/1040956952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc70024d0 msgr2=0x561fc70028f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8f27fc640 1 --2- 172.21.15.77:0/1040956952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc70024d0 0x561fc70028f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8f27fc640 1 -- 172.21.15.77:0/1040956952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7008030 msgr2=0x561fc700a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8f27fc640 1 --2- 172.21.15.77:0/1040956952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7008030 0x561fc700a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8f27fc640 1 -- 172.21.15.77:0/1040956952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fc6f28460 con 0x561fc7002ec0 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8f27fc640 1 --2- 172.21.15.77:0/1040956952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7002ec0 0x561fc70079c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80114f0 tx=0x7fa8e80119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd9d5efef735f46 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8d97fa640 1 -- 172.21.15.77:0/1040956952 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e801e070 con 0x561fc7002ec0 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8d97fa640 1 -- 172.21.15.77:0/1040956952 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8e80035f0 con 0x561fc7002ec0 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.773+0000 7fa8d97fa640 1 -- 172.21.15.77:0/1040956952 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e800f820 con 0x561fc7002ec0 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.774+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/1040956952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7002ec0 msgr2=0x561fc70079c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.774+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/1040956952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7002ec0 0x561fc70079c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80114f0 tx=0x7fa8e80119b0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.774+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/1040956952 shutdown_connections 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.774+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/1040956952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7008030 0x561fc700a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.774+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/1040956952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7002ec0 0x561fc70079c0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.774+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/1040956952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc70024d0 0x561fc70028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.774+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/1040956952 >> 172.21.15.77:0/1040956952 conn(0x561fc6ffda40 msgr2=0x561fc6ffde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.774+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/1040956952 shutdown_connections 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.774+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/1040956952 wait complete. 2024-03-16T10:17:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8fc3e1740 1 -- start start 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8fc3e1740 1 Processor -- start 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8fc3e1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc70024d0 0x561fc700c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8fc3e1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7002ec0 0x561fc700cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8fc3e1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7008030 0x561fc7013db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8fc3e1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561fc700afe0 con 0x561fc7008030 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8fc3e1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561fc6f23d70 con 0x561fc70024d0 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8fc3e1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561fc700b160 con 0x561fc7002ec0 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8fc3e1740 10 client.0 _ll_register_callbacks cb 0x561fc6c7ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8f37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7008030 0x561fc7013db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8f2ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc70024d0 0x561fc700c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8f27fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7002ec0 0x561fc700cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8f37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7008030 0x561fc7013db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52664/0 (socket says 172.21.15.77:52664) 2024-03-16T10:17:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8f37fe640 1 -- 172.21.15.77:0/3807574216 learned_addr learned my addr 172.21.15.77:0/3807574216 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8f37fe640 1 -- 172.21.15.77:0/3807574216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7002ec0 msgr2=0x561fc700cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8f37fe640 1 --2- 172.21.15.77:0/3807574216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7002ec0 0x561fc700cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8f37fe640 1 -- 172.21.15.77:0/3807574216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc70024d0 msgr2=0x561fc700c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8f37fe640 1 --2- 172.21.15.77:0/3807574216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc70024d0 0x561fc700c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8f37fe640 1 -- 172.21.15.77:0/3807574216 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fc6c7c210 con 0x561fc7008030 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.775+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/3807574216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc70024d0 0x561fc700c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8f37fe640 1 --2- 172.21.15.77:0/3807574216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7008030 0x561fc7013db0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec09b890 tx=0x7fa8ec09b320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990dd312d1042508 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8d6ff5640 1 -- 172.21.15.77:0/3807574216 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec0b3070 con 0x561fc7008030 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8d6ff5640 1 -- 172.21.15.77:0/3807574216 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa8ec0860b0 con 0x561fc7008030 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/3807574216 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8e80035f0 con 0x561fc7008030 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8d6ff5640 1 -- 172.21.15.77:0/3807574216 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa8ec0a40a0 con 0x561fc7008030 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8fc3e1740 10 client.15723 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/3807574216 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561fc6d48620 con 0x561fc7008030 2024-03-16T10:17:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8d6ff5640 1 -- 172.21.15.77:0/3807574216 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8e80035f0 con 0x561fc7008030 2024-03-16T10:17:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8d6ff5640 1 -- 172.21.15.77:0/3807574216 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa8ec0afce0 con 0x561fc7008030 2024-03-16T10:17:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8fc3e1740 10 client.15723 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8fc3e1740 10 client.15723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8fc3e1740 20 client.15723 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/3807574216 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561fc70186e0 0x561fc701ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8fc3e1740 4 client.15723 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:17:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.776+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/3807574216 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fc6f74f80 con 0x561fc70186e0 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.777+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/3807574216 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561fc70186e0 0x561fc701ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.777+0000 7fa8d6ff5640 10 client.15723 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.777+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/3807574216 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561fc70186e0 0x561fc701ab50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1d7b7016b03c09c server_cookie=2163fd10d1673dd1 in_seq=0 out_seq=0 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8d6ff5640 1 -- 172.21.15.77:0/3807574216 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561fc6f74f80 con 0x561fc70186e0 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8d6ff5640 10 client.15723 handle_command_reply: tid=0 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8fc3e1740 1 client.15723 shutdown 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/3807574216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7008030 msgr2=0x561fc7013db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/3807574216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7008030 0x561fc7013db0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec09b890 tx=0x7fa8ec09b320 comp rx=0 tx=0).stop 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/3807574216 shutdown_connections 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/3807574216 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561fc70186e0 0x561fc701ab50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/3807574216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc7008030 0x561fc7013db0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/3807574216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7002ec0 0x561fc700cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8d6ff5640 0 client.15723 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/3807574216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc70024d0 0x561fc700c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/3807574216 >> 172.21.15.77:0/3807574216 conn(0x561fc6ffda40 msgr2=0x561fc6ffcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/3807574216 shutdown_connections 2024-03-16T10:17:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/3807574216 wait complete. 2024-03-16T10:17:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.778+0000 7fa8fc3e1740 20 client.15723 trim_cache size 0 max 16384 2024-03-16T10:17:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.794+0000 7fa8fc3e1740 1 Processor -- start 2024-03-16T10:17:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.794+0000 7fa8fc3e1740 1 -- start start 2024-03-16T10:17:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8fc3e1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 0x561fc6ff8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8fc3e1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7002ec0 0x561fc7013d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8fc3e1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7008030 0x561fc70013b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8fc3e1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f41a8a10 con 0x561fc70024d0 2024-03-16T10:17:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8fc3e1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f41aa550 con 0x561fc7002ec0 2024-03-16T10:17:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8fc3e1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f41a90e0 con 0x561fc7008030 2024-03-16T10:17:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8f2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 0x561fc6ff8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8f2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 0x561fc6ff8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52676/0 (socket says 172.21.15.77:52676) 2024-03-16T10:17:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8f2ffd640 1 -- 172.21.15.77:0/1993082169 learned_addr learned my addr 172.21.15.77:0/1993082169 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8f37fe640 1 --2- 172.21.15.77:0/1993082169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7008030 0x561fc70013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8f27fc640 1 --2- 172.21.15.77:0/1993082169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7002ec0 0x561fc7013d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8f2ffd640 1 -- 172.21.15.77:0/1993082169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7008030 msgr2=0x561fc70013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/1993082169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7008030 0x561fc70013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8f2ffd640 1 -- 172.21.15.77:0/1993082169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7002ec0 msgr2=0x561fc7013d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/1993082169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7002ec0 0x561fc7013d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8f2ffd640 1 -- 172.21.15.77:0/1993082169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fc6ff92c0 con 0x561fc70024d0 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8f27fc640 1 --2- 172.21.15.77:0/1993082169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7002ec0 0x561fc7013d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/1993082169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 0x561fc6ff8af0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4068c60 tx=0x7fa8e403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef7aa20efa9e835 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.795+0000 7fa8d8ff9640 1 -- 172.21.15.77:0/1993082169 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4056050 con 0x561fc70024d0 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.796+0000 7fa8d8ff9640 1 -- 172.21.15.77:0/1993082169 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8e40566d0 con 0x561fc70024d0 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.796+0000 7fa8d8ff9640 1 -- 172.21.15.77:0/1993082169 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e40035f0 con 0x561fc70024d0 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.796+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/1993082169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 msgr2=0x561fc6ff8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.796+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/1993082169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 0x561fc6ff8af0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4068c60 tx=0x7fa8e403e000 comp rx=0 tx=0).stop 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.796+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/1993082169 shutdown_connections 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.796+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/1993082169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7008030 0x561fc70013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.796+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/1993082169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7002ec0 0x561fc7013d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.796+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/1993082169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 0x561fc6ff8af0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.796+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/1993082169 >> 172.21.15.77:0/1993082169 conn(0x561fc6ffda40 msgr2=0x561fc7000ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.796+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/1993082169 shutdown_connections 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/1993082169 wait complete. 2024-03-16T10:17:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8fc3e1740 1 -- start start 2024-03-16T10:17:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8fc3e1740 1 Processor -- start 2024-03-16T10:17:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8fc3e1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 0x561fc70137a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8fc3e1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7002ec0 0x561fc7013ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8fc3e1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7008030 0x561fc7011eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8fc3e1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f41a9e80 con 0x561fc70024d0 2024-03-16T10:17:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8fc3e1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f41a97b0 con 0x561fc7002ec0 2024-03-16T10:17:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8fc3e1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f41a6a10 con 0x561fc7008030 2024-03-16T10:17:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8fc3e1740 10 client.0 _ll_register_callbacks cb 0x7fa8ec005bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:17:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8f2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 0x561fc70137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8f2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 0x561fc70137a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52688/0 (socket says 172.21.15.77:52688) 2024-03-16T10:17:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8f2ffd640 1 -- 172.21.15.77:0/4116684821 learned_addr learned my addr 172.21.15.77:0/4116684821 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:17:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8f27fc640 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7002ec0 0x561fc7013ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8f37fe640 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7008030 0x561fc7011eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8f2ffd640 1 -- 172.21.15.77:0/4116684821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7008030 msgr2=0x561fc7011eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7008030 0x561fc7011eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8f2ffd640 1 -- 172.21.15.77:0/4116684821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7002ec0 msgr2=0x561fc7013ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:17:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7002ec0 0x561fc7013ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8f2ffd640 1 -- 172.21.15.77:0/4116684821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fc70126e0 con 0x561fc70024d0 2024-03-16T10:17:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.797+0000 7fa8f27fc640 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7002ec0 0x561fc7013ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:17:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8f2ffd640 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 0x561fc70137a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4006010 tx=0x7fa8e4004f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431c8bb281be0489 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:17:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8d6ff5640 1 -- 172.21.15.77:0/4116684821 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e40046c0 con 0x561fc70024d0 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8d6ff5640 1 -- 172.21.15.77:0/4116684821 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa8e4004860 con 0x561fc70024d0 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/4116684821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8e40566d0 con 0x561fc70024d0 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8d6ff5640 1 -- 172.21.15.77:0/4116684821 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa8e4056020 con 0x561fc70024d0 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8fc3e1740 10 client.15729 fetch_fsmap learned FSMap version 27 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/4116684821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561fc707f620 con 0x561fc70024d0 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8d6ff5640 1 -- 172.21.15.77:0/4116684821 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8e40566d0 con 0x561fc70024d0 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8d6ff5640 1 -- 172.21.15.77:0/4116684821 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa8e4034440 con 0x561fc70024d0 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8fc3e1740 10 client.15729 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8fc3e1740 10 client.15729 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8fc3e1740 20 client.15729 populate_metadata read hostname 'smithi077' 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561fc707fff0 0x561fc7082410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8fc3e1740 4 client.15729 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:17:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.798+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/4116684821 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561fc6f6f2d0 con 0x561fc707fff0 2024-03-16T10:17:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.799+0000 7fa8f27fc640 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561fc707fff0 0x561fc7082410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:17:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.799+0000 7fa8d6ff5640 10 client.15729 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.799+0000 7fa8f27fc640 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561fc707fff0 0x561fc7082410 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d178d824e38cd10d server_cookie=96d19bf55b681bb6 in_seq=0 out_seq=0 2024-03-16T10:17:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8d6ff5640 1 -- 172.21.15.77:0/4116684821 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561fc6f6f2d0 con 0x561fc707fff0 2024-03-16T10:17:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8d6ff5640 10 client.15729 handle_command_reply: tid=0 2024-03-16T10:17:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8fc3e1740 1 client.15729 shutdown 2024-03-16T10:17:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/4116684821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 msgr2=0x561fc70137a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 0x561fc70137a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4006010 tx=0x7fa8e4004f40 comp rx=0 tx=0).stop 2024-03-16T10:17:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/4116684821 shutdown_connections 2024-03-16T10:17:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561fc7008030 0x561fc7011eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561fc7002ec0 0x561fc7013ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561fc707fff0 0x561fc7082410 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8d6ff5640 0 client.15729 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:17:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8fc3e1740 1 --2- 172.21.15.77:0/4116684821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561fc70024d0 0x561fc70137a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/4116684821 >> 172.21.15.77:0/4116684821 conn(0x561fc6ffda40 msgr2=0x561fc7001530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/4116684821 shutdown_connections 2024-03-16T10:17:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.805+0000 7fa8fc3e1740 1 -- 172.21.15.77:0/4116684821 wait complete. 2024-03-16T10:17:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8fc3e1740 20 client.15729 trim_cache size 0 max 16384 2024-03-16T10:17:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:17:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:17:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:17:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:17:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2784439892 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa8dc0a2900 msgr2=0x7fa8dc0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8f97ff640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa8dc0a2900 0x7fa8dc0a4dc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa8e402c7b0 tx=0x7fa8e4000bc0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2784439892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f41032b0 msgr2=0x7fa8f419ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:17:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8f97ff640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f41032b0 0x7fa8f419ef60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec00eb10 tx=0x7fa8ec00c2d0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2784439892 shutdown_connections 2024-03-16T10:17:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8f97ff640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f41032b0 0x7fa8f419ef60 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8f97ff640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa8dc0a2900 0x7fa8dc0a4dc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8f97ff640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8f41028f0 0x7fa8f4197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8f97ff640 1 --2- 172.21.15.77:0/2784439892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8f40ff920 0x7fa8f41979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:17:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2784439892 >> 172.21.15.77:0/2784439892 conn(0x7fa8f40fb250 msgr2=0x7fa8f4105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:17:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2784439892 shutdown_connections 2024-03-16T10:17:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:17:49.806+0000 7fa8f97ff640 1 -- 172.21.15.77:0/2784439892 wait complete. 2024-03-16T10:17:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:49 smithi192 ceph-mon[36354]: pgmap v372: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-16T10:17:50.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:49 smithi077 ceph-mon[32400]: pgmap v372: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-16T10:17:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:49 smithi193 ceph-mon[36311]: pgmap v372: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-16T10:17:52.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:51 smithi192 ceph-mon[36354]: pgmap v373: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 938 B/s wr, 9 op/s 2024-03-16T10:17:52.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:51 smithi077 ceph-mon[32400]: pgmap v373: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 938 B/s wr, 9 op/s 2024-03-16T10:17:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:51 smithi193 ceph-mon[36311]: pgmap v373: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 938 B/s wr, 9 op/s 2024-03-16T10:17:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:53 smithi192 ceph-mon[36354]: pgmap v374: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-16T10:17:54.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:53 smithi077 ceph-mon[32400]: pgmap v374: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-16T10:17:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:53 smithi193 ceph-mon[36311]: pgmap v374: 97 pgs: 97 active+clean; 669 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-16T10:17:56.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:55 smithi192 ceph-mon[36354]: pgmap v375: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-16T10:17:56.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:55 smithi077 ceph-mon[32400]: pgmap v375: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-16T10:17:56.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:55 smithi193 ceph-mon[36311]: pgmap v375: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-16T10:17:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:57 smithi192 ceph-mon[36354]: pgmap v376: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-16T10:17:58.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:57 smithi077 ceph-mon[32400]: pgmap v376: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-16T10:17:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:57 smithi193 ceph-mon[36311]: pgmap v376: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-16T10:18:00.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:17:59 smithi192 ceph-mon[36354]: pgmap v377: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 767 B/s wr, 14 op/s 2024-03-16T10:18:00.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:17:59 smithi077 ceph-mon[32400]: pgmap v377: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 767 B/s wr, 14 op/s 2024-03-16T10:18:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:17:59 smithi193 ceph-mon[36311]: pgmap v377: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 767 B/s wr, 14 op/s 2024-03-16T10:18:02.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:01 smithi192 ceph-mon[36354]: pgmap v378: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-16T10:18:02.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:01 smithi077 ceph-mon[32400]: pgmap v378: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-16T10:18:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:01 smithi193 ceph-mon[36311]: pgmap v378: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-16T10:18:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.916+0000 7f67d637e640 1 -- 172.21.15.77:0/2744127115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67d01018f0 msgr2=0x7f67d0101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.916+0000 7f67d637e640 1 --2- 172.21.15.77:0/2744127115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67d01018f0 0x7f67d0101d70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009900 tx=0x7f67c402bc30 comp rx=0 tx=0).stop 2024-03-16T10:18:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.916+0000 7f67d637e640 1 -- 172.21.15.77:0/2744127115 shutdown_connections 2024-03-16T10:18:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.916+0000 7f67d637e640 1 --2- 172.21.15.77:0/2744127115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67d01022b0 0x7f67d0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.916+0000 7f67d637e640 1 --2- 172.21.15.77:0/2744127115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67d01018f0 0x7f67d0101d70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.916+0000 7f67d637e640 1 --2- 172.21.15.77:0/2744127115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67d0100700 0x7f67d0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.916+0000 7f67d637e640 1 -- 172.21.15.77:0/2744127115 >> 172.21.15.77:0/2744127115 conn(0x7f67d00fbc70 msgr2=0x7f67d00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.917+0000 7f67d637e640 1 -- 172.21.15.77:0/2744127115 shutdown_connections 2024-03-16T10:18:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.917+0000 7f67d637e640 1 -- 172.21.15.77:0/2744127115 wait complete. 2024-03-16T10:18:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.917+0000 7f67d637e640 1 Processor -- start 2024-03-16T10:18:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.917+0000 7f67d637e640 1 -- start start 2024-03-16T10:18:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67d637e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67d0100700 0x7f67d0199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67d637e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67d01018f0 0x7f67d019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67d637e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67d01022b0 0x7f67d01a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67d637e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f67d010b280 con 0x7f67d01018f0 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67d637e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f67d010b100 con 0x7f67d01022b0 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67cf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67d01018f0 0x7f67d019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67cf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67d01018f0 0x7f67d019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51692/0 (socket says 172.21.15.77:51692) 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67cffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67d0100700 0x7f67d0199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67cf7fe640 1 -- 172.21.15.77:0/730294908 learned_addr learned my addr 172.21.15.77:0/730294908 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67d637e640 1 -- 172.21.15.77:0/730294908 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f67d010b400 con 0x7f67d0100700 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67d491b640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67d01022b0 0x7f67d01a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67cffff640 1 -- 172.21.15.77:0/730294908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67d01022b0 msgr2=0x7f67d01a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67cffff640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67d01022b0 0x7f67d01a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67cffff640 1 -- 172.21.15.77:0/730294908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67d01018f0 msgr2=0x7f67d019a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67cffff640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67d01018f0 0x7f67d019a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67cffff640 1 -- 172.21.15.77:0/730294908 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d01a1870 con 0x7f67d0100700 2024-03-16T10:18:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67cf7fe640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67d01018f0 0x7f67d019a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67d491b640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67d01022b0 0x7f67d01a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.918+0000 7f67cffff640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67d0100700 0x7f67d0199c60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f67c0009730 tx=0x7f67c0009bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.919+0000 7f67cd7fa640 1 -- 172.21.15.77:0/730294908 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0009e40 con 0x7f67d0100700 2024-03-16T10:18:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.919+0000 7f67cd7fa640 1 -- 172.21.15.77:0/730294908 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c0004020 con 0x7f67d0100700 2024-03-16T10:18:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.919+0000 7f67cd7fa640 1 -- 172.21.15.77:0/730294908 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0003da0 con 0x7f67d0100700 2024-03-16T10:18:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.919+0000 7f67d637e640 1 -- 172.21.15.77:0/730294908 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d01a1b60 con 0x7f67d0100700 2024-03-16T10:18:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.919+0000 7f67d637e640 1 -- 172.21.15.77:0/730294908 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d01a2060 con 0x7f67d0100700 2024-03-16T10:18:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.921+0000 7f67cd7fa640 1 -- 172.21.15.77:0/730294908 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f67c001d020 con 0x7f67d0100700 2024-03-16T10:18:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.922+0000 7f67cd7fa640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f67b80a2a80 0x7f67b80a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.922+0000 7f67d8f6c740 1 Processor -- start 2024-03-16T10:18:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.922+0000 7f67d8f6c740 1 -- start start 2024-03-16T10:18:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67cf7fe640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f67b80a2a80 0x7f67b80a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67cd7fa640 1 -- 172.21.15.77:0/730294908 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f67c00cdf30 con 0x7f67d0100700 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67d8f6c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706340 0x562037706760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67d8f6c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 0x56203770b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67d8f6c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56203770bea0 0x56203770e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67d8f6c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56203766ba40 con 0x562037706340 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67d8f6c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x562037614920 con 0x56203770bea0 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67d8f6c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562037667ae0 con 0x562037706d30 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67cffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706340 0x562037706760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67cffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706340 0x562037706760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51696/0 (socket says 172.21.15.77:51696) 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67cffff640 1 -- 172.21.15.77:0/815980903 learned_addr learned my addr 172.21.15.77:0/815980903 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67cf7fe640 1 --2- 172.21.15.77:0/815980903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 0x56203770b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67d491b640 1 --2- 172.21.15.77:0/815980903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56203770bea0 0x56203770e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67cf7fe640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f67b80a2a80 0x7f67b80a4f40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f67c4000c00 tx=0x7f67c402c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67cffff640 1 -- 172.21.15.77:0/815980903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 msgr2=0x56203770b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67cffff640 1 --2- 172.21.15.77:0/815980903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 0x56203770b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67cffff640 1 -- 172.21.15.77:0/815980903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56203770bea0 msgr2=0x56203770e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67cffff640 1 --2- 172.21.15.77:0/815980903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56203770bea0 0x56203770e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67cffff640 1 -- 172.21.15.77:0/815980903 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56203762c460 con 0x562037706340 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.923+0000 7f67d491b640 1 --2- 172.21.15.77:0/815980903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56203770bea0 0x56203770e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.924+0000 7f67cffff640 1 --2- 172.21.15.77:0/815980903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706340 0x562037706760 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f67c00a2ad0 tx=0x7f67c0098d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d206409be8e1e904 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.924+0000 7f67b67fc640 1 -- 172.21.15.77:0/815980903 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c009a860 con 0x562037706340 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.924+0000 7f67b67fc640 1 -- 172.21.15.77:0/815980903 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c009e9b0 con 0x562037706340 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.924+0000 7f67b67fc640 1 -- 172.21.15.77:0/815980903 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c009bb90 con 0x562037706340 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.925+0000 7f67d8f6c740 1 -- 172.21.15.77:0/815980903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706340 msgr2=0x562037706760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.925+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/815980903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706340 0x562037706760 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f67c00a2ad0 tx=0x7f67c0098d70 comp rx=0 tx=0).stop 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.925+0000 7f67d8f6c740 1 -- 172.21.15.77:0/815980903 shutdown_connections 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.925+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/815980903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56203770bea0 0x56203770e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.925+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/815980903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 0x56203770b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.925+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/815980903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706340 0x562037706760 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.925+0000 7f67d8f6c740 1 -- 172.21.15.77:0/815980903 >> 172.21.15.77:0/815980903 conn(0x5620377018b0 msgr2=0x562037701cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.925+0000 7f67d8f6c740 1 -- 172.21.15.77:0/815980903 shutdown_connections 2024-03-16T10:18:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.925+0000 7f67d8f6c740 1 -- 172.21.15.77:0/815980903 wait complete. 2024-03-16T10:18:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.926+0000 7f67d8f6c740 1 -- start start 2024-03-16T10:18:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.926+0000 7f67d8f6c740 1 Processor -- start 2024-03-16T10:18:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.926+0000 7f67d8f6c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 0x562037710730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.926+0000 7f67d8f6c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 0x562037710c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.926+0000 7f67cffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 0x562037710730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.926+0000 7f67cffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 0x562037710730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:37380/0 (socket says 172.21.15.77:37380) 2024-03-16T10:18:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67cffff640 1 -- 172.21.15.77:0/3302474765 learned_addr learned my addr 172.21.15.77:0/3302474765 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56203770bea0 0x562037717cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67cf7fe640 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 0x562037710c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67d491b640 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56203770bea0 0x562037717cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3302474765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56203770ee50 con 0x56203770bea0 2024-03-16T10:18:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3302474765 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x562037627d70 con 0x562037706340 2024-03-16T10:18:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3302474765 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56203770efd0 con 0x562037706d30 2024-03-16T10:18:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67d8f6c740 10 client.0 _ll_register_callbacks cb 0x56203737fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67cffff640 1 -- 172.21.15.77:0/3302474765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 msgr2=0x562037710c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67cffff640 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 0x562037710c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67cffff640 1 -- 172.21.15.77:0/3302474765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56203770bea0 msgr2=0x562037717cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67cffff640 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56203770bea0 0x562037717cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67cffff640 1 -- 172.21.15.77:0/3302474765 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562037380210 con 0x562037706340 2024-03-16T10:18:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67cffff640 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 0x562037710730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f67c00ce2a0 tx=0x7f67c0097070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5ae3529a993aa49 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.927+0000 7f67af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:18:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.928+0000 7f67af7fe640 1 -- 172.21.15.77:0/3302474765 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c00b7070 con 0x562037706340 2024-03-16T10:18:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.928+0000 7f67af7fe640 1 -- 172.21.15.77:0/3302474765 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f67c0099eb0 con 0x562037706340 2024-03-16T10:18:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.928+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3302474765 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67c009e9b0 con 0x562037706340 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.928+0000 7f67af7fe640 1 -- 172.21.15.77:0/3302474765 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f67c009a860 con 0x562037706340 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.928+0000 7f67d8f6c740 10 client.25547 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.928+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3302474765 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56203744c620 con 0x562037706340 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.928+0000 7f67af7fe640 1 -- 172.21.15.77:0/3302474765 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c009e9b0 con 0x562037706340 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.928+0000 7f67af7fe640 1 -- 172.21.15.77:0/3302474765 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f67c00a5060 con 0x562037706340 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.928+0000 7f67d8f6c740 10 client.25547 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.929+0000 7f67d8f6c740 10 client.25547 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.929+0000 7f67d8f6c740 20 client.25547 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.929+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56203771bf90 0x56203771e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.929+0000 7f67d8f6c740 4 client.25547 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.929+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3302474765 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562037678f80 con 0x56203771bf90 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.929+0000 7f67cf7fe640 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56203771bf90 0x56203771e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.929+0000 7f67cf7fe640 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56203771bf90 0x56203771e380 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a019990160bd26c server_cookie=3bf322c3bf131973 in_seq=0 out_seq=0 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.929+0000 7f67af7fe640 10 client.25547 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.930+0000 7f67af7fe640 1 -- 172.21.15.77:0/3302474765 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562037678f80 con 0x56203771bf90 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.930+0000 7f67af7fe640 10 client.25547 handle_command_reply: tid=0 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.930+0000 7f67d8f6c740 1 client.25547 shutdown 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.931+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3302474765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 msgr2=0x562037710730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.931+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 0x562037710730 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f67c00ce2a0 tx=0x7f67c0097070 comp rx=0 tx=0).stop 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.931+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3302474765 shutdown_connections 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.931+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56203771bf90 0x56203771e380 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.931+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56203770bea0 0x562037717cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.931+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 0x562037710c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.931+0000 7f67af7fe640 0 client.25547 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.931+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3302474765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 0x562037710730 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.931+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3302474765 >> 172.21.15.77:0/3302474765 conn(0x5620377018b0 msgr2=0x562037700bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.931+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3302474765 shutdown_connections 2024-03-16T10:18:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.931+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3302474765 wait complete. 2024-03-16T10:18:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.932+0000 7f67d8f6c740 20 client.25547 trim_cache size 0 max 16384 2024-03-16T10:18:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67d8f6c740 1 Processor -- start 2024-03-16T10:18:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67d8f6c740 1 -- start start 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67d8f6c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706340 0x562037710bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67d8f6c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706d30 0x562037717c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67d8f6c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56203770bea0 0x562037705220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67d8f6c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f67d01aac40 con 0x562037706d30 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67d8f6c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f67d01ac780 con 0x56203770bea0 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67d8f6c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f67d01ab310 con 0x562037706340 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67cffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706340 0x562037710bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67cf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706d30 0x562037717c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67cffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706340 0x562037710bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:34848/0 (socket says 172.21.15.77:34848) 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67cf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706d30 0x562037717c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51716/0 (socket says 172.21.15.77:51716) 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67cffff640 1 -- 172.21.15.77:0/3252818846 learned_addr learned my addr 172.21.15.77:0/3252818846 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67d491b640 1 --2- 172.21.15.77:0/3252818846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56203770bea0 0x562037705220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67cf7fe640 1 -- 172.21.15.77:0/3252818846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706340 msgr2=0x562037710bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67cf7fe640 1 --2- 172.21.15.77:0/3252818846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706340 0x562037710bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67cf7fe640 1 -- 172.21.15.77:0/3252818846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56203770bea0 msgr2=0x562037705220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67cf7fe640 1 --2- 172.21.15.77:0/3252818846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56203770bea0 0x562037705220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67cf7fe640 1 -- 172.21.15.77:0/3252818846 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620376f3140 con 0x562037706d30 2024-03-16T10:18:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.959+0000 7f67cffff640 1 --2- 172.21.15.77:0/3252818846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706340 0x562037710bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.960+0000 7f67cf7fe640 1 --2- 172.21.15.77:0/3252818846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706d30 0x562037717c70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f67c402c7b0 tx=0x7f67c4044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01ee81339b39056 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.960+0000 7f67b5ffb640 1 -- 172.21.15.77:0/3252818846 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c40048c0 con 0x562037706d30 2024-03-16T10:18:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.960+0000 7f67b5ffb640 1 -- 172.21.15.77:0/3252818846 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c4044070 con 0x562037706d30 2024-03-16T10:18:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.960+0000 7f67b5ffb640 1 -- 172.21.15.77:0/3252818846 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c40051e0 con 0x562037706d30 2024-03-16T10:18:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.961+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3252818846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706d30 msgr2=0x562037717c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.961+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3252818846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706d30 0x562037717c70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f67c402c7b0 tx=0x7f67c4044000 comp rx=0 tx=0).stop 2024-03-16T10:18:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.961+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3252818846 shutdown_connections 2024-03-16T10:18:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.961+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3252818846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56203770bea0 0x562037705220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.961+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3252818846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562037706d30 0x562037717c70 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.961+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3252818846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706340 0x562037710bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.961+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3252818846 >> 172.21.15.77:0/3252818846 conn(0x5620377018b0 msgr2=0x562037704b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.961+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3252818846 shutdown_connections 2024-03-16T10:18:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.961+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3252818846 wait complete. 2024-03-16T10:18:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.961+0000 7f67d8f6c740 1 -- start start 2024-03-16T10:18:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67d8f6c740 1 Processor -- start 2024-03-16T10:18:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67d8f6c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 0x562037715310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67d8f6c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 0x562037715850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67d8f6c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56203770bea0 0x5620377167f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67cffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 0x562037715310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67cffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 0x562037715310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:37396/0 (socket says 172.21.15.77:37396) 2024-03-16T10:18:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67d491b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56203770bea0 0x5620377167f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67d491b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56203770bea0 0x5620377167f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51730/0 (socket says 172.21.15.77:51730) 2024-03-16T10:18:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67cf7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 0x562037715850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67cffff640 1 -- 172.21.15.77:0/3970612916 learned_addr learned my addr 172.21.15.77:0/3970612916 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3970612916 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f67d01ac0b0 con 0x56203770bea0 2024-03-16T10:18:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3970612916 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f67d01ab9e0 con 0x562037706340 2024-03-16T10:18:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3970612916 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f67d01a8c20 con 0x562037706d30 2024-03-16T10:18:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.962+0000 7f67d8f6c740 10 client.0 _ll_register_callbacks cb 0x7f67c8009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67cffff640 1 -- 172.21.15.77:0/3970612916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 msgr2=0x562037715850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67cffff640 1 --2- 172.21.15.77:0/3970612916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 0x562037715850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67cffff640 1 -- 172.21.15.77:0/3970612916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56203770bea0 msgr2=0x5620377167f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67cffff640 1 --2- 172.21.15.77:0/3970612916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56203770bea0 0x5620377167f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67cffff640 1 -- 172.21.15.77:0/3970612916 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562037705f80 con 0x562037706340 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67d491b640 1 --2- 172.21.15.77:0/3970612916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56203770bea0 0x5620377167f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67cffff640 1 --2- 172.21.15.77:0/3970612916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 0x562037715310 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f67c0096c30 tx=0x7f67c0099ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6055f318543f37ff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67af7fe640 1 -- 172.21.15.77:0/3970612916 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c00b7070 con 0x562037706340 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67af7fe640 1 -- 172.21.15.77:0/3970612916 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f67c0096520 con 0x562037706340 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3970612916 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67c4044070 con 0x562037706340 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67af7fe640 1 -- 172.21.15.77:0/3970612916 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f67c009e9b0 con 0x562037706340 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67d8f6c740 10 client.25553 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.963+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3970612916 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562037705f80 con 0x562037706340 2024-03-16T10:18:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.964+0000 7f67af7fe640 1 -- 172.21.15.77:0/3970612916 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c4044070 con 0x562037706340 2024-03-16T10:18:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.964+0000 7f67af7fe640 1 -- 172.21.15.77:0/3970612916 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f67c0096720 con 0x562037706340 2024-03-16T10:18:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.964+0000 7f67d8f6c740 10 client.25553 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.964+0000 7f67d8f6c740 10 client.25553 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.964+0000 7f67d8f6c740 20 client.25553 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.964+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3970612916 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56203771bf90 0x562037784510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.964+0000 7f67d8f6c740 4 client.25553 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:18:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.964+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3970612916 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5620376732d0 con 0x56203771bf90 2024-03-16T10:18:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.964+0000 7f67cf7fe640 1 --2- 172.21.15.77:0/3970612916 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56203771bf90 0x562037784510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.964+0000 7f67af7fe640 10 client.25553 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.965+0000 7f67cf7fe640 1 --2- 172.21.15.77:0/3970612916 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56203771bf90 0x562037784510 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4e319319f8cf741 server_cookie=d494359a4a48f8b4 in_seq=0 out_seq=0 2024-03-16T10:18:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67af7fe640 1 -- 172.21.15.77:0/3970612916 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5620376732d0 con 0x56203771bf90 2024-03-16T10:18:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67af7fe640 10 client.25553 handle_command_reply: tid=0 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67d8f6c740 1 client.25553 shutdown 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3970612916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 msgr2=0x562037715310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3970612916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 0x562037715310 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f67c0096c30 tx=0x7f67c0099ee0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3970612916 shutdown_connections 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3970612916 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56203771bf90 0x562037784510 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3970612916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56203770bea0 0x5620377167f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3970612916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562037706d30 0x562037715850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67d8f6c740 1 --2- 172.21.15.77:0/3970612916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562037706340 0x562037715310 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3970612916 >> 172.21.15.77:0/3970612916 conn(0x5620377018b0 msgr2=0x5620377053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67af7fe640 0 client.25553 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3970612916 shutdown_connections 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67d8f6c740 1 -- 172.21.15.77:0/3970612916 wait complete. 2024-03-16T10:18:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.968+0000 7f67d8f6c740 20 client.25553 trim_cache size 0 max 16384 2024-03-16T10:18:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:18:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:18:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:18:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:18:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.969+0000 7f67d637e640 1 -- 172.21.15.77:0/730294908 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f67b80a2a80 msgr2=0x7f67b80a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.969+0000 7f67d637e640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f67b80a2a80 0x7f67b80a4f40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f67c4000c00 tx=0x7f67c402c880 comp rx=0 tx=0).stop 2024-03-16T10:18:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.969+0000 7f67d637e640 1 -- 172.21.15.77:0/730294908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67d0100700 msgr2=0x7f67d0199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.969+0000 7f67d637e640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67d0100700 0x7f67d0199c60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f67c0009730 tx=0x7f67c0009bf0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.969+0000 7f67d637e640 1 -- 172.21.15.77:0/730294908 shutdown_connections 2024-03-16T10:18:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.969+0000 7f67d637e640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67d01022b0 0x7f67d01a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.969+0000 7f67d637e640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67d01018f0 0x7f67d019a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.969+0000 7f67d637e640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f67b80a2a80 0x7f67b80a4f40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.969+0000 7f67d637e640 1 --2- 172.21.15.77:0/730294908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67d0100700 0x7f67d0199c60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.969+0000 7f67d637e640 1 -- 172.21.15.77:0/730294908 >> 172.21.15.77:0/730294908 conn(0x7f67d00fbc70 msgr2=0x7f67d0104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.969+0000 7f67d637e640 1 -- 172.21.15.77:0/730294908 shutdown_connections 2024-03-16T10:18:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:02.969+0000 7f67d637e640 1 -- 172.21.15.77:0/730294908 wait complete. 2024-03-16T10:18:04.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:03 smithi192 ceph-mon[36354]: pgmap v379: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-16T10:18:04.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:03 smithi077 ceph-mon[32400]: pgmap v379: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-16T10:18:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:03 smithi193 ceph-mon[36311]: pgmap v379: 97 pgs: 97 active+clean; 670 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-16T10:18:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:05 smithi192 ceph-mon[36354]: pgmap v380: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-16T10:18:06.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:05 smithi077 ceph-mon[32400]: pgmap v380: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-16T10:18:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:05 smithi193 ceph-mon[36311]: pgmap v380: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-16T10:18:08.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:07 smithi192 ceph-mon[36354]: pgmap v381: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-16T10:18:08.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:07 smithi077 ceph-mon[32400]: pgmap v381: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-16T10:18:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:07 smithi193 ceph-mon[36311]: pgmap v381: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-16T10:18:10.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:09 smithi192 ceph-mon[36354]: pgmap v382: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-16T10:18:10.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:09 smithi077 ceph-mon[32400]: pgmap v382: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-16T10:18:10.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:09 smithi193 ceph-mon[36311]: pgmap v382: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-16T10:18:12.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:11 smithi192 ceph-mon[36354]: pgmap v383: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T10:18:12.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:11 smithi077 ceph-mon[32400]: pgmap v383: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T10:18:12.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:11 smithi193 ceph-mon[36311]: pgmap v383: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-16T10:18:13.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:18:13.268 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:18:13.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.412+0000 7eff6b675640 1 -- 172.21.15.77:0/368435700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff64101930 msgr2=0x7eff64101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:13.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.412+0000 7eff6b675640 1 --2- 172.21.15.77:0/368435700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff64101930 0x7eff64101db0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7eff54009860 tx=0x7eff5402bb70 comp rx=0 tx=0).stop 2024-03-16T10:18:13.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.412+0000 7eff6b675640 1 -- 172.21.15.77:0/368435700 shutdown_connections 2024-03-16T10:18:13.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.412+0000 7eff6b675640 1 --2- 172.21.15.77:0/368435700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff641022f0 0x7eff64108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.412+0000 7eff6b675640 1 --2- 172.21.15.77:0/368435700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff64101930 0x7eff64101db0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.412+0000 7eff6b675640 1 --2- 172.21.15.77:0/368435700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff64100c20 0x7eff64101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.412+0000 7eff6b675640 1 -- 172.21.15.77:0/368435700 >> 172.21.15.77:0/368435700 conn(0x7eff640fb3f0 msgr2=0x7eff640fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:13.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.412+0000 7eff6b675640 1 -- 172.21.15.77:0/368435700 shutdown_connections 2024-03-16T10:18:13.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.412+0000 7eff6b675640 1 -- 172.21.15.77:0/368435700 wait complete. 2024-03-16T10:18:13.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.413+0000 7eff6b675640 1 Processor -- start 2024-03-16T10:18:13.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.413+0000 7eff6b675640 1 -- start start 2024-03-16T10:18:13.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.413+0000 7eff6b675640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff64100c20 0x7eff641958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.413+0000 7eff6b675640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff64101930 0x7eff64195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.416 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.413+0000 7eff6b675640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff641022f0 0x7eff6419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.413+0000 7eff69411640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff64100c20 0x7eff641958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.413+0000 7eff69411640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff64100c20 0x7eff641958d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33044/0 (socket says 172.21.15.77:33044) 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff68c10640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff64101930 0x7eff64195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff68c10640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff64101930 0x7eff64195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50406/0 (socket says 172.21.15.77:50406) 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff69c12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff641022f0 0x7eff6419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff6b675640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff6410b280 con 0x7eff641022f0 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff69c12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff641022f0 0x7eff6419ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54206/0 (socket says 172.21.15.77:54206) 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff69c12640 1 -- 172.21.15.77:0/1117231636 learned_addr learned my addr 172.21.15.77:0/1117231636 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff6b675640 1 -- 172.21.15.77:0/1117231636 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7eff6410b100 con 0x7eff64101930 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff6b675640 1 -- 172.21.15.77:0/1117231636 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff6410b400 con 0x7eff64100c20 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff69411640 1 -- 172.21.15.77:0/1117231636 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff64101930 msgr2=0x7eff64195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff69411640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff64101930 0x7eff64195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff69411640 1 -- 172.21.15.77:0/1117231636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff641022f0 msgr2=0x7eff6419ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:13.417 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff69411640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff641022f0 0x7eff6419ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff69411640 1 -- 172.21.15.77:0/1117231636 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6419d4f0 con 0x7eff64100c20 2024-03-16T10:18:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff69c12640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff641022f0 0x7eff6419ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff69411640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff64100c20 0x7eff641958d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7eff60005fd0 tx=0x7eff6000ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff5a7fc640 1 -- 172.21.15.77:0/1117231636 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6000ccc0 con 0x7eff64100c20 2024-03-16T10:18:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff5a7fc640 1 -- 172.21.15.77:0/1117231636 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff6000ee70 con 0x7eff64100c20 2024-03-16T10:18:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.414+0000 7eff5a7fc640 1 -- 172.21.15.77:0/1117231636 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff60010430 con 0x7eff64100c20 2024-03-16T10:18:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.415+0000 7eff6b675640 1 -- 172.21.15.77:0/1117231636 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6419d7e0 con 0x7eff64100c20 2024-03-16T10:18:13.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.415+0000 7eff6b675640 1 -- 172.21.15.77:0/1117231636 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff6419dc50 con 0x7eff64100c20 2024-03-16T10:18:13.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.418+0000 7eff6e25b740 1 Processor -- start 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.418+0000 7eff6e25b740 1 -- start start 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.418+0000 7eff5a7fc640 1 -- 172.21.15.77:0/1117231636 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7eff60003ef0 con 0x7eff64100c20 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.418+0000 7eff6e25b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da32575590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.418+0000 7eff6e25b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da32575b60 0x55da3257a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.418+0000 7eff6e25b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da3257acd0 0x55da3257d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.418+0000 7eff6e25b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55da324da500 con 0x55da32575170 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.418+0000 7eff69c12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da32575590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.418+0000 7eff69c12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da32575590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54218/0 (socket says 172.21.15.77:54218) 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.418+0000 7eff69c12640 1 -- 172.21.15.77:0/2230305464 learned_addr learned my addr 172.21.15.77:0/2230305464 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.418+0000 7eff69411640 1 --2- 172.21.15.77:0/2230305464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da32575b60 0x55da3257a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.418+0000 7eff68c10640 1 --2- 172.21.15.77:0/2230305464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da3257acd0 0x55da3257d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.419+0000 7eff6e25b740 1 -- 172.21.15.77:0/2230305464 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55da32482be0 con 0x55da3257acd0 2024-03-16T10:18:13.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.419+0000 7eff5a7fc640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff500a2910 0x7eff500a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.419+0000 7eff6e25b740 1 -- 172.21.15.77:0/2230305464 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55da324d65a0 con 0x55da32575b60 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.419+0000 7eff5a7fc640 1 -- 172.21.15.77:0/1117231636 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7eff600cee00 con 0x7eff64100c20 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.419+0000 7eff69411640 1 -- 172.21.15.77:0/2230305464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da3257acd0 msgr2=0x55da3257d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.419+0000 7eff69411640 1 --2- 172.21.15.77:0/2230305464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da3257acd0 0x55da3257d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.419+0000 7eff69411640 1 -- 172.21.15.77:0/2230305464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 msgr2=0x55da32575590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.419+0000 7eff69411640 1 --2- 172.21.15.77:0/2230305464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da32575590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.419+0000 7eff69411640 1 -- 172.21.15.77:0/2230305464 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da324964b0 con 0x55da32575b60 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.419+0000 7eff69c12640 1 --2- 172.21.15.77:0/2230305464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da32575590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.419+0000 7eff69411640 1 --2- 172.21.15.77:0/2230305464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da32575b60 0x55da3257a660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7eff600cc7a0 tx=0x7eff60009690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5456feb0e57e1fd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.419+0000 7eff69c12640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff500a2910 0x7eff500a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.420+0000 7eff4f7fe640 1 -- 172.21.15.77:0/2230305464 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6009bd30 con 0x55da32575b60 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.420+0000 7eff4f7fe640 1 -- 172.21.15.77:0/2230305464 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff6009b630 con 0x55da32575b60 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.420+0000 7eff4f7fe640 1 -- 172.21.15.77:0/2230305464 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff600a3ba0 con 0x55da32575b60 2024-03-16T10:18:13.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.420+0000 7eff6e25b740 1 -- 172.21.15.77:0/2230305464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da32575b60 msgr2=0x55da3257a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.420+0000 7eff69c12640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff500a2910 0x7eff500a4dd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00bfd0 tx=0x7eff5c010920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.420+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2230305464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da32575b60 0x55da3257a660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7eff600cc7a0 tx=0x7eff60009690 comp rx=0 tx=0).stop 2024-03-16T10:18:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.421+0000 7eff6e25b740 1 -- 172.21.15.77:0/2230305464 shutdown_connections 2024-03-16T10:18:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.421+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2230305464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da3257acd0 0x55da3257d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.421+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2230305464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da32575b60 0x55da3257a660 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.421+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2230305464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da32575590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.421+0000 7eff6e25b740 1 -- 172.21.15.77:0/2230305464 >> 172.21.15.77:0/2230305464 conn(0x55da325706e0 msgr2=0x55da32570ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.421+0000 7eff6e25b740 1 -- 172.21.15.77:0/2230305464 shutdown_connections 2024-03-16T10:18:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.421+0000 7eff6e25b740 1 -- 172.21.15.77:0/2230305464 wait complete. 2024-03-16T10:18:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.421+0000 7eff6e25b740 1 -- start start 2024-03-16T10:18:13.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.421+0000 7eff6e25b740 1 Processor -- start 2024-03-16T10:18:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff6e25b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da3257f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff6e25b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575b60 0x55da3257fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff6e25b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da32586bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff6e25b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55da3257dbc0 con 0x55da32575170 2024-03-16T10:18:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff6e25b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55da32496160 con 0x55da32575b60 2024-03-16T10:18:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff6e25b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55da3257dd40 con 0x55da3257acd0 2024-03-16T10:18:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff6e25b740 10 client.0 _ll_register_callbacks cb 0x55da3247a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff69411640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da3257f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff69411640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da3257f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54234/0 (socket says 172.21.15.77:54234) 2024-03-16T10:18:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff69411640 1 -- 172.21.15.77:0/3022378233 learned_addr learned my addr 172.21.15.77:0/3022378233 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:13.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff69c12640 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da32586bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff68c10640 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575b60 0x55da3257fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff69c12640 1 -- 172.21.15.77:0/3022378233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575b60 msgr2=0x55da3257fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff69c12640 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575b60 0x55da3257fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff69c12640 1 -- 172.21.15.77:0/3022378233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 msgr2=0x55da3257f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff69c12640 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da3257f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff69c12640 1 -- 172.21.15.77:0/3022378233 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da322b87d0 con 0x55da3257acd0 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.422+0000 7eff69411640 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da3257f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff69c12640 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da32586bd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0049d0 tx=0x7eff5c00b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b79fd0e8dc1c67e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff4cff9640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff4cff9640 1 -- 172.21.15.77:0/3022378233 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c004e00 con 0x55da3257acd0 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff4cff9640 1 -- 172.21.15.77:0/3022378233 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7eff5c009ca0 con 0x55da3257acd0 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff6e25b740 1 -- 172.21.15.77:0/3022378233 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff6009b630 con 0x55da3257acd0 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff4cff9640 1 -- 172.21.15.77:0/3022378233 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7eff5c009ea0 con 0x55da3257acd0 2024-03-16T10:18:13.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff6e25b740 10 client.25549 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff6e25b740 1 -- 172.21.15.77:0/3022378233 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55da321ee3f0 con 0x55da3257acd0 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff4cff9640 1 -- 172.21.15.77:0/3022378233 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff6009b630 con 0x55da3257acd0 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff4cff9640 1 -- 172.21.15.77:0/3022378233 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7eff5c01d930 con 0x55da3257acd0 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff6e25b740 10 client.25549 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff6e25b740 10 client.25549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff6e25b740 20 client.25549 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff6e25b740 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55da32589f90 0x55da3258a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff6e25b740 4 client.25549 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.423+0000 7eff6e25b740 1 -- 172.21.15.77:0/3022378233 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da324e7a40 con 0x55da32589f90 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.424+0000 7eff69411640 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55da32589f90 0x55da3258a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.424+0000 7eff4cff9640 10 client.25549 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.424+0000 7eff69411640 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55da32589f90 0x55da3258a3f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea11e8f7d367c046 server_cookie=8b4d515bd4fdf307 in_seq=0 out_seq=0 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff4cff9640 1 -- 172.21.15.77:0/3022378233 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55da324e7a40 con 0x55da32589f90 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff4cff9640 10 client.25549 handle_command_reply: tid=0 2024-03-16T10:18:13.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff6e25b740 1 client.25549 shutdown 2024-03-16T10:18:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff6e25b740 1 -- 172.21.15.77:0/3022378233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 msgr2=0x55da32586bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff6e25b740 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da32586bd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0049d0 tx=0x7eff5c00b710 comp rx=0 tx=0).stop 2024-03-16T10:18:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff6e25b740 1 -- 172.21.15.77:0/3022378233 shutdown_connections 2024-03-16T10:18:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff6e25b740 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da32586bd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff6e25b740 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575b60 0x55da3257fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff6e25b740 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55da32589f90 0x55da3258a3f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff6e25b740 1 --2- 172.21.15.77:0/3022378233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da3257f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff6e25b740 1 -- 172.21.15.77:0/3022378233 >> 172.21.15.77:0/3022378233 conn(0x55da325706e0 msgr2=0x55da3256f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff4cff9640 0 client.25549 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff6e25b740 1 -- 172.21.15.77:0/3022378233 shutdown_connections 2024-03-16T10:18:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff6e25b740 1 -- 172.21.15.77:0/3022378233 wait complete. 2024-03-16T10:18:13.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.425+0000 7eff6e25b740 20 client.25549 trim_cache size 0 max 16384 2024-03-16T10:18:13.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.443+0000 7eff6e25b740 1 Processor -- start 2024-03-16T10:18:13.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.443+0000 7eff6e25b740 1 -- start start 2024-03-16T10:18:13.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.443+0000 7eff6e25b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575170 0x55da32570ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.443+0000 7eff6e25b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575b60 0x55da32586b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.443+0000 7eff6e25b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da325770f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.443+0000 7eff6e25b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff641a69e0 con 0x55da32575b60 2024-03-16T10:18:13.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.443+0000 7eff6e25b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7eff641a8520 con 0x55da32575170 2024-03-16T10:18:13.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.443+0000 7eff6e25b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff641a70b0 con 0x55da3257acd0 2024-03-16T10:18:13.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.443+0000 7eff68c10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575b60 0x55da32586b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.443+0000 7eff68c10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575b60 0x55da32586b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54242/0 (socket says 172.21.15.77:54242) 2024-03-16T10:18:13.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.443+0000 7eff68c10640 1 -- 172.21.15.77:0/2688612921 learned_addr learned my addr 172.21.15.77:0/2688612921 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:13.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.443+0000 7eff69411640 1 --2- 172.21.15.77:0/2688612921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575170 0x55da32570ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.444+0000 7eff69c12640 1 --2- 172.21.15.77:0/2688612921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da325770f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.444+0000 7eff69411640 1 -- 172.21.15.77:0/2688612921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 msgr2=0x55da325770f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.444+0000 7eff69411640 1 --2- 172.21.15.77:0/2688612921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da325770f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.444+0000 7eff69411640 1 -- 172.21.15.77:0/2688612921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575b60 msgr2=0x55da32586b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.444+0000 7eff69411640 1 --2- 172.21.15.77:0/2688612921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575b60 0x55da32586b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.444+0000 7eff69411640 1 -- 172.21.15.77:0/2688612921 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da321ae0f0 con 0x55da32575170 2024-03-16T10:18:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.444+0000 7eff69c12640 1 --2- 172.21.15.77:0/2688612921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da325770f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:18:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.444+0000 7eff68c10640 1 --2- 172.21.15.77:0/2688612921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575b60 0x55da32586b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.444+0000 7eff69411640 1 --2- 172.21.15.77:0/2688612921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575170 0x55da32570ac0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7eff60097a90 tx=0x7eff600cc710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66e39d82b7670101 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.444+0000 7eff4effd640 1 -- 172.21.15.77:0/2688612921 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff600a3dd0 con 0x55da32575170 2024-03-16T10:18:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.444+0000 7eff4effd640 1 -- 172.21.15.77:0/2688612921 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff6009f780 con 0x55da32575170 2024-03-16T10:18:13.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.444+0000 7eff4effd640 1 -- 172.21.15.77:0/2688612921 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff60084e20 con 0x55da32575170 2024-03-16T10:18:13.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.445+0000 7eff6e25b740 1 -- 172.21.15.77:0/2688612921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575170 msgr2=0x55da32570ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:13.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.445+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2688612921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575170 0x55da32570ac0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7eff60097a90 tx=0x7eff600cc710 comp rx=0 tx=0).stop 2024-03-16T10:18:13.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.445+0000 7eff6e25b740 1 -- 172.21.15.77:0/2688612921 shutdown_connections 2024-03-16T10:18:13.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.445+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2688612921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da325770f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.445+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2688612921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575b60 0x55da32586b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.454 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.445+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2688612921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575170 0x55da32570ac0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.445+0000 7eff6e25b740 1 -- 172.21.15.77:0/2688612921 >> 172.21.15.77:0/2688612921 conn(0x55da325706e0 msgr2=0x55da32576560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:13.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.445+0000 7eff6e25b740 1 -- 172.21.15.77:0/2688612921 shutdown_connections 2024-03-16T10:18:13.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.445+0000 7eff6e25b740 1 -- 172.21.15.77:0/2688612921 wait complete. 2024-03-16T10:18:13.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.446+0000 7eff6e25b740 1 -- start start 2024-03-16T10:18:13.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.446+0000 7eff6e25b740 1 Processor -- start 2024-03-16T10:18:13.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.446+0000 7eff6e25b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da325838c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.446+0000 7eff6e25b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575b60 0x55da32583e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.446+0000 7eff6e25b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da325f85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.446+0000 7eff6e25b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff641a7e50 con 0x55da32575170 2024-03-16T10:18:13.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.446+0000 7eff6e25b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7eff641a7780 con 0x55da32575b60 2024-03-16T10:18:13.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.446+0000 7eff6e25b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff641a49c0 con 0x55da3257acd0 2024-03-16T10:18:13.455 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff6e25b740 10 client.0 _ll_register_callbacks cb 0x7eff5c00bef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:13.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff68c10640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575b60 0x55da32583e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff68c10640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575b60 0x55da32583e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50446/0 (socket says 172.21.15.77:50446) 2024-03-16T10:18:13.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff68c10640 1 -- 172.21.15.77:0/2449628512 learned_addr learned my addr 172.21.15.77:0/2449628512 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:13.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff69411640 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da325838c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff69c12640 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da325f85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff68c10640 1 -- 172.21.15.77:0/2449628512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 msgr2=0x55da325f85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:13.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff68c10640 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da325f85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff68c10640 1 -- 172.21.15.77:0/2449628512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 msgr2=0x55da325838c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:13.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff68c10640 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da325838c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff68c10640 1 -- 172.21.15.77:0/2449628512 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da324a19d0 con 0x55da32575b60 2024-03-16T10:18:13.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff69411640 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da325838c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:13.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff68c10640 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575b60 0x55da32583e00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7eff540349e0 tx=0x7eff54034440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6f9fbd0502144a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:13.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff4cff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:18:13.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff4cff9640 1 -- 172.21.15.77:0/2449628512 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54039040 con 0x55da32575b60 2024-03-16T10:18:13.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff6e25b740 1 -- 172.21.15.77:0/2449628512 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff6009f780 con 0x55da32575b60 2024-03-16T10:18:13.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.447+0000 7eff4cff9640 1 -- 172.21.15.77:0/2449628512 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7eff54004240 con 0x55da32575b60 2024-03-16T10:18:13.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.448+0000 7eff4cff9640 1 -- 172.21.15.77:0/2449628512 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7eff54004440 con 0x55da32575b60 2024-03-16T10:18:13.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.448+0000 7eff6e25b740 10 client.25571 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:13.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.448+0000 7eff6e25b740 1 -- 172.21.15.77:0/2449628512 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55da325f9a60 con 0x55da32575b60 2024-03-16T10:18:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.448+0000 7eff4cff9640 1 -- 172.21.15.77:0/2449628512 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff6009f780 con 0x55da32575b60 2024-03-16T10:18:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.448+0000 7eff4cff9640 1 -- 172.21.15.77:0/2449628512 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7eff54032b10 con 0x55da32575b60 2024-03-16T10:18:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.448+0000 7eff6e25b740 10 client.25571 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.448+0000 7eff6e25b740 10 client.25571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.448+0000 7eff6e25b740 20 client.25571 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.448+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55da325fa3f0 0x55da325fc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.448+0000 7eff6e25b740 4 client.25571 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:18:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.448+0000 7eff6e25b740 1 -- 172.21.15.77:0/2449628512 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55da324e1d90 con 0x55da325fa3f0 2024-03-16T10:18:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.448+0000 7eff69411640 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55da325fa3f0 0x55da325fc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff4cff9640 10 client.25571 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:13.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff69411640 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55da325fa3f0 0x55da325fc810 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd4be8834feff3ff server_cookie=bded4583d12b742a in_seq=0 out_seq=0 2024-03-16T10:18:13.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff4cff9640 1 -- 172.21.15.77:0/2449628512 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55da324e1d90 con 0x55da325fa3f0 2024-03-16T10:18:13.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff4cff9640 10 client.25571 handle_command_reply: tid=0 2024-03-16T10:18:13.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff6e25b740 1 client.25571 shutdown 2024-03-16T10:18:13.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff6e25b740 1 -- 172.21.15.77:0/2449628512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575b60 msgr2=0x55da32583e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:13.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575b60 0x55da32583e00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7eff540349e0 tx=0x7eff54034440 comp rx=0 tx=0).stop 2024-03-16T10:18:13.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff6e25b740 1 -- 172.21.15.77:0/2449628512 shutdown_connections 2024-03-16T10:18:13.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da3257acd0 0x55da325f85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55da32575b60 0x55da32583e00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55da325fa3f0 0x55da325fc810 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff6e25b740 1 --2- 172.21.15.77:0/2449628512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55da32575170 0x55da325838c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff4cff9640 0 client.25571 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:13.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.449+0000 7eff6e25b740 1 -- 172.21.15.77:0/2449628512 >> 172.21.15.77:0/2449628512 conn(0x55da325706e0 msgr2=0x55da32570ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:13.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.450+0000 7eff6e25b740 1 -- 172.21.15.77:0/2449628512 shutdown_connections 2024-03-16T10:18:13.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.450+0000 7eff6e25b740 1 -- 172.21.15.77:0/2449628512 wait complete. 2024-03-16T10:18:13.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.450+0000 7eff6e25b740 20 client.25571 trim_cache size 0 max 16384 2024-03-16T10:18:13.461 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:18:13.461 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T10:18:13.461 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T10:18:13.461 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:18:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.450+0000 7eff6b675640 1 -- 172.21.15.77:0/1117231636 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff500a2910 msgr2=0x7eff500a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.450+0000 7eff6b675640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff500a2910 0x7eff500a4dd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00bfd0 tx=0x7eff5c010920 comp rx=0 tx=0).stop 2024-03-16T10:18:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.450+0000 7eff6b675640 1 -- 172.21.15.77:0/1117231636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff64100c20 msgr2=0x7eff641958d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.450+0000 7eff6b675640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff64100c20 0x7eff641958d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7eff60005fd0 tx=0x7eff6000ec20 comp rx=0 tx=0).stop 2024-03-16T10:18:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.451+0000 7eff6b675640 1 -- 172.21.15.77:0/1117231636 shutdown_connections 2024-03-16T10:18:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.451+0000 7eff6b675640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff641022f0 0x7eff6419ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.451+0000 7eff6b675640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff64101930 0x7eff64195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.451+0000 7eff6b675640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff500a2910 0x7eff500a4dd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.451+0000 7eff6b675640 1 --2- 172.21.15.77:0/1117231636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff64100c20 0x7eff641958d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.451+0000 7eff6b675640 1 -- 172.21.15.77:0/1117231636 >> 172.21.15.77:0/1117231636 conn(0x7eff640fb3f0 msgr2=0x7eff64104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:13.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.451+0000 7eff6b675640 1 -- 172.21.15.77:0/1117231636 shutdown_connections 2024-03-16T10:18:13.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.451+0000 7eff6b675640 1 -- 172.21.15.77:0/1117231636 wait complete. 2024-03-16T10:18:13.467 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T10:18:13.467 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:18:13.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.598+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2246524480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c1033d0 msgr2=0x7fbf9c109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:13.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.598+0000 7fbfa0c63640 1 --2- 172.21.15.77:0/2246524480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c1033d0 0x7fbf9c109c60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009ca0 tx=0x7fbf9402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.598+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2246524480 shutdown_connections 2024-03-16T10:18:13.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.598+0000 7fbfa0c63640 1 --2- 172.21.15.77:0/2246524480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c1033d0 0x7fbf9c109c60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.599+0000 7fbfa0c63640 1 --2- 172.21.15.77:0/2246524480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf9c102a10 0x7fbf9c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.599+0000 7fbfa0c63640 1 --2- 172.21.15.77:0/2246524480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf9c101820 0x7fbf9c101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.599+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2246524480 >> 172.21.15.77:0/2246524480 conn(0x7fbf9c0fcd90 msgr2=0x7fbf9c0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:13.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.599+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2246524480 shutdown_connections 2024-03-16T10:18:13.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.599+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2246524480 wait complete. 2024-03-16T10:18:13.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.599+0000 7fbfa0c63640 1 Processor -- start 2024-03-16T10:18:13.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbfa0c63640 1 -- start start 2024-03-16T10:18:13.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbfa0c63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf9c101820 0x7fbf9c075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbfa0c63640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c102a10 0x7fbf9c073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbfa0c63640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf9c1033d0 0x7fbf9c074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbfa0c63640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c10d480 con 0x7fbf9c101820 2024-03-16T10:18:13.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbfa0c63640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c10d300 con 0x7fbf9c102a10 2024-03-16T10:18:13.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbfa0c63640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c10d600 con 0x7fbf9c1033d0 2024-03-16T10:18:13.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbf93fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c102a10 0x7fbf9c073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbf93fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c102a10 0x7fbf9c073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50476/0 (socket says 172.21.15.77:50476) 2024-03-16T10:18:13.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbf93fff640 1 -- 172.21.15.77:0/2138458961 learned_addr learned my addr 172.21.15.77:0/2138458961 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:13.604 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbf9a59c640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf9c101820 0x7fbf9c075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbf9ad9d640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf9c1033d0 0x7fbf9c074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbf9ad9d640 1 -- 172.21.15.77:0/2138458961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c102a10 msgr2=0x7fbf9c073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:13.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbf9ad9d640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c102a10 0x7fbf9c073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbf9ad9d640 1 -- 172.21.15.77:0/2138458961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf9c101820 msgr2=0x7fbf9c075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:13.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbf9ad9d640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf9c101820 0x7fbf9c075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:13.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbf9ad9d640 1 -- 172.21.15.77:0/2138458961 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9c074650 con 0x7fbf9c1033d0 2024-03-16T10:18:13.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.600+0000 7fbf9a59c640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf9c101820 0x7fbf9c075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:13.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.601+0000 7fbf93fff640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c102a10 0x7fbf9c073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:13.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.601+0000 7fbf9ad9d640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf9c1033d0 0x7fbf9c074110 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009bf0 tx=0x7fbf94032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:13.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.601+0000 7fbf937fe640 1 -- 172.21.15.77:0/2138458961 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94041020 con 0x7fbf9c1033d0 2024-03-16T10:18:13.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.601+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2138458961 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf9c0748e0 con 0x7fbf9c1033d0 2024-03-16T10:18:13.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.601+0000 7fbf937fe640 1 -- 172.21.15.77:0/2138458961 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf94034cd0 con 0x7fbf9c1033d0 2024-03-16T10:18:13.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.601+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2138458961 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf9c1a6560 con 0x7fbf9c1033d0 2024-03-16T10:18:13.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.601+0000 7fbf937fe640 1 -- 172.21.15.77:0/2138458961 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94039040 con 0x7fbf9c1033d0 2024-03-16T10:18:13.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.603+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2138458961 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf9c0685d0 con 0x7fbf9c1033d0 2024-03-16T10:18:13.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.603+0000 7fbf937fe640 1 -- 172.21.15.77:0/2138458961 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fbf94044020 con 0x7fbf9c1033d0 2024-03-16T10:18:13.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.604+0000 7fbf937fe640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf880a2890 0x7fbf880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:13.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.604+0000 7fbf937fe640 1 -- 172.21.15.77:0/2138458961 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fbf940f0830 con 0x7fbf9c1033d0 2024-03-16T10:18:13.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.604+0000 7fbf9a59c640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf880a2890 0x7fbf880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:13.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.605+0000 7fbf9a59c640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf880a2890 0x7fbf880a4d50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c102870 tx=0x7fbf8c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:13.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:13.611+0000 7fbf937fe640 1 -- 172.21.15.77:0/2138458961 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbf9404b020 con 0x7fbf9c1033d0 2024-03-16T10:18:14.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.049+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2138458961 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf9c02d1b0 con 0x7fbf9c1033d0 2024-03-16T10:18:14.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.050+0000 7fbf937fe640 1 -- 172.21.15.77:0/2138458961 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11938 (secure 0 0 0) 0x7fbf940bd970 con 0x7fbf9c1033d0 2024-03-16T10:18:14.051 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:18:14.051 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:17:02.900684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24475,24454,24482,14658,24455]},"id":1}]} 2024-03-16T10:18:14.051 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 27 2024-03-16T10:18:14.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.052+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2138458961 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf880a2890 msgr2=0x7fbf880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.052+0000 7fbfa0c63640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf880a2890 0x7fbf880a4d50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c102870 tx=0x7fbf8c00b390 comp rx=0 tx=0).stop 2024-03-16T10:18:14.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.052+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2138458961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf9c1033d0 msgr2=0x7fbf9c074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.052+0000 7fbfa0c63640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf9c1033d0 0x7fbf9c074110 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009bf0 tx=0x7fbf94032ea0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.052+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2138458961 shutdown_connections 2024-03-16T10:18:14.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.052+0000 7fbfa0c63640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf880a2890 0x7fbf880a4d50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.052+0000 7fbfa0c63640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf9c1033d0 0x7fbf9c074110 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.052+0000 7fbfa0c63640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c102a10 0x7fbf9c073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.052+0000 7fbfa0c63640 1 --2- 172.21.15.77:0/2138458961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf9c101820 0x7fbf9c075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.052+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2138458961 >> 172.21.15.77:0/2138458961 conn(0x7fbf9c0fcd90 msgr2=0x7fbf9c105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.053+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2138458961 shutdown_connections 2024-03-16T10:18:14.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.053+0000 7fbfa0c63640 1 -- 172.21.15.77:0/2138458961 wait complete. 2024-03-16T10:18:14.067 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-16T10:18:14.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:13 smithi192 ceph-mon[36354]: pgmap v384: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-16T10:18:14.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.174+0000 7f038ccdc640 1 -- 172.21.15.77:0/956362236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0388102290 msgr2=0x7f0388108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.174+0000 7f038ccdc640 1 --2- 172.21.15.77:0/956362236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0388102290 0x7f0388108b20 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0380009ca0 tx=0x7f038002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.174+0000 7f038ccdc640 1 -- 172.21.15.77:0/956362236 shutdown_connections 2024-03-16T10:18:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.174+0000 7f038ccdc640 1 --2- 172.21.15.77:0/956362236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0388102290 0x7f0388108b20 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.174+0000 7f038ccdc640 1 --2- 172.21.15.77:0/956362236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03881018d0 0x7f0388101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.174+0000 7f038ccdc640 1 --2- 172.21.15.77:0/956362236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03881006e0 0x7f0388100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.174+0000 7f038ccdc640 1 -- 172.21.15.77:0/956362236 >> 172.21.15.77:0/956362236 conn(0x7f03880fbc70 msgr2=0x7f03880fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.174+0000 7f038ccdc640 1 -- 172.21.15.77:0/956362236 shutdown_connections 2024-03-16T10:18:14.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.174+0000 7f038ccdc640 1 -- 172.21.15.77:0/956362236 wait complete. 2024-03-16T10:18:14.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.175+0000 7f038ccdc640 1 Processor -- start 2024-03-16T10:18:14.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.175+0000 7f038ccdc640 1 -- start start 2024-03-16T10:18:14.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.175+0000 7f038ccdc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03881006e0 0x7f0388197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.175+0000 7f038ccdc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03881018d0 0x7f0388198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.175+0000 7f038ccdc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0388102290 0x7f038819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.175+0000 7f038ccdc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f038810d440 con 0x7f0388102290 2024-03-16T10:18:14.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.175+0000 7f038ccdc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f038810d2c0 con 0x7f03881018d0 2024-03-16T10:18:14.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.175+0000 7f038ccdc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f038810d5c0 con 0x7f03881006e0 2024-03-16T10:18:14.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.175+0000 7f038659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03881006e0 0x7f0388197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.175+0000 7f0385d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03881018d0 0x7f0388198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.175+0000 7f038659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03881006e0 0x7f0388197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33120/0 (socket says 172.21.15.77:33120) 2024-03-16T10:18:14.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.175+0000 7f038659c640 1 -- 172.21.15.77:0/1433433546 learned_addr learned my addr 172.21.15.77:0/1433433546 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.176+0000 7f038659c640 1 -- 172.21.15.77:0/1433433546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03881018d0 msgr2=0x7f0388198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.176+0000 7f038659c640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03881018d0 0x7f0388198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.176+0000 7f038659c640 1 -- 172.21.15.77:0/1433433546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0388102290 msgr2=0x7f038819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.176+0000 7f0386d9d640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0388102290 0x7f038819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.176+0000 7f038659c640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0388102290 0x7f038819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.176+0000 7f038659c640 1 -- 172.21.15.77:0/1433433546 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038819f700 con 0x7f03881006e0 2024-03-16T10:18:14.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.176+0000 7f0385d9b640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03881018d0 0x7f0388198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:14.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.176+0000 7f0386d9d640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0388102290 0x7f038819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:18:14.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.176+0000 7f038659c640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03881006e0 0x7f0388197ae0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f037800d950 tx=0x7f037800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.176+0000 7f03777fe640 1 -- 172.21.15.77:0/1433433546 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0378017020 con 0x7f03881006e0 2024-03-16T10:18:14.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.177+0000 7f038ccdc640 1 -- 172.21.15.77:0/1433433546 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0388109540 con 0x7f03881006e0 2024-03-16T10:18:14.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.177+0000 7f038ccdc640 1 -- 172.21.15.77:0/1433433546 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0388109ae0 con 0x7f03881006e0 2024-03-16T10:18:14.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.178+0000 7f03777fe640 1 -- 172.21.15.77:0/1433433546 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0378009c30 con 0x7f03881006e0 2024-03-16T10:18:14.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.178+0000 7f03777fe640 1 -- 172.21.15.77:0/1433433546 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037800f740 con 0x7f03881006e0 2024-03-16T10:18:14.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.180+0000 7f03777fe640 1 -- 172.21.15.77:0/1433433546 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f037801a020 con 0x7f03881006e0 2024-03-16T10:18:14.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f038f99e740 1 Processor -- start 2024-03-16T10:18:14.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f038f99e740 1 -- start start 2024-03-16T10:18:14.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f038f99e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b4166c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f038f99e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4167240 0x55b0b416bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f038f99e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 0x55b0b416e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f038f99e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b0b40cb490 con 0x55b0b416c3b0 2024-03-16T10:18:14.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f038f99e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55b0b4073c90 con 0x55b0b4166850 2024-03-16T10:18:14.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f038f99e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b0b40c7530 con 0x55b0b4167240 2024-03-16T10:18:14.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f0385d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 0x55b0b416e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f038659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4167240 0x55b0b416bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f0385d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 0x55b0b416e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54288/0 (socket says 172.21.15.77:54288) 2024-03-16T10:18:14.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f038659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4167240 0x55b0b416bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33126/0 (socket says 172.21.15.77:33126) 2024-03-16T10:18:14.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.181+0000 7f0385d9b640 1 -- 172.21.15.77:0/4046170989 learned_addr learned my addr 172.21.15.77:0/4046170989 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.182+0000 7f0386d9d640 1 --2- 172.21.15.77:0/4046170989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b4166c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.182+0000 7f038659c640 1 -- 172.21.15.77:0/4046170989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 msgr2=0x55b0b4166c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.183+0000 7f038659c640 1 --2- 172.21.15.77:0/4046170989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b4166c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.183+0000 7f038659c640 1 -- 172.21.15.77:0/4046170989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 msgr2=0x55b0b416e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.183+0000 7f038659c640 1 --2- 172.21.15.77:0/4046170989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 0x55b0b416e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.183+0000 7f038659c640 1 -- 172.21.15.77:0/4046170989 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0b412e500 con 0x55b0b4167240 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.183+0000 7f0385d9b640 1 --2- 172.21.15.77:0/4046170989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 0x55b0b416e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.183+0000 7f038659c640 1 --2- 172.21.15.77:0/4046170989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4167240 0x55b0b416bd40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f03780cc470 tx=0x7f03780d7580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c565857c82aaa675 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.183+0000 7f0374ff9640 1 -- 172.21.15.77:0/4046170989 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03780dc630 con 0x55b0b4167240 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.183+0000 7f0374ff9640 1 -- 172.21.15.77:0/4046170989 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03780d2e10 con 0x55b0b4167240 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.183+0000 7f0374ff9640 1 -- 172.21.15.77:0/4046170989 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03780dba50 con 0x55b0b4167240 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.183+0000 7f03777fe640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f03700a2910 0x7f03700a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.183+0000 7f0386d9d640 1 --2- 172.21.15.77:0/4046170989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b4166c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.184+0000 7f03777fe640 1 -- 172.21.15.77:0/1433433546 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f03780ceac0 con 0x7f03881006e0 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.184+0000 7f0386d9d640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f03700a2910 0x7f03700a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.184+0000 7f038f99e740 1 -- 172.21.15.77:0/4046170989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4167240 msgr2=0x55b0b416bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.184+0000 7f038f99e740 1 --2- 172.21.15.77:0/4046170989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4167240 0x55b0b416bd40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f03780cc470 tx=0x7f03780d7580 comp rx=0 tx=0).stop 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.184+0000 7f038f99e740 1 -- 172.21.15.77:0/4046170989 shutdown_connections 2024-03-16T10:18:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.184+0000 7f038f99e740 1 --2- 172.21.15.77:0/4046170989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 0x55b0b416e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.184+0000 7f038f99e740 1 --2- 172.21.15.77:0/4046170989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4167240 0x55b0b416bd40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.184+0000 7f038f99e740 1 --2- 172.21.15.77:0/4046170989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b4166c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.184+0000 7f038f99e740 1 -- 172.21.15.77:0/4046170989 >> 172.21.15.77:0/4046170989 conn(0x55b0b4161dc0 msgr2=0x55b0b41621c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.184+0000 7f038f99e740 1 -- 172.21.15.77:0/4046170989 shutdown_connections 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.185+0000 7f0386d9d640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f03700a2910 0x7f03700a4dd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0380037040 tx=0x7f0380004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.185+0000 7f038f99e740 1 -- 172.21.15.77:0/4046170989 wait complete. 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.185+0000 7f038f99e740 1 -- start start 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.185+0000 7f038f99e740 1 Processor -- start 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.185+0000 7f038f99e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b4170c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.185+0000 7f038f99e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 0x55b0b4171150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.185+0000 7f038f99e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b416c3b0 0x55b0b4178190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.185+0000 7f038f99e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b0b416f360 con 0x55b0b4167240 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.185+0000 7f038f99e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55b0b4087230 con 0x55b0b4166850 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.185+0000 7f038f99e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b0b416f4e0 con 0x55b0b416c3b0 2024-03-16T10:18:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.185+0000 7f038f99e740 10 client.0 _ll_register_callbacks cb 0x55b0b40c3010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.185+0000 7f0385d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 0x55b0b4171150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f0385d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 0x55b0b4171150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54298/0 (socket says 172.21.15.77:54298) 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f0385d9b640 1 -- 172.21.15.77:0/474737208 learned_addr learned my addr 172.21.15.77:0/474737208 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f0386d9d640 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b416c3b0 0x55b0b4178190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f0386d9d640 1 -- 172.21.15.77:0/474737208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 msgr2=0x55b0b4170c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f038659c640 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b4170c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f0386d9d640 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b4170c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f0386d9d640 1 -- 172.21.15.77:0/474737208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 msgr2=0x55b0b4171150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f0386d9d640 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 0x55b0b4171150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f0386d9d640 1 -- 172.21.15.77:0/474737208 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0b4096100 con 0x55b0b416c3b0 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f0385d9b640 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 0x55b0b4171150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f0386d9d640 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b416c3b0 0x55b0b4178190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0380004af0 tx=0x7f03800095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d85f6acf5f407cea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f036dffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f036dffb640 1 -- 172.21.15.77:0/474737208 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0380046050 con 0x55b0b416c3b0 2024-03-16T10:18:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.186+0000 7f036dffb640 1 -- 172.21.15.77:0/474737208 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f0380004450 con 0x55b0b416c3b0 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.187+0000 7f038f99e740 1 -- 172.21.15.77:0/474737208 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03780d2e10 con 0x55b0b416c3b0 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.187+0000 7f036dffb640 1 -- 172.21.15.77:0/474737208 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0380039040 con 0x55b0b416c3b0 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.187+0000 7f038f99e740 10 client.25570 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.187+0000 7f038f99e740 1 -- 172.21.15.77:0/474737208 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b0b3ddf3f0 con 0x55b0b416c3b0 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.187+0000 7f036dffb640 1 -- 172.21.15.77:0/474737208 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03780d2e10 con 0x55b0b416c3b0 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.187+0000 7f036dffb640 1 -- 172.21.15.77:0/474737208 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f038004b440 con 0x55b0b416c3b0 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.187+0000 7f038f99e740 10 client.25570 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.187+0000 7f038f99e740 10 client.25570 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.187+0000 7f038f99e740 20 client.25570 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.188+0000 7f038f99e740 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b0b417caf0 0x55b0b417ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.188+0000 7f038f99e740 4 client.25570 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.188+0000 7f038f99e740 1 -- 172.21.15.77:0/474737208 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0b40d89d0 con 0x55b0b417caf0 2024-03-16T10:18:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.188+0000 7f038659c640 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b0b417caf0 0x55b0b417ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.188+0000 7f038659c640 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b0b417caf0 0x55b0b417ef10 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9d7b1297001911ba server_cookie=5dbe304debb3474d in_seq=0 out_seq=0 2024-03-16T10:18:14.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.188+0000 7f036dffb640 10 client.25570 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:18:14.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.189+0000 7f036dffb640 1 -- 172.21.15.77:0/474737208 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b0b40d89d0 con 0x55b0b417caf0 2024-03-16T10:18:14.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.189+0000 7f036dffb640 10 client.25570 handle_command_reply: tid=0 2024-03-16T10:18:14.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.189+0000 7f038f99e740 1 client.25570 shutdown 2024-03-16T10:18:14.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.189+0000 7f038f99e740 1 -- 172.21.15.77:0/474737208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b416c3b0 msgr2=0x55b0b4178190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.189+0000 7f038f99e740 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b416c3b0 0x55b0b4178190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0380004af0 tx=0x7f03800095c0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.189+0000 7f038f99e740 1 -- 172.21.15.77:0/474737208 shutdown_connections 2024-03-16T10:18:14.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.189+0000 7f038f99e740 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b0b417caf0 0x55b0b417ef10 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.190+0000 7f036dffb640 0 client.25570 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:18:14.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.190+0000 7f038f99e740 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b416c3b0 0x55b0b4178190 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.190+0000 7f038f99e740 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 0x55b0b4171150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.190+0000 7f038f99e740 1 --2- 172.21.15.77:0/474737208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b4170c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.190+0000 7f038f99e740 1 -- 172.21.15.77:0/474737208 >> 172.21.15.77:0/474737208 conn(0x55b0b4161dc0 msgr2=0x55b0b4160fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.190+0000 7f038f99e740 1 -- 172.21.15.77:0/474737208 shutdown_connections 2024-03-16T10:18:14.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.190+0000 7f038f99e740 1 -- 172.21.15.77:0/474737208 wait complete. 2024-03-16T10:18:14.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.190+0000 7f038f99e740 20 client.25570 trim_cache size 0 max 16384 2024-03-16T10:18:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f038f99e740 1 Processor -- start 2024-03-16T10:18:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f038f99e740 1 -- start start 2024-03-16T10:18:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f038f99e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b416e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f038f99e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 0x55b0b4178150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f038f99e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b416c3b0 0x55b0b416fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f038f99e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f036800e330 con 0x55b0b4167240 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f038659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b416e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f0385d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 0x55b0b4178150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f038659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b416e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50518/0 (socket says 172.21.15.77:50518) 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f038659c640 1 -- 172.21.15.77:0/739131754 learned_addr learned my addr 172.21.15.77:0/739131754 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f038f99e740 1 -- 172.21.15.77:0/739131754 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f036800fe70 con 0x55b0b4166850 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f038f99e740 1 -- 172.21.15.77:0/739131754 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f036800ea00 con 0x55b0b416c3b0 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.206+0000 7f0386d9d640 1 --2- 172.21.15.77:0/739131754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b416c3b0 0x55b0b416fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.207+0000 7f038659c640 1 -- 172.21.15.77:0/739131754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b416c3b0 msgr2=0x55b0b416fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.207+0000 7f038659c640 1 --2- 172.21.15.77:0/739131754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b416c3b0 0x55b0b416fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.207+0000 7f038659c640 1 -- 172.21.15.77:0/739131754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 msgr2=0x55b0b4178150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.207+0000 7f038659c640 1 --2- 172.21.15.77:0/739131754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 0x55b0b4178150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.207+0000 7f038659c640 1 -- 172.21.15.77:0/739131754 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0b3ea97d0 con 0x55b0b4166850 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.207+0000 7f0385d9b640 1 --2- 172.21.15.77:0/739131754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 0x55b0b4178150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.207+0000 7f038659c640 1 --2- 172.21.15.77:0/739131754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b416e7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0378009610 tx=0x7f0378010e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e1874b216cb2ec0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.207+0000 7f036ffff640 1 -- 172.21.15.77:0/739131754 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0378098610 con 0x55b0b4166850 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.207+0000 7f036ffff640 1 -- 172.21.15.77:0/739131754 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f037809ca70 con 0x55b0b4166850 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.207+0000 7f036ffff640 1 -- 172.21.15.77:0/739131754 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037809bcd0 con 0x55b0b4166850 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.208+0000 7f038f99e740 1 -- 172.21.15.77:0/739131754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 msgr2=0x55b0b416e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.208+0000 7f038f99e740 1 --2- 172.21.15.77:0/739131754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b416e7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0378009610 tx=0x7f0378010e10 comp rx=0 tx=0).stop 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.208+0000 7f038f99e740 1 -- 172.21.15.77:0/739131754 shutdown_connections 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.208+0000 7f038f99e740 1 --2- 172.21.15.77:0/739131754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b416c3b0 0x55b0b416fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.208+0000 7f038f99e740 1 --2- 172.21.15.77:0/739131754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b4167240 0x55b0b4178150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.208+0000 7f038f99e740 1 --2- 172.21.15.77:0/739131754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4166850 0x55b0b416e7a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.208+0000 7f038f99e740 1 -- 172.21.15.77:0/739131754 >> 172.21.15.77:0/739131754 conn(0x55b0b4161dc0 msgr2=0x55b0b4168560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.208+0000 7f038f99e740 1 -- 172.21.15.77:0/739131754 shutdown_connections 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.208+0000 7f038f99e740 1 -- 172.21.15.77:0/739131754 wait complete. 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.209+0000 7f038f99e740 1 -- start start 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.209+0000 7f038f99e740 1 Processor -- start 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.209+0000 7f038f99e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4166850 0x55b0b4174d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.209+0000 7f038f99e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4167240 0x55b0b4175280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.209+0000 7f038f99e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 0x55b0b41e0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.209+0000 7f038f99e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f036800f7a0 con 0x55b0b416c3b0 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.209+0000 7f038f99e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f036800f0d0 con 0x55b0b4167240 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.209+0000 7f038f99e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f036800c310 con 0x55b0b4166850 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.209+0000 7f038f99e740 10 client.0 _ll_register_callbacks cb 0x7f0380037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.209+0000 7f038659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4166850 0x55b0b4174d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.209+0000 7f038659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4166850 0x55b0b4174d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33142/0 (socket says 172.21.15.77:33142) 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.209+0000 7f038659c640 1 -- 172.21.15.77:0/3847582476 learned_addr learned my addr 172.21.15.77:0/3847582476 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f0386d9d640 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 0x55b0b41e0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f038659c640 1 -- 172.21.15.77:0/3847582476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4167240 msgr2=0x55b0b4175280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f0385d9b640 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4167240 0x55b0b4175280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f038659c640 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4167240 0x55b0b4175280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f038659c640 1 -- 172.21.15.77:0/3847582476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 msgr2=0x55b0b41e0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f038659c640 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 0x55b0b41e0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f038659c640 1 -- 172.21.15.77:0/3847582476 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0b4178350 con 0x55b0b4166850 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f0386d9d640 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 0x55b0b41e0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f0385d9b640 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4167240 0x55b0b4175280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f038659c640 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4166850 0x55b0b4174d40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f037809c070 tx=0x7f03780a04c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18305ed73e840b42 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f036dffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f036dffb640 1 -- 172.21.15.77:0/3847582476 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037809beb0 con 0x55b0b4166850 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.210+0000 7f036dffb640 1 -- 172.21.15.77:0/3847582476 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f03780b51a0 con 0x55b0b4166850 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.211+0000 7f038f99e740 1 -- 172.21.15.77:0/3847582476 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f037809ca70 con 0x55b0b4166850 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.211+0000 7f036dffb640 1 -- 172.21.15.77:0/3847582476 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f037809c140 con 0x55b0b4166850 2024-03-16T10:18:14.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.211+0000 7f038f99e740 10 client.25576 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.211+0000 7f038f99e740 1 -- 172.21.15.77:0/3847582476 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b0b41e2310 con 0x55b0b4166850 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.211+0000 7f036dffb640 1 -- 172.21.15.77:0/3847582476 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f037809ca70 con 0x55b0b4166850 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.211+0000 7f036dffb640 1 -- 172.21.15.77:0/3847582476 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f03780989e0 con 0x55b0b4166850 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.211+0000 7f038f99e740 10 client.25576 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.211+0000 7f038f99e740 10 client.25576 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.211+0000 7f038f99e740 20 client.25576 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.212+0000 7f038f99e740 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b0b417caf0 0x55b0b41e4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.212+0000 7f038f99e740 4 client.25576 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.212+0000 7f038f99e740 1 -- 172.21.15.77:0/3847582476 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b0b40d2d20 con 0x55b0b417caf0 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.212+0000 7f0385d9b640 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b0b417caf0 0x55b0b41e4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.213+0000 7f036dffb640 10 client.25576 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.213+0000 7f0385d9b640 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b0b417caf0 0x55b0b41e4810 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e45f4255427e67f5 server_cookie=eb1dc8bc345738c in_seq=0 out_seq=0 2024-03-16T10:18:14.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.214+0000 7f036dffb640 1 -- 172.21.15.77:0/3847582476 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b0b40d2d20 con 0x55b0b417caf0 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.214+0000 7f036dffb640 10 client.25576 handle_command_reply: tid=0 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.214+0000 7f038f99e740 1 client.25576 shutdown 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.214+0000 7f038f99e740 1 -- 172.21.15.77:0/3847582476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4166850 msgr2=0x55b0b4174d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.214+0000 7f038f99e740 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4166850 0x55b0b4174d40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f037809c070 tx=0x7f03780a04c0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.215+0000 7f038f99e740 1 -- 172.21.15.77:0/3847582476 shutdown_connections 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.215+0000 7f038f99e740 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b0b416c3b0 0x55b0b41e0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.215+0000 7f038f99e740 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b0b4167240 0x55b0b4175280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.215+0000 7f038f99e740 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b0b417caf0 0x55b0b41e4810 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.215+0000 7f038f99e740 1 --2- 172.21.15.77:0/3847582476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0b4166850 0x55b0b4174d40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.215+0000 7f038f99e740 1 -- 172.21.15.77:0/3847582476 >> 172.21.15.77:0/3847582476 conn(0x55b0b4161dc0 msgr2=0x55b0b4170c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.215+0000 7f036dffb640 0 client.25576 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.216+0000 7f038f99e740 1 -- 172.21.15.77:0/3847582476 shutdown_connections 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.216+0000 7f038f99e740 1 -- 172.21.15.77:0/3847582476 wait complete. 2024-03-16T10:18:14.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.216+0000 7f038f99e740 20 client.25576 trim_cache size 0 max 16384 2024-03-16T10:18:14.224 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:18:14.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.216+0000 7f038ccdc640 1 -- 172.21.15.77:0/1433433546 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f03700a2910 msgr2=0x7f03700a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.216+0000 7f038ccdc640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f03700a2910 0x7f03700a4dd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0380037040 tx=0x7f0380004a40 comp rx=0 tx=0).stop 2024-03-16T10:18:14.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.216+0000 7f038ccdc640 1 -- 172.21.15.77:0/1433433546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03881006e0 msgr2=0x7f0388197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.216+0000 7f038ccdc640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03881006e0 0x7f0388197ae0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f037800d950 tx=0x7f037800de10 comp rx=0 tx=0).stop 2024-03-16T10:18:14.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.216+0000 7f038ccdc640 1 -- 172.21.15.77:0/1433433546 shutdown_connections 2024-03-16T10:18:14.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.216+0000 7f038ccdc640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0388102290 0x7f038819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.216+0000 7f038ccdc640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03881018d0 0x7f0388198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.216+0000 7f038ccdc640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f03700a2910 0x7f03700a4dd0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.217+0000 7f038ccdc640 1 --2- 172.21.15.77:0/1433433546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03881006e0 0x7f0388197ae0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.217+0000 7f038ccdc640 1 -- 172.21.15.77:0/1433433546 >> 172.21.15.77:0/1433433546 conn(0x7f03880fbc70 msgr2=0x7f0388104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.217+0000 7f038ccdc640 1 -- 172.21.15.77:0/1433433546 shutdown_connections 2024-03-16T10:18:14.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.217+0000 7f038ccdc640 1 -- 172.21.15.77:0/1433433546 wait complete. 2024-03-16T10:18:14.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:13 smithi077 ceph-mon[32400]: pgmap v384: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-16T10:18:14.232 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-16T10:18:14.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:13 smithi193 ceph-mon[36311]: pgmap v384: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-16T10:18:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.383+0000 7f1c74f77640 1 -- 172.21.15.77:0/1922112534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c701022b0 msgr2=0x7f1c70108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.383+0000 7f1c74f77640 1 --2- 172.21.15.77:0/1922112534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c701022b0 0x7f1c70108b40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009be0 tx=0x7f1c6802bef0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.384+0000 7f1c74f77640 1 -- 172.21.15.77:0/1922112534 shutdown_connections 2024-03-16T10:18:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.384+0000 7f1c74f77640 1 --2- 172.21.15.77:0/1922112534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c701022b0 0x7f1c70108b40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.384+0000 7f1c74f77640 1 --2- 172.21.15.77:0/1922112534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c701018f0 0x7f1c70101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.384+0000 7f1c74f77640 1 --2- 172.21.15.77:0/1922112534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70100700 0x7f1c70100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.384+0000 7f1c74f77640 1 -- 172.21.15.77:0/1922112534 >> 172.21.15.77:0/1922112534 conn(0x7f1c700fbc90 msgr2=0x7f1c700fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.384+0000 7f1c74f77640 1 -- 172.21.15.77:0/1922112534 shutdown_connections 2024-03-16T10:18:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.384+0000 7f1c74f77640 1 -- 172.21.15.77:0/1922112534 wait complete. 2024-03-16T10:18:14.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.385+0000 7f1c74f77640 1 Processor -- start 2024-03-16T10:18:14.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.385+0000 7f1c74f77640 1 -- start start 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c74f77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70100700 0x7f1c70195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c74f77640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c701018f0 0x7f1c70195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c74f77640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c701022b0 0x7f1c7019ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c74f77640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7010b2d0 con 0x7f1c70100700 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c74f77640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7010b150 con 0x7f1c701018f0 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c74f77640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7010b450 con 0x7f1c701022b0 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70100700 0x7f1c70195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70100700 0x7f1c70195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54336/0 (socket says 172.21.15.77:54336) 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c6e59c640 1 -- 172.21.15.77:0/2226878521 learned_addr learned my addr 172.21.15.77:0/2226878521 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c6ed9d640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c701022b0 0x7f1c7019ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c701018f0 0x7f1c70195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c6ed9d640 1 -- 172.21.15.77:0/2226878521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c701018f0 msgr2=0x7f1c70195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c6ed9d640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c701018f0 0x7f1c70195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c6ed9d640 1 -- 172.21.15.77:0/2226878521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70100700 msgr2=0x7f1c70195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c6ed9d640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70100700 0x7f1c70195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c6ed9d640 1 -- 172.21.15.77:0/2226878521 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c7019d540 con 0x7f1c701022b0 2024-03-16T10:18:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c6e59c640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70100700 0x7f1c70195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.386+0000 7f1c6ed9d640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c701022b0 0x7f1c7019ced0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1c6802c430 tx=0x7f1c680337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.387+0000 7f1c5f7fe640 1 -- 172.21.15.77:0/2226878521 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68043020 con 0x7f1c701022b0 2024-03-16T10:18:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.387+0000 7f1c5f7fe640 1 -- 172.21.15.77:0/2226878521 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c6802ca60 con 0x7f1c701022b0 2024-03-16T10:18:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.387+0000 7f1c74f77640 1 -- 172.21.15.77:0/2226878521 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c7019d7d0 con 0x7f1c701022b0 2024-03-16T10:18:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.387+0000 7f1c5f7fe640 1 -- 172.21.15.77:0/2226878521 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68034690 con 0x7f1c701022b0 2024-03-16T10:18:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.387+0000 7f1c74f77640 1 -- 172.21.15.77:0/2226878521 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c7019dd70 con 0x7f1c701022b0 2024-03-16T10:18:14.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.390+0000 7f1c5f7fe640 1 -- 172.21.15.77:0/2226878521 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f1c68046020 con 0x7f1c701022b0 2024-03-16T10:18:14.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.390+0000 7f1c777fd740 1 Processor -- start 2024-03-16T10:18:14.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c777fd740 1 -- start start 2024-03-16T10:18:14.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c777fd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a469c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c777fd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46a240 0x55fd0a46ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c777fd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46f3b0 0x55fd0a471840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.392 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c777fd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fd0a3ce490 con 0x55fd0a46a240 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c777fd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55fd0a376c90 con 0x55fd0a46f3b0 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c777fd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fd0a3ca530 con 0x55fd0a469850 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c6dd9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a469c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46a240 0x55fd0a46ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c6dd9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a469c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33158/0 (socket says 172.21.15.77:33158) 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c6dd9b640 1 -- 172.21.15.77:0/2342589789 learned_addr learned my addr 172.21.15.77:0/2342589789 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46a240 0x55fd0a46ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54350/0 (socket says 172.21.15.77:54350) 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/2342589789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46f3b0 0x55fd0a471840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.391+0000 7f1c5f7fe640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1c580a2910 0x7f1c580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c6ed9d640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1c580a2910 0x7f1c580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c6dd9b640 1 -- 172.21.15.77:0/2342589789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46f3b0 msgr2=0x55fd0a471840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/2342589789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46f3b0 0x55fd0a471840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c6dd9b640 1 -- 172.21.15.77:0/2342589789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46a240 msgr2=0x55fd0a46ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/2342589789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46a240 0x55fd0a46ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c6dd9b640 1 -- 172.21.15.77:0/2342589789 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd0a431500 con 0x55fd0a469850 2024-03-16T10:18:14.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c6e59c640 1 --2- 172.21.15.77:0/2342589789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46a240 0x55fd0a46ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:14.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c5f7fe640 1 -- 172.21.15.77:0/2226878521 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f1c680f0f30 con 0x7f1c701022b0 2024-03-16T10:18:14.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c6ed9d640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1c580a2910 0x7f1c580a4dd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1c680c5be0 tx=0x7f1c680ecf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/2342589789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a469c70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1c640058c0 tx=0x7f1c64005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3032f5e1028deac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c5cff9640 1 -- 172.21.15.77:0/2342589789 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6400f7c0 con 0x55fd0a469850 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c5cff9640 1 -- 172.21.15.77:0/2342589789 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c6400fe40 con 0x55fd0a469850 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.392+0000 7f1c5cff9640 1 -- 172.21.15.77:0/2342589789 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6400eda0 con 0x55fd0a469850 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.393+0000 7f1c777fd740 1 -- 172.21.15.77:0/2342589789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 msgr2=0x55fd0a469c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.393+0000 7f1c777fd740 1 --2- 172.21.15.77:0/2342589789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a469c70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1c640058c0 tx=0x7f1c64005850 comp rx=0 tx=0).stop 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.394+0000 7f1c777fd740 1 -- 172.21.15.77:0/2342589789 shutdown_connections 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.394+0000 7f1c777fd740 1 --2- 172.21.15.77:0/2342589789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46f3b0 0x55fd0a471840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.394+0000 7f1c777fd740 1 --2- 172.21.15.77:0/2342589789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46a240 0x55fd0a46ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.394+0000 7f1c777fd740 1 --2- 172.21.15.77:0/2342589789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a469c70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.394+0000 7f1c777fd740 1 -- 172.21.15.77:0/2342589789 >> 172.21.15.77:0/2342589789 conn(0x55fd0a464dc0 msgr2=0x55fd0a4651c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.394+0000 7f1c777fd740 1 -- 172.21.15.77:0/2342589789 shutdown_connections 2024-03-16T10:18:14.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.394+0000 7f1c777fd740 1 -- 172.21.15.77:0/2342589789 wait complete. 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.394+0000 7f1c777fd740 1 -- start start 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.394+0000 7f1c777fd740 1 Processor -- start 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.394+0000 7f1c777fd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a469850 0x55fd0a478230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c777fd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a46a240 0x55fd0a478770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c777fd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a47f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c6dd9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a469850 0x55fd0a478230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c6dd9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a469850 0x55fd0a478230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50560/0 (socket says 172.21.15.77:50560) 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c6dd9b640 1 -- 172.21.15.77:0/3601726702 learned_addr learned my addr 172.21.15.77:0/3601726702 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c777fd740 1 -- 172.21.15.77:0/3601726702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fd0a476770 con 0x55fd0a46f3b0 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c777fd740 1 -- 172.21.15.77:0/3601726702 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55fd0a38a230 con 0x55fd0a469850 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c6e59c640 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a46a240 0x55fd0a478770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c777fd740 1 -- 172.21.15.77:0/3601726702 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fd0a4768f0 con 0x55fd0a46a240 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a47f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c777fd740 10 client.0 _ll_register_callbacks cb 0x55fd0a3c6010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c6e59c640 1 -- 172.21.15.77:0/3601726702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a469850 msgr2=0x55fd0a478230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c6e59c640 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a469850 0x55fd0a478230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c6e59c640 1 -- 172.21.15.77:0/3601726702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 msgr2=0x55fd0a47f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c6e59c640 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a47f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c6e59c640 1 -- 172.21.15.77:0/3601726702 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd0a399100 con 0x55fd0a46a240 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.395+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a47f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.396+0000 7f1c6e59c640 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a46a240 0x55fd0a478770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x55fd0a46a640 tx=0x7f1c6000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b34b19bfc1e5769a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.396+0000 7f1c55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.396+0000 7f1c55ffb640 1 -- 172.21.15.77:0/3601726702 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c600120d0 con 0x55fd0a46a240 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.396+0000 7f1c777fd740 1 -- 172.21.15.77:0/3601726702 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c6400fe40 con 0x55fd0a46a240 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.396+0000 7f1c55ffb640 1 -- 172.21.15.77:0/3601726702 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f1c60004030 con 0x55fd0a46a240 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.396+0000 7f1c55ffb640 1 -- 172.21.15.77:0/3601726702 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1c60012270 con 0x55fd0a46a240 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.396+0000 7f1c777fd740 10 client.25588 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.396+0000 7f1c777fd740 1 -- 172.21.15.77:0/3601726702 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fd0a0e23f0 con 0x55fd0a46a240 2024-03-16T10:18:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.397+0000 7f1c55ffb640 1 -- 172.21.15.77:0/3601726702 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c6400fe40 con 0x55fd0a46a240 2024-03-16T10:18:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.398+0000 7f1c55ffb640 1 -- 172.21.15.77:0/3601726702 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1c600035f0 con 0x55fd0a46a240 2024-03-16T10:18:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.398+0000 7f1c777fd740 10 client.25588 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.398+0000 7f1c777fd740 10 client.25588 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:18:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.398+0000 7f1c777fd740 20 client.25588 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.398+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55fd0a484570 0x55fd0a4869e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.398+0000 7f1c777fd740 4 client.25588 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.398+0000 7f1c777fd740 1 -- 172.21.15.77:0/3601726702 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd0a3db9d0 con 0x55fd0a484570 2024-03-16T10:18:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.398+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55fd0a484570 0x55fd0a4869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.398+0000 7f1c55ffb640 10 client.25588 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:18:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.398+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55fd0a484570 0x55fd0a4869e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aabea7df6b06a170 server_cookie=bf2212302f2ae4c4 in_seq=0 out_seq=0 2024-03-16T10:18:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.399+0000 7f1c55ffb640 1 -- 172.21.15.77:0/3601726702 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fd0a3db9d0 con 0x55fd0a484570 2024-03-16T10:18:14.404 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.399+0000 7f1c55ffb640 10 client.25588 handle_command_reply: tid=0 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.399+0000 7f1c777fd740 1 client.25588 shutdown 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.400+0000 7f1c777fd740 1 -- 172.21.15.77:0/3601726702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a46a240 msgr2=0x55fd0a478770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.400+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a46a240 0x55fd0a478770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x55fd0a46a640 tx=0x7f1c6000f790 comp rx=0 tx=0).stop 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.400+0000 7f1c777fd740 1 -- 172.21.15.77:0/3601726702 shutdown_connections 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.400+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55fd0a484570 0x55fd0a4869e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.400+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a47f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.400+0000 7f1c55ffb640 0 client.25588 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.400+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a46a240 0x55fd0a478770 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.400+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3601726702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a469850 0x55fd0a478230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.400+0000 7f1c777fd740 1 -- 172.21.15.77:0/3601726702 >> 172.21.15.77:0/3601726702 conn(0x55fd0a464dc0 msgr2=0x55fd0a463fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.400+0000 7f1c777fd740 1 -- 172.21.15.77:0/3601726702 shutdown_connections 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.400+0000 7f1c777fd740 1 -- 172.21.15.77:0/3601726702 wait complete. 2024-03-16T10:18:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.400+0000 7f1c777fd740 20 client.25588 trim_cache size 0 max 16384 2024-03-16T10:18:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c777fd740 1 Processor -- start 2024-03-16T10:18:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c777fd740 1 -- start start 2024-03-16T10:18:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c777fd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a476c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c777fd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46a240 0x55fd0a47f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c777fd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a4771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c777fd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701a6940 con 0x55fd0a46f3b0 2024-03-16T10:18:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c777fd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701a8480 con 0x55fd0a46a240 2024-03-16T10:18:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c777fd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701a7010 con 0x55fd0a469850 2024-03-16T10:18:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46a240 0x55fd0a47f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c6dd9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a476c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.418 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46a240 0x55fd0a47f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50576/0 (socket says 172.21.15.77:50576) 2024-03-16T10:18:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c6e59c640 1 -- 172.21.15.77:0/3890179274 learned_addr learned my addr 172.21.15.77:0/3890179274 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.416+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/3890179274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a4771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.417+0000 7f1c6dd9b640 1 -- 172.21.15.77:0/3890179274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46a240 msgr2=0x55fd0a47f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.417+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/3890179274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46a240 0x55fd0a47f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.417+0000 7f1c6dd9b640 1 -- 172.21.15.77:0/3890179274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 msgr2=0x55fd0a4771d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.417+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/3890179274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a4771d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.417+0000 7f1c6dd9b640 1 -- 172.21.15.77:0/3890179274 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd0a1ac7d0 con 0x55fd0a469850 2024-03-16T10:18:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.417+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/3890179274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a476c90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55fd0a487260 tx=0x7f1c64005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eea7b2ceccfd63c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.417+0000 7f1c57fff640 1 -- 172.21.15.77:0/3890179274 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c64037070 con 0x55fd0a469850 2024-03-16T10:18:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.417+0000 7f1c57fff640 1 -- 172.21.15.77:0/3890179274 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c64005510 con 0x55fd0a469850 2024-03-16T10:18:14.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.417+0000 7f1c57fff640 1 -- 172.21.15.77:0/3890179274 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6400fcd0 con 0x55fd0a469850 2024-03-16T10:18:14.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.418+0000 7f1c777fd740 1 -- 172.21.15.77:0/3890179274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 msgr2=0x55fd0a476c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.418+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3890179274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a476c90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55fd0a487260 tx=0x7f1c64005850 comp rx=0 tx=0).stop 2024-03-16T10:18:14.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.418+0000 7f1c777fd740 1 -- 172.21.15.77:0/3890179274 shutdown_connections 2024-03-16T10:18:14.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.418+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3890179274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a4771d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.418+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3890179274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46a240 0x55fd0a47f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.418+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3890179274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a476c90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.418+0000 7f1c777fd740 1 -- 172.21.15.77:0/3890179274 >> 172.21.15.77:0/3890179274 conn(0x55fd0a464dc0 msgr2=0x55fd0a46b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.418+0000 7f1c777fd740 1 -- 172.21.15.77:0/3890179274 shutdown_connections 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.418+0000 7f1c777fd740 1 -- 172.21.15.77:0/3890179274 wait complete. 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.419+0000 7f1c777fd740 1 -- start start 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.419+0000 7f1c777fd740 1 Processor -- start 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.419+0000 7f1c777fd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a47f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.419+0000 7f1c777fd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46a240 0x55fd0a47f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.419+0000 7f1c777fd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a47bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c6dd9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a47f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c6dd9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a47f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33188/0 (socket says 172.21.15.77:33188) 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c6dd9b640 1 -- 172.21.15.77:0/3516558791 learned_addr learned my addr 172.21.15.77:0/3516558791 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c6e59c640 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46a240 0x55fd0a47f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c777fd740 1 -- 172.21.15.77:0/3516558791 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701a7db0 con 0x55fd0a46f3b0 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c777fd740 1 -- 172.21.15.77:0/3516558791 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701a76e0 con 0x55fd0a46a240 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c777fd740 1 -- 172.21.15.77:0/3516558791 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701a4920 con 0x55fd0a469850 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a47bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c6dd9b640 1 -- 172.21.15.77:0/3516558791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46a240 msgr2=0x55fd0a47f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46a240 0x55fd0a47f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c6dd9b640 1 -- 172.21.15.77:0/3516558791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 msgr2=0x55fd0a47bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a47bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.420+0000 7f1c6dd9b640 1 -- 172.21.15.77:0/3516558791 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd0a47c820 con 0x55fd0a469850 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.421+0000 7f1c777fd740 10 client.0 _ll_register_callbacks cb 0x7f1c60009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.421+0000 7f1c6dd9b640 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a47f300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1c640057c0 tx=0x7f1c64005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac09adf22ed1db9c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.421+0000 7f1c55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.421+0000 7f1c777fd740 1 -- 172.21.15.77:0/3516558791 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c64005510 con 0x55fd0a469850 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.421+0000 7f1c55ffb640 1 -- 172.21.15.77:0/3516558791 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c64037040 con 0x55fd0a469850 2024-03-16T10:18:14.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.421+0000 7f1c55ffb640 1 -- 172.21.15.77:0/3516558791 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f1c6400fc50 con 0x55fd0a469850 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.421+0000 7f1c55ffb640 1 -- 172.21.15.77:0/3516558791 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1c64044050 con 0x55fd0a469850 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.421+0000 7f1c777fd740 10 client.25594 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.421+0000 7f1c777fd740 1 -- 172.21.15.77:0/3516558791 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55fd0a4e95d0 con 0x55fd0a469850 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.422+0000 7f1c55ffb640 1 -- 172.21.15.77:0/3516558791 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1c6401d020 con 0x55fd0a469850 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.422+0000 7f1c777fd740 10 client.25594 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.422+0000 7f1c777fd740 10 client.25594 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.422+0000 7f1c777fd740 20 client.25594 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.422+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55fd0a484570 0x55fd0a4ebba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.422+0000 7f1c777fd740 4 client.25594 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.422+0000 7f1c777fd740 1 -- 172.21.15.77:0/3516558791 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd0a478020 con 0x55fd0a484570 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.422+0000 7f1c6e59c640 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55fd0a484570 0x55fd0a4ebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.422+0000 7f1c55ffb640 10 client.25594 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.422+0000 7f1c6e59c640 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55fd0a484570 0x55fd0a4ebba0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fd39d25b495f76d5 server_cookie=2c1006f770335a8a in_seq=0 out_seq=0 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c55ffb640 1 -- 172.21.15.77:0/3516558791 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd0a478020 con 0x55fd0a484570 2024-03-16T10:18:14.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c55ffb640 10 client.25594 handle_command_reply: tid=0 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c777fd740 1 client.25594 shutdown 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c777fd740 1 -- 172.21.15.77:0/3516558791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 msgr2=0x55fd0a47f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a47f300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1c640057c0 tx=0x7f1c64005fd0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c777fd740 1 -- 172.21.15.77:0/3516558791 shutdown_connections 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55fd0a484570 0x55fd0a4ebba0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd0a46f3b0 0x55fd0a47bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c55ffb640 0 client.25594 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fd0a46a240 0x55fd0a47f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c777fd740 1 --2- 172.21.15.77:0/3516558791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd0a469850 0x55fd0a47f300 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c777fd740 1 -- 172.21.15.77:0/3516558791 >> 172.21.15.77:0/3516558791 conn(0x55fd0a464dc0 msgr2=0x55fd0a46b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.423+0000 7f1c777fd740 1 -- 172.21.15.77:0/3516558791 shutdown_connections 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.424+0000 7f1c777fd740 1 -- 172.21.15.77:0/3516558791 wait complete. 2024-03-16T10:18:14.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.424+0000 7f1c777fd740 20 client.25594 trim_cache size 0 max 16384 2024-03-16T10:18:14.429 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:18:14.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.424+0000 7f1c74f77640 1 -- 172.21.15.77:0/2226878521 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1c580a2910 msgr2=0x7f1c580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.424+0000 7f1c74f77640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1c580a2910 0x7f1c580a4dd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1c680c5be0 tx=0x7f1c680ecf90 comp rx=0 tx=0).stop 2024-03-16T10:18:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.425+0000 7f1c74f77640 1 -- 172.21.15.77:0/2226878521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c701022b0 msgr2=0x7f1c7019ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.425+0000 7f1c74f77640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c701022b0 0x7f1c7019ced0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1c6802c430 tx=0x7f1c680337b0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.425+0000 7f1c74f77640 1 -- 172.21.15.77:0/2226878521 shutdown_connections 2024-03-16T10:18:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.425+0000 7f1c74f77640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1c580a2910 0x7f1c580a4dd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.425+0000 7f1c74f77640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c701022b0 0x7f1c7019ced0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.425+0000 7f1c74f77640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c701018f0 0x7f1c70195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.425+0000 7f1c74f77640 1 --2- 172.21.15.77:0/2226878521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c70100700 0x7f1c70195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.425+0000 7f1c74f77640 1 -- 172.21.15.77:0/2226878521 >> 172.21.15.77:0/2226878521 conn(0x7f1c700fbc90 msgr2=0x7f1c70104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.425+0000 7f1c74f77640 1 -- 172.21.15.77:0/2226878521 shutdown_connections 2024-03-16T10:18:14.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.425+0000 7f1c74f77640 1 -- 172.21.15.77:0/2226878521 wait complete. 2024-03-16T10:18:14.440 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T10:18:14.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.552+0000 7f48e7f87640 1 -- 172.21.15.77:0/3807359527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e01006e0 msgr2=0x7f48e0100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.552+0000 7f48e7f87640 1 --2- 172.21.15.77:0/3807359527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e01006e0 0x7f48e0100b00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f48dc00d910 tx=0x7f48dc02dbb0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.552+0000 7f48e7f87640 1 -- 172.21.15.77:0/3807359527 shutdown_connections 2024-03-16T10:18:14.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.552+0000 7f48e7f87640 1 --2- 172.21.15.77:0/3807359527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e0102290 0x7f48e0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.552+0000 7f48e7f87640 1 --2- 172.21.15.77:0/3807359527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e01018d0 0x7f48e0101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.552+0000 7f48e7f87640 1 --2- 172.21.15.77:0/3807359527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e01006e0 0x7f48e0100b00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.552+0000 7f48e7f87640 1 -- 172.21.15.77:0/3807359527 >> 172.21.15.77:0/3807359527 conn(0x7f48e00fbc70 msgr2=0x7f48e00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.552+0000 7f48e7f87640 1 -- 172.21.15.77:0/3807359527 shutdown_connections 2024-03-16T10:18:14.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.552+0000 7f48e7f87640 1 -- 172.21.15.77:0/3807359527 wait complete. 2024-03-16T10:18:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.553+0000 7f48e7f87640 1 Processor -- start 2024-03-16T10:18:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.553+0000 7f48e7f87640 1 -- start start 2024-03-16T10:18:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.553+0000 7f48e7f87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e01006e0 0x7f48e0195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.553+0000 7f48e7f87640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e01018d0 0x7f48e0195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.553+0000 7f48e5d23640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e01006e0 0x7f48e0195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e5d23640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e01006e0 0x7f48e0195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54402/0 (socket says 172.21.15.77:54402) 2024-03-16T10:18:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e5522640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e01018d0 0x7f48e0195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e5d23640 1 -- 172.21.15.77:0/1288390695 learned_addr learned my addr 172.21.15.77:0/1288390695 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e7f87640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e0102290 0x7f48e019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e7f87640 1 -- 172.21.15.77:0/1288390695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f48e010b2b0 con 0x7f48e01006e0 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e7f87640 1 -- 172.21.15.77:0/1288390695 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f48e010b130 con 0x7f48e01018d0 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e7f87640 1 -- 172.21.15.77:0/1288390695 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f48e010b430 con 0x7f48e0102290 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e6524640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e0102290 0x7f48e019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e5d23640 1 -- 172.21.15.77:0/1288390695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e0102290 msgr2=0x7f48e019cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e5d23640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e0102290 0x7f48e019cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e5d23640 1 -- 172.21.15.77:0/1288390695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e01018d0 msgr2=0x7f48e0195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e5d23640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e01018d0 0x7f48e0195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e5d23640 1 -- 172.21.15.77:0/1288390695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e019d590 con 0x7f48e01006e0 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e5522640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e01018d0 0x7f48e0195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.554+0000 7f48e5d23640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e01006e0 0x7f48e0195970 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f48dc004c20 tx=0x7f48dc004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.555+0000 7f48d6ffd640 1 -- 172.21.15.77:0/1288390695 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc044070 con 0x7f48e01006e0 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.555+0000 7f48d6ffd640 1 -- 172.21.15.77:0/1288390695 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48dc040660 con 0x7f48e01006e0 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.555+0000 7f48e7f87640 1 -- 172.21.15.77:0/1288390695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48e019d820 con 0x7f48e01006e0 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.555+0000 7f48e7f87640 1 -- 172.21.15.77:0/1288390695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48e019dd60 con 0x7f48e01006e0 2024-03-16T10:18:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.555+0000 7f48d6ffd640 1 -- 172.21.15.77:0/1288390695 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc02e8d0 con 0x7f48e01006e0 2024-03-16T10:18:14.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.558+0000 7f48d6ffd640 1 -- 172.21.15.77:0/1288390695 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f48dc047070 con 0x7f48e01006e0 2024-03-16T10:18:14.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.559+0000 7f48d6ffd640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f48cc0a2990 0x7f48cc0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.559+0000 7f48e5522640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f48cc0a2990 0x7f48cc0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.560+0000 7f48d6ffd640 1 -- 172.21.15.77:0/1288390695 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f48dc0f3410 con 0x7f48e01006e0 2024-03-16T10:18:14.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.560+0000 7f48e5522640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f48cc0a2990 0x7f48cc0a4e50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f48e0196e80 tx=0x7f48d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.563+0000 7f48eab70740 1 Processor -- start 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.563+0000 7f48eab70740 1 -- start start 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.563+0000 7f48eab70740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d3850 0x564e5b0d3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.563+0000 7f48eab70740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d4240 0x564e5b0d8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.563+0000 7f48eab70740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d93b0 0x564e5b0db840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.563+0000 7f48eab70740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564e5b038490 con 0x564e5b0d93b0 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.563+0000 7f48eab70740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564e5afe0c90 con 0x564e5b0d4240 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.563+0000 7f48eab70740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564e5b034530 con 0x564e5b0d3850 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.563+0000 7f48e5d23640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d3850 0x564e5b0d3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.563+0000 7f48e5d23640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d3850 0x564e5b0d3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33206/0 (socket says 172.21.15.77:33206) 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.563+0000 7f48e5d23640 1 -- 172.21.15.77:0/2618673132 learned_addr learned my addr 172.21.15.77:0/2618673132 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48e5522640 1 --2- 172.21.15.77:0/2618673132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d4240 0x564e5b0d8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48e6524640 1 --2- 172.21.15.77:0/2618673132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d93b0 0x564e5b0db840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48e5d23640 1 -- 172.21.15.77:0/2618673132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d4240 msgr2=0x564e5b0d8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48e5d23640 1 --2- 172.21.15.77:0/2618673132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d4240 0x564e5b0d8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48e5d23640 1 -- 172.21.15.77:0/2618673132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d93b0 msgr2=0x564e5b0db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48e5d23640 1 --2- 172.21.15.77:0/2618673132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d93b0 0x564e5b0db840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48e5d23640 1 -- 172.21.15.77:0/2618673132 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e5b09b500 con 0x564e5b0d3850 2024-03-16T10:18:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48e5522640 1 --2- 172.21.15.77:0/2618673132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d4240 0x564e5b0d8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48e6524640 1 --2- 172.21.15.77:0/2618673132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d93b0 0x564e5b0db840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48e5d23640 1 --2- 172.21.15.77:0/2618673132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d3850 0x564e5b0d3c70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0bfce0 tx=0x7f48dc02dbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=876248d36daf9e0d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48cbfff640 1 -- 172.21.15.77:0/2618673132 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc0bfd50 con 0x564e5b0d3850 2024-03-16T10:18:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48cbfff640 1 -- 172.21.15.77:0/2618673132 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48dc032070 con 0x564e5b0d3850 2024-03-16T10:18:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.564+0000 7f48cbfff640 1 -- 172.21.15.77:0/2618673132 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc0bc560 con 0x564e5b0d3850 2024-03-16T10:18:14.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.565+0000 7f48eab70740 1 -- 172.21.15.77:0/2618673132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d3850 msgr2=0x564e5b0d3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.565+0000 7f48eab70740 1 --2- 172.21.15.77:0/2618673132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d3850 0x564e5b0d3c70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0bfce0 tx=0x7f48dc02dbb0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.565+0000 7f48eab70740 1 -- 172.21.15.77:0/2618673132 shutdown_connections 2024-03-16T10:18:14.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.565+0000 7f48eab70740 1 --2- 172.21.15.77:0/2618673132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d93b0 0x564e5b0db840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.565+0000 7f48eab70740 1 --2- 172.21.15.77:0/2618673132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d4240 0x564e5b0d8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.565+0000 7f48eab70740 1 --2- 172.21.15.77:0/2618673132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d3850 0x564e5b0d3c70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.565+0000 7f48eab70740 1 -- 172.21.15.77:0/2618673132 >> 172.21.15.77:0/2618673132 conn(0x564e5b0cedc0 msgr2=0x564e5b0cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.565+0000 7f48eab70740 1 -- 172.21.15.77:0/2618673132 shutdown_connections 2024-03-16T10:18:14.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.565+0000 7f48eab70740 1 -- 172.21.15.77:0/2618673132 wait complete. 2024-03-16T10:18:14.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.565+0000 7f48eab70740 1 -- start start 2024-03-16T10:18:14.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48eab70740 1 Processor -- start 2024-03-16T10:18:14.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48eab70740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d3850 0x564e5b0ddaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48eab70740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d4240 0x564e5b0ddfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48eab70740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d93b0 0x564e5b0e5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48e5d23640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d3850 0x564e5b0ddaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48e5d23640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d3850 0x564e5b0ddaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54432/0 (socket says 172.21.15.77:54432) 2024-03-16T10:18:14.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48e5d23640 1 -- 172.21.15.77:0/2350860759 learned_addr learned my addr 172.21.15.77:0/2350860759 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48eab70740 1 -- 172.21.15.77:0/2350860759 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564e5b0dc120 con 0x564e5b0d3850 2024-03-16T10:18:14.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48eab70740 1 -- 172.21.15.77:0/2350860759 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564e5aff4230 con 0x564e5b0d4240 2024-03-16T10:18:14.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48eab70740 1 -- 172.21.15.77:0/2350860759 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564e5b0dc2a0 con 0x564e5b0d93b0 2024-03-16T10:18:14.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48e5522640 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d4240 0x564e5b0ddfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48e6524640 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d93b0 0x564e5b0e5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.566+0000 7f48eab70740 10 client.0 _ll_register_callbacks cb 0x564e5b030010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:14.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48e5522640 1 -- 172.21.15.77:0/2350860759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d93b0 msgr2=0x564e5b0e5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48e5522640 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d93b0 0x564e5b0e5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48e5522640 1 -- 172.21.15.77:0/2350860759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d3850 msgr2=0x564e5b0ddaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48e5522640 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d3850 0x564e5b0ddaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48e5522640 1 -- 172.21.15.77:0/2350860759 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e5b003100 con 0x564e5b0d4240 2024-03-16T10:18:14.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48e5d23640 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d3850 0x564e5b0ddaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:14.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48e5522640 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d4240 0x564e5b0ddfe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x564e5b0d4640 tx=0x7f48d0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=747e5764533f704a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:18:14.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48c97fa640 1 -- 172.21.15.77:0/2350860759 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d0012e60 con 0x564e5b0d4240 2024-03-16T10:18:14.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48eab70740 1 -- 172.21.15.77:0/2350860759 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48dc032070 con 0x564e5b0d4240 2024-03-16T10:18:14.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48c97fa640 1 -- 172.21.15.77:0/2350860759 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f48d001fb10 con 0x564e5b0d4240 2024-03-16T10:18:14.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48c97fa640 1 -- 172.21.15.77:0/2350860759 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f48d001fd10 con 0x564e5b0d4240 2024-03-16T10:18:14.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48eab70740 10 client.25616 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:14.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.567+0000 7f48eab70740 1 -- 172.21.15.77:0/2350860759 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564e5ad4c3f0 con 0x564e5b0d4240 2024-03-16T10:18:14.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.568+0000 7f48c97fa640 1 -- 172.21.15.77:0/2350860759 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48dc032070 con 0x564e5b0d4240 2024-03-16T10:18:14.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.568+0000 7f48c97fa640 1 -- 172.21.15.77:0/2350860759 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f48d0032050 con 0x564e5b0d4240 2024-03-16T10:18:14.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.568+0000 7f48eab70740 10 client.25616 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:14.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.568+0000 7f48eab70740 10 client.25616 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:18:14.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.568+0000 7f48eab70740 20 client.25616 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:14.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.568+0000 7f48eab70740 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564e5b0e9840 0x564e5b0ebd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.568+0000 7f48eab70740 4 client.25616 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:14.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.568+0000 7f48eab70740 1 -- 172.21.15.77:0/2350860759 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e5b0459d0 con 0x564e5b0e9840 2024-03-16T10:18:14.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.568+0000 7f48e5d23640 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564e5b0e9840 0x564e5b0ebd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.568+0000 7f48c97fa640 10 client.25616 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:18:14.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.568+0000 7f48e5d23640 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564e5b0e9840 0x564e5b0ebd00 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4d7ebde741d16fdc server_cookie=822f8038705a0944 in_seq=0 out_seq=0 2024-03-16T10:18:14.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.569+0000 7f48c97fa640 1 -- 172.21.15.77:0/2350860759 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564e5b0459d0 con 0x564e5b0e9840 2024-03-16T10:18:14.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.569+0000 7f48c97fa640 10 client.25616 handle_command_reply: tid=0 2024-03-16T10:18:14.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.569+0000 7f48eab70740 1 client.25616 shutdown 2024-03-16T10:18:14.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.570+0000 7f48eab70740 1 -- 172.21.15.77:0/2350860759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d4240 msgr2=0x564e5b0ddfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.570+0000 7f48eab70740 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d4240 0x564e5b0ddfe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x564e5b0d4640 tx=0x7f48d0004730 comp rx=0 tx=0).stop 2024-03-16T10:18:14.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.570+0000 7f48eab70740 1 -- 172.21.15.77:0/2350860759 shutdown_connections 2024-03-16T10:18:14.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.570+0000 7f48eab70740 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d93b0 0x564e5b0e5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.570+0000 7f48eab70740 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564e5b0e9840 0x564e5b0ebd00 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.570+0000 7f48eab70740 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d4240 0x564e5b0ddfe0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.570+0000 7f48c97fa640 0 client.25616 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:18:14.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.570+0000 7f48eab70740 1 --2- 172.21.15.77:0/2350860759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d3850 0x564e5b0ddaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.570+0000 7f48eab70740 1 -- 172.21.15.77:0/2350860759 >> 172.21.15.77:0/2350860759 conn(0x564e5b0cedc0 msgr2=0x564e5b0cdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.570+0000 7f48eab70740 1 -- 172.21.15.77:0/2350860759 shutdown_connections 2024-03-16T10:18:14.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.570+0000 7f48eab70740 1 -- 172.21.15.77:0/2350860759 wait complete. 2024-03-16T10:18:14.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.570+0000 7f48eab70740 20 client.25616 trim_cache size 0 max 16384 2024-03-16T10:18:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.595+0000 7f48eab70740 1 Processor -- start 2024-03-16T10:18:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.595+0000 7f48eab70740 1 -- start start 2024-03-16T10:18:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48eab70740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d3850 0x564e5b0d2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48eab70740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d4240 0x564e5b0e4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48eab70740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d93b0 0x564e5b0f05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48eab70740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f48e01a6a60 con 0x564e5b0d93b0 2024-03-16T10:18:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48eab70740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f48e01a85a0 con 0x564e5b0d3850 2024-03-16T10:18:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48eab70740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f48e01a7130 con 0x564e5b0d4240 2024-03-16T10:18:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48e5d23640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d3850 0x564e5b0d2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48e5d23640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d3850 0x564e5b0d2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50618/0 (socket says 172.21.15.77:50618) 2024-03-16T10:18:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48e5d23640 1 -- 172.21.15.77:0/1993287487 learned_addr learned my addr 172.21.15.77:0/1993287487 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48e5522640 1 --2- 172.21.15.77:0/1993287487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d4240 0x564e5b0e4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48e6524640 1 --2- 172.21.15.77:0/1993287487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d93b0 0x564e5b0f05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48e5522640 1 -- 172.21.15.77:0/1993287487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d3850 msgr2=0x564e5b0d2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48e5522640 1 --2- 172.21.15.77:0/1993287487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d3850 0x564e5b0d2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48e5522640 1 -- 172.21.15.77:0/1993287487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d93b0 msgr2=0x564e5b0f05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48e5522640 1 --2- 172.21.15.77:0/1993287487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d93b0 0x564e5b0f05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.596+0000 7f48e5522640 1 -- 172.21.15.77:0/1993287487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e5ae167d0 con 0x564e5b0d4240 2024-03-16T10:18:14.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.597+0000 7f48e5522640 1 --2- 172.21.15.77:0/1993287487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d4240 0x564e5b0e4fe0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f48d00206a0 tx=0x7f48d00097b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=430dde9d7aee3d68 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.597+0000 7f48cb7fe640 1 -- 172.21.15.77:0/1993287487 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d0020430 con 0x564e5b0d4240 2024-03-16T10:18:14.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.597+0000 7f48cb7fe640 1 -- 172.21.15.77:0/1993287487 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48d0012450 con 0x564e5b0d4240 2024-03-16T10:18:14.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.597+0000 7f48cb7fe640 1 -- 172.21.15.77:0/1993287487 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d00299b0 con 0x564e5b0d4240 2024-03-16T10:18:14.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.598+0000 7f48eab70740 1 -- 172.21.15.77:0/1993287487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d4240 msgr2=0x564e5b0e4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.598+0000 7f48eab70740 1 --2- 172.21.15.77:0/1993287487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d4240 0x564e5b0e4fe0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f48d00206a0 tx=0x7f48d00097b0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.598+0000 7f48eab70740 1 -- 172.21.15.77:0/1993287487 shutdown_connections 2024-03-16T10:18:14.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.598+0000 7f48eab70740 1 --2- 172.21.15.77:0/1993287487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d93b0 0x564e5b0f05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.598+0000 7f48eab70740 1 --2- 172.21.15.77:0/1993287487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d4240 0x564e5b0e4fe0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.598+0000 7f48eab70740 1 --2- 172.21.15.77:0/1993287487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d3850 0x564e5b0d2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.598+0000 7f48eab70740 1 -- 172.21.15.77:0/1993287487 >> 172.21.15.77:0/1993287487 conn(0x564e5b0cedc0 msgr2=0x564e5b0d2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.598+0000 7f48eab70740 1 -- 172.21.15.77:0/1993287487 shutdown_connections 2024-03-16T10:18:14.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.598+0000 7f48eab70740 1 -- 172.21.15.77:0/1993287487 wait complete. 2024-03-16T10:18:14.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.598+0000 7f48eab70740 1 -- start start 2024-03-16T10:18:14.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.598+0000 7f48eab70740 1 Processor -- start 2024-03-16T10:18:14.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.599+0000 7f48eab70740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d3850 0x564e5b0e2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.599+0000 7f48eab70740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d4240 0x564e5b0e2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.599+0000 7f48eab70740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d93b0 0x564e5b0e3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.599+0000 7f48eab70740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f48e01a7ed0 con 0x564e5b0d4240 2024-03-16T10:18:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.599+0000 7f48e5522640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d4240 0x564e5b0e2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.599+0000 7f48eab70740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f48e01a7800 con 0x564e5b0d3850 2024-03-16T10:18:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.599+0000 7f48e5d23640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d3850 0x564e5b0e2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.599+0000 7f48e5522640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d4240 0x564e5b0e2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54456/0 (socket says 172.21.15.77:54456) 2024-03-16T10:18:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.599+0000 7f48e6524640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d93b0 0x564e5b0e3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.599+0000 7f48e5522640 1 -- 172.21.15.77:0/3334453379 learned_addr learned my addr 172.21.15.77:0/3334453379 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.599+0000 7f48eab70740 1 -- 172.21.15.77:0/3334453379 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f48e01a4a40 con 0x564e5b0d93b0 2024-03-16T10:18:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48eab70740 10 client.0 _ll_register_callbacks cb 0x7f48d00099f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48e6524640 1 -- 172.21.15.77:0/3334453379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d3850 msgr2=0x564e5b0e2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48e6524640 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d3850 0x564e5b0e2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48e6524640 1 -- 172.21.15.77:0/3334453379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d4240 msgr2=0x564e5b0e2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48e6524640 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d4240 0x564e5b0e2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48e6524640 1 -- 172.21.15.77:0/3334453379 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e5b0e4410 con 0x564e5b0d93b0 2024-03-16T10:18:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48e5d23640 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d3850 0x564e5b0e2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48e5522640 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d4240 0x564e5b0e2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48e6524640 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d93b0 0x564e5b0e3e20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x564e5b0cfd00 tx=0x7f48d800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2722f33d32933213 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:18:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48c97fa640 1 -- 172.21.15.77:0/3334453379 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d8023060 con 0x564e5b0d93b0 2024-03-16T10:18:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48c97fa640 1 -- 172.21.15.77:0/3334453379 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f48d8003b70 con 0x564e5b0d93b0 2024-03-16T10:18:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.600+0000 7f48eab70740 1 -- 172.21.15.77:0/3334453379 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564e5b03fd20 con 0x564e5b0d93b0 2024-03-16T10:18:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.601+0000 7f48c97fa640 1 -- 172.21.15.77:0/3334453379 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f48d8003d70 con 0x564e5b0d93b0 2024-03-16T10:18:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.601+0000 7f48eab70740 10 client.25612 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.601+0000 7f48eab70740 1 -- 172.21.15.77:0/3334453379 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564e5b14fec0 con 0x564e5b0d93b0 2024-03-16T10:18:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.601+0000 7f48c97fa640 1 -- 172.21.15.77:0/3334453379 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48d8004530 con 0x564e5b0d93b0 2024-03-16T10:18:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.602+0000 7f48c97fa640 1 -- 172.21.15.77:0/3334453379 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f48d801e6f0 con 0x564e5b0d93b0 2024-03-16T10:18:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.602+0000 7f48eab70740 10 client.25612 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:14.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.602+0000 7f48eab70740 10 client.25612 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:18:14.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.602+0000 7f48eab70740 20 client.25612 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:14.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.602+0000 7f48eab70740 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564e5b0e9840 0x564e5b152900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.602+0000 7f48eab70740 4 client.25612 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:18:14.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.602+0000 7f48eab70740 1 -- 172.21.15.77:0/3334453379 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e5b03fd20 con 0x564e5b0e9840 2024-03-16T10:18:14.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.602+0000 7f48e5d23640 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564e5b0e9840 0x564e5b152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48c97fa640 10 client.25612 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:18:14.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48e5d23640 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564e5b0e9840 0x564e5b152900 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cc14d361a634258 server_cookie=21529c51e7df8989 in_seq=0 out_seq=0 2024-03-16T10:18:14.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48c97fa640 1 -- 172.21.15.77:0/3334453379 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e5b03fd20 con 0x564e5b0e9840 2024-03-16T10:18:14.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48c97fa640 10 client.25612 handle_command_reply: tid=0 2024-03-16T10:18:14.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48eab70740 1 client.25612 shutdown 2024-03-16T10:18:14.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48eab70740 1 -- 172.21.15.77:0/3334453379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d93b0 msgr2=0x564e5b0e3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48eab70740 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d93b0 0x564e5b0e3e20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x564e5b0cfd00 tx=0x7f48d800f790 comp rx=0 tx=0).stop 2024-03-16T10:18:14.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48eab70740 1 -- 172.21.15.77:0/3334453379 shutdown_connections 2024-03-16T10:18:14.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48eab70740 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e5b0d93b0 0x564e5b0e3e20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48eab70740 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564e5b0d4240 0x564e5b0e2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48eab70740 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x564e5b0e9840 0x564e5b152900 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48eab70740 1 --2- 172.21.15.77:0/3334453379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564e5b0d3850 0x564e5b0e2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48eab70740 1 -- 172.21.15.77:0/3334453379 >> 172.21.15.77:0/3334453379 conn(0x564e5b0cedc0 msgr2=0x564e5b0d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48c97fa640 0 client.25612 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:18:14.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48eab70740 1 -- 172.21.15.77:0/3334453379 shutdown_connections 2024-03-16T10:18:14.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.603+0000 7f48eab70740 1 -- 172.21.15.77:0/3334453379 wait complete. 2024-03-16T10:18:14.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48eab70740 20 client.25612 trim_cache size 0 max 16384 2024-03-16T10:18:14.616 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:18:14.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48e7f87640 1 -- 172.21.15.77:0/1288390695 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f48cc0a2990 msgr2=0x7f48cc0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48e7f87640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f48cc0a2990 0x7f48cc0a4e50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f48e0196e80 tx=0x7f48d0005f90 comp rx=0 tx=0).stop 2024-03-16T10:18:14.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48e7f87640 1 -- 172.21.15.77:0/1288390695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e01006e0 msgr2=0x7f48e0195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48e7f87640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e01006e0 0x7f48e0195970 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f48dc004c20 tx=0x7f48dc004e70 comp rx=0 tx=0).stop 2024-03-16T10:18:14.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48e7f87640 1 -- 172.21.15.77:0/1288390695 shutdown_connections 2024-03-16T10:18:14.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48e7f87640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f48cc0a2990 0x7f48cc0a4e50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48e7f87640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48e0102290 0x7f48e019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48e7f87640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48e01018d0 0x7f48e0195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48e7f87640 1 --2- 172.21.15.77:0/1288390695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48e01006e0 0x7f48e0195970 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48e7f87640 1 -- 172.21.15.77:0/1288390695 >> 172.21.15.77:0/1288390695 conn(0x7f48e00fbc70 msgr2=0x7f48e0104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48e7f87640 1 -- 172.21.15.77:0/1288390695 shutdown_connections 2024-03-16T10:18:14.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.604+0000 7f48e7f87640 1 -- 172.21.15.77:0/1288390695 wait complete. 2024-03-16T10:18:14.620 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T10:18:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.758+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1142778081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4102290 msgr2=0x7fbfa4108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.758+0000 7fbfaaf69640 1 --2- 172.21.15.77:0/1142778081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4102290 0x7fbfa4108b20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009be0 tx=0x7fbf9c02bef0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.758+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1142778081 shutdown_connections 2024-03-16T10:18:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.758+0000 7fbfaaf69640 1 --2- 172.21.15.77:0/1142778081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4102290 0x7fbfa4108b20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.758+0000 7fbfaaf69640 1 --2- 172.21.15.77:0/1142778081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfa41018d0 0x7fbfa4101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.758+0000 7fbfaaf69640 1 --2- 172.21.15.77:0/1142778081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa41006e0 0x7fbfa4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.758+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1142778081 >> 172.21.15.77:0/1142778081 conn(0x7fbfa40fbc70 msgr2=0x7fbfa40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.758+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1142778081 shutdown_connections 2024-03-16T10:18:14.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.758+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1142778081 wait complete. 2024-03-16T10:18:14.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.758+0000 7fbfaaf69640 1 Processor -- start 2024-03-16T10:18:14.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfaaf69640 1 -- start start 2024-03-16T10:18:14.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfaaf69640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfa41006e0 0x7fbfa4197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfaaf69640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa41018d0 0x7fbfa4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfaaf69640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4102290 0x7fbfa419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfaaf69640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa410d480 con 0x7fbfa41018d0 2024-03-16T10:18:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfaaf69640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa410d300 con 0x7fbfa41006e0 2024-03-16T10:18:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfaaf69640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa410d600 con 0x7fbfa4102290 2024-03-16T10:18:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfa3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa41018d0 0x7fbfa4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfa8d05640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfa41006e0 0x7fbfa4197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfa3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa41018d0 0x7fbfa4198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54474/0 (socket says 172.21.15.77:54474) 2024-03-16T10:18:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfa3fff640 1 -- 172.21.15.77:0/1674539543 learned_addr learned my addr 172.21.15.77:0/1674539543 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfa9506640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4102290 0x7fbfa419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfa3fff640 1 -- 172.21.15.77:0/1674539543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4102290 msgr2=0x7fbfa419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4102290 0x7fbfa419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfa3fff640 1 -- 172.21.15.77:0/1674539543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfa41006e0 msgr2=0x7fbfa4197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfa41006e0 0x7fbfa4197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfa3fff640 1 -- 172.21.15.77:0/1674539543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa4109560 con 0x7fbfa41018d0 2024-03-16T10:18:14.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.759+0000 7fbfa8d05640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfa41006e0 0x7fbfa4197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:14.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.760+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa41018d0 0x7fbfa4198010 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbf9000e9f0 tx=0x7fbf9000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.760+0000 7fbfa1ffb640 1 -- 172.21.15.77:0/1674539543 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9000ccc0 con 0x7fbfa41018d0 2024-03-16T10:18:14.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.760+0000 7fbfa1ffb640 1 -- 172.21.15.77:0/1674539543 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf9000ce60 con 0x7fbfa41018d0 2024-03-16T10:18:14.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.760+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1674539543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa4109850 con 0x7fbfa41018d0 2024-03-16T10:18:14.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.760+0000 7fbfa1ffb640 1 -- 172.21.15.77:0/1674539543 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf90010690 con 0x7fbfa41018d0 2024-03-16T10:18:14.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.761+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1674539543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa4109da0 con 0x7fbfa41018d0 2024-03-16T10:18:14.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.764+0000 7fbfa1ffb640 1 -- 172.21.15.77:0/1674539543 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fbf900046c0 con 0x7fbfa41018d0 2024-03-16T10:18:14.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfa1ffb640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf940a2990 0x7fbf940a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfadc32740 1 Processor -- start 2024-03-16T10:18:14.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfadc32740 1 -- start start 2024-03-16T10:18:14.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfadc32740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e20480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfadc32740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e20a50 0x557e52e25550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfadc32740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 0x557e52e28050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfadc32740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557e52d7c490 con 0x557e52e25bc0 2024-03-16T10:18:14.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfadc32740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x557e52d24c90 con 0x557e52e20a50 2024-03-16T10:18:14.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfadc32740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557e52d78530 con 0x557e52e20060 2024-03-16T10:18:14.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfa8d05640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf940a2990 0x7fbf940a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfa9506640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 0x557e52e28050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfa9506640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 0x557e52e28050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54478/0 (socket says 172.21.15.77:54478) 2024-03-16T10:18:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfa9506640 1 -- 172.21.15.77:0/1474575944 learned_addr learned my addr 172.21.15.77:0/1474575944 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.765+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/1474575944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e20a50 0x557e52e25550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.766+0000 7fbfa1ffb640 1 -- 172.21.15.77:0/1674539543 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fbf900cf0b0 con 0x7fbfa41018d0 2024-03-16T10:18:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.766+0000 7fbfa9506640 1 -- 172.21.15.77:0/1474575944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 msgr2=0x557e52e20480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.766+0000 7fbfa9506640 1 --2- 172.21.15.77:0/1474575944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e20480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.766+0000 7fbfa9506640 1 -- 172.21.15.77:0/1474575944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e20a50 msgr2=0x557e52e25550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.766+0000 7fbfa9506640 1 --2- 172.21.15.77:0/1474575944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e20a50 0x557e52e25550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.766+0000 7fbfa9506640 1 -- 172.21.15.77:0/1474575944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e52ddf500 con 0x557e52e25bc0 2024-03-16T10:18:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.766+0000 7fbfa9506640 1 --2- 172.21.15.77:0/1474575944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 0x557e52e28050 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c035000 tx=0x7fbf9c005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af39fbdf52ac438 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.767+0000 7fbfa8d05640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf940a2990 0x7fbf940a4e50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4101730 tx=0x7fbf98011860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.767+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/1474575944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e20a50 0x557e52e25550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:18:14.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.767+0000 7fbf8effd640 1 -- 172.21.15.77:0/1474575944 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c043070 con 0x557e52e25bc0 2024-03-16T10:18:14.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.767+0000 7fbf8effd640 1 -- 172.21.15.77:0/1474575944 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf9c033b90 con 0x557e52e25bc0 2024-03-16T10:18:14.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.768+0000 7fbf8effd640 1 -- 172.21.15.77:0/1474575944 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c02caf0 con 0x557e52e25bc0 2024-03-16T10:18:14.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.768+0000 7fbfadc32740 1 -- 172.21.15.77:0/1474575944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 msgr2=0x557e52e28050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.768+0000 7fbfadc32740 1 --2- 172.21.15.77:0/1474575944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 0x557e52e28050 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c035000 tx=0x7fbf9c005c70 comp rx=0 tx=0).stop 2024-03-16T10:18:14.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.768+0000 7fbfadc32740 1 -- 172.21.15.77:0/1474575944 shutdown_connections 2024-03-16T10:18:14.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.768+0000 7fbfadc32740 1 --2- 172.21.15.77:0/1474575944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 0x557e52e28050 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.768+0000 7fbfadc32740 1 --2- 172.21.15.77:0/1474575944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e20a50 0x557e52e25550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.768+0000 7fbfadc32740 1 --2- 172.21.15.77:0/1474575944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e20480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.768+0000 7fbfadc32740 1 -- 172.21.15.77:0/1474575944 >> 172.21.15.77:0/1474575944 conn(0x557e52e12dc0 msgr2=0x557e52e131c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.768+0000 7fbfadc32740 1 -- 172.21.15.77:0/1474575944 shutdown_connections 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.768+0000 7fbfadc32740 1 -- 172.21.15.77:0/1474575944 wait complete. 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.768+0000 7fbfadc32740 1 -- start start 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.768+0000 7fbfadc32740 1 Processor -- start 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfadc32740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e20060 0x557e52e2a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfadc32740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e20a50 0x557e52e2aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfadc32740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e25bc0 0x557e52e31a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfadc32740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557e52e28bb0 con 0x557e52e20a50 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfadc32740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x557e52d38230 con 0x557e52e20060 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfadc32740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557e52e28d30 con 0x557e52e25bc0 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfadc32740 10 client.0 _ll_register_callbacks cb 0x557e52d74010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:14.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfa9506640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e25bc0 0x557e52e31a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfa3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e20a50 0x557e52e2aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfa8d05640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e20060 0x557e52e2a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfa3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e20a50 0x557e52e2aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54494/0 (socket says 172.21.15.77:54494) 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfa3fff640 1 -- 172.21.15.77:0/4199418738 learned_addr learned my addr 172.21.15.77:0/4199418738 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfa8d05640 1 --2- 172.21.15.77:0/4199418738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e20060 0x557e52e2a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50662/0 (socket says 172.21.15.77:50662) 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfa3fff640 1 -- 172.21.15.77:0/4199418738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e25bc0 msgr2=0x557e52e31a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/4199418738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e25bc0 0x557e52e31a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfa3fff640 1 -- 172.21.15.77:0/4199418738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e20060 msgr2=0x557e52e2a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.769+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/4199418738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e20060 0x557e52e2a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.770+0000 7fbfa3fff640 1 -- 172.21.15.77:0/4199418738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e52d47100 con 0x557e52e20a50 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.770+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/4199418738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e20a50 0x557e52e2aa30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbf9009b5e0 tx=0x7fbf9009b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ceb64de30aa17ab server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.770+0000 7fbf83fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.770+0000 7fbf83fff640 1 -- 172.21.15.77:0/4199418738 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf900b4060 con 0x557e52e20a50 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.770+0000 7fbf83fff640 1 -- 172.21.15.77:0/4199418738 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fbf900d10b0 con 0x557e52e20a50 2024-03-16T10:18:14.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.770+0000 7fbfadc32740 1 -- 172.21.15.77:0/4199418738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf9c033b90 con 0x557e52e20a50 2024-03-16T10:18:14.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.770+0000 7fbf83fff640 1 -- 172.21.15.77:0/4199418738 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbf900d12b0 con 0x557e52e20a50 2024-03-16T10:18:14.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.770+0000 7fbfadc32740 10 client.15834 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:14.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.770+0000 7fbfadc32740 1 -- 172.21.15.77:0/4199418738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557e52a903f0 con 0x557e52e20a50 2024-03-16T10:18:14.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.770+0000 7fbf83fff640 1 -- 172.21.15.77:0/4199418738 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf9c033b90 con 0x557e52e20a50 2024-03-16T10:18:14.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.771+0000 7fbf83fff640 1 -- 172.21.15.77:0/4199418738 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbf9009ad30 con 0x557e52e20a50 2024-03-16T10:18:14.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.771+0000 7fbfadc32740 10 client.15834 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:14.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.771+0000 7fbfadc32740 10 client.15834 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:18:14.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.771+0000 7fbfadc32740 20 client.15834 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:14.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.771+0000 7fbfadc32740 1 --2- 172.21.15.77:0/4199418738 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x557e52e362c0 0x557e52e386d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.771+0000 7fbfadc32740 4 client.15834 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.772+0000 7fbfadc32740 1 -- 172.21.15.77:0/4199418738 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e52d899d0 con 0x557e52e362c0 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.772+0000 7fbfa8d05640 1 --2- 172.21.15.77:0/4199418738 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x557e52e362c0 0x557e52e386d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.772+0000 7fbf83fff640 10 client.15834 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.772+0000 7fbfa8d05640 1 --2- 172.21.15.77:0/4199418738 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x557e52e362c0 0x557e52e386d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=afecb280f9c7e7e server_cookie=74a9f269302dcee9 in_seq=0 out_seq=0 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbf83fff640 1 -- 172.21.15.77:0/4199418738 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557e52d899d0 con 0x557e52e362c0 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbf83fff640 10 client.15834 handle_command_reply: tid=0 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbfadc32740 1 client.15834 shutdown 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbfadc32740 1 -- 172.21.15.77:0/4199418738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e20a50 msgr2=0x557e52e2aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbfadc32740 1 --2- 172.21.15.77:0/4199418738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e20a50 0x557e52e2aa30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbf9009b5e0 tx=0x7fbf9009b330 comp rx=0 tx=0).stop 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbfadc32740 1 -- 172.21.15.77:0/4199418738 shutdown_connections 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbfadc32740 1 --2- 172.21.15.77:0/4199418738 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x557e52e362c0 0x557e52e386d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbfadc32740 1 --2- 172.21.15.77:0/4199418738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e25bc0 0x557e52e31a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbfadc32740 1 --2- 172.21.15.77:0/4199418738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e20a50 0x557e52e2aa30 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbfadc32740 1 --2- 172.21.15.77:0/4199418738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e20060 0x557e52e2a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbfadc32740 1 -- 172.21.15.77:0/4199418738 >> 172.21.15.77:0/4199418738 conn(0x557e52e12dc0 msgr2=0x557e52e11fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbf83fff640 0 client.15834 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:18:14.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbfadc32740 1 -- 172.21.15.77:0/4199418738 shutdown_connections 2024-03-16T10:18:14.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.773+0000 7fbfadc32740 1 -- 172.21.15.77:0/4199418738 wait complete. 2024-03-16T10:18:14.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.774+0000 7fbfadc32740 20 client.15834 trim_cache size 0 max 16384 2024-03-16T10:18:14.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.799+0000 7fbfadc32740 1 Processor -- start 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.799+0000 7fbfadc32740 1 -- start start 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.799+0000 7fbfadc32740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e29b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.799+0000 7fbfadc32740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e25bc0 0x557e52e31a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.799+0000 7fbfadc32740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e16730 0x557e52e3cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.799+0000 7fbfadc32740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa41a6890 con 0x557e52e16730 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.799+0000 7fbfadc32740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa41a83d0 con 0x557e52e25bc0 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.799+0000 7fbfadc32740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa41a6f60 con 0x557e52e20060 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.799+0000 7fbfa8d05640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e29b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.799+0000 7fbfa8d05640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e29b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33262/0 (socket says 172.21.15.77:33262) 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.799+0000 7fbfa8d05640 1 -- 172.21.15.77:0/215645596 learned_addr learned my addr 172.21.15.77:0/215645596 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.799+0000 7fbfa9506640 1 --2- 172.21.15.77:0/215645596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e16730 0x557e52e3cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.800+0000 7fbfa8d05640 1 -- 172.21.15.77:0/215645596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e25bc0 msgr2=0x557e52e31a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.800+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/215645596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e25bc0 0x557e52e31a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.800+0000 7fbfa8d05640 1 --2- 172.21.15.77:0/215645596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e25bc0 0x557e52e31a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.800+0000 7fbfa8d05640 1 -- 172.21.15.77:0/215645596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e16730 msgr2=0x557e52e3cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.800+0000 7fbfa8d05640 1 --2- 172.21.15.77:0/215645596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e16730 0x557e52e3cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.800+0000 7fbfa8d05640 1 -- 172.21.15.77:0/215645596 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e52b5a7d0 con 0x557e52e20060 2024-03-16T10:18:14.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.800+0000 7fbfa9506640 1 --2- 172.21.15.77:0/215645596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e16730 0x557e52e3cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:14.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.800+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/215645596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e25bc0 0x557e52e31a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:18:14.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.800+0000 7fbfa8d05640 1 --2- 172.21.15.77:0/215645596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e29b10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbf98005fd0 tx=0x7fbf9801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=836a87cd004b7a9a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.800+0000 7fbf8e7fc640 1 -- 172.21.15.77:0/215645596 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9802c070 con 0x557e52e20060 2024-03-16T10:18:14.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.800+0000 7fbf8e7fc640 1 -- 172.21.15.77:0/215645596 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf98019050 con 0x557e52e20060 2024-03-16T10:18:14.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.800+0000 7fbf8e7fc640 1 -- 172.21.15.77:0/215645596 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf98017030 con 0x557e52e20060 2024-03-16T10:18:14.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.801+0000 7fbfadc32740 1 -- 172.21.15.77:0/215645596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 msgr2=0x557e52e29b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.801+0000 7fbfadc32740 1 --2- 172.21.15.77:0/215645596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e29b10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbf98005fd0 tx=0x7fbf9801a000 comp rx=0 tx=0).stop 2024-03-16T10:18:14.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.801+0000 7fbfadc32740 1 -- 172.21.15.77:0/215645596 shutdown_connections 2024-03-16T10:18:14.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.801+0000 7fbfadc32740 1 --2- 172.21.15.77:0/215645596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e16730 0x557e52e3cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.801+0000 7fbfadc32740 1 --2- 172.21.15.77:0/215645596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e25bc0 0x557e52e31a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.801+0000 7fbfadc32740 1 --2- 172.21.15.77:0/215645596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e29b10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.801+0000 7fbfadc32740 1 -- 172.21.15.77:0/215645596 >> 172.21.15.77:0/215645596 conn(0x557e52e12dc0 msgr2=0x557e52e16020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.801+0000 7fbfadc32740 1 -- 172.21.15.77:0/215645596 shutdown_connections 2024-03-16T10:18:14.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.801+0000 7fbfadc32740 1 -- 172.21.15.77:0/215645596 wait complete. 2024-03-16T10:18:14.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.802+0000 7fbfadc32740 1 -- start start 2024-03-16T10:18:14.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.802+0000 7fbfadc32740 1 Processor -- start 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.802+0000 7fbfadc32740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e16730 0x557e52e31520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.802+0000 7fbfadc32740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e2fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.802+0000 7fbfadc32740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 0x557e52e30070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa8d05640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e16730 0x557e52e31520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e2fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa9506640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 0x557e52e30070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa9506640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 0x557e52e30070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54516/0 (socket says 172.21.15.77:54516) 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa8d05640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e16730 0x557e52e31520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50692/0 (socket says 172.21.15.77:50692) 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e2fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33264/0 (socket says 172.21.15.77:33264) 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa9506640 1 -- 172.21.15.77:0/446837434 learned_addr learned my addr 172.21.15.77:0/446837434 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfadc32740 1 -- 172.21.15.77:0/446837434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa41a7d00 con 0x557e52e25bc0 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfadc32740 1 -- 172.21.15.77:0/446837434 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa41a7630 con 0x557e52e16730 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfadc32740 1 -- 172.21.15.77:0/446837434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa41a4940 con 0x557e52e20060 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfadc32740 10 client.0 _ll_register_callbacks cb 0x7fbf90009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:14.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa8d05640 1 -- 172.21.15.77:0/446837434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 msgr2=0x557e52e2fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa8d05640 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e2fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa8d05640 1 -- 172.21.15.77:0/446837434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 msgr2=0x557e52e30070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa8d05640 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 0x557e52e30070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa8d05640 1 -- 172.21.15.77:0/446837434 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e52e305b0 con 0x557e52e16730 2024-03-16T10:18:14.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e2fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:18:14.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa9506640 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 0x557e52e30070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:14.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbfa8d05640 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e16730 0x557e52e31520 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbf98009320 tx=0x7fbf9800fec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4324f787535e84a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.803+0000 7fbf83fff640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:18:14.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbf83fff640 1 -- 172.21.15.77:0/446837434 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9802c070 con 0x557e52e16730 2024-03-16T10:18:14.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbf83fff640 1 -- 172.21.15.77:0/446837434 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fbf980047c0 con 0x557e52e16730 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbfadc32740 1 -- 172.21.15.77:0/446837434 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf98019050 con 0x557e52e16730 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbf83fff640 1 -- 172.21.15.77:0/446837434 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbf980055c0 con 0x557e52e16730 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbfadc32740 10 client.25634 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbfadc32740 1 -- 172.21.15.77:0/446837434 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557e52e9d1b0 con 0x557e52e16730 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbf83fff640 1 -- 172.21.15.77:0/446837434 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf98019050 con 0x557e52e16730 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbf83fff640 1 -- 172.21.15.77:0/446837434 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbf9803a420 con 0x557e52e16730 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbfadc32740 10 client.25634 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbfadc32740 10 client.25634 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbfadc32740 20 client.25634 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbfadc32740 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x557e52e9da30 0x557e52e9fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbfadc32740 4 client.25634 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.804+0000 7fbfadc32740 1 -- 172.21.15.77:0/446837434 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557e52e2a2e0 con 0x557e52e9da30 2024-03-16T10:18:14.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.805+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x557e52e9da30 0x557e52e9fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.805+0000 7fbf83fff640 10 client.25634 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.805+0000 7fbfa3fff640 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x557e52e9da30 0x557e52e9fe50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b18dedebf3c94d89 server_cookie=3649ebdf05120672 in_seq=0 out_seq=0 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.805+0000 7fbf83fff640 1 -- 172.21.15.77:0/446837434 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557e52e2a2e0 con 0x557e52e9da30 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.805+0000 7fbf83fff640 10 client.25634 handle_command_reply: tid=0 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.805+0000 7fbfadc32740 1 client.25634 shutdown 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.806+0000 7fbfadc32740 1 -- 172.21.15.77:0/446837434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e16730 msgr2=0x557e52e31520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.806+0000 7fbfadc32740 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e16730 0x557e52e31520 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbf98009320 tx=0x7fbf9800fec0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.806+0000 7fbfadc32740 1 -- 172.21.15.77:0/446837434 shutdown_connections 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.806+0000 7fbfadc32740 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x557e52e9da30 0x557e52e9fe50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.806+0000 7fbfadc32740 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557e52e25bc0 0x557e52e30070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.806+0000 7fbfadc32740 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e52e20060 0x557e52e2fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.806+0000 7fbfadc32740 1 --2- 172.21.15.77:0/446837434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557e52e16730 0x557e52e31520 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.806+0000 7fbfadc32740 1 -- 172.21.15.77:0/446837434 >> 172.21.15.77:0/446837434 conn(0x557e52e12dc0 msgr2=0x557e52e11750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.806+0000 7fbf83fff640 0 client.25634 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:18:14.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.806+0000 7fbfadc32740 1 -- 172.21.15.77:0/446837434 shutdown_connections 2024-03-16T10:18:14.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.806+0000 7fbfadc32740 1 -- 172.21.15.77:0/446837434 wait complete. 2024-03-16T10:18:14.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.806+0000 7fbfadc32740 20 client.25634 trim_cache size 0 max 16384 2024-03-16T10:18:14.813 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:18:14.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.807+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1674539543 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf940a2990 msgr2=0x7fbf940a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.807+0000 7fbfaaf69640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf940a2990 0x7fbf940a4e50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4101730 tx=0x7fbf98011860 comp rx=0 tx=0).stop 2024-03-16T10:18:14.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.807+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1674539543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa41018d0 msgr2=0x7fbfa4198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.807+0000 7fbfaaf69640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa41018d0 0x7fbfa4198010 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbf9000e9f0 tx=0x7fbf9000edf0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.807+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1674539543 shutdown_connections 2024-03-16T10:18:14.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.807+0000 7fbfaaf69640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbf940a2990 0x7fbf940a4e50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.807+0000 7fbfaaf69640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4102290 0x7fbfa419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.807+0000 7fbfaaf69640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfa41018d0 0x7fbfa4198010 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.808+0000 7fbfaaf69640 1 --2- 172.21.15.77:0/1674539543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbfa41006e0 0x7fbfa4197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.808+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1674539543 >> 172.21.15.77:0/1674539543 conn(0x7fbfa40fbc70 msgr2=0x7fbfa4104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.808+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1674539543 shutdown_connections 2024-03-16T10:18:14.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.808+0000 7fbfaaf69640 1 -- 172.21.15.77:0/1674539543 wait complete. 2024-03-16T10:18:14.821 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T10:18:14.872 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:14 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2138458961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:18:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.963+0000 7f47b4234640 1 -- 172.21.15.77:0/325946967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47ac107e20 msgr2=0x7f47ac108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.963+0000 7f47b4234640 1 --2- 172.21.15.77:0/325946967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47ac107e20 0x7f47ac108240 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009900 tx=0x7f47a802bc30 comp rx=0 tx=0).stop 2024-03-16T10:18:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.963+0000 7f47b4234640 1 -- 172.21.15.77:0/325946967 shutdown_connections 2024-03-16T10:18:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.963+0000 7f47b4234640 1 --2- 172.21.15.77:0/325946967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47ac109930 0x7f47ac110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.963+0000 7f47b4234640 1 --2- 172.21.15.77:0/325946967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47ac108f70 0x7f47ac1093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.963+0000 7f47b4234640 1 --2- 172.21.15.77:0/325946967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47ac107e20 0x7f47ac108240 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.963+0000 7f47b4234640 1 -- 172.21.15.77:0/325946967 >> 172.21.15.77:0/325946967 conn(0x7f47ac076db0 msgr2=0x7f47ac0791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.964+0000 7f47b4234640 1 -- 172.21.15.77:0/325946967 shutdown_connections 2024-03-16T10:18:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.964+0000 7f47b4234640 1 -- 172.21.15.77:0/325946967 wait complete. 2024-03-16T10:18:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.964+0000 7f47b4234640 1 Processor -- start 2024-03-16T10:18:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.964+0000 7f47b4234640 1 -- start start 2024-03-16T10:18:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b4234640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47ac107e20 0x7f47ac19e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b4234640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47ac108f70 0x7f47ac19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b4234640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47ac109930 0x7f47ac1a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b4234640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac113af0 con 0x7f47ac109930 2024-03-16T10:18:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b4234640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac113970 con 0x7f47ac108f70 2024-03-16T10:18:14.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b4234640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac113c70 con 0x7f47ac107e20 2024-03-16T10:18:14.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b1fd0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47ac107e20 0x7f47ac19e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b1fd0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47ac107e20 0x7f47ac19e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33276/0 (socket says 172.21.15.77:33276) 2024-03-16T10:18:14.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b1fd0640 1 -- 172.21.15.77:0/516474655 learned_addr learned my addr 172.21.15.77:0/516474655 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b27d1640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47ac109930 0x7f47ac1a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b17cf640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47ac108f70 0x7f47ac19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b1fd0640 1 -- 172.21.15.77:0/516474655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47ac108f70 msgr2=0x7f47ac19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b1fd0640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47ac108f70 0x7f47ac19e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b1fd0640 1 -- 172.21.15.77:0/516474655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47ac109930 msgr2=0x7f47ac1a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b1fd0640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47ac109930 0x7f47ac1a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b1fd0640 1 -- 172.21.15.77:0/516474655 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ac1a5d50 con 0x7f47ac107e20 2024-03-16T10:18:14.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b27d1640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47ac109930 0x7f47ac1a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.965+0000 7f47b17cf640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47ac108f70 0x7f47ac19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:18:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.966+0000 7f47b1fd0640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47ac107e20 0x7f47ac19e130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f47a8005ab0 tx=0x7f47a802c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.966+0000 7f47a2ffd640 1 -- 172.21.15.77:0/516474655 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8043020 con 0x7f47ac107e20 2024-03-16T10:18:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.966+0000 7f47a2ffd640 1 -- 172.21.15.77:0/516474655 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47a80054e0 con 0x7f47ac107e20 2024-03-16T10:18:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.966+0000 7f47a2ffd640 1 -- 172.21.15.77:0/516474655 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8003c70 con 0x7f47ac107e20 2024-03-16T10:18:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.966+0000 7f47b4234640 1 -- 172.21.15.77:0/516474655 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47ac1a5fe0 con 0x7f47ac107e20 2024-03-16T10:18:14.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.966+0000 7f47b4234640 1 -- 172.21.15.77:0/516474655 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47ac1a63e0 con 0x7f47ac107e20 2024-03-16T10:18:14.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.969+0000 7f47a2ffd640 1 -- 172.21.15.77:0/516474655 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f47a8046020 con 0x7f47ac107e20 2024-03-16T10:18:14.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.969+0000 7f47a2ffd640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f47980a2910 0x7f47980a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.969+0000 7f47b6e1b740 1 Processor -- start 2024-03-16T10:18:14.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.969+0000 7f47a2ffd640 1 -- 172.21.15.77:0/516474655 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f47a80f0a80 con 0x7f47ac107e20 2024-03-16T10:18:14.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.969+0000 7f47b17cf640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f47980a2910 0x7f47980a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b6e1b740 1 -- start start 2024-03-16T10:18:14.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b6e1b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 0x55c732f5fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b6e1b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f60240 0x55c732f64d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b6e1b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f653b0 0x55c732f67840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b6e1b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c732ec4490 con 0x55c732f653b0 2024-03-16T10:18:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b6e1b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c732e6cc90 con 0x55c732f5f850 2024-03-16T10:18:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b6e1b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c732ec0530 con 0x55c732f60240 2024-03-16T10:18:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b27d1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f653b0 0x55c732f67840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b27d1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f653b0 0x55c732f67840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54546/0 (socket says 172.21.15.77:54546) 2024-03-16T10:18:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b27d1640 1 -- 172.21.15.77:0/1121979512 learned_addr learned my addr 172.21.15.77:0/1121979512 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b1fd0640 1 --2- 172.21.15.77:0/1121979512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 0x55c732f5fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b27d1640 1 -- 172.21.15.77:0/1121979512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f60240 msgr2=0x55c732f64d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:18:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b27d1640 1 --2- 172.21.15.77:0/1121979512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f60240 0x55c732f64d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b27d1640 1 -- 172.21.15.77:0/1121979512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 msgr2=0x55c732f5fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b27d1640 1 --2- 172.21.15.77:0/1121979512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 0x55c732f5fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b27d1640 1 -- 172.21.15.77:0/1121979512 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c732f27500 con 0x55c732f653b0 2024-03-16T10:18:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.970+0000 7f47b1fd0640 1 --2- 172.21.15.77:0/1121979512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 0x55c732f5fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:18:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.971+0000 7f47b27d1640 1 --2- 172.21.15.77:0/1121979512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f653b0 0x55c732f67840 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f47a4011790 tx=0x7f47a4011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc400353cc3812a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.971+0000 7f4797fff640 1 -- 172.21.15.77:0/1121979512 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a4004d20 con 0x55c732f653b0 2024-03-16T10:18:14.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.971+0000 7f4797fff640 1 -- 172.21.15.77:0/1121979512 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47a4003970 con 0x55c732f653b0 2024-03-16T10:18:14.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.972+0000 7f4797fff640 1 -- 172.21.15.77:0/1121979512 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a4004580 con 0x55c732f653b0 2024-03-16T10:18:14.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.972+0000 7f47b6e1b740 1 -- 172.21.15.77:0/1121979512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f653b0 msgr2=0x55c732f67840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.972+0000 7f47b17cf640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f47980a2910 0x7f47980a4dd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f479c005d00 tx=0x7f479c005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.972+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/1121979512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f653b0 0x55c732f67840 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f47a4011790 tx=0x7f47a4011c50 comp rx=0 tx=0).stop 2024-03-16T10:18:14.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.972+0000 7f47b6e1b740 1 -- 172.21.15.77:0/1121979512 shutdown_connections 2024-03-16T10:18:14.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.972+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/1121979512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f653b0 0x55c732f67840 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.972+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/1121979512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f60240 0x55c732f64d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.972+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/1121979512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 0x55c732f5fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.972+0000 7f47b6e1b740 1 -- 172.21.15.77:0/1121979512 >> 172.21.15.77:0/1121979512 conn(0x55c732f5adc0 msgr2=0x55c732f5b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.972+0000 7f47b6e1b740 1 -- 172.21.15.77:0/1121979512 shutdown_connections 2024-03-16T10:18:14.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.972+0000 7f47b6e1b740 1 -- 172.21.15.77:0/1121979512 wait complete. 2024-03-16T10:18:14.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.973+0000 7f47b6e1b740 1 -- start start 2024-03-16T10:18:14.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.973+0000 7f47b6e1b740 1 Processor -- start 2024-03-16T10:18:14.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.973+0000 7f47b6e1b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f5f850 0x55c732f69c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.973+0000 7f47b6e1b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 0x55c732f6a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.973+0000 7f47b17cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 0x55c732f6a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b17cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 0x55c732f6a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54560/0 (socket says 172.21.15.77:54560) 2024-03-16T10:18:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b17cf640 1 -- 172.21.15.77:0/862048436 learned_addr learned my addr 172.21.15.77:0/862048436 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b1fd0640 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f5f850 0x55c732f69c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f653b0 0x55c732f71160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b6e1b740 1 -- 172.21.15.77:0/862048436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c732f683b0 con 0x55c732f60240 2024-03-16T10:18:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b6e1b740 1 -- 172.21.15.77:0/862048436 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c732e80230 con 0x55c732f653b0 2024-03-16T10:18:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b6e1b740 1 -- 172.21.15.77:0/862048436 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c732f68530 con 0x55c732f5f850 2024-03-16T10:18:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b27d1640 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f653b0 0x55c732f71160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b6e1b740 10 client.0 _ll_register_callbacks cb 0x55c732ebc010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b17cf640 1 -- 172.21.15.77:0/862048436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f5f850 msgr2=0x55c732f69c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b17cf640 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f5f850 0x55c732f69c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b17cf640 1 -- 172.21.15.77:0/862048436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f653b0 msgr2=0x55c732f71160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b17cf640 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f653b0 0x55c732f71160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b17cf640 1 -- 172.21.15.77:0/862048436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c732e8f100 con 0x55c732f60240 2024-03-16T10:18:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b17cf640 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 0x55c732f6a190 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f479c011770 tx=0x7f479c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565c908554b77646 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47957fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.974+0000 7f47b6e1b740 1 -- 172.21.15.77:0/862048436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47a4003970 con 0x55c732f60240 2024-03-16T10:18:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.975+0000 7f47957fa640 1 -- 172.21.15.77:0/862048436 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c022030 con 0x55c732f60240 2024-03-16T10:18:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.975+0000 7f47957fa640 1 -- 172.21.15.77:0/862048436 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f479c0047a0 con 0x55c732f60240 2024-03-16T10:18:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.975+0000 7f47957fa640 1 -- 172.21.15.77:0/862048436 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f479c0049e0 con 0x55c732f60240 2024-03-16T10:18:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.975+0000 7f47b6e1b740 10 client.15852 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.975+0000 7f47b6e1b740 1 -- 172.21.15.77:0/862048436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55c732bd83f0 con 0x55c732f60240 2024-03-16T10:18:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.975+0000 7f47957fa640 1 -- 172.21.15.77:0/862048436 <== mon.0 v2:172.21.15.77:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f479c00f040 con 0x55c732f60240 2024-03-16T10:18:14.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.975+0000 7f47b6e1b740 10 client.15852 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:14.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.975+0000 7f47b6e1b740 10 client.15852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:14.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.975+0000 7f47b6e1b740 20 client.15852 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:14.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.975+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c732f756b0 0x55c732f77b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:14.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.975+0000 7f47b6e1b740 4 client.15852 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:14.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.975+0000 7f47b6e1b740 1 -- 172.21.15.77:0/862048436 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f47a4003970 con 0x55c732f756b0 2024-03-16T10:18:14.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.976+0000 7f47b1fd0640 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c732f756b0 0x55c732f77b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:14.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.976+0000 7f47b1fd0640 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c732f756b0 0x55c732f77b10 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cc7c4b7eb6a5a8b server_cookie=2c91ce1ffea0a644 in_seq=0 out_seq=0 2024-03-16T10:18:14.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.976+0000 7f47957fa640 10 client.15852 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:14.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47957fa640 1 -- 172.21.15.77:0/862048436 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f47a4003970 con 0x55c732f756b0 2024-03-16T10:18:14.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47957fa640 10 client.15852 handle_command_reply: tid=0 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47b6e1b740 1 client.15852 shutdown 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47b6e1b740 1 -- 172.21.15.77:0/862048436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 msgr2=0x55c732f6a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 0x55c732f6a190 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f479c011770 tx=0x7f479c0046c0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47b6e1b740 1 -- 172.21.15.77:0/862048436 shutdown_connections 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f653b0 0x55c732f71160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c732f756b0 0x55c732f77b10 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 0x55c732f6a190 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/862048436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f5f850 0x55c732f69c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47b6e1b740 1 -- 172.21.15.77:0/862048436 >> 172.21.15.77:0/862048436 conn(0x55c732f5adc0 msgr2=0x55c732f59fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47957fa640 0 client.15852 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47b6e1b740 1 -- 172.21.15.77:0/862048436 shutdown_connections 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47b6e1b740 1 -- 172.21.15.77:0/862048436 wait complete. 2024-03-16T10:18:14.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:14.977+0000 7f47b6e1b740 20 client.15852 trim_cache size 0 max 16384 2024-03-16T10:18:15.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.001+0000 7f47b6e1b740 1 Processor -- start 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.001+0000 7f47b6e1b740 1 -- start start 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.002+0000 7f47b6e1b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f5f850 0x55c732f55e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.002+0000 7f47b6e1b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f60240 0x55c732f71120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.002+0000 7f47b6e1b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f653b0 0x55c732f69270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.002+0000 7f47b6e1b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1af2a0 con 0x55c732f5f850 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.002+0000 7f47b1fd0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f5f850 0x55c732f55e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.002+0000 7f47b17cf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f60240 0x55c732f71120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.002+0000 7f47b1fd0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f5f850 0x55c732f55e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54574/0 (socket says 172.21.15.77:54574) 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.002+0000 7f47b1fd0640 1 -- 172.21.15.77:0/3121319966 learned_addr learned my addr 172.21.15.77:0/3121319966 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.002+0000 7f47b17cf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f60240 0x55c732f71120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50714/0 (socket says 172.21.15.77:50714) 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.002+0000 7f47b27d1640 1 --2- 172.21.15.77:0/3121319966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f653b0 0x55c732f69270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.002+0000 7f47b6e1b740 1 -- 172.21.15.77:0/3121319966 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1b0de0 con 0x55c732f60240 2024-03-16T10:18:15.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.002+0000 7f47b6e1b740 1 -- 172.21.15.77:0/3121319966 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1af970 con 0x55c732f653b0 2024-03-16T10:18:15.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.003+0000 7f47b17cf640 1 -- 172.21.15.77:0/3121319966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f653b0 msgr2=0x55c732f69270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:15.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.003+0000 7f47b17cf640 1 --2- 172.21.15.77:0/3121319966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f653b0 0x55c732f69270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.003+0000 7f47b17cf640 1 -- 172.21.15.77:0/3121319966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f5f850 msgr2=0x55c732f55e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:15.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.003+0000 7f47b17cf640 1 --2- 172.21.15.77:0/3121319966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f5f850 0x55c732f55e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.003+0000 7f47b17cf640 1 -- 172.21.15.77:0/3121319966 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c732ca27d0 con 0x55c732f60240 2024-03-16T10:18:15.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.003+0000 7f47b27d1640 1 --2- 172.21.15.77:0/3121319966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f653b0 0x55c732f69270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:15.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.003+0000 7f47b1fd0640 1 --2- 172.21.15.77:0/3121319966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f5f850 0x55c732f55e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:15.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.003+0000 7f47b17cf640 1 --2- 172.21.15.77:0/3121319966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f60240 0x55c732f71120 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f479c00ebc0 tx=0x7f479c01e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=937a2c68a60cb286 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:15.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.003+0000 7f47977fe640 1 -- 172.21.15.77:0/3121319966 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c022070 con 0x55c732f60240 2024-03-16T10:18:15.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.003+0000 7f47977fe640 1 -- 172.21.15.77:0/3121319966 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f479c023040 con 0x55c732f60240 2024-03-16T10:18:15.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.003+0000 7f47977fe640 1 -- 172.21.15.77:0/3121319966 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c027450 con 0x55c732f60240 2024-03-16T10:18:15.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.004+0000 7f47b6e1b740 1 -- 172.21.15.77:0/3121319966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f60240 msgr2=0x55c732f71120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:15.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.004+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/3121319966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f60240 0x55c732f71120 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f479c00ebc0 tx=0x7f479c01e000 comp rx=0 tx=0).stop 2024-03-16T10:18:15.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.004+0000 7f47b6e1b740 1 -- 172.21.15.77:0/3121319966 shutdown_connections 2024-03-16T10:18:15.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.004+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/3121319966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f653b0 0x55c732f69270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.004+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/3121319966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f60240 0x55c732f71120 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.004+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/3121319966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f5f850 0x55c732f55e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.004+0000 7f47b6e1b740 1 -- 172.21.15.77:0/3121319966 >> 172.21.15.77:0/3121319966 conn(0x55c732f5adc0 msgr2=0x55c732f5e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:15.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.004+0000 7f47b6e1b740 1 -- 172.21.15.77:0/3121319966 shutdown_connections 2024-03-16T10:18:15.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.004+0000 7f47b6e1b740 1 -- 172.21.15.77:0/3121319966 wait complete. 2024-03-16T10:18:15.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.005+0000 7f47b6e1b740 1 -- start start 2024-03-16T10:18:15.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.005+0000 7f47b6e1b740 1 Processor -- start 2024-03-16T10:18:15.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.005+0000 7f47b6e1b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 0x55c732f6ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:15.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b6e1b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 0x55c732f6efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:15.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b6e1b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f653b0 0x55c732f6ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:15.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b1fd0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 0x55c732f6ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:15.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b1fd0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 0x55c732f6ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50716/0 (socket says 172.21.15.77:50716) 2024-03-16T10:18:15.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b1fd0640 1 -- 172.21.15.77:0/80731620 learned_addr learned my addr 172.21.15.77:0/80731620 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:15.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b6e1b740 1 -- 172.21.15.77:0/80731620 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1b0710 con 0x55c732f60240 2024-03-16T10:18:15.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b6e1b740 1 -- 172.21.15.77:0/80731620 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1b0040 con 0x55c732f5f850 2024-03-16T10:18:15.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b6e1b740 1 -- 172.21.15.77:0/80731620 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac1ad280 con 0x55c732f653b0 2024-03-16T10:18:15.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b27d1640 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f653b0 0x55c732f6ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:15.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b17cf640 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 0x55c732f6efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:15.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b1fd0640 1 -- 172.21.15.77:0/80731620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f653b0 msgr2=0x55c732f6ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:15.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b1fd0640 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f653b0 0x55c732f6ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b1fd0640 1 -- 172.21.15.77:0/80731620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 msgr2=0x55c732f6efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:15.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b6e1b740 10 client.0 _ll_register_callbacks cb 0x7f479c00eef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:15.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b1fd0640 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 0x55c732f6efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b1fd0640 1 -- 172.21.15.77:0/80731620 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c732f705d0 con 0x55c732f5f850 2024-03-16T10:18:15.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.006+0000 7f47b6e1b740 1 -- 172.21.15.77:0/80731620 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f479c023040 con 0x55c732f5f850 2024-03-16T10:18:15.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47b17cf640 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 0x55c732f6efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:15.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47b1fd0640 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 0x55c732f6ea90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f47a80bd1c0 tx=0x7f47a80a8040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b21cba6f2fb774f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:15.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47957fa640 10 client.25646 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:18:15.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47957fa640 1 -- 172.21.15.77:0/80731620 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a80a6030 con 0x55c732f5f850 2024-03-16T10:18:15.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47957fa640 1 -- 172.21.15.77:0/80731620 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f47a80bb7c0 con 0x55c732f5f850 2024-03-16T10:18:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47957fa640 1 -- 172.21.15.77:0/80731620 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f47a80bd4c0 con 0x55c732f5f850 2024-03-16T10:18:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47b6e1b740 10 client.25646 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47b6e1b740 1 -- 172.21.15.77:0/80731620 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55c732fdba80 con 0x55c732f5f850 2024-03-16T10:18:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47957fa640 1 -- 172.21.15.77:0/80731620 <== mon.2 v2:172.21.15.192:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f47a80d6200 con 0x55c732f5f850 2024-03-16T10:18:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47b6e1b740 10 client.25646 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47b6e1b740 10 client.25646 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47b6e1b740 20 client.25646 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c732fdc290 0x55c732fde6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47b6e1b740 4 client.25646 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:18:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.007+0000 7f47b6e1b740 1 -- 172.21.15.77:0/80731620 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c732ed19d0 con 0x55c732fdc290 2024-03-16T10:18:15.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.008+0000 7f47b17cf640 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c732fdc290 0x55c732fde6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:15.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.008+0000 7f47957fa640 10 client.25646 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:15.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.008+0000 7f47b17cf640 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c732fdc290 0x55c732fde6f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8781d9139da010ad server_cookie=995a35317d4e1da7 in_seq=0 out_seq=0 2024-03-16T10:18:15.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.008+0000 7f47957fa640 1 -- 172.21.15.77:0/80731620 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c732ed19d0 con 0x55c732fdc290 2024-03-16T10:18:15.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.008+0000 7f47957fa640 10 client.25646 handle_command_reply: tid=0 2024-03-16T10:18:15.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.008+0000 7f47b6e1b740 1 client.25646 shutdown 2024-03-16T10:18:15.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.009+0000 7f47b6e1b740 1 -- 172.21.15.77:0/80731620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 msgr2=0x55c732f6ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:15.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.009+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 0x55c732f6ea90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f47a80bd1c0 tx=0x7f47a80a8040 comp rx=0 tx=0).stop 2024-03-16T10:18:15.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.009+0000 7f47b6e1b740 1 -- 172.21.15.77:0/80731620 shutdown_connections 2024-03-16T10:18:15.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.009+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c732f653b0 0x55c732f6ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.009+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c732fdc290 0x55c732fde6f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.009+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c732f60240 0x55c732f6efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.009+0000 7f47b6e1b740 1 --2- 172.21.15.77:0/80731620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c732f5f850 0x55c732f6ea90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.009+0000 7f47b6e1b740 1 -- 172.21.15.77:0/80731620 >> 172.21.15.77:0/80731620 conn(0x55c732f5adc0 msgr2=0x55c732f59750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:15.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.009+0000 7f47957fa640 0 client.25646 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:15.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.009+0000 7f47b6e1b740 1 -- 172.21.15.77:0/80731620 shutdown_connections 2024-03-16T10:18:15.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.009+0000 7f47b6e1b740 1 -- 172.21.15.77:0/80731620 wait complete. 2024-03-16T10:18:15.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.009+0000 7f47b6e1b740 20 client.25646 trim_cache size 0 max 16384 2024-03-16T10:18:15.016 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:18:15.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.010+0000 7f47b4234640 1 -- 172.21.15.77:0/516474655 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f47980a2910 msgr2=0x7f47980a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.010+0000 7f47b4234640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f47980a2910 0x7f47980a4dd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f479c005d00 tx=0x7f479c005ef0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.010+0000 7f47b4234640 1 -- 172.21.15.77:0/516474655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47ac107e20 msgr2=0x7f47ac19e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.010+0000 7f47b4234640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47ac107e20 0x7f47ac19e130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f47a8005ab0 tx=0x7f47a802c6c0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.010+0000 7f47b4234640 1 -- 172.21.15.77:0/516474655 shutdown_connections 2024-03-16T10:18:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.010+0000 7f47b4234640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47ac109930 0x7f47ac1a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.010+0000 7f47b4234640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47ac108f70 0x7f47ac19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.010+0000 7f47b4234640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f47980a2910 0x7f47980a4dd0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.010+0000 7f47b4234640 1 --2- 172.21.15.77:0/516474655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47ac107e20 0x7f47ac19e130 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:15.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.010+0000 7f47b4234640 1 -- 172.21.15.77:0/516474655 >> 172.21.15.77:0/516474655 conn(0x7f47ac076db0 msgr2=0x7f47ac10c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:15.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.010+0000 7f47b4234640 1 -- 172.21.15.77:0/516474655 shutdown_connections 2024-03-16T10:18:15.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:15.010+0000 7f47b4234640 1 -- 172.21.15.77:0/516474655 wait complete. 2024-03-16T10:18:15.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:14 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2138458961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:18:15.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:14 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2138458961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:18:16.028 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T10:18:16.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:18:16.028 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T10:18:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.104+0000 7fc1b570a640 1 -- 172.21.15.77:0/266965799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b01033f0 msgr2=0x7fc1b0109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.104+0000 7fc1b570a640 1 --2- 172.21.15.77:0/266965799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b01033f0 0x7fc1b0109c80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4009900 tx=0x7fc1a402bc60 comp rx=0 tx=0).stop 2024-03-16T10:18:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.104+0000 7fc1b570a640 1 -- 172.21.15.77:0/266965799 shutdown_connections 2024-03-16T10:18:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.104+0000 7fc1b570a640 1 --2- 172.21.15.77:0/266965799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b01033f0 0x7fc1b0109c80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.104+0000 7fc1b570a640 1 --2- 172.21.15.77:0/266965799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1b0102a30 0x7fc1b0102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.104+0000 7fc1b570a640 1 --2- 172.21.15.77:0/266965799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1b0101840 0x7fc1b0101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.104+0000 7fc1b570a640 1 -- 172.21.15.77:0/266965799 >> 172.21.15.77:0/266965799 conn(0x7fc1b00fcdb0 msgr2=0x7fc1b00ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.104+0000 7fc1b570a640 1 -- 172.21.15.77:0/266965799 shutdown_connections 2024-03-16T10:18:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.104+0000 7fc1b570a640 1 -- 172.21.15.77:0/266965799 wait complete. 2024-03-16T10:18:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.105+0000 7fc1b570a640 1 Processor -- start 2024-03-16T10:18:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.105+0000 7fc1b570a640 1 -- start start 2024-03-16T10:18:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1b570a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1b0101840 0x7fc1b0075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1b570a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1b0102a30 0x7fc1b0073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1b570a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b01033f0 0x7fc1b0074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1b570a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b010d4f0 con 0x7fc1b0101840 2024-03-16T10:18:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1b570a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b010d370 con 0x7fc1b01033f0 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1b570a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b010d670 con 0x7fc1b0102a30 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1aeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1b0101840 0x7fc1b0075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1af7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b01033f0 0x7fc1b0074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1af7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b01033f0 0x7fc1b0074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50730/0 (socket says 172.21.15.77:50730) 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1af7fe640 1 -- 172.21.15.77:0/2174104422 learned_addr learned my addr 172.21.15.77:0/2174104422 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1b0102a30 0x7fc1b0073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1af7fe640 1 -- 172.21.15.77:0/2174104422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1b0102a30 msgr2=0x7fc1b0073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1af7fe640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1b0102a30 0x7fc1b0073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1af7fe640 1 -- 172.21.15.77:0/2174104422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1b0101840 msgr2=0x7fc1b0075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1af7fe640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1b0101840 0x7fc1b0075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.106+0000 7fc1af7fe640 1 -- 172.21.15.77:0/2174104422 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b00749a0 con 0x7fc1b01033f0 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.107+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1b0102a30 0x7fc1b0073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.107+0000 7fc1af7fe640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b01033f0 0x7fc1b0074110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc1a402c8d0 tx=0x7fc1a4004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.107+0000 7fc197fff640 1 -- 172.21.15.77:0/2174104422 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a4043020 con 0x7fc1b01033f0 2024-03-16T10:18:16.109 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.107+0000 7fc197fff640 1 -- 172.21.15.77:0/2174104422 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1a4004440 con 0x7fc1b01033f0 2024-03-16T10:18:16.109 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.107+0000 7fc197fff640 1 -- 172.21.15.77:0/2174104422 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a4003da0 con 0x7fc1b01033f0 2024-03-16T10:18:16.109 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.108+0000 7fc1b570a640 1 -- 172.21.15.77:0/2174104422 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1b01a1aa0 con 0x7fc1b01033f0 2024-03-16T10:18:16.109 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.108+0000 7fc1b570a640 1 -- 172.21.15.77:0/2174104422 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1b01a1fa0 con 0x7fc1b01033f0 2024-03-16T10:18:16.109 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.108+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1b0101840 0x7fc1b0075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:18:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.111+0000 7fc1b83cd740 1 Processor -- start 2024-03-16T10:18:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.111+0000 7fc197fff640 1 -- 172.21.15.77:0/2174104422 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7fc1a4046020 con 0x7fc1b01033f0 2024-03-16T10:18:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.111+0000 7fc1b83cd740 1 -- start start 2024-03-16T10:18:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.111+0000 7fc1b83cd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd44f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.111+0000 7fc1b83cd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 0x561aecd4a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.111+0000 7fc1b83cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd4a6b0 0x561aecd4cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.111+0000 7fc1b83cd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561aecca1a40 con 0x561aecd44b50 2024-03-16T10:18:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.111+0000 7fc1b83cd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561aecc4a920 con 0x561aecd45540 2024-03-16T10:18:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.111+0000 7fc1b83cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561aecc9dae0 con 0x561aecd4a6b0 2024-03-16T10:18:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc1aeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 0x561aecd4a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc1aeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 0x561aecd4a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50736/0 (socket says 172.21.15.77:50736) 2024-03-16T10:18:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc1aeffd640 1 -- 172.21.15.77:0/3633588090 learned_addr learned my addr 172.21.15.77:0/3633588090 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/3633588090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd44f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc197fff640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc1980a2a40 0x7fc1980a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/3633588090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd4a6b0 0x561aecd4cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc1aeffd640 1 -- 172.21.15.77:0/3633588090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd4a6b0 msgr2=0x561aecd4cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/3633588090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd4a6b0 0x561aecd4cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc1aeffd640 1 -- 172.21.15.77:0/3633588090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 msgr2=0x561aecd44f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/3633588090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd44f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc1aeffd640 1 -- 172.21.15.77:0/3633588090 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aecc62460 con 0x561aecd45540 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/3633588090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd44f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.112+0000 7fc197fff640 1 -- 172.21.15.77:0/2174104422 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc1a40f0790 con 0x7fc1b01033f0 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.113+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/3633588090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 0x561aecd4a040 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8008ef0 tx=0x7fc1a800d710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6321f9270a873a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.113+0000 7fc1af7fe640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc1980a2a40 0x7fc1980a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.113+0000 7fc1957fa640 1 -- 172.21.15.77:0/3633588090 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8003b70 con 0x561aecd45540 2024-03-16T10:18:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.113+0000 7fc1957fa640 1 -- 172.21.15.77:0/3633588090 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1a8003d10 con 0x561aecd45540 2024-03-16T10:18:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.113+0000 7fc1957fa640 1 -- 172.21.15.77:0/3633588090 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800fbe0 con 0x561aecd45540 2024-03-16T10:18:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 -- 172.21.15.77:0/3633588090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 msgr2=0x561aecd4a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1af7fe640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc1980a2a40 0x7fc1980a4f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4005ab0 tx=0x7fc1a40dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/3633588090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 0x561aecd4a040 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8008ef0 tx=0x7fc1a800d710 comp rx=0 tx=0).stop 2024-03-16T10:18:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 -- 172.21.15.77:0/3633588090 shutdown_connections 2024-03-16T10:18:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/3633588090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd4a6b0 0x561aecd4cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/3633588090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 0x561aecd4a040 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/3633588090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd44f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 -- 172.21.15.77:0/3633588090 >> 172.21.15.77:0/3633588090 conn(0x561aecd400c0 msgr2=0x561aecd404c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 -- 172.21.15.77:0/3633588090 shutdown_connections 2024-03-16T10:18:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 -- 172.21.15.77:0/3633588090 wait complete. 2024-03-16T10:18:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 -- start start 2024-03-16T10:18:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 Processor -- start 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd4ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd45540 0x561aecd4f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd4a6b0 0x561aecd564a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561aecd4d700 con 0x561aecd44b50 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561aecc5dd70 con 0x561aecd4a6b0 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.114+0000 7fc1b83cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561aecd4d880 con 0x561aecd45540 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1b83cd740 10 client.0 _ll_register_callbacks cb 0x561aec9b5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1ae7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd4ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1ae7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd4ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54634/0 (socket says 172.21.15.77:54634) 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1ae7fc640 1 -- 172.21.15.77:0/2580565450 learned_addr learned my addr 172.21.15.77:0/2580565450 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1aeffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd45540 0x561aecd4f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd4a6b0 0x561aecd564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1aeffd640 1 -- 172.21.15.77:0/2580565450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd4a6b0 msgr2=0x561aecd564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd4a6b0 0x561aecd564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1aeffd640 1 -- 172.21.15.77:0/2580565450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 msgr2=0x561aecd4ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd4ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1aeffd640 1 -- 172.21.15.77:0/2580565450 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aec9b6210 con 0x561aecd45540 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd4a6b0 0x561aecd564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd45540 0x561aecd4f460 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x561aecd41030 tx=0x7fc1a8008c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c102455bf429356a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc192ff5640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc192ff5640 1 -- 172.21.15.77:0/2580565450 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800fbf0 con 0x561aecd45540 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2580565450 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1a8003d10 con 0x561aecd45540 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.115+0000 7fc192ff5640 1 -- 172.21.15.77:0/2580565450 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc1a800fd90 con 0x561aecd45540 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.116+0000 7fc192ff5640 1 -- 172.21.15.77:0/2580565450 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc1a8003ec0 con 0x561aecd45540 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.116+0000 7fc1b83cd740 10 client.25645 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.116+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2580565450 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561aeca82620 con 0x561aecd45540 2024-03-16T10:18:16.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.116+0000 7fc192ff5640 1 -- 172.21.15.77:0/2580565450 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1a8003d10 con 0x561aecd45540 2024-03-16T10:18:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.116+0000 7fc192ff5640 1 -- 172.21.15.77:0/2580565450 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc1a802b050 con 0x561aecd45540 2024-03-16T10:18:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.116+0000 7fc1b83cd740 10 client.25645 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.116+0000 7fc1b83cd740 10 client.25645 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.116+0000 7fc1b83cd740 20 client.25645 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.116+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561aecd5aaa0 0x561aecd5ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.116+0000 7fc1b83cd740 4 client.25645 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.116+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2580565450 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561aeccaef80 con 0x561aecd5aaa0 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.116+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561aecd5aaa0 0x561aecd5ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc192ff5640 10 client.25645 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561aecd5aaa0 0x561aecd5ce90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d488fe2f89c167d server_cookie=f5f96bdb3c4e4e65 in_seq=0 out_seq=0 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc192ff5640 1 -- 172.21.15.77:0/2580565450 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561aeccaef80 con 0x561aecd5aaa0 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc192ff5640 10 client.25645 handle_command_reply: tid=0 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc1b83cd740 1 client.25645 shutdown 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2580565450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd45540 msgr2=0x561aecd4f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd45540 0x561aecd4f460 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x561aecd41030 tx=0x7fc1a8008c30 comp rx=0 tx=0).stop 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2580565450 shutdown_connections 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd4a6b0 0x561aecd564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd45540 0x561aecd4f460 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561aecd5aaa0 0x561aecd5ce90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/2580565450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd4ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc192ff5640 0 client.25645 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2580565450 >> 172.21.15.77:0/2580565450 conn(0x561aecd400c0 msgr2=0x561aecd36bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2580565450 shutdown_connections 2024-03-16T10:18:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.117+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2580565450 wait complete. 2024-03-16T10:18:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.118+0000 7fc1b83cd740 20 client.25645 trim_cache size 0 max 16384 2024-03-16T10:18:16.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.146+0000 7fc1b83cd740 1 Processor -- start 2024-03-16T10:18:16.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.146+0000 7fc1b83cd740 1 -- start start 2024-03-16T10:18:16.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.146+0000 7fc1b83cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd44b50 0x561aecd32a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.146+0000 7fc1b83cd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 0x561aecd56460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.146+0000 7fc1b83cd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd4a6b0 0x561aecd36320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.148 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.146+0000 7fc1b83cd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b01a68c0 con 0x561aecd4a6b0 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.146+0000 7fc1b83cd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b01a8400 con 0x561aecd45540 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.146+0000 7fc1b83cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b01a6f90 con 0x561aecd44b50 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1aeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 0x561aecd56460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1ae7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd44b50 0x561aecd32a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1aeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 0x561aecd56460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50750/0 (socket says 172.21.15.77:50750) 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1ae7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd44b50 0x561aecd32a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33342/0 (socket says 172.21.15.77:33342) 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1aeffd640 1 -- 172.21.15.77:0/2106600641 learned_addr learned my addr 172.21.15.77:0/2106600641 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/2106600641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd4a6b0 0x561aecd36320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1aeffd640 1 -- 172.21.15.77:0/2106600641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd44b50 msgr2=0x561aecd32a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/2106600641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd44b50 0x561aecd32a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1aeffd640 1 -- 172.21.15.77:0/2106600641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd4a6b0 msgr2=0x561aecd36320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/2106600641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd4a6b0 0x561aecd36320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1aeffd640 1 -- 172.21.15.77:0/2106600641 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aecd5d200 con 0x561aecd45540 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/2106600641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd44b50 0x561aecd32a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/2106600641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 0x561aecd56460 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8002230 tx=0x7fc1a800d710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d160358b41394415 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc194ff9640 1 -- 172.21.15.77:0/2106600641 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8003cf0 con 0x561aecd45540 2024-03-16T10:18:16.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc194ff9640 1 -- 172.21.15.77:0/2106600641 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1a8031020 con 0x561aecd45540 2024-03-16T10:18:16.150 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.147+0000 7fc194ff9640 1 -- 172.21.15.77:0/2106600641 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8004060 con 0x561aecd45540 2024-03-16T10:18:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.148+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2106600641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 msgr2=0x561aecd56460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.148+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/2106600641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 0x561aecd56460 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8002230 tx=0x7fc1a800d710 comp rx=0 tx=0).stop 2024-03-16T10:18:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.148+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2106600641 shutdown_connections 2024-03-16T10:18:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.148+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/2106600641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd4a6b0 0x561aecd36320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.148+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/2106600641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd45540 0x561aecd56460 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.148+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/2106600641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd44b50 0x561aecd32a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.148+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2106600641 >> 172.21.15.77:0/2106600641 conn(0x561aecd400c0 msgr2=0x561aecd404a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.148+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2106600641 shutdown_connections 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.148+0000 7fc1b83cd740 1 -- 172.21.15.77:0/2106600641 wait complete. 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.149+0000 7fc1b83cd740 1 -- start start 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.149+0000 7fc1b83cd740 1 Processor -- start 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.149+0000 7fc1b83cd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd539b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.149+0000 7fc1b83cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd45540 0x561aecd53ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.149+0000 7fc1ae7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.149+0000 7fc1ae7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54652/0 (socket says 172.21.15.77:54652) 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.149+0000 7fc1ae7fc640 1 -- 172.21.15.77:0/1696149734 learned_addr learned my addr 172.21.15.77:0/1696149734 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.149+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd4a6b0 0x561aecd54eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.149+0000 7fc1b83cd740 1 -- 172.21.15.77:0/1696149734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b01a7d30 con 0x561aecd44b50 2024-03-16T10:18:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.149+0000 7fc1b83cd740 1 -- 172.21.15.77:0/1696149734 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b01a7660 con 0x561aecd4a6b0 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.149+0000 7fc1b83cd740 1 -- 172.21.15.77:0/1696149734 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b01a48c0 con 0x561aecd45540 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc1b83cd740 10 client.0 _ll_register_callbacks cb 0x7fc1a8008d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd4a6b0 0x561aecd54eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd45540 0x561aecd53ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc1ae7fc640 1 -- 172.21.15.77:0/1696149734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd45540 msgr2=0x561aecd53ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd45540 0x561aecd53ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc1ae7fc640 1 -- 172.21.15.77:0/1696149734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd4a6b0 msgr2=0x561aecd54eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd4a6b0 0x561aecd54eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc1ae7fc640 1 -- 172.21.15.77:0/1696149734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aecd554a0 con 0x561aecd44b50 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc1ae7fc640 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd539b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0004700 tx=0x7fc1a8003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1dff3ace4a8d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc192ff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc192ff5640 1 -- 172.21.15.77:0/1696149734 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0004020 con 0x561aecd44b50 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc192ff5640 1 -- 172.21.15.77:0/1696149734 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc1a00041c0 con 0x561aecd44b50 2024-03-16T10:18:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.150+0000 7fc1b83cd740 1 -- 172.21.15.77:0/1696149734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1a8031020 con 0x561aecd44b50 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.151+0000 7fc192ff5640 1 -- 172.21.15.77:0/1696149734 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc1a00043c0 con 0x561aecd44b50 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.151+0000 7fc1b83cd740 10 client.15873 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.151+0000 7fc1b83cd740 1 -- 172.21.15.77:0/1696149734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561aecdc0dd0 con 0x561aecd44b50 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.151+0000 7fc192ff5640 1 -- 172.21.15.77:0/1696149734 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1a0005500 con 0x561aecd44b50 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.151+0000 7fc192ff5640 1 -- 172.21.15.77:0/1696149734 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc1a00035f0 con 0x561aecd44b50 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.151+0000 7fc1b83cd740 10 client.15873 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.151+0000 7fc1b83cd740 10 client.15873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.151+0000 7fc1b83cd740 20 client.15873 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.151+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561aecdc1590 0x561aecdc39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.151+0000 7fc1b83cd740 4 client.15873 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.151+0000 7fc1b83cd740 1 -- 172.21.15.77:0/1696149734 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fc1a8031020 con 0x561aecdc1590 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.151+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561aecdc1590 0x561aecdc39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.152+0000 7fc1aeffd640 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561aecdc1590 0x561aecdc39b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=770a97ef54d13c15 server_cookie=cc9aa55d4755c63e in_seq=0 out_seq=0 2024-03-16T10:18:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.152+0000 7fc192ff5640 10 client.15873 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:16.156 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:15 smithi193 ceph-mon[36311]: pgmap v385: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-16T10:18:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc192ff5640 1 -- 172.21.15.77:0/1696149734 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fc1a8031020 con 0x561aecdc1590 2024-03-16T10:18:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc192ff5640 10 client.15873 handle_command_reply: tid=0 2024-03-16T10:18:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc1b83cd740 1 client.15873 shutdown 2024-03-16T10:18:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc1b83cd740 1 -- 172.21.15.77:0/1696149734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 msgr2=0x561aecd539b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd539b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0004700 tx=0x7fc1a8003c80 comp rx=0 tx=0).stop 2024-03-16T10:18:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc1b83cd740 1 -- 172.21.15.77:0/1696149734 shutdown_connections 2024-03-16T10:18:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561aecd4a6b0 0x561aecd54eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561aecd45540 0x561aecd53ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561aecdc1590 0x561aecdc39b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc192ff5640 0 client.15873 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc1b83cd740 1 --2- 172.21.15.77:0/1696149734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561aecd44b50 0x561aecd539b0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc1b83cd740 1 -- 172.21.15.77:0/1696149734 >> 172.21.15.77:0/1696149734 conn(0x561aecd400c0 msgr2=0x561aecd32a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc1b83cd740 1 -- 172.21.15.77:0/1696149734 shutdown_connections 2024-03-16T10:18:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc1b83cd740 1 -- 172.21.15.77:0/1696149734 wait complete. 2024-03-16T10:18:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.155+0000 7fc1b83cd740 20 client.15873 trim_cache size 0 max 16384 2024-03-16T10:18:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:18:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:18:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:18:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:18:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.156+0000 7fc1b570a640 1 -- 172.21.15.77:0/2174104422 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc1980a2a40 msgr2=0x7fc1980a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.156+0000 7fc1b570a640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc1980a2a40 0x7fc1980a4f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4005ab0 tx=0x7fc1a40dd040 comp rx=0 tx=0).stop 2024-03-16T10:18:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.156+0000 7fc1b570a640 1 -- 172.21.15.77:0/2174104422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b01033f0 msgr2=0x7fc1b0074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.156+0000 7fc1b570a640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b01033f0 0x7fc1b0074110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc1a402c8d0 tx=0x7fc1a4004040 comp rx=0 tx=0).stop 2024-03-16T10:18:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.156+0000 7fc1b570a640 1 -- 172.21.15.77:0/2174104422 shutdown_connections 2024-03-16T10:18:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.156+0000 7fc1b570a640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b01033f0 0x7fc1b0074110 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.156+0000 7fc1b570a640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc1980a2a40 0x7fc1980a4f00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.156+0000 7fc1b570a640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1b0102a30 0x7fc1b0073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.156+0000 7fc1b570a640 1 --2- 172.21.15.77:0/2174104422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1b0101840 0x7fc1b0075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.156+0000 7fc1b570a640 1 -- 172.21.15.77:0/2174104422 >> 172.21.15.77:0/2174104422 conn(0x7fc1b00fcdb0 msgr2=0x7fc1b0105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.157+0000 7fc1b570a640 1 -- 172.21.15.77:0/2174104422 shutdown_connections 2024-03-16T10:18:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:16.157+0000 7fc1b570a640 1 -- 172.21.15.77:0/2174104422 wait complete. 2024-03-16T10:18:16.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:15 smithi192 ceph-mon[36354]: pgmap v385: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-16T10:18:16.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.189+0000 7f7e0f590640 1 -- 172.21.15.77:0/2709011878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e081018f0 msgr2=0x7f7e08101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.189+0000 7f7e0f590640 1 --2- 172.21.15.77:0/2709011878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e081018f0 0x7f7e08101d70 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009900 tx=0x7f7df802bc30 comp rx=0 tx=0).stop 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.189+0000 7f7e0f590640 1 -- 172.21.15.77:0/2709011878 shutdown_connections 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.189+0000 7f7e0f590640 1 --2- 172.21.15.77:0/2709011878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e081022b0 0x7f7e08108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.189+0000 7f7e0f590640 1 --2- 172.21.15.77:0/2709011878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e081018f0 0x7f7e08101d70 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.190+0000 7f7e0f590640 1 --2- 172.21.15.77:0/2709011878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e08100700 0x7f7e08100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.190+0000 7f7e0f590640 1 -- 172.21.15.77:0/2709011878 >> 172.21.15.77:0/2709011878 conn(0x7f7e080fbc90 msgr2=0x7f7e080fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.190+0000 7f7e0f590640 1 -- 172.21.15.77:0/2709011878 shutdown_connections 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.190+0000 7f7e0f590640 1 -- 172.21.15.77:0/2709011878 wait complete. 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.190+0000 7f7e0f590640 1 Processor -- start 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.190+0000 7f7e0f590640 1 -- start start 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0f590640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e08100700 0x7f7e08104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0f590640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e081018f0 0x7f7e08102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0f590640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e081022b0 0x7f7e08103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0f590640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0810c3b0 con 0x7f7e081022b0 2024-03-16T10:18:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0f590640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0810c230 con 0x7f7e081018f0 2024-03-16T10:18:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0f590640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0810c530 con 0x7f7e08100700 2024-03-16T10:18:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0cb2b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e081018f0 0x7f7e08102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0db2d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e081022b0 0x7f7e08103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0cb2b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e081018f0 0x7f7e08102cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50786/0 (socket says 172.21.15.77:50786) 2024-03-16T10:18:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0cb2b640 1 -- 172.21.15.77:0/2851616776 learned_addr learned my addr 172.21.15.77:0/2851616776 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0db2d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e081022b0 0x7f7e08103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54658/0 (socket says 172.21.15.77:54658) 2024-03-16T10:18:16.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0cb2b640 1 -- 172.21.15.77:0/2851616776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e08100700 msgr2=0x7f7e08104690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T10:18:16.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0cb2b640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e08100700 0x7f7e08104690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0cb2b640 1 -- 172.21.15.77:0/2851616776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e081022b0 msgr2=0x7f7e08103210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0cb2b640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e081022b0 0x7f7e08103210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0cb2b640 1 -- 172.21.15.77:0/2851616776 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e08103750 con 0x7f7e081018f0 2024-03-16T10:18:16.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0db2d640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e081022b0 0x7f7e08103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:16.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7e0cb2b640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e081018f0 0x7f7e08102cd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009850 tx=0x7f7df8005e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.191+0000 7f7dfe7fc640 1 -- 172.21.15.77:0/2851616776 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8043020 con 0x7f7e081018f0 2024-03-16T10:18:16.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.192+0000 7f7e0f590640 1 -- 172.21.15.77:0/2851616776 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e081039e0 con 0x7f7e081018f0 2024-03-16T10:18:16.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.192+0000 7f7dfe7fc640 1 -- 172.21.15.77:0/2851616776 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df80054e0 con 0x7f7e081018f0 2024-03-16T10:18:16.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.192+0000 7f7dfe7fc640 1 -- 172.21.15.77:0/2851616776 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df803e450 con 0x7f7e081018f0 2024-03-16T10:18:16.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.192+0000 7f7e0f590640 1 -- 172.21.15.77:0/2851616776 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e081a63f0 con 0x7f7e081018f0 2024-03-16T10:18:16.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.194+0000 7f7dfe7fc640 1 -- 172.21.15.77:0/2851616776 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f7df8046020 con 0x7f7e081018f0 2024-03-16T10:18:16.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.195+0000 7f7e1217f740 1 Processor -- start 2024-03-16T10:18:16.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.195+0000 7f7e1217f740 1 -- start start 2024-03-16T10:18:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.195+0000 7f7dfe7fc640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7df40a2910 0x7f7df40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.195+0000 7f7e1217f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 0x560d7dd28db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.195+0000 7f7e1217f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd29380 0x560d7dd2de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.195+0000 7f7e1217f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd2e4f0 0x560d7dd30980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.195+0000 7f7e1217f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560d7dc8d0d0 con 0x560d7dd28990 2024-03-16T10:18:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.195+0000 7f7e1217f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x560d7dc35c70 con 0x560d7dd29380 2024-03-16T10:18:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.195+0000 7f7e1217f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560d7dc89170 con 0x560d7dd2e4f0 2024-03-16T10:18:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.196+0000 7f7dfe7fc640 1 -- 172.21.15.77:0/2851616776 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7df80f0ea0 con 0x7f7e081018f0 2024-03-16T10:18:16.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.196+0000 7f7e0d32c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 0x560d7dd28db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.196+0000 7f7e0db2d640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7df40a2910 0x7f7df40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.196+0000 7f7e0cb2b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd29380 0x560d7dd2de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.196+0000 7f7e0d32c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 0x560d7dd28db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54666/0 (socket says 172.21.15.77:54666) 2024-03-16T10:18:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.196+0000 7f7e0d32c640 1 -- 172.21.15.77:0/3189811223 learned_addr learned my addr 172.21.15.77:0/3189811223 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.196+0000 7f7e0d32c640 1 -- 172.21.15.77:0/3189811223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd2e4f0 msgr2=0x560d7dd30980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.196+0000 7f7e0d32c640 1 --2- 172.21.15.77:0/3189811223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd2e4f0 0x560d7dd30980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.196+0000 7f7e0d32c640 1 -- 172.21.15.77:0/3189811223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd29380 msgr2=0x560d7dd2de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.196+0000 7f7e0d32c640 1 --2- 172.21.15.77:0/3189811223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd29380 0x560d7dd2de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.196+0000 7f7e0d32c640 1 -- 172.21.15.77:0/3189811223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d7da697e0 con 0x560d7dd28990 2024-03-16T10:18:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.196+0000 7f7e0db2d640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7df40a2910 0x7f7df40a4dd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7e00005fd0 tx=0x7f7e000098e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.197+0000 7f7e0d32c640 1 --2- 172.21.15.77:0/3189811223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 0x560d7dd28db0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400bad0 tx=0x7f7e0400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ebe746d38a3cc78 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.197+0000 7f7df37fe640 1 -- 172.21.15.77:0/3189811223 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04016070 con 0x560d7dd28990 2024-03-16T10:18:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.197+0000 7f7df37fe640 1 -- 172.21.15.77:0/3189811223 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e0400dca0 con 0x560d7dd28990 2024-03-16T10:18:16.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.197+0000 7f7df37fe640 1 -- 172.21.15.77:0/3189811223 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0400f7a0 con 0x560d7dd28990 2024-03-16T10:18:16.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.198+0000 7f7e1217f740 1 -- 172.21.15.77:0/3189811223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 msgr2=0x560d7dd28db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.198+0000 7f7e1217f740 1 --2- 172.21.15.77:0/3189811223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 0x560d7dd28db0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400bad0 tx=0x7f7e0400bf90 comp rx=0 tx=0).stop 2024-03-16T10:18:16.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.198+0000 7f7e1217f740 1 -- 172.21.15.77:0/3189811223 shutdown_connections 2024-03-16T10:18:16.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.198+0000 7f7e1217f740 1 --2- 172.21.15.77:0/3189811223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd2e4f0 0x560d7dd30980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.198+0000 7f7e1217f740 1 --2- 172.21.15.77:0/3189811223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd29380 0x560d7dd2de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.198+0000 7f7e1217f740 1 --2- 172.21.15.77:0/3189811223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 0x560d7dd28db0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.198+0000 7f7e1217f740 1 -- 172.21.15.77:0/3189811223 >> 172.21.15.77:0/3189811223 conn(0x560d7dcf0c50 msgr2=0x560d7dd24300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.198+0000 7f7e1217f740 1 -- 172.21.15.77:0/3189811223 shutdown_connections 2024-03-16T10:18:16.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.198+0000 7f7e1217f740 1 -- 172.21.15.77:0/3189811223 wait complete. 2024-03-16T10:18:16.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.198+0000 7f7e1217f740 1 -- start start 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.199+0000 7f7e1217f740 1 Processor -- start 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.199+0000 7f7e1217f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 0x560d7dd43ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.199+0000 7f7e1217f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd29380 0x560d7dd44430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.199+0000 7f7e1217f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd2e4f0 0x560d7dd4b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.199+0000 7f7e1217f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560d7dd42550 con 0x560d7dd29380 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.199+0000 7f7e0d32c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 0x560d7dd43ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.199+0000 7f7e1217f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x560d7dc490d0 con 0x560d7dd2e4f0 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.199+0000 7f7e1217f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560d7dd426d0 con 0x560d7dd28990 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.199+0000 7f7e0cb2b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd29380 0x560d7dd44430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.199+0000 7f7e0d32c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 0x560d7dd43ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33396/0 (socket says 172.21.15.77:33396) 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.199+0000 7f7e0d32c640 1 -- 172.21.15.77:0/1351694967 learned_addr learned my addr 172.21.15.77:0/1351694967 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.199+0000 7f7e1217f740 10 client.0 _ll_register_callbacks cb 0x560d7dc12760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.200+0000 7f7e0db2d640 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd2e4f0 0x560d7dd4b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.200+0000 7f7e0db2d640 1 -- 172.21.15.77:0/1351694967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 msgr2=0x560d7dd43ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.200+0000 7f7e0db2d640 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 0x560d7dd43ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.200+0000 7f7e0db2d640 1 -- 172.21.15.77:0/1351694967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd29380 msgr2=0x560d7dd44430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.200+0000 7f7e0db2d640 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd29380 0x560d7dd44430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.200+0000 7f7e0db2d640 1 -- 172.21.15.77:0/1351694967 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d7dac9f70 con 0x560d7dd2e4f0 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.200+0000 7f7e0cb2b640 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd29380 0x560d7dd44430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.200+0000 7f7e0db2d640 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd2e4f0 0x560d7dd4b400 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7e00013830 tx=0x7f7e00004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=672b07d2d772ac6a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.200+0000 7f7df0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.200+0000 7f7df0ff9640 1 -- 172.21.15.77:0/1351694967 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e00004510 con 0x560d7dd2e4f0 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.200+0000 7f7e1217f740 1 -- 172.21.15.77:0/1351694967 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e0400dca0 con 0x560d7dd2e4f0 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.200+0000 7f7df0ff9640 1 -- 172.21.15.77:0/1351694967 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7e000046b0 con 0x560d7dd2e4f0 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.201+0000 7f7df0ff9640 1 -- 172.21.15.77:0/1351694967 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7e00010430 con 0x560d7dd2e4f0 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.201+0000 7f7e1217f740 10 client.25673 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.201+0000 7f7e1217f740 1 -- 172.21.15.77:0/1351694967 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560d7d961170 con 0x560d7dd2e4f0 2024-03-16T10:18:16.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.201+0000 7f7df0ff9640 1 -- 172.21.15.77:0/1351694967 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e0400dca0 con 0x560d7dd2e4f0 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.201+0000 7f7df0ff9640 1 -- 172.21.15.77:0/1351694967 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7e0001d490 con 0x560d7dd2e4f0 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.201+0000 7f7e1217f740 10 client.25673 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.201+0000 7f7e1217f740 10 client.25673 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.201+0000 7f7e1217f740 20 client.25673 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.201+0000 7f7e1217f740 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560d7dd4f9e0 0x560d7dd51e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.201+0000 7f7e1217f740 4 client.25673 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.201+0000 7f7e1217f740 1 -- 172.21.15.77:0/1351694967 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d7dc9a610 con 0x560d7dd4f9e0 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.201+0000 7f7e0d32c640 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560d7dd4f9e0 0x560d7dd51e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.202+0000 7f7e0d32c640 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560d7dd4f9e0 0x560d7dd51e40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27995e3aed8e19f9 server_cookie=a9db9d7a7b13f055 in_seq=0 out_seq=0 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.202+0000 7f7df0ff9640 10 client.25673 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.202+0000 7f7df0ff9640 1 -- 172.21.15.77:0/1351694967 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d7dc9a610 con 0x560d7dd4f9e0 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.202+0000 7f7df0ff9640 10 client.25673 handle_command_reply: tid=0 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.202+0000 7f7e1217f740 1 client.25673 shutdown 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.202+0000 7f7e1217f740 1 -- 172.21.15.77:0/1351694967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd2e4f0 msgr2=0x560d7dd4b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.202+0000 7f7e1217f740 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd2e4f0 0x560d7dd4b400 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7e00013830 tx=0x7f7e00004ab0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.203+0000 7f7e1217f740 1 -- 172.21.15.77:0/1351694967 shutdown_connections 2024-03-16T10:18:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.203+0000 7f7e1217f740 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd2e4f0 0x560d7dd4b400 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.203+0000 7f7e1217f740 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560d7dd4f9e0 0x560d7dd51e40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.203+0000 7f7df0ff9640 0 client.25673 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.203+0000 7f7e1217f740 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd29380 0x560d7dd44430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.203+0000 7f7e1217f740 1 --2- 172.21.15.77:0/1351694967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 0x560d7dd43ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.203+0000 7f7e1217f740 1 -- 172.21.15.77:0/1351694967 >> 172.21.15.77:0/1351694967 conn(0x560d7dcf0c50 msgr2=0x560d7dd234e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.203+0000 7f7e1217f740 1 -- 172.21.15.77:0/1351694967 shutdown_connections 2024-03-16T10:18:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.203+0000 7f7e1217f740 1 -- 172.21.15.77:0/1351694967 wait complete. 2024-03-16T10:18:16.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.203+0000 7f7e1217f740 20 client.25673 trim_cache size 0 max 16384 2024-03-16T10:18:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.220+0000 7f7e1217f740 1 Processor -- start 2024-03-16T10:18:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.220+0000 7f7e1217f740 1 -- start start 2024-03-16T10:18:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.220+0000 7f7e1217f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 0x560d7dd23360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.220+0000 7f7e1217f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd29380 0x560d7dd238a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.220+0000 7f7e1217f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd2e4f0 0x560d7dd4b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.220+0000 7f7e0d32c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 0x560d7dd23360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.220+0000 7f7e1217f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e081a66e0 con 0x560d7dd28990 2024-03-16T10:18:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e0d32c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 0x560d7dd23360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54680/0 (socket says 172.21.15.77:54680) 2024-03-16T10:18:16.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e0d32c640 1 -- 172.21.15.77:0/2730339050 learned_addr learned my addr 172.21.15.77:0/2730339050 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e1217f740 1 -- 172.21.15.77:0/2730339050 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7e081a8200 con 0x560d7dd29380 2024-03-16T10:18:16.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e1217f740 1 -- 172.21.15.77:0/2730339050 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7e081a6d90 con 0x560d7dd2e4f0 2024-03-16T10:18:16.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e0cb2b640 1 --2- 172.21.15.77:0/2730339050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd29380 0x560d7dd238a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e0db2d640 1 --2- 172.21.15.77:0/2730339050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd2e4f0 0x560d7dd4b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e0d32c640 1 -- 172.21.15.77:0/2730339050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd2e4f0 msgr2=0x560d7dd4b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e0d32c640 1 --2- 172.21.15.77:0/2730339050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd2e4f0 0x560d7dd4b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e0d32c640 1 -- 172.21.15.77:0/2730339050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd29380 msgr2=0x560d7dd238a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e0d32c640 1 --2- 172.21.15.77:0/2730339050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd29380 0x560d7dd238a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e0d32c640 1 -- 172.21.15.77:0/2730339050 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d7dd29760 con 0x560d7dd28990 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e0db2d640 1 --2- 172.21.15.77:0/2730339050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd2e4f0 0x560d7dd4b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e0cb2b640 1 --2- 172.21.15.77:0/2730339050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd29380 0x560d7dd238a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7e0d32c640 1 --2- 172.21.15.77:0/2730339050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 0x560d7dd23360 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7e04000c00 tx=0x7f7e0400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5824625a18d9dd06 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7df2ffd640 1 -- 172.21.15.77:0/2730339050 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04016040 con 0x560d7dd28990 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7df2ffd640 1 -- 172.21.15.77:0/2730339050 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e0400dca0 con 0x560d7dd28990 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.221+0000 7f7df2ffd640 1 -- 172.21.15.77:0/2730339050 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04004680 con 0x560d7dd28990 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.222+0000 7f7e1217f740 1 -- 172.21.15.77:0/2730339050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 msgr2=0x560d7dd23360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.222+0000 7f7e1217f740 1 --2- 172.21.15.77:0/2730339050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 0x560d7dd23360 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7e04000c00 tx=0x7f7e0400b7c0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.222+0000 7f7e1217f740 1 -- 172.21.15.77:0/2730339050 shutdown_connections 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.222+0000 7f7e1217f740 1 --2- 172.21.15.77:0/2730339050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd2e4f0 0x560d7dd4b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.222+0000 7f7e1217f740 1 --2- 172.21.15.77:0/2730339050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd29380 0x560d7dd238a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.222+0000 7f7e1217f740 1 --2- 172.21.15.77:0/2730339050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd28990 0x560d7dd23360 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.222+0000 7f7e1217f740 1 -- 172.21.15.77:0/2730339050 >> 172.21.15.77:0/2730339050 conn(0x560d7dcf0c50 msgr2=0x560d7dd22c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.222+0000 7f7e1217f740 1 -- 172.21.15.77:0/2730339050 shutdown_connections 2024-03-16T10:18:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.222+0000 7f7e1217f740 1 -- 172.21.15.77:0/2730339050 wait complete. 2024-03-16T10:18:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.222+0000 7f7e1217f740 1 -- start start 2024-03-16T10:18:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e1217f740 1 Processor -- start 2024-03-16T10:18:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e1217f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 0x560d7dd489d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e1217f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd29380 0x560d7dd48f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e1217f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd2e4f0 0x560d7dd49f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e1217f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e081a7b30 con 0x560d7dd29380 2024-03-16T10:18:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e1217f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7e081a7460 con 0x560d7dd2e4f0 2024-03-16T10:18:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e1217f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7e08071b70 con 0x560d7dd28990 2024-03-16T10:18:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e1217f740 10 client.0 _ll_register_callbacks cb 0x7f7e0000f580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:16.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e0d32c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 0x560d7dd489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e0d32c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 0x560d7dd489d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33418/0 (socket says 172.21.15.77:33418) 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e0d32c640 1 -- 172.21.15.77:0/92983903 learned_addr learned my addr 172.21.15.77:0/92983903 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e0cb2b640 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd29380 0x560d7dd48f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e0db2d640 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd2e4f0 0x560d7dd49f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e0d32c640 1 -- 172.21.15.77:0/92983903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd2e4f0 msgr2=0x560d7dd49f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e0d32c640 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd2e4f0 0x560d7dd49f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e0d32c640 1 -- 172.21.15.77:0/92983903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd29380 msgr2=0x560d7dd48f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e0d32c640 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd29380 0x560d7dd48f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e0d32c640 1 -- 172.21.15.77:0/92983903 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d7dd4a510 con 0x560d7dd28990 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.223+0000 7f7e0cb2b640 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd29380 0x560d7dd48f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.224+0000 7f7e0d32c640 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 0x560d7dd489d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400b790 tx=0x7f7e0400dc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dcdcab10c35dc72 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.224+0000 7f7df0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.224+0000 7f7df0ff9640 1 -- 172.21.15.77:0/92983903 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04038020 con 0x560d7dd28990 2024-03-16T10:18:16.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.224+0000 7f7e1217f740 1 -- 172.21.15.77:0/92983903 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e0400dca0 con 0x560d7dd28990 2024-03-16T10:18:16.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.224+0000 7f7df0ff9640 1 -- 172.21.15.77:0/92983903 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7e04028ba0 con 0x560d7dd28990 2024-03-16T10:18:16.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.224+0000 7f7df0ff9640 1 -- 172.21.15.77:0/92983903 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7e04043050 con 0x560d7dd28990 2024-03-16T10:18:16.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.224+0000 7f7e1217f740 10 client.25663 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:16.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.224+0000 7f7e1217f740 1 -- 172.21.15.77:0/92983903 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560d7dd4b7c0 con 0x560d7dd28990 2024-03-16T10:18:16.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.224+0000 7f7df0ff9640 1 -- 172.21.15.77:0/92983903 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e0400dca0 con 0x560d7dd28990 2024-03-16T10:18:16.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.225+0000 7f7df0ff9640 1 -- 172.21.15.77:0/92983903 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7e04010a10 con 0x560d7dd28990 2024-03-16T10:18:16.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.225+0000 7f7e1217f740 10 client.25663 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:16.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.225+0000 7f7e1217f740 10 client.25663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:16.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.225+0000 7f7e1217f740 20 client.25663 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:16.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.225+0000 7f7e1217f740 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560d7dd4f9e0 0x560d7ddb8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.230 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.225+0000 7f7e1217f740 4 client.25663 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.225+0000 7f7e1217f740 1 -- 172.21.15.77:0/92983903 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560d7dd245f0 con 0x560d7dd4f9e0 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.225+0000 7f7e0cb2b640 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560d7dd4f9e0 0x560d7ddb8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.225+0000 7f7e0cb2b640 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560d7dd4f9e0 0x560d7ddb8320 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51913c62e5e377e2 server_cookie=4ebf71b23a864ad3 in_seq=0 out_seq=0 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.225+0000 7f7df0ff9640 10 client.25663 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.226+0000 7f7df0ff9640 1 -- 172.21.15.77:0/92983903 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560d7dd245f0 con 0x560d7dd4f9e0 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.226+0000 7f7df0ff9640 10 client.25663 handle_command_reply: tid=0 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.226+0000 7f7e1217f740 1 client.25663 shutdown 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e1217f740 1 -- 172.21.15.77:0/92983903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 msgr2=0x560d7dd489d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e1217f740 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 0x560d7dd489d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400b790 tx=0x7f7e0400dc00 comp rx=0 tx=0).stop 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e1217f740 1 -- 172.21.15.77:0/92983903 shutdown_connections 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e1217f740 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560d7dd2e4f0 0x560d7dd49f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e1217f740 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560d7dd4f9e0 0x560d7ddb8320 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e1217f740 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560d7dd29380 0x560d7dd48f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e1217f740 1 --2- 172.21.15.77:0/92983903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d7dd28990 0x560d7dd489d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e1217f740 1 -- 172.21.15.77:0/92983903 >> 172.21.15.77:0/92983903 conn(0x560d7dcf0c50 msgr2=0x560d7dd234e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7df0ff9640 0 client.25663 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:16.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e1217f740 1 -- 172.21.15.77:0/92983903 shutdown_connections 2024-03-16T10:18:16.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e1217f740 1 -- 172.21.15.77:0/92983903 wait complete. 2024-03-16T10:18:16.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e1217f740 20 client.25663 trim_cache size 0 max 16384 2024-03-16T10:18:16.232 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:18:16.232 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T10:18:16.232 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "0d67ab24-63e9-4452-a4d3-cefbddd66f0b", 2024-03-16T10:18:16.232 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T10:18:16.232 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:18:16.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e0f590640 1 -- 172.21.15.77:0/2851616776 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7df40a2910 msgr2=0x7f7df40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.227+0000 7f7e0f590640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7df40a2910 0x7f7df40a4dd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7e00005fd0 tx=0x7f7e000098e0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.228+0000 7f7e0f590640 1 -- 172.21.15.77:0/2851616776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e081018f0 msgr2=0x7f7e08102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.228+0000 7f7e0f590640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e081018f0 0x7f7e08102cd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009850 tx=0x7f7df8005e10 comp rx=0 tx=0).stop 2024-03-16T10:18:16.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.228+0000 7f7e0f590640 1 -- 172.21.15.77:0/2851616776 shutdown_connections 2024-03-16T10:18:16.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.228+0000 7f7e0f590640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e081022b0 0x7f7e08103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.228+0000 7f7e0f590640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e081018f0 0x7f7e08102cd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.228+0000 7f7e0f590640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7df40a2910 0x7f7df40a4dd0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.228+0000 7f7e0f590640 1 --2- 172.21.15.77:0/2851616776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e08100700 0x7f7e08104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.228+0000 7f7e0f590640 1 -- 172.21.15.77:0/2851616776 >> 172.21.15.77:0/2851616776 conn(0x7f7e080fbc90 msgr2=0x7f7e080fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.228+0000 7f7e0f590640 1 -- 172.21.15.77:0/2851616776 shutdown_connections 2024-03-16T10:18:16.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.229+0000 7f7e0f590640 1 -- 172.21.15.77:0/2851616776 wait complete. 2024-03-16T10:18:16.239 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:15 smithi077 ceph-mon[32400]: pgmap v385: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-16T10:18:16.241 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:18:16.241 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:18:16.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.409+0000 7f2c380dc640 1 -- 172.21.15.77:0/4055662478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c301029c0 msgr2=0x7f2c30104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.409+0000 7f2c380dc640 1 --2- 172.21.15.77:0/4055662478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c301029c0 0x7f2c30104e30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009900 tx=0x7f2c2002bc30 comp rx=0 tx=0).stop 2024-03-16T10:18:16.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.409+0000 7f2c380dc640 1 -- 172.21.15.77:0/4055662478 shutdown_connections 2024-03-16T10:18:16.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.409+0000 7f2c380dc640 1 --2- 172.21.15.77:0/4055662478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c30105370 0x7f2c301057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.409+0000 7f2c380dc640 1 --2- 172.21.15.77:0/4055662478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c301029c0 0x7f2c30104e30 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.409+0000 7f2c380dc640 1 --2- 172.21.15.77:0/4055662478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c30100050 0x7f2c30102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.409+0000 7f2c380dc640 1 -- 172.21.15.77:0/4055662478 >> 172.21.15.77:0/4055662478 conn(0x7f2c300fbc70 msgr2=0x7f2c300fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.410+0000 7f2c380dc640 1 -- 172.21.15.77:0/4055662478 shutdown_connections 2024-03-16T10:18:16.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.410+0000 7f2c380dc640 1 -- 172.21.15.77:0/4055662478 wait complete. 2024-03-16T10:18:16.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.410+0000 7f2c380dc640 1 Processor -- start 2024-03-16T10:18:16.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.410+0000 7f2c380dc640 1 -- start start 2024-03-16T10:18:16.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.410+0000 7f2c380dc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c30100050 0x7f2c3019e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c380dc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c301029c0 0x7f2c3019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c380dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c30105370 0x7f2c301a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c380dc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c30113a30 con 0x7f2c30105370 2024-03-16T10:18:16.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c380dc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2c301138b0 con 0x7f2c30100050 2024-03-16T10:18:16.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c380dc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2c30113bb0 con 0x7f2c301029c0 2024-03-16T10:18:16.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c35e78640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c30100050 0x7f2c3019e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c35e78640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c30100050 0x7f2c3019e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50838/0 (socket says 172.21.15.77:50838) 2024-03-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c35e78640 1 -- 172.21.15.77:0/149071926 learned_addr learned my addr 172.21.15.77:0/149071926 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c35677640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c301029c0 0x7f2c3019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c36679640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c30105370 0x7f2c301a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c35e78640 1 -- 172.21.15.77:0/149071926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c301029c0 msgr2=0x7f2c3019e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c35e78640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c301029c0 0x7f2c3019e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c35e78640 1 -- 172.21.15.77:0/149071926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c30105370 msgr2=0x7f2c301a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c35e78640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c30105370 0x7f2c301a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c35e78640 1 -- 172.21.15.77:0/149071926 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c301a5d00 con 0x7f2c30100050 2024-03-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c35e78640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c30100050 0x7f2c3019e0e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c00e9f0 tx=0x7f2c2c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.411+0000 7f2c26ffd640 1 -- 172.21.15.77:0/149071926 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c00cd80 con 0x7f2c30100050 2024-03-16T10:18:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.412+0000 7f2c26ffd640 1 -- 172.21.15.77:0/149071926 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2c010910 con 0x7f2c30100050 2024-03-16T10:18:16.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.412+0000 7f2c380dc640 1 -- 172.21.15.77:0/149071926 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c301a5ff0 con 0x7f2c30100050 2024-03-16T10:18:16.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.412+0000 7f2c26ffd640 1 -- 172.21.15.77:0/149071926 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c0188f0 con 0x7f2c30100050 2024-03-16T10:18:16.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.412+0000 7f2c380dc640 1 -- 172.21.15.77:0/149071926 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c301a6450 con 0x7f2c30100050 2024-03-16T10:18:16.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.415+0000 7f2c26ffd640 1 -- 172.21.15.77:0/149071926 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f2c2c010ac0 con 0x7f2c30100050 2024-03-16T10:18:16.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.415+0000 7f2c3ad9e740 1 Processor -- start 2024-03-16T10:18:16.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.415+0000 7f2c3ad9e740 1 -- start start 2024-03-16T10:18:16.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.415+0000 7f2c3ad9e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334170 0x56375b334590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.415+0000 7f2c3ad9e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334b60 0x56375b339660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.415+0000 7f2c3ad9e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b339cd0 0x56375b33c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c3ad9e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56375b299500 con 0x56375b334b60 2024-03-16T10:18:16.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c3ad9e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56375b241be0 con 0x56375b334170 2024-03-16T10:18:16.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c3ad9e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56375b2955a0 con 0x56375b339cd0 2024-03-16T10:18:16.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c35677640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b339cd0 0x56375b33c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c35677640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b339cd0 0x56375b33c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33432/0 (socket says 172.21.15.77:33432) 2024-03-16T10:18:16.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c35677640 1 -- 172.21.15.77:0/1337427321 learned_addr learned my addr 172.21.15.77:0/1337427321 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c36679640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334170 0x56375b334590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c35e78640 1 --2- 172.21.15.77:0/1337427321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334b60 0x56375b339660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c26ffd640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2c1c0a2910 0x7f2c1c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c35677640 1 -- 172.21.15.77:0/1337427321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334170 msgr2=0x56375b334590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c35677640 1 --2- 172.21.15.77:0/1337427321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334170 0x56375b334590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c35677640 1 -- 172.21.15.77:0/1337427321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334b60 msgr2=0x56375b339660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c35677640 1 --2- 172.21.15.77:0/1337427321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334b60 0x56375b339660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c35677640 1 -- 172.21.15.77:0/1337427321 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56375b2554b0 con 0x56375b339cd0 2024-03-16T10:18:16.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c36679640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2c1c0a2910 0x7f2c1c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c26ffd640 1 -- 172.21.15.77:0/149071926 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2c2c0cef90 con 0x7f2c30100050 2024-03-16T10:18:16.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.416+0000 7f2c35677640 1 --2- 172.21.15.77:0/1337427321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b339cd0 0x56375b33c160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009900 tx=0x7f2c20009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3aa22f92f13ab8ef server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.417+0000 7f2c1bfff640 1 -- 172.21.15.77:0/1337427321 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2002cb80 con 0x56375b339cd0 2024-03-16T10:18:16.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.417+0000 7f2c1bfff640 1 -- 172.21.15.77:0/1337427321 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c20005040 con 0x56375b339cd0 2024-03-16T10:18:16.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.417+0000 7f2c1bfff640 1 -- 172.21.15.77:0/1337427321 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20004900 con 0x56375b339cd0 2024-03-16T10:18:16.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.418+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1337427321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b339cd0 msgr2=0x56375b33c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.418+0000 7f2c36679640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2c1c0a2910 0x7f2c1c0a4dd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009fd0 tx=0x7f2c28004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.418+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1337427321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b339cd0 0x56375b33c160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009900 tx=0x7f2c20009560 comp rx=0 tx=0).stop 2024-03-16T10:18:16.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.418+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1337427321 shutdown_connections 2024-03-16T10:18:16.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.418+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1337427321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b339cd0 0x56375b33c160 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.418+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1337427321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334b60 0x56375b339660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.418+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1337427321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334170 0x56375b334590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.418+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1337427321 >> 172.21.15.77:0/1337427321 conn(0x56375b32f6e0 msgr2=0x56375b32fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.418+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1337427321 shutdown_connections 2024-03-16T10:18:16.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.418+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1337427321 wait complete. 2024-03-16T10:18:16.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.418+0000 7f2c3ad9e740 1 -- start start 2024-03-16T10:18:16.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.418+0000 7f2c3ad9e740 1 Processor -- start 2024-03-16T10:18:16.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c3ad9e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 0x56375b33e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c3ad9e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334b60 0x56375b33eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c3ad9e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b339cd0 0x56375b345b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c3ad9e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56375b33cc20 con 0x56375b334170 2024-03-16T10:18:16.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c3ad9e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56375b255160 con 0x56375b334b60 2024-03-16T10:18:16.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c3ad9e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56375b33cda0 con 0x56375b339cd0 2024-03-16T10:18:16.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c35e78640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 0x56375b33e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c3ad9e740 10 client.0 _ll_register_callbacks cb 0x56375b239150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:16.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c35e78640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 0x56375b33e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54728/0 (socket says 172.21.15.77:54728) 2024-03-16T10:18:16.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c35e78640 1 -- 172.21.15.77:0/1041500761 learned_addr learned my addr 172.21.15.77:0/1041500761 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c35677640 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334b60 0x56375b33eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c36679640 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b339cd0 0x56375b345b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c35e78640 1 -- 172.21.15.77:0/1041500761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b339cd0 msgr2=0x56375b345b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c35e78640 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b339cd0 0x56375b345b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c35e78640 1 -- 172.21.15.77:0/1041500761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334b60 msgr2=0x56375b33eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c35e78640 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334b60 0x56375b33eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c35e78640 1 -- 172.21.15.77:0/1041500761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56375b0777d0 con 0x56375b334170 2024-03-16T10:18:16.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.419+0000 7f2c35677640 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334b60 0x56375b33eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:16.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c35e78640 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 0x56375b33e610 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c00c6a0 tx=0x7f2c2c099d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20004943e5935c22 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c197fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:16.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c197fa640 1 -- 172.21.15.77:0/1041500761 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c09b870 con 0x56375b334170 2024-03-16T10:18:16.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c197fa640 1 -- 172.21.15.77:0/1041500761 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2c2c0860f0 con 0x56375b334170 2024-03-16T10:18:16.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1041500761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c20005040 con 0x56375b334170 2024-03-16T10:18:16.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c197fa640 1 -- 172.21.15.77:0/1041500761 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2c2c09f910 con 0x56375b334170 2024-03-16T10:18:16.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c3ad9e740 10 client.15903 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:16.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1041500761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56375afad3f0 con 0x56375b334170 2024-03-16T10:18:16.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c197fa640 1 -- 172.21.15.77:0/1041500761 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c20005040 con 0x56375b334170 2024-03-16T10:18:16.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c197fa640 1 -- 172.21.15.77:0/1041500761 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2c2c018b50 con 0x56375b334170 2024-03-16T10:18:16.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c3ad9e740 10 client.15903 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:16.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c3ad9e740 10 client.15903 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:16.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c3ad9e740 20 client.15903 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:16.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56375b34a080 0x56375b34c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c3ad9e740 4 client.15903 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:16.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.420+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1041500761 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56375b2a6a40 con 0x56375b34a080 2024-03-16T10:18:16.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.421+0000 7f2c35677640 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56375b34a080 0x56375b34c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.421+0000 7f2c197fa640 10 client.15903 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:16.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.421+0000 7f2c35677640 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56375b34a080 0x56375b34c510 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e924897e16eef9b4 server_cookie=63d8e6e3dadad6ec in_seq=0 out_seq=0 2024-03-16T10:18:16.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c197fa640 1 -- 172.21.15.77:0/1041500761 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f2c200509c0 con 0x56375b34a080 2024-03-16T10:18:16.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c197fa640 10 client.15903 handle_command_reply: tid=0 2024-03-16T10:18:16.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c3ad9e740 1 client.15903 shutdown 2024-03-16T10:18:16.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1041500761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 msgr2=0x56375b33e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 0x56375b33e610 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c00c6a0 tx=0x7f2c2c099d80 comp rx=0 tx=0).stop 2024-03-16T10:18:16.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1041500761 shutdown_connections 2024-03-16T10:18:16.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b339cd0 0x56375b345b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334b60 0x56375b33eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56375b34a080 0x56375b34c510 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1041500761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 0x56375b33e610 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c197fa640 0 client.15903 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:16.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1041500761 >> 172.21.15.77:0/1041500761 conn(0x56375b32f6e0 msgr2=0x56375b32e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1041500761 shutdown_connections 2024-03-16T10:18:16.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1041500761 wait complete. 2024-03-16T10:18:16.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.422+0000 7f2c3ad9e740 20 client.15903 trim_cache size 0 max 16384 2024-03-16T10:18:16.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.452+0000 7f2c3ad9e740 1 Processor -- start 2024-03-16T10:18:16.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.452+0000 7f2c3ad9e740 1 -- start start 2024-03-16T10:18:16.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c3ad9e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334170 0x56375b32fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c3ad9e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b334b60 0x56375b345b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c3ad9e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b339cd0 0x56375b33e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c3ad9e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c301af0a0 con 0x56375b339cd0 2024-03-16T10:18:16.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c3ad9e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2c301b0be0 con 0x56375b334170 2024-03-16T10:18:16.456 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c3ad9e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2c301af770 con 0x56375b334b60 2024-03-16T10:18:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c36679640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b339cd0 0x56375b33e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c35677640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b334b60 0x56375b345b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c36679640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b339cd0 0x56375b33e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54742/0 (socket says 172.21.15.77:54742) 2024-03-16T10:18:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c36679640 1 -- 172.21.15.77:0/1225282562 learned_addr learned my addr 172.21.15.77:0/1225282562 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c35e78640 1 --2- 172.21.15.77:0/1225282562 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334170 0x56375b32fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c35677640 1 -- 172.21.15.77:0/1225282562 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334170 msgr2=0x56375b32fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c35677640 1 --2- 172.21.15.77:0/1225282562 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334170 0x56375b32fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c35677640 1 -- 172.21.15.77:0/1225282562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b339cd0 msgr2=0x56375b33e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c35677640 1 --2- 172.21.15.77:0/1225282562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b339cd0 0x56375b33e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c35677640 1 -- 172.21.15.77:0/1225282562 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56375af6d0f0 con 0x56375b334b60 2024-03-16T10:18:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c36679640 1 --2- 172.21.15.77:0/1225282562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b339cd0 0x56375b33e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.453+0000 7f2c35677640 1 --2- 172.21.15.77:0/1225282562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b334b60 0x56375b345b50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2c2002c7b0 tx=0x7f2c2002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28fd49ee760d17c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.454+0000 7f2c1b7fe640 1 -- 172.21.15.77:0/1225282562 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20005d50 con 0x56375b334b60 2024-03-16T10:18:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.454+0000 7f2c1b7fe640 1 -- 172.21.15.77:0/1225282562 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2004b070 con 0x56375b334b60 2024-03-16T10:18:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.454+0000 7f2c1b7fe640 1 -- 172.21.15.77:0/1225282562 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c200039e0 con 0x56375b334b60 2024-03-16T10:18:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.455+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1225282562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b334b60 msgr2=0x56375b345b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.455+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1225282562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b334b60 0x56375b345b50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2c2002c7b0 tx=0x7f2c2002bc30 comp rx=0 tx=0).stop 2024-03-16T10:18:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.455+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1225282562 shutdown_connections 2024-03-16T10:18:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.455+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1225282562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b339cd0 0x56375b33e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.455+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1225282562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b334b60 0x56375b345b50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.455+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/1225282562 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b334170 0x56375b32fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.455+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1225282562 >> 172.21.15.77:0/1225282562 conn(0x56375b32f6e0 msgr2=0x56375b33e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.455+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1225282562 shutdown_connections 2024-03-16T10:18:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.455+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/1225282562 wait complete. 2024-03-16T10:18:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.455+0000 7f2c3ad9e740 1 -- start start 2024-03-16T10:18:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.455+0000 7f2c3ad9e740 1 Processor -- start 2024-03-16T10:18:16.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c3ad9e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 0x56375b342b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c3ad9e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b334b60 0x56375b343090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c3ad9e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b339cd0 0x56375b3ae400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c35e78640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 0x56375b342b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c35e78640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 0x56375b342b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54756/0 (socket says 172.21.15.77:54756) 2024-03-16T10:18:16.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c35e78640 1 -- 172.21.15.77:0/3759660880 learned_addr learned my addr 172.21.15.77:0/3759660880 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:16.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/3759660880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c301b0510 con 0x56375b334170 2024-03-16T10:18:16.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c35677640 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b334b60 0x56375b343090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/3759660880 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2c301afe40 con 0x56375b339cd0 2024-03-16T10:18:16.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/3759660880 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2c301ad080 con 0x56375b334b60 2024-03-16T10:18:16.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c36679640 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b339cd0 0x56375b3ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c3ad9e740 10 client.0 _ll_register_callbacks cb 0x7f2c2c097450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:16.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c35e78640 1 -- 172.21.15.77:0/3759660880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b334b60 msgr2=0x56375b343090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c35e78640 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b334b60 0x56375b343090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c35e78640 1 -- 172.21.15.77:0/3759660880 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b339cd0 msgr2=0x56375b3ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:16.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c35e78640 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b339cd0 0x56375b3ae400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.456+0000 7f2c35e78640 1 -- 172.21.15.77:0/3759660880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56375b2609d0 con 0x56375b334170 2024-03-16T10:18:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.457+0000 7f2c35e78640 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 0x56375b342b50 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c09b570 tx=0x7f2c2c086320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac8e3f48abb7315 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.457+0000 7f2c197fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.457+0000 7f2c197fa640 1 -- 172.21.15.77:0/3759660880 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c09b3b0 con 0x56375b334170 2024-03-16T10:18:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.457+0000 7f2c197fa640 1 -- 172.21.15.77:0/3759660880 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2c2c0afdb0 con 0x56375b334170 2024-03-16T10:18:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.457+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/3759660880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c2004b070 con 0x56375b334170 2024-03-16T10:18:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.457+0000 7f2c197fa640 1 -- 172.21.15.77:0/3759660880 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2c2c09b870 con 0x56375b334170 2024-03-16T10:18:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.457+0000 7f2c3ad9e740 10 client.15909 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.457+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/3759660880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56375b3afad0 con 0x56375b334170 2024-03-16T10:18:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.458+0000 7f2c197fa640 1 -- 172.21.15.77:0/3759660880 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2004b070 con 0x56375b334170 2024-03-16T10:18:16.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.458+0000 7f2c197fa640 1 -- 172.21.15.77:0/3759660880 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2c2c0975c0 con 0x56375b334170 2024-03-16T10:18:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.458+0000 7f2c3ad9e740 10 client.15909 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.459+0000 7f2c3ad9e740 10 client.15909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.459+0000 7f2c3ad9e740 20 client.15909 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.459+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56375b3b02f0 0x56375b3b2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.459+0000 7f2c3ad9e740 4 client.15909 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:18:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.459+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/3759660880 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56375b2a0d90 con 0x56375b3b02f0 2024-03-16T10:18:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.459+0000 7f2c35677640 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56375b3b02f0 0x56375b3b2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.459+0000 7f2c197fa640 10 client.15909 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.459+0000 7f2c35677640 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56375b3b02f0 0x56375b3b2710 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86f9cd2358308b1c server_cookie=77da350e4e3b84e0 in_seq=0 out_seq=0 2024-03-16T10:18:16.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.459+0000 7f2c197fa640 1 -- 172.21.15.77:0/3759660880 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56375b2a0d90 con 0x56375b3b02f0 2024-03-16T10:18:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.459+0000 7f2c197fa640 10 client.15909 handle_command_reply: tid=0 2024-03-16T10:18:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c3ad9e740 1 client.15909 shutdown 2024-03-16T10:18:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/3759660880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 msgr2=0x56375b342b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 0x56375b342b50 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c09b570 tx=0x7f2c2c086320 comp rx=0 tx=0).stop 2024-03-16T10:18:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/3759660880 shutdown_connections 2024-03-16T10:18:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56375b339cd0 0x56375b3ae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56375b334b60 0x56375b343090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56375b3b02f0 0x56375b3b2710 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c3ad9e740 1 --2- 172.21.15.77:0/3759660880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56375b334170 0x56375b342b50 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c197fa640 0 client.15909 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/3759660880 >> 172.21.15.77:0/3759660880 conn(0x56375b32f6e0 msgr2=0x56375b332940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/3759660880 shutdown_connections 2024-03-16T10:18:16.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c3ad9e740 1 -- 172.21.15.77:0/3759660880 wait complete. 2024-03-16T10:18:16.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.460+0000 7f2c3ad9e740 20 client.15909 trim_cache size 0 max 16384 2024-03-16T10:18:16.469 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:18:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:18:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:18:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "0d67ab24-63e9-4452-a4d3-cefbddd66f0b": { 2024-03-16T10:18:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:18:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "0d67ab24-63e9-4452-a4d3-cefbddd66f0b", 2024-03-16T10:18:16.470 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:18:16.470 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:18:16.470 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:18:16.470 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:18:16.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.461+0000 7f2c380dc640 1 -- 172.21.15.77:0/149071926 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2c1c0a2910 msgr2=0x7f2c1c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.461+0000 7f2c380dc640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2c1c0a2910 0x7f2c1c0a4dd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009fd0 tx=0x7f2c28004a40 comp rx=0 tx=0).stop 2024-03-16T10:18:16.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.461+0000 7f2c380dc640 1 -- 172.21.15.77:0/149071926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c30100050 msgr2=0x7f2c3019e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:16.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.461+0000 7f2c380dc640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c30100050 0x7f2c3019e0e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c00e9f0 tx=0x7f2c2c00eeb0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.461+0000 7f2c380dc640 1 -- 172.21.15.77:0/149071926 shutdown_connections 2024-03-16T10:18:16.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.461+0000 7f2c380dc640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c30105370 0x7f2c301a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.461+0000 7f2c380dc640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2c1c0a2910 0x7f2c1c0a4dd0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.461+0000 7f2c380dc640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c301029c0 0x7f2c3019e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.461+0000 7f2c380dc640 1 --2- 172.21.15.77:0/149071926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c30100050 0x7f2c3019e0e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.461+0000 7f2c380dc640 1 -- 172.21.15.77:0/149071926 >> 172.21.15.77:0/149071926 conn(0x7f2c300fbc70 msgr2=0x7f2c301009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.461+0000 7f2c380dc640 1 -- 172.21.15.77:0/149071926 shutdown_connections 2024-03-16T10:18:16.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:16.461+0000 7f2c380dc640 1 -- 172.21.15.77:0/149071926 wait complete. 2024-03-16T10:18:16.473 INFO:tasks.cephfs.filesystem:scrub status for tag:0d67ab24-63e9-4452-a4d3-cefbddd66f0b - {'path': '/', 'tag': '0d67ab24-63e9-4452-a4d3-cefbddd66f0b', 'options': 'recursive,force'} 2024-03-16T10:18:18.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:17 smithi192 ceph-mon[36354]: scrub queued for path: / 2024-03-16T10:18:18.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:17 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:18.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:17 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:18:18.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:17 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:18.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:17 smithi192 ceph-mon[36354]: pgmap v386: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:18:18.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:17 smithi077 ceph-mon[32400]: scrub queued for path: / 2024-03-16T10:18:18.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:17 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:18.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:17 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:18:18.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:17 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:18.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:17 smithi077 ceph-mon[32400]: pgmap v386: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:18:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:17 smithi193 ceph-mon[36311]: scrub queued for path: / 2024-03-16T10:18:18.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:17 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:18.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:17 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:18:18.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:17 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:18.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:17 smithi193 ceph-mon[36311]: pgmap v386: 97 pgs: 97 active+clean; 671 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T10:18:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:19 smithi192 ceph-mon[36354]: pgmap v387: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-16T10:18:20.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:19 smithi077 ceph-mon[32400]: pgmap v387: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-16T10:18:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:19 smithi193 ceph-mon[36311]: pgmap v387: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-16T10:18:21.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:20 smithi192 ceph-mon[36354]: pgmap v388: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-16T10:18:21.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:20 smithi077 ceph-mon[32400]: pgmap v388: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-16T10:18:21.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:20 smithi193 ceph-mon[36311]: pgmap v388: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-16T10:18:22.146 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:21 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:18:22.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:21 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:18:22.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:21 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:18:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:18:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:22 smithi192 ceph-mon[36354]: pgmap v389: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-16T10:18:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:23.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:22 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:18:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:22 smithi077 ceph-mon[32400]: pgmap v389: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-16T10:18:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:22 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:23.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:18:23.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:22 smithi193 ceph-mon[36311]: pgmap v389: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-16T10:18:23.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:23.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:22 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:23 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:18:24.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:23 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:18:24.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:23 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:24.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:23 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:18:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:23 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:18:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:23 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:24.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:23 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:18:24.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:23 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:18:24.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:23 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:18:25.168 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:24 smithi192 ceph-mon[36354]: pgmap v390: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-16T10:18:25.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:24 smithi077 ceph-mon[32400]: pgmap v390: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-16T10:18:25.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:24 smithi193 ceph-mon[36311]: pgmap v390: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-16T10:18:28.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:27 smithi192 ceph-mon[36354]: pgmap v391: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-16T10:18:28.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:27 smithi077 ceph-mon[32400]: pgmap v391: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-16T10:18:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:27 smithi193 ceph-mon[36311]: pgmap v391: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-16T10:18:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.272+0000 7f8c821e1640 1 -- 172.21.15.77:0/1395948710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c100700 msgr2=0x7f8c7c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.272+0000 7f8c821e1640 1 --2- 172.21.15.77:0/1395948710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c100700 0x7f8c7c100b20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009900 tx=0x7f8c6c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:18:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.272+0000 7f8c821e1640 1 -- 172.21.15.77:0/1395948710 shutdown_connections 2024-03-16T10:18:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.272+0000 7f8c821e1640 1 --2- 172.21.15.77:0/1395948710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c7c1022b0 0x7f8c7c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.272+0000 7f8c821e1640 1 --2- 172.21.15.77:0/1395948710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c7c1018f0 0x7f8c7c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.272+0000 7f8c821e1640 1 --2- 172.21.15.77:0/1395948710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c100700 0x7f8c7c100b20 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.272+0000 7f8c821e1640 1 -- 172.21.15.77:0/1395948710 >> 172.21.15.77:0/1395948710 conn(0x7f8c7c0fbc70 msgr2=0x7f8c7c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.272+0000 7f8c821e1640 1 -- 172.21.15.77:0/1395948710 shutdown_connections 2024-03-16T10:18:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.272+0000 7f8c821e1640 1 -- 172.21.15.77:0/1395948710 wait complete. 2024-03-16T10:18:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c821e1640 1 Processor -- start 2024-03-16T10:18:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c821e1640 1 -- start start 2024-03-16T10:18:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c821e1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c7c1018f0 0x7f8c7c196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c821e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1022b0 0x7f8c7c197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c821e1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c7c19c160 0x7f8c7c19e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c821e1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c10c3c0 con 0x7f8c7c1022b0 2024-03-16T10:18:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c7affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1022b0 0x7f8c7c197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c7b7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c7c1018f0 0x7f8c7c196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c7affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1022b0 0x7f8c7c197100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52410/0 (socket says 172.21.15.77:52410) 2024-03-16T10:18:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c7affd640 1 -- 172.21.15.77:0/448430003 learned_addr learned my addr 172.21.15.77:0/448430003 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c7bfff640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c7c19c160 0x7f8c7c19e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c821e1640 1 -- 172.21.15.77:0/448430003 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c10c240 con 0x7f8c7c1018f0 2024-03-16T10:18:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.273+0000 7f8c821e1640 1 -- 172.21.15.77:0/448430003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c10c540 con 0x7f8c7c19c160 2024-03-16T10:18:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.274+0000 7f8c7affd640 1 -- 172.21.15.77:0/448430003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c7c19c160 msgr2=0x7f8c7c19e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.274+0000 7f8c7affd640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c7c19c160 0x7f8c7c19e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.274+0000 7f8c7affd640 1 -- 172.21.15.77:0/448430003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c7c1018f0 msgr2=0x7f8c7c196bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.274+0000 7f8c7affd640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c7c1018f0 0x7f8c7c196bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.274+0000 7f8c7affd640 1 -- 172.21.15.77:0/448430003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7c19ebc0 con 0x7f8c7c1022b0 2024-03-16T10:18:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.274+0000 7f8c7b7fe640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c7c1018f0 0x7f8c7c196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.274+0000 7f8c7affd640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1022b0 0x7f8c7c197100 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000e9f0 tx=0x7f8c7000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.274+0000 7f8c78ff9640 1 -- 172.21.15.77:0/448430003 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7000cd80 con 0x7f8c7c1022b0 2024-03-16T10:18:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.274+0000 7f8c78ff9640 1 -- 172.21.15.77:0/448430003 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c70005500 con 0x7f8c7c1022b0 2024-03-16T10:18:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.274+0000 7f8c78ff9640 1 -- 172.21.15.77:0/448430003 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c70010690 con 0x7f8c7c1022b0 2024-03-16T10:18:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.274+0000 7f8c821e1640 1 -- 172.21.15.77:0/448430003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7c19eeb0 con 0x7f8c7c1022b0 2024-03-16T10:18:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.274+0000 7f8c821e1640 1 -- 172.21.15.77:0/448430003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c7c19f320 con 0x7f8c7c1022b0 2024-03-16T10:18:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.277+0000 7f8c78ff9640 1 -- 172.21.15.77:0/448430003 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 137341+0+0 (secure 0 0 0) 0x7f8c700046c0 con 0x7f8c7c1022b0 2024-03-16T10:18:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.278+0000 7f8c84dc5740 1 Processor -- start 2024-03-16T10:18:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.278+0000 7f8c84dc5740 1 -- start start 2024-03-16T10:18:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.278+0000 7f8c84dc5740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d6550f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.278+0000 7f8c84dc5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d655630 0x56215d655ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.278+0000 7f8c84dc5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d6619f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.278+0000 7f8c84dc5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56215d5baa40 con 0x56215d655630 2024-03-16T10:18:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.278+0000 7f8c84dc5740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56215d563920 con 0x56215d6569a0 2024-03-16T10:18:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.278+0000 7f8c84dc5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56215d5b6ae0 con 0x56215d65f600 2024-03-16T10:18:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.278+0000 7f8c78ff9640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8c640a2940 0x7f8c640a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.278+0000 7f8c7b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d655630 0x56215d655ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.278+0000 7f8c7affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d6619f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.279+0000 7f8c7b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d655630 0x56215d655ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52414/0 (socket says 172.21.15.77:52414) 2024-03-16T10:18:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.279+0000 7f8c7b7fe640 1 -- 172.21.15.77:0/3559618181 learned_addr learned my addr 172.21.15.77:0/3559618181 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.279+0000 7f8c7bfff640 1 --2- 172.21.15.77:0/3559618181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d6550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.279+0000 7f8c78ff9640 1 -- 172.21.15.77:0/448430003 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8c700cef10 con 0x7f8c7c1022b0 2024-03-16T10:18:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.279+0000 7f8c7b7fe640 1 -- 172.21.15.77:0/3559618181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 msgr2=0x56215d6619f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.279+0000 7f8c7b7fe640 1 --2- 172.21.15.77:0/3559618181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d6619f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.279+0000 7f8c7b7fe640 1 -- 172.21.15.77:0/3559618181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 msgr2=0x56215d6550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.279+0000 7f8c7b7fe640 1 --2- 172.21.15.77:0/3559618181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d6550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.279+0000 7f8c7b7fe640 1 -- 172.21.15.77:0/3559618181 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56215d57b460 con 0x56215d655630 2024-03-16T10:18:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.279+0000 7f8c7bfff640 1 --2- 172.21.15.77:0/3559618181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d6550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.279+0000 7f8c7bfff640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8c640a2940 0x7f8c640a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.279+0000 7f8c7b7fe640 1 --2- 172.21.15.77:0/3559618181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d655630 0x56215d655ab0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c02c240 tx=0x7f8c6c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d36c1481ebbfdcef server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.280+0000 7f8c7bfff640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8c640a2940 0x7f8c640a4e00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x56215d656530 tx=0x7f8c74004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.280+0000 7f8c61ffb640 1 -- 172.21.15.77:0/3559618181 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c045070 con 0x56215d655630 2024-03-16T10:18:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.280+0000 7f8c61ffb640 1 -- 172.21.15.77:0/3559618181 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c6c02cd30 con 0x56215d655630 2024-03-16T10:18:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.280+0000 7f8c61ffb640 1 -- 172.21.15.77:0/3559618181 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c0050d0 con 0x56215d655630 2024-03-16T10:18:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.281+0000 7f8c84dc5740 1 -- 172.21.15.77:0/3559618181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d655630 msgr2=0x56215d655ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.281+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/3559618181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d655630 0x56215d655ab0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c02c240 tx=0x7f8c6c009560 comp rx=0 tx=0).stop 2024-03-16T10:18:29.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.281+0000 7f8c84dc5740 1 -- 172.21.15.77:0/3559618181 shutdown_connections 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.281+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/3559618181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d6619f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.281+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/3559618181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d655630 0x56215d655ab0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.281+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/3559618181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d6550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.281+0000 7f8c84dc5740 1 -- 172.21.15.77:0/3559618181 >> 172.21.15.77:0/3559618181 conn(0x56215d6508b0 msgr2=0x56215d650cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.281+0000 7f8c84dc5740 1 -- 172.21.15.77:0/3559618181 shutdown_connections 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.281+0000 7f8c84dc5740 1 -- 172.21.15.77:0/3559618181 wait complete. 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.282+0000 7f8c84dc5740 1 -- start start 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.282+0000 7f8c84dc5740 1 Processor -- start 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.282+0000 7f8c84dc5740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d65cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.282+0000 7f8c84dc5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d65d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.282+0000 7f8c84dc5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 0x56215d65e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.282+0000 7f8c84dc5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56215d6573b0 con 0x56215d65e2a0 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.282+0000 7f8c84dc5740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56215d576d70 con 0x56215d6569a0 2024-03-16T10:18:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.282+0000 7f8c84dc5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56215d657530 con 0x56215d65f600 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.282+0000 7f8c84dc5740 10 client.0 _ll_register_callbacks cb 0x56215d2cea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.282+0000 7f8c7b7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d65cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.282+0000 7f8c7affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d65d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c7b7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d65cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:48800/0 (socket says 172.21.15.77:48800) 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c7b7fe640 1 -- 172.21.15.77:0/599120327 learned_addr learned my addr 172.21.15.77:0/599120327 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c7affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d65d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:54340/0 (socket says 172.21.15.77:54340) 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c7bfff640 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 0x56215d65e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c7b7fe640 1 -- 172.21.15.77:0/599120327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 msgr2=0x56215d65d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c7b7fe640 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d65d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c7b7fe640 1 -- 172.21.15.77:0/599120327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 msgr2=0x56215d65e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c7b7fe640 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 0x56215d65e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c7b7fe640 1 -- 172.21.15.77:0/599120327 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56215d2cf210 con 0x56215d6569a0 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c7bfff640 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 0x56215d65e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c7b7fe640 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d65cd50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c006010 tx=0x7f8c6c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74edad436c5f6032 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c5affd640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c5affd640 1 -- 172.21.15.77:0/599120327 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c045050 con 0x56215d6569a0 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c84dc5740 1 -- 172.21.15.77:0/599120327 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c6c02cd30 con 0x56215d6569a0 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.283+0000 7f8c5affd640 1 -- 172.21.15.77:0/599120327 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8c6c005070 con 0x56215d6569a0 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.284+0000 7f8c5affd640 1 -- 172.21.15.77:0/599120327 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8c6c005270 con 0x56215d6569a0 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.284+0000 7f8c84dc5740 10 client.25709 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.284+0000 7f8c84dc5740 1 -- 172.21.15.77:0/599120327 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56215d39b620 con 0x56215d6569a0 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.284+0000 7f8c5affd640 1 -- 172.21.15.77:0/599120327 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c6c02cd30 con 0x56215d6569a0 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.284+0000 7f8c5affd640 1 -- 172.21.15.77:0/599120327 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8c6c003c90 con 0x56215d6569a0 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.284+0000 7f8c84dc5740 10 client.25709 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.284+0000 7f8c84dc5740 10 client.25709 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.284+0000 7f8c84dc5740 20 client.25709 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.284+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56215d673dc0 0x56215d6761b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.284+0000 7f8c84dc5740 4 client.25709 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:29.288 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.284+0000 7f8c84dc5740 1 -- 172.21.15.77:0/599120327 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56215d5c7f80 con 0x56215d673dc0 2024-03-16T10:18:29.289 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.284+0000 7f8c7affd640 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56215d673dc0 0x56215d6761b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.289 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.285+0000 7f8c5affd640 10 client.25709 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:29.289 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.285+0000 7f8c7affd640 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56215d673dc0 0x56215d6761b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d8515ca42a062f9 server_cookie=23e168beb7c2ac6 in_seq=0 out_seq=0 2024-03-16T10:18:29.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c5affd640 1 -- 172.21.15.77:0/599120327 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56215d5c7f80 con 0x56215d673dc0 2024-03-16T10:18:29.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c5affd640 10 client.25709 handle_command_reply: tid=0 2024-03-16T10:18:29.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c84dc5740 1 client.25709 shutdown 2024-03-16T10:18:29.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c84dc5740 1 -- 172.21.15.77:0/599120327 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 msgr2=0x56215d65cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:29.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d65cd50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c006010 tx=0x7f8c6c004270 comp rx=0 tx=0).stop 2024-03-16T10:18:29.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c84dc5740 1 -- 172.21.15.77:0/599120327 shutdown_connections 2024-03-16T10:18:29.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56215d673dc0 0x56215d6761b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 0x56215d65e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d65d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/599120327 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d65cd50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.291 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c84dc5740 1 -- 172.21.15.77:0/599120327 >> 172.21.15.77:0/599120327 conn(0x56215d6508b0 msgr2=0x56215d64fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:29.291 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c5affd640 0 client.25709 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:29.291 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c84dc5740 1 -- 172.21.15.77:0/599120327 shutdown_connections 2024-03-16T10:18:29.291 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c84dc5740 1 -- 172.21.15.77:0/599120327 wait complete. 2024-03-16T10:18:29.291 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.286+0000 7f8c84dc5740 20 client.25709 trim_cache size 0 max 16384 2024-03-16T10:18:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c84dc5740 1 Processor -- start 2024-03-16T10:18:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c84dc5740 1 -- start start 2024-03-16T10:18:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c84dc5740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d65bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c84dc5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 0x56215d654220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c84dc5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d65c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c84dc5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c1a6de0 con 0x56215d65e2a0 2024-03-16T10:18:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c84dc5740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c1a8920 con 0x56215d6569a0 2024-03-16T10:18:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c84dc5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c1a74b0 con 0x56215d65f600 2024-03-16T10:18:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c7affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 0x56215d654220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c7affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 0x56215d654220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52426/0 (socket says 172.21.15.77:52426) 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c7affd640 1 -- 172.21.15.77:0/3365732599 learned_addr learned my addr 172.21.15.77:0/3365732599 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c7b7fe640 1 --2- 172.21.15.77:0/3365732599 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d65bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.305+0000 7f8c7bfff640 1 --2- 172.21.15.77:0/3365732599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d65c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.306+0000 7f8c7affd640 1 -- 172.21.15.77:0/3365732599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 msgr2=0x56215d65c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.306+0000 7f8c7affd640 1 --2- 172.21.15.77:0/3365732599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d65c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.306+0000 7f8c7affd640 1 -- 172.21.15.77:0/3365732599 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 msgr2=0x56215d65bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.306+0000 7f8c7affd640 1 --2- 172.21.15.77:0/3365732599 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d65bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.306+0000 7f8c7affd640 1 -- 172.21.15.77:0/3365732599 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56215d65f430 con 0x56215d65e2a0 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.306+0000 7f8c7bfff640 1 --2- 172.21.15.77:0/3365732599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d65c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.306+0000 7f8c7affd640 1 --2- 172.21.15.77:0/3365732599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 0x56215d654220 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8c70004c00 tx=0x7f8c700962a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=323ed0f82b180686 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.306+0000 7f8c617fa640 1 -- 172.21.15.77:0/3365732599 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7009bf40 con 0x56215d65e2a0 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.306+0000 7f8c617fa640 1 -- 172.21.15.77:0/3365732599 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c7009b840 con 0x56215d65e2a0 2024-03-16T10:18:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.306+0000 7f8c617fa640 1 -- 172.21.15.77:0/3365732599 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7009cb70 con 0x56215d65e2a0 2024-03-16T10:18:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.307+0000 7f8c84dc5740 1 -- 172.21.15.77:0/3365732599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 msgr2=0x56215d654220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.307+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/3365732599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 0x56215d654220 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8c70004c00 tx=0x7f8c700962a0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.307+0000 7f8c84dc5740 1 -- 172.21.15.77:0/3365732599 shutdown_connections 2024-03-16T10:18:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.307+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/3365732599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d65f600 0x56215d65c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.307+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/3365732599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65e2a0 0x56215d654220 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.307+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/3365732599 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d6569a0 0x56215d65bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.307+0000 7f8c84dc5740 1 -- 172.21.15.77:0/3365732599 >> 172.21.15.77:0/3365732599 conn(0x56215d6508b0 msgr2=0x56215d653b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.307+0000 7f8c84dc5740 1 -- 172.21.15.77:0/3365732599 shutdown_connections 2024-03-16T10:18:29.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.307+0000 7f8c84dc5740 1 -- 172.21.15.77:0/3365732599 wait complete. 2024-03-16T10:18:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.307+0000 7f8c84dc5740 1 -- start start 2024-03-16T10:18:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c84dc5740 1 Processor -- start 2024-03-16T10:18:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c84dc5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d6569a0 0x56215d66d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c84dc5740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d65e2a0 0x56215d66de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c84dc5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65f600 0x56215d66edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c84dc5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c1a8250 con 0x56215d65f600 2024-03-16T10:18:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c84dc5740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c1a7b80 con 0x56215d65e2a0 2024-03-16T10:18:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c84dc5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c1a4dc0 con 0x56215d6569a0 2024-03-16T10:18:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c7b7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d6569a0 0x56215d66d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c7bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65f600 0x56215d66edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c7b7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d6569a0 0x56215d66d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:54368/0 (socket says 172.21.15.77:54368) 2024-03-16T10:18:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c7b7fe640 1 -- 172.21.15.77:0/664942482 learned_addr learned my addr 172.21.15.77:0/664942482 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c7affd640 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d65e2a0 0x56215d66de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c7b7fe640 1 -- 172.21.15.77:0/664942482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d65e2a0 msgr2=0x56215d66de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c84dc5740 10 client.0 _ll_register_callbacks cb 0x7f8c7000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c7b7fe640 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d65e2a0 0x56215d66de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c7b7fe640 1 -- 172.21.15.77:0/664942482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65f600 msgr2=0x56215d66edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.308+0000 7f8c7b7fe640 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65f600 0x56215d66edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.309+0000 7f8c7b7fe640 1 -- 172.21.15.77:0/664942482 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56215d66f3e0 con 0x56215d6569a0 2024-03-16T10:18:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.309+0000 7f8c7bfff640 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65f600 0x56215d66edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.309+0000 7f8c84dc5740 1 -- 172.21.15.77:0/664942482 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c7009b840 con 0x56215d6569a0 2024-03-16T10:18:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.309+0000 7f8c7b7fe640 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d6569a0 0x56215d66d8f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x56215d655030 tx=0x7f8c6c005640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31e84c36cc26e364 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.309+0000 7f8c5affd640 10 client.25699 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.309+0000 7f8c5affd640 1 -- 172.21.15.77:0/664942482 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c045030 con 0x56215d6569a0 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.309+0000 7f8c5affd640 1 -- 172.21.15.77:0/664942482 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8c6c004bd0 con 0x56215d6569a0 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.309+0000 7f8c5affd640 1 -- 172.21.15.77:0/664942482 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8c6c005d50 con 0x56215d6569a0 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.309+0000 7f8c84dc5740 10 client.25699 fetch_fsmap learned FSMap version 27 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.309+0000 7f8c84dc5740 1 -- 172.21.15.77:0/664942482 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=27}) -- 0x56215d6705d0 con 0x56215d6569a0 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.309+0000 7f8c5affd640 1 -- 172.21.15.77:0/664942482 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8c6c0035f0 con 0x56215d6569a0 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.310+0000 7f8c84dc5740 10 client.25699 fetch_fsmap finished waiting for FSMap version 27 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.310+0000 7f8c84dc5740 10 client.25699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.310+0000 7f8c84dc5740 20 client.25699 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.310+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56215d6da630 0x56215d6dca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.310+0000 7f8c84dc5740 4 client.25699 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.310+0000 7f8c84dc5740 1 -- 172.21.15.77:0/664942482 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56215d5c22d0 con 0x56215d6da630 2024-03-16T10:18:29.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.310+0000 7f8c7affd640 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56215d6da630 0x56215d6dca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.310+0000 7f8c7affd640 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56215d6da630 0x56215d6dca50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebddbb0aa4d2349c server_cookie=3d8cf55eec4f4ce5 in_seq=0 out_seq=0 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.310+0000 7f8c5affd640 10 client.25699 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.314+0000 7f8c5affd640 1 -- 172.21.15.77:0/664942482 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56215d5c22d0 con 0x56215d6da630 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c5affd640 10 client.25699 handle_command_reply: tid=0 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c84dc5740 1 client.25699 shutdown 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c84dc5740 1 -- 172.21.15.77:0/664942482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d6569a0 msgr2=0x56215d66d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d6569a0 0x56215d66d8f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x56215d655030 tx=0x7f8c6c005640 comp rx=0 tx=0).stop 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c84dc5740 1 -- 172.21.15.77:0/664942482 shutdown_connections 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56215d6da630 0x56215d6dca50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56215d65f600 0x56215d66edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c5affd640 0 client.25699 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56215d65e2a0 0x56215d66de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c84dc5740 1 --2- 172.21.15.77:0/664942482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56215d6569a0 0x56215d66d8f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.322 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c84dc5740 1 -- 172.21.15.77:0/664942482 >> 172.21.15.77:0/664942482 conn(0x56215d6508b0 msgr2=0x56215d64f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:29.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c84dc5740 1 -- 172.21.15.77:0/664942482 shutdown_connections 2024-03-16T10:18:29.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c84dc5740 1 -- 172.21.15.77:0/664942482 wait complete. 2024-03-16T10:18:29.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.315+0000 7f8c84dc5740 20 client.25699 trim_cache size 0 max 16384 2024-03-16T10:18:29.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:18:29.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:18:29.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:18:29.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:18:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.316+0000 7f8c821e1640 1 -- 172.21.15.77:0/448430003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8c640a2940 msgr2=0x7f8c640a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.316+0000 7f8c821e1640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8c640a2940 0x7f8c640a4e00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x56215d656530 tx=0x7f8c74004a40 comp rx=0 tx=0).stop 2024-03-16T10:18:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.316+0000 7f8c821e1640 1 -- 172.21.15.77:0/448430003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1022b0 msgr2=0x7f8c7c197100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.316+0000 7f8c821e1640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1022b0 0x7f8c7c197100 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000e9f0 tx=0x7f8c7000edf0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.316+0000 7f8c821e1640 1 -- 172.21.15.77:0/448430003 shutdown_connections 2024-03-16T10:18:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.316+0000 7f8c821e1640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8c640a2940 0x7f8c640a4e00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.316+0000 7f8c821e1640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c7c19c160 0x7f8c7c19e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.316+0000 7f8c821e1640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c7c1022b0 0x7f8c7c197100 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.316+0000 7f8c821e1640 1 --2- 172.21.15.77:0/448430003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c7c1018f0 0x7f8c7c196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.316+0000 7f8c821e1640 1 -- 172.21.15.77:0/448430003 >> 172.21.15.77:0/448430003 conn(0x7f8c7c0fbc70 msgr2=0x7f8c7c104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.316+0000 7f8c821e1640 1 -- 172.21.15.77:0/448430003 shutdown_connections 2024-03-16T10:18:29.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:29.316+0000 7f8c821e1640 1 -- 172.21.15.77:0/448430003 wait complete. 2024-03-16T10:18:30.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:29 smithi192 ceph-mon[36354]: pgmap v392: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-16T10:18:30.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:29 smithi077 ceph-mon[32400]: pgmap v392: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-16T10:18:30.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:29 smithi193 ceph-mon[36311]: pgmap v392: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-16T10:18:31.929 INFO:tasks.workunit.client.0.smithi077.stderr:Updating files: 96% (11868/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-16T10:18:32.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:31 smithi192 ceph-mon[36354]: pgmap v393: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-16T10:18:32.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:31 smithi192 ceph-mon[36354]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-16T10:18:32.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:31 smithi077 ceph-mon[32400]: pgmap v393: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-16T10:18:32.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:31 smithi077 ceph-mon[32400]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-16T10:18:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:31 smithi193 ceph-mon[36311]: pgmap v393: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-16T10:18:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:31 smithi193 ceph-mon[36311]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-16T10:18:32.714 INFO:tasks.workunit.client.0.smithi077.stderr:Note: switching to '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'. 2024-03-16T10:18:32.714 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T10:18:32.714 INFO:tasks.workunit.client.0.smithi077.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-16T10:18:32.714 INFO:tasks.workunit.client.0.smithi077.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-16T10:18:32.714 INFO:tasks.workunit.client.0.smithi077.stderr:state without impacting any branches by switching back to a branch. 2024-03-16T10:18:32.714 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T10:18:32.714 INFO:tasks.workunit.client.0.smithi077.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-16T10:18:32.714 INFO:tasks.workunit.client.0.smithi077.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-16T10:18:32.715 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T10:18:32.715 INFO:tasks.workunit.client.0.smithi077.stderr: git switch -c 2024-03-16T10:18:32.715 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T10:18:32.715 INFO:tasks.workunit.client.0.smithi077.stderr:Or undo this operation with: 2024-03-16T10:18:32.715 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T10:18:32.715 INFO:tasks.workunit.client.0.smithi077.stderr: git switch - 2024-03-16T10:18:32.715 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T10:18:32.715 INFO:tasks.workunit.client.0.smithi077.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-16T10:18:32.715 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T10:18:32.716 INFO:tasks.workunit.client.0.smithi077.stderr:HEAD is now at 3d78f5ba07b Merge PR #54581 into wip-batrick-testing-20240314.183243 2024-03-16T10:18:32.719 DEBUG:teuthology.orchestra.run.smithi077:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-16T10:18:32.743 INFO:tasks.workunit.client.0.smithi077.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-16T10:18:32.745 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T10:18:32.745 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-16T10:18:33.466 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-16T10:18:33.534 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-16T10:18:33.573 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T10:18:33.574 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T10:18:33.574 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-16T10:18:33.614 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T10:18:33.617 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:18:33.617 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-16T10:18:33.673 INFO:tasks.workunit:Running workunits matching suites/blogbench.sh on client.0... 2024-03-16T10:18:33.675 INFO:tasks.workunit:Running workunit suites/blogbench.sh... 2024-03-16T10:18:33.675 DEBUG:teuthology.orchestra.run.smithi077:workunit test suites/blogbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-03-16T10:18:33.734 INFO:tasks.workunit.client.0.smithi077.stdout:getting blogbench 2024-03-16T10:18:33.735 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 'getting blogbench' 2024-03-16T10:18:33.735 INFO:tasks.workunit.client.0.smithi077.stderr:+ wget http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-03-16T10:18:33.762 INFO:tasks.workunit.client.0.smithi077.stderr:--2024-03-16 10:18:33-- http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-03-16T10:18:33.762 INFO:tasks.workunit.client.0.smithi077.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-16T10:18:33.784 INFO:tasks.workunit.client.0.smithi077.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-16T10:18:33.804 INFO:tasks.workunit.client.0.smithi077.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-16T10:18:33.804 INFO:tasks.workunit.client.0.smithi077.stderr:Length: 103355 (101K) [application/octet-stream] 2024-03-16T10:18:33.805 INFO:tasks.workunit.client.0.smithi077.stderr:Saving to: ‘blogbench-1.0.tar.bz2’ 2024-03-16T10:18:33.807 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T10:18:33.846 INFO:tasks.workunit.client.0.smithi077.stderr: 0K .......... .......... .......... .......... .......... 49% 1.22M 0s 2024-03-16T10:18:33.866 INFO:tasks.workunit.client.0.smithi077.stderr: 50K .......... .......... .......... .......... .......... 99% 2.38M 0s 2024-03-16T10:18:33.867 INFO:tasks.workunit.client.0.smithi077.stderr: 100K 100% 1.74T=0.06s 2024-03-16T10:18:33.867 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T10:18:33.867 INFO:tasks.workunit.client.0.smithi077.stderr:2024-03-16 10:18:33 (1.63 MB/s) - ‘blogbench-1.0.tar.bz2’ saved [103355/103355] 2024-03-16T10:18:33.867 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-03-16T10:18:33.867 INFO:tasks.workunit.client.0.smithi077.stderr:+ tar -xvf blogbench-1.0.tar.bz2 2024-03-16T10:18:33.888 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/ 2024-03-16T10:18:33.888 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/m4/ 2024-03-16T10:18:33.889 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/m4/acx_pthread.m4 2024-03-16T10:18:33.890 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/man/ 2024-03-16T10:18:33.891 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/man/blogbench.8 2024-03-16T10:18:33.892 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/man/Makefile.am 2024-03-16T10:18:33.892 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/man/Makefile.in 2024-03-16T10:18:33.893 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/ 2024-03-16T10:18:33.894 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/mysnprintf.c 2024-03-16T10:18:33.895 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/mysnprintf.h 2024-03-16T10:18:33.896 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/blogbench.c 2024-03-16T10:18:33.896 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/blogbench.h 2024-03-16T10:18:33.897 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/fakesnprintf.c 2024-03-16T10:18:33.898 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/fakesnprintf.h 2024-03-16T10:18:33.899 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/blogbench_p.h 2024-03-16T10:18:33.900 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/bsd-getopt_long.c 2024-03-16T10:18:33.900 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/bsd-getopt_long.h 2024-03-16T10:18:33.901 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/Makefile.am 2024-03-16T10:18:33.902 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/Makefile.in 2024-03-16T10:18:33.903 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/process.c 2024-03-16T10:18:33.904 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/globals.h 2024-03-16T10:18:33.905 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/commenter.c 2024-03-16T10:18:33.906 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/rewriter.c 2024-03-16T10:18:33.907 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/reader.c 2024-03-16T10:18:33.907 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/writer.c 2024-03-16T10:18:33.908 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/src/helpers.c 2024-03-16T10:18:33.909 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/NEWS 2024-03-16T10:18:33.910 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/compile 2024-03-16T10:18:33.911 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/depcomp 2024-03-16T10:18:33.911 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/aclocal.m4 2024-03-16T10:18:33.912 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/README 2024-03-16T10:18:33.913 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/configure 2024-03-16T10:18:33.918 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/configure.ac 2024-03-16T10:18:33.918 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/config.guess 2024-03-16T10:18:33.918 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/install-sh 2024-03-16T10:18:33.919 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/config.sub 2024-03-16T10:18:33.919 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/missing 2024-03-16T10:18:33.920 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/mkinstalldirs 2024-03-16T10:18:33.921 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/Makefile.am 2024-03-16T10:18:33.922 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/Makefile.in 2024-03-16T10:18:33.922 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/config.h.in 2024-03-16T10:18:33.923 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/AUTHORS 2024-03-16T10:18:33.924 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/INSTALL 2024-03-16T10:18:33.925 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/ChangeLog 2024-03-16T10:18:33.926 INFO:tasks.workunit.client.0.smithi077.stdout:blogbench-1.0/COPYING 2024-03-16T10:18:33.927 INFO:tasks.workunit.client.0.smithi077.stderr:+ cd blogbench-1.0/ 2024-03-16T10:18:33.927 INFO:tasks.workunit.client.0.smithi077.stdout:making blogbench 2024-03-16T10:18:33.927 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 'making blogbench' 2024-03-16T10:18:33.927 INFO:tasks.workunit.client.0.smithi077.stderr:+ ./configure 2024-03-16T10:18:34.042 INFO:tasks.workunit.client.0.smithi077.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-16T10:18:34.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:33 smithi192 ceph-mon[36354]: pgmap v394: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-16T10:18:34.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:33 smithi077 ceph-mon[32400]: pgmap v394: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-16T10:18:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:33 smithi193 ceph-mon[36311]: pgmap v394: 97 pgs: 97 active+clean; 672 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-16T10:18:35.048 INFO:tasks.workunit.client.0.smithi077.stdout:checking whether build environment is sane... yes 2024-03-16T10:18:35.056 INFO:tasks.workunit.client.0.smithi077.stdout:checking for gawk... gawk 2024-03-16T10:18:35.073 INFO:tasks.workunit.client.0.smithi077.stdout:checking whether make sets $(MAKE)... yes 2024-03-16T10:18:35.091 INFO:tasks.workunit.client.0.smithi077.stdout:checking for gcc... gcc 2024-03-16T10:18:35.138 INFO:tasks.workunit.client.0.smithi077.stdout:checking for C compiler default output file name... a.out 2024-03-16T10:18:35.139 INFO:tasks.workunit.client.0.smithi077.stdout:checking whether the C compiler works... yes 2024-03-16T10:18:35.142 INFO:tasks.workunit.client.0.smithi077.stdout:checking whether we are cross compiling... no 2024-03-16T10:18:35.178 INFO:tasks.workunit.client.0.smithi077.stdout:checking for suffix of executables... 2024-03-16T10:18:35.220 INFO:tasks.workunit.client.0.smithi077.stdout:checking for suffix of object files... o 2024-03-16T10:18:35.251 INFO:tasks.workunit.client.0.smithi077.stdout:checking whether we are using the GNU C compiler... yes 2024-03-16T10:18:35.287 INFO:tasks.workunit.client.0.smithi077.stdout:checking whether gcc accepts -g... yes 2024-03-16T10:18:35.355 INFO:tasks.workunit.client.0.smithi077.stdout:checking for gcc option to accept ANSI C... none needed 2024-03-16T10:18:35.408 INFO:tasks.workunit.client.0.smithi077.stdout:checking for style of include used by make... GNU 2024-03-16T10:18:35.509 INFO:tasks.workunit.client.0.smithi077.stdout:checking dependency style of gcc... gcc3 2024-03-16T10:18:35.509 INFO:tasks.workunit.client.0.smithi077.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-16T10:18:35.565 INFO:tasks.workunit.client.0.smithi077.stdout:checking how to run the C preprocessor... gcc -E 2024-03-16T10:18:35.603 INFO:tasks.workunit.client.0.smithi077.stdout:checking for egrep... grep -E 2024-03-16T10:18:35.612 INFO:tasks.workunit.client.0.smithi077.stdout:checking for AIX... no 2024-03-16T10:18:35.695 INFO:tasks.workunit.client.0.smithi077.stdout:checking for library containing strerror... none required 2024-03-16T10:18:35.925 INFO:tasks.workunit.client.0.smithi077.stdout:checking for ANSI C header files... yes 2024-03-16T10:18:35.972 INFO:tasks.workunit.client.0.smithi077.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-16T10:18:36.024 INFO:tasks.workunit.client.0.smithi077.stdout:checking for sys/types.h... yes 2024-03-16T10:18:36.071 INFO:tasks.workunit.client.0.smithi077.stdout:checking for sys/stat.h... yes 2024-03-16T10:18:36.121 INFO:tasks.workunit.client.0.smithi077.stdout:checking for stdlib.h... yes 2024-03-16T10:18:36.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:35 smithi192 ceph-mon[36354]: pgmap v395: 97 pgs: 97 active+clean; 673 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-16T10:18:36.176 INFO:tasks.workunit.client.0.smithi077.stdout:checking for string.h... yes 2024-03-16T10:18:36.232 INFO:tasks.workunit.client.0.smithi077.stdout:checking for memory.h... yes 2024-03-16T10:18:36.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:35 smithi077 ceph-mon[32400]: pgmap v395: 97 pgs: 97 active+clean; 673 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-16T10:18:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:35 smithi193 ceph-mon[36311]: pgmap v395: 97 pgs: 97 active+clean; 673 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-16T10:18:36.289 INFO:tasks.workunit.client.0.smithi077.stdout:checking for strings.h... yes 2024-03-16T10:18:36.347 INFO:tasks.workunit.client.0.smithi077.stdout:checking for inttypes.h... yes 2024-03-16T10:18:36.395 INFO:tasks.workunit.client.0.smithi077.stdout:checking for stdint.h... yes 2024-03-16T10:18:36.454 INFO:tasks.workunit.client.0.smithi077.stdout:checking for unistd.h... yes 2024-03-16T10:18:36.466 INFO:tasks.workunit.client.0.smithi077.stdout:checking for memory.h... (cached) yes 2024-03-16T10:18:36.471 INFO:tasks.workunit.client.0.smithi077.stdout:checking for strings.h... (cached) yes 2024-03-16T10:18:36.475 INFO:tasks.workunit.client.0.smithi077.stdout:checking for unistd.h... (cached) yes 2024-03-16T10:18:36.526 INFO:tasks.workunit.client.0.smithi077.stdout:checking locale.h usability... yes 2024-03-16T10:18:36.545 INFO:tasks.workunit.client.0.smithi077.stdout:checking locale.h presence... yes 2024-03-16T10:18:36.546 INFO:tasks.workunit.client.0.smithi077.stdout:checking for locale.h... yes 2024-03-16T10:18:36.563 INFO:tasks.workunit.client.0.smithi077.stdout:checking for stdlib.h... (cached) yes 2024-03-16T10:18:36.568 INFO:tasks.workunit.client.0.smithi077.stdout:checking for string.h... (cached) yes 2024-03-16T10:18:36.627 INFO:tasks.workunit.client.0.smithi077.stdout:checking sys/socket.h usability... yes 2024-03-16T10:18:36.647 INFO:tasks.workunit.client.0.smithi077.stdout:checking sys/socket.h presence... yes 2024-03-16T10:18:36.648 INFO:tasks.workunit.client.0.smithi077.stdout:checking for sys/socket.h... yes 2024-03-16T10:18:36.700 INFO:tasks.workunit.client.0.smithi077.stdout:checking sys/time.h usability... yes 2024-03-16T10:18:36.719 INFO:tasks.workunit.client.0.smithi077.stdout:checking sys/time.h presence... yes 2024-03-16T10:18:36.720 INFO:tasks.workunit.client.0.smithi077.stdout:checking for sys/time.h... yes 2024-03-16T10:18:36.771 INFO:tasks.workunit.client.0.smithi077.stdout:checking fcntl.h usability... yes 2024-03-16T10:18:36.793 INFO:tasks.workunit.client.0.smithi077.stdout:checking fcntl.h presence... yes 2024-03-16T10:18:36.793 INFO:tasks.workunit.client.0.smithi077.stdout:checking for fcntl.h... yes 2024-03-16T10:18:36.869 INFO:tasks.workunit.client.0.smithi077.stdout:checking sys/fcntl.h usability... yes 2024-03-16T10:18:38.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:37 smithi192 ceph-mon[36354]: pgmap v396: 97 pgs: 97 active+clean; 673 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-16T10:18:38.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:37 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:18:38.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:37 smithi077 ceph-mon[32400]: pgmap v396: 97 pgs: 97 active+clean; 673 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-16T10:18:38.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:37 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:18:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:37 smithi193 ceph-mon[36311]: pgmap v396: 97 pgs: 97 active+clean; 673 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-16T10:18:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:37 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:18:40.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:39 smithi192 ceph-mon[36354]: pgmap v397: 97 pgs: 97 active+clean; 1.7 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 18 op/s 2024-03-16T10:18:40.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:39 smithi077 ceph-mon[32400]: pgmap v397: 97 pgs: 97 active+clean; 1.7 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 18 op/s 2024-03-16T10:18:40.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:39 smithi193 ceph-mon[36311]: pgmap v397: 97 pgs: 97 active+clean; 1.7 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 18 op/s 2024-03-16T10:18:41.117 INFO:tasks.workunit.client.0.smithi077.stdout:checking sys/fcntl.h presence... yes 2024-03-16T10:18:41.118 INFO:tasks.workunit.client.0.smithi077.stdout:checking for sys/fcntl.h... yes 2024-03-16T10:18:41.198 INFO:tasks.workunit.client.0.smithi077.stdout:checking sys/param.h usability... yes 2024-03-16T10:18:41.223 INFO:tasks.workunit.client.0.smithi077.stdout:checking sys/param.h presence... yes 2024-03-16T10:18:41.224 INFO:tasks.workunit.client.0.smithi077.stdout:checking for sys/param.h... yes 2024-03-16T10:18:41.275 INFO:tasks.workunit.client.0.smithi077.stdout:checking ioctl.h usability... no 2024-03-16T10:18:41.294 INFO:tasks.workunit.client.0.smithi077.stdout:checking ioctl.h presence... no 2024-03-16T10:18:41.295 INFO:tasks.workunit.client.0.smithi077.stdout:checking for ioctl.h... no 2024-03-16T10:18:41.347 INFO:tasks.workunit.client.0.smithi077.stdout:checking sys/ioctl.h usability... yes 2024-03-16T10:18:41.365 INFO:tasks.workunit.client.0.smithi077.stdout:checking sys/ioctl.h presence... yes 2024-03-16T10:18:41.366 INFO:tasks.workunit.client.0.smithi077.stdout:checking for sys/ioctl.h... yes 2024-03-16T10:18:41.416 INFO:tasks.workunit.client.0.smithi077.stdout:checking return type of signal handlers... void 2024-03-16T10:18:41.461 INFO:tasks.workunit.client.0.smithi077.stdout:checking for size_t... yes 2024-03-16T10:18:41.501 INFO:tasks.workunit.client.0.smithi077.stdout:checking for dev_t... yes 2024-03-16T10:18:41.544 INFO:tasks.workunit.client.0.smithi077.stdout:checking for short... yes 2024-03-16T10:18:41.648 INFO:tasks.workunit.client.0.smithi077.stdout:checking size of short... 2 2024-03-16T10:18:41.692 INFO:tasks.workunit.client.0.smithi077.stdout:checking for int... yes 2024-03-16T10:18:41.762 INFO:tasks.workunit.client.0.smithi077.stdout:checking size of int... 4 2024-03-16T10:18:41.805 INFO:tasks.workunit.client.0.smithi077.stdout:checking for long... yes 2024-03-16T10:18:41.883 INFO:tasks.workunit.client.0.smithi077.stdout:checking size of long... 8 2024-03-16T10:18:41.926 INFO:tasks.workunit.client.0.smithi077.stdout:checking for long long... yes 2024-03-16T10:18:42.001 INFO:tasks.workunit.client.0.smithi077.stdout:checking size of long long... 8 2024-03-16T10:18:42.047 INFO:tasks.workunit.client.0.smithi077.stdout:checking for ssize_t... yes 2024-03-16T10:18:42.094 INFO:tasks.workunit.client.0.smithi077.stdout:checking for sig_atomic_t... yes 2024-03-16T10:18:42.162 INFO:tasks.workunit.client.0.smithi077.stdout:checking whether gcc needs -traditional... no 2024-03-16T10:18:42.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:41 smithi192 ceph-mon[36354]: pgmap v398: 97 pgs: 97 active+clean; 1.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 110 KiB/s wr, 19 op/s 2024-03-16T10:18:42.202 INFO:tasks.workunit.client.0.smithi077.stdout:checking for an ANSI C-conforming const... yes 2024-03-16T10:18:42.228 INFO:tasks.workunit.client.0.smithi077.stdout:checking for inline... inline 2024-03-16T10:18:42.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:41 smithi077 ceph-mon[32400]: pgmap v398: 97 pgs: 97 active+clean; 1.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 110 KiB/s wr, 19 op/s 2024-03-16T10:18:42.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:41 smithi193 ceph-mon[36311]: pgmap v398: 97 pgs: 97 active+clean; 1.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 110 KiB/s wr, 19 op/s 2024-03-16T10:18:42.299 INFO:tasks.workunit.client.0.smithi077.stdout:checking for working alloca.h... yes 2024-03-16T10:18:42.350 INFO:tasks.workunit.client.0.smithi077.stdout:checking for alloca... yes 2024-03-16T10:18:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 -- 172.21.15.77:0/866602646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24b01022b0 msgr2=0x7f24b0108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 --2- 172.21.15.77:0/866602646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24b01022b0 0x7f24b0108b40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009ca0 tx=0x7f24a802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 -- 172.21.15.77:0/866602646 shutdown_connections 2024-03-16T10:18:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 --2- 172.21.15.77:0/866602646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24b01022b0 0x7f24b0108b40 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 --2- 172.21.15.77:0/866602646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24b01018f0 0x7f24b0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 --2- 172.21.15.77:0/866602646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24b0100700 0x7f24b0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 -- 172.21.15.77:0/866602646 >> 172.21.15.77:0/866602646 conn(0x7f24b00fbc70 msgr2=0x7f24b00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 -- 172.21.15.77:0/866602646 shutdown_connections 2024-03-16T10:18:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 -- 172.21.15.77:0/866602646 wait complete. 2024-03-16T10:18:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 Processor -- start 2024-03-16T10:18:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 -- start start 2024-03-16T10:18:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24b0100700 0x7f24b00699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.406+0000 7f24b7b77640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24b01018f0 0x7f24b019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b7b77640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24b01022b0 0x7f24b019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b7b77640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f24b010c380 con 0x7f24b0100700 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b7b77640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f24b010c200 con 0x7f24b01018f0 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b7b77640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f24b010c500 con 0x7f24b01022b0 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b5913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24b0100700 0x7f24b00699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b5913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24b0100700 0x7f24b00699d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60738/0 (socket says 172.21.15.77:60738) 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b5913640 1 -- 172.21.15.77:0/3844624957 learned_addr learned my addr 172.21.15.77:0/3844624957 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b5112640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24b01018f0 0x7f24b019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b6114640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24b01022b0 0x7f24b019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b5913640 1 -- 172.21.15.77:0/3844624957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24b01022b0 msgr2=0x7f24b019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b5913640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24b01022b0 0x7f24b019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b5913640 1 -- 172.21.15.77:0/3844624957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24b01018f0 msgr2=0x7f24b019c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b5913640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24b01018f0 0x7f24b019c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b5913640 1 -- 172.21.15.77:0/3844624957 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b019f6b0 con 0x7f24b0100700 2024-03-16T10:18:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b5913640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24b0100700 0x7f24b00699d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00e9c0 tx=0x7f24ac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24a6ffd640 1 -- 172.21.15.77:0/3844624957 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac00cd80 con 0x7f24b0100700 2024-03-16T10:18:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24a6ffd640 1 -- 172.21.15.77:0/3844624957 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24ac0035f0 con 0x7f24b0100700 2024-03-16T10:18:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24a6ffd640 1 -- 172.21.15.77:0/3844624957 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac010690 con 0x7f24b0100700 2024-03-16T10:18:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b7b77640 1 -- 172.21.15.77:0/3844624957 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b019f940 con 0x7f24b0100700 2024-03-16T10:18:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.407+0000 7f24b7b77640 1 -- 172.21.15.77:0/3844624957 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b019fe90 con 0x7f24b0100700 2024-03-16T10:18:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.409+0000 7f24a6ffd640 1 -- 172.21.15.77:0/3844624957 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f24ac0037a0 con 0x7f24b0100700 2024-03-16T10:18:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.410+0000 7f24a6ffd640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f249c0a28a0 0x7f249c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.410+0000 7f24b5112640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f249c0a28a0 0x7f249c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.410+0000 7f24a6ffd640 1 -- 172.21.15.77:0/3844624957 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f24ac0ced80 con 0x7f24b0100700 2024-03-16T10:18:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.410+0000 7f24b5112640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f249c0a28a0 0x7f249c0a4d60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f24b0069740 tx=0x7f24a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.411+0000 7f24ba765740 1 Processor -- start 2024-03-16T10:18:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.411+0000 7f24ba765740 1 -- start start 2024-03-16T10:18:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.411+0000 7f24ba765740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0ef760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.411+0000 7f24ba765740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0efd30 0x56025b0f4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.411+0000 7f24ba765740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0f4ea0 0x56025b0f7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.411+0000 7f24ba765740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56025b054a40 con 0x56025b0ef340 2024-03-16T10:18:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.411+0000 7f24ba765740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56025affd920 con 0x56025b0f4ea0 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.411+0000 7f24ba765740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56025b050ae0 con 0x56025b0efd30 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.411+0000 7f24b5913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0ef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.411+0000 7f24b5913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0ef760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60746/0 (socket says 172.21.15.77:60746) 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.411+0000 7f24b5112640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0efd30 0x56025b0f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.411+0000 7f24b5913640 1 -- 172.21.15.77:0/395519981 learned_addr learned my addr 172.21.15.77:0/395519981 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.412+0000 7f24b6114640 1 --2- 172.21.15.77:0/395519981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0f4ea0 0x56025b0f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.412+0000 7f24b5913640 1 -- 172.21.15.77:0/395519981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0efd30 msgr2=0x56025b0f4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.412+0000 7f24b5913640 1 --2- 172.21.15.77:0/395519981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0efd30 0x56025b0f4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.412+0000 7f24b5913640 1 -- 172.21.15.77:0/395519981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0f4ea0 msgr2=0x56025b0f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.412+0000 7f24b5913640 1 --2- 172.21.15.77:0/395519981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0f4ea0 0x56025b0f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.412+0000 7f24b5913640 1 -- 172.21.15.77:0/395519981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56025b015460 con 0x56025b0ef340 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.412+0000 7f24b5913640 1 --2- 172.21.15.77:0/395519981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0ef760 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f24ac009830 tx=0x7f24ac09beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8176d40ebadf0808 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.412+0000 7f249bfff640 1 -- 172.21.15.77:0/395519981 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0a3f50 con 0x56025b0ef340 2024-03-16T10:18:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.412+0000 7f249bfff640 1 -- 172.21.15.77:0/395519981 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24ac099ab0 con 0x56025b0ef340 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.412+0000 7f249bfff640 1 -- 172.21.15.77:0/395519981 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0afb90 con 0x56025b0ef340 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.413+0000 7f24ba765740 1 -- 172.21.15.77:0/395519981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 msgr2=0x56025b0ef760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.413+0000 7f24ba765740 1 --2- 172.21.15.77:0/395519981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0ef760 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f24ac009830 tx=0x7f24ac09beb0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.413+0000 7f24ba765740 1 -- 172.21.15.77:0/395519981 shutdown_connections 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.413+0000 7f24ba765740 1 --2- 172.21.15.77:0/395519981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0f4ea0 0x56025b0f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.413+0000 7f24ba765740 1 --2- 172.21.15.77:0/395519981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0efd30 0x56025b0f4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.413+0000 7f24ba765740 1 --2- 172.21.15.77:0/395519981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0ef760 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.413+0000 7f24ba765740 1 -- 172.21.15.77:0/395519981 >> 172.21.15.77:0/395519981 conn(0x56025b0ea8b0 msgr2=0x56025b0eacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.413+0000 7f24ba765740 1 -- 172.21.15.77:0/395519981 shutdown_connections 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.413+0000 7f24ba765740 1 -- 172.21.15.77:0/395519981 wait complete. 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.413+0000 7f24ba765740 1 -- start start 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.413+0000 7f24ba765740 1 Processor -- start 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24ba765740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0f95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24ba765740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0efd30 0x56025b0f9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24ba765740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0f4ea0 0x56025b100b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24ba765740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56025b0f7df0 con 0x56025b0ef340 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24ba765740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56025b010d70 con 0x56025b0f4ea0 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24ba765740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56025b0f7f70 con 0x56025b0efd30 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24ba765740 10 client.0 _ll_register_callbacks cb 0x56025ad68a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24b5913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0f95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24b5913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0f95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60756/0 (socket says 172.21.15.77:60756) 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24b5913640 1 -- 172.21.15.77:0/3053894060 learned_addr learned my addr 172.21.15.77:0/3053894060 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24b5112640 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0efd30 0x56025b0f9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24b6114640 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0f4ea0 0x56025b100b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24b5913640 1 -- 172.21.15.77:0/3053894060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0efd30 msgr2=0x56025b0f9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24b5913640 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0efd30 0x56025b0f9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24b5913640 1 -- 172.21.15.77:0/3053894060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0f4ea0 msgr2=0x56025b100b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24b5913640 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0f4ea0 0x56025b100b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24b5913640 1 -- 172.21.15.77:0/3053894060 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56025ad69210 con 0x56025b0ef340 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24b5913640 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0f95b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f24ac09b740 tx=0x7f24ac099c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6231ef4befe47b4a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24997fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24997fa640 1 -- 172.21.15.77:0/3053894060 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac07f060 con 0x56025b0ef340 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.414+0000 7f24997fa640 1 -- 172.21.15.77:0/3053894060 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f24ac097f00 con 0x56025b0ef340 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24ba765740 1 -- 172.21.15.77:0/3053894060 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24ac099ab0 con 0x56025b0ef340 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24997fa640 1 -- 172.21.15.77:0/3053894060 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f24ac0ba070 con 0x56025b0ef340 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24ba765740 10 client.15939 fetch_fsmap learned FSMap version 28 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24ba765740 1 -- 172.21.15.77:0/3053894060 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56025ae35620 con 0x56025b0ef340 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24997fa640 1 -- 172.21.15.77:0/3053894060 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24ac099ab0 con 0x56025b0ef340 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24997fa640 1 -- 172.21.15.77:0/3053894060 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f24ac09a670 con 0x56025b0ef340 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24ba765740 10 client.15939 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24ba765740 10 client.15939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24ba765740 20 client.15939 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24ba765740 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56025b105020 0x56025b107410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24ba765740 4 client.15939 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24ba765740 1 -- 172.21.15.77:0/3053894060 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56025b061f80 con 0x56025b105020 2024-03-16T10:18:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.415+0000 7f24b5112640 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56025b105020 0x56025b107410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.416+0000 7f24b5112640 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56025b105020 0x56025b107410 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf01b49e15ee226f server_cookie=377a890b3d8d3c08 in_seq=0 out_seq=0 2024-03-16T10:18:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.416+0000 7f24997fa640 10 client.15939 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24997fa640 1 -- 172.21.15.77:0/3053894060 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56025b061f80 con 0x56025b105020 2024-03-16T10:18:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24997fa640 10 client.15939 handle_command_reply: tid=0 2024-03-16T10:18:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24ba765740 1 client.15939 shutdown 2024-03-16T10:18:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24ba765740 1 -- 172.21.15.77:0/3053894060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 msgr2=0x56025b0f95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24ba765740 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0f95b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f24ac09b740 tx=0x7f24ac099c60 comp rx=0 tx=0).stop 2024-03-16T10:18:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24ba765740 1 -- 172.21.15.77:0/3053894060 shutdown_connections 2024-03-16T10:18:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24ba765740 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0f4ea0 0x56025b100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24ba765740 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0efd30 0x56025b0f9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24ba765740 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56025b105020 0x56025b107410 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24ba765740 1 --2- 172.21.15.77:0/3053894060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0f95b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24997fa640 0 client.15939 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24ba765740 1 -- 172.21.15.77:0/3053894060 >> 172.21.15.77:0/3053894060 conn(0x56025b0ea8b0 msgr2=0x56025b0e9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24ba765740 1 -- 172.21.15.77:0/3053894060 shutdown_connections 2024-03-16T10:18:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24ba765740 1 -- 172.21.15.77:0/3053894060 wait complete. 2024-03-16T10:18:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.417+0000 7f24ba765740 20 client.15939 trim_cache size 0 max 16384 2024-03-16T10:18:42.431 INFO:tasks.workunit.client.0.smithi077.stdout:checking for working memcmp... yes 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24ba765740 1 Processor -- start 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24ba765740 1 -- start start 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24ba765740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0eac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24ba765740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0efd30 0x56025b0f91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24ba765740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0f4ea0 0x56025b0f9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24ba765740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f24b01a8a80 con 0x56025b0ef340 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24ba765740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f24b01aa5f0 con 0x56025b0efd30 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24ba765740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f24b01a9150 con 0x56025b0f4ea0 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24b5913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0eac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24b5913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0eac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60764/0 (socket says 172.21.15.77:60764) 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24b5112640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0efd30 0x56025b0f91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24b5913640 1 -- 172.21.15.77:0/1812775975 learned_addr learned my addr 172.21.15.77:0/1812775975 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:42.436 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24b6114640 1 --2- 172.21.15.77:0/1812775975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0f4ea0 0x56025b0f9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24b5913640 1 -- 172.21.15.77:0/1812775975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0f4ea0 msgr2=0x56025b0f9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24b5913640 1 --2- 172.21.15.77:0/1812775975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0f4ea0 0x56025b0f9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24b5913640 1 -- 172.21.15.77:0/1812775975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0efd30 msgr2=0x56025b0f91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.434+0000 7f24b5913640 1 --2- 172.21.15.77:0/1812775975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0efd30 0x56025b0f91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.435+0000 7f24b5913640 1 -- 172.21.15.77:0/1812775975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56025b0f80d0 con 0x56025b0ef340 2024-03-16T10:18:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.435+0000 7f24b5913640 1 --2- 172.21.15.77:0/1812775975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0eac90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0058c0 tx=0x7f24ac09beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bcb24317c59161 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.435+0000 7f249b7fe640 1 -- 172.21.15.77:0/1812775975 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac099ab0 con 0x56025b0ef340 2024-03-16T10:18:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.435+0000 7f249b7fe640 1 -- 172.21.15.77:0/1812775975 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24ac09f900 con 0x56025b0ef340 2024-03-16T10:18:42.437 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.435+0000 7f249b7fe640 1 -- 172.21.15.77:0/1812775975 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac097500 con 0x56025b0ef340 2024-03-16T10:18:42.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.436+0000 7f24ba765740 1 -- 172.21.15.77:0/1812775975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 msgr2=0x56025b0eac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:42.439 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.436+0000 7f24ba765740 1 --2- 172.21.15.77:0/1812775975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0eac90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0058c0 tx=0x7f24ac09beb0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.436+0000 7f24ba765740 1 -- 172.21.15.77:0/1812775975 shutdown_connections 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.436+0000 7f24ba765740 1 --2- 172.21.15.77:0/1812775975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0f4ea0 0x56025b0f9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.436+0000 7f24ba765740 1 --2- 172.21.15.77:0/1812775975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0efd30 0x56025b0f91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.436+0000 7f24ba765740 1 --2- 172.21.15.77:0/1812775975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0eac90 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.436+0000 7f24ba765740 1 -- 172.21.15.77:0/1812775975 >> 172.21.15.77:0/1812775975 conn(0x56025b0ea8b0 msgr2=0x56025b0e5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.436+0000 7f24ba765740 1 -- 172.21.15.77:0/1812775975 shutdown_connections 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.436+0000 7f24ba765740 1 -- 172.21.15.77:0/1812775975 wait complete. 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.436+0000 7f24ba765740 1 -- start start 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.436+0000 7f24ba765740 1 Processor -- start 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24ba765740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0fe4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24ba765740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0efd30 0x56025b0fea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24ba765740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0f4ea0 0x56025b0ffa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.440 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24ba765740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f24b01a9ef0 con 0x56025b0ef340 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24ba765740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f24b01a9820 con 0x56025b0efd30 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24ba765740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f24b01a6a60 con 0x56025b0f4ea0 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24ba765740 10 client.0 _ll_register_callbacks cb 0x7f24ac005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b5913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0fe4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b5112640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0efd30 0x56025b0fea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b6114640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0f4ea0 0x56025b0ffa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b6114640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0f4ea0 0x56025b0ffa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:40096/0 (socket says 172.21.15.77:40096) 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b5913640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0fe4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60766/0 (socket says 172.21.15.77:60766) 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b6114640 1 -- 172.21.15.77:0/2364889049 learned_addr learned my addr 172.21.15.77:0/2364889049 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b5112640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0efd30 0x56025b0fea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:55922/0 (socket says 172.21.15.77:55922) 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b5112640 1 -- 172.21.15.77:0/2364889049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0f4ea0 msgr2=0x56025b0ffa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b5112640 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0f4ea0 0x56025b0ffa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b5112640 1 -- 172.21.15.77:0/2364889049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 msgr2=0x56025b0fe4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b5112640 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0fe4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.441 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b5112640 1 -- 172.21.15.77:0/2364889049 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56025b100020 con 0x56025b0efd30 2024-03-16T10:18:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b6114640 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0f4ea0 0x56025b0ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b5913640 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0fe4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24b5112640 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0efd30 0x56025b0fea20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x56025b107bc0 tx=0x7f24a0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a63025d4ad5ead7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:42.442 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.437+0000 7f24997fa640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:18:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24997fa640 1 -- 172.21.15.77:0/2364889049 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a002a410 con 0x56025b0efd30 2024-03-16T10:18:42.443 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24997fa640 1 -- 172.21.15.77:0/2364889049 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f24a002a5b0 con 0x56025b0efd30 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24ba765740 1 -- 172.21.15.77:0/2364889049 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24ac09f900 con 0x56025b0efd30 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24997fa640 1 -- 172.21.15.77:0/2364889049 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f24a003a050 con 0x56025b0efd30 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24ba765740 10 client.25733 fetch_fsmap learned FSMap version 28 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24ba765740 1 -- 172.21.15.77:0/2364889049 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56025b16b8c0 con 0x56025b0efd30 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24997fa640 1 -- 172.21.15.77:0/2364889049 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24ac09f900 con 0x56025b0efd30 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24997fa640 1 -- 172.21.15.77:0/2364889049 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f24a0033410 con 0x56025b0efd30 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24ba765740 10 client.25733 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24ba765740 10 client.25733 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24ba765740 20 client.25733 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24ba765740 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56025b16c240 0x56025b16e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24ba765740 4 client.25733 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24ba765740 1 -- 172.21.15.77:0/2364889049 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56025b05c2d0 con 0x56025b16c240 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.438+0000 7f24b5913640 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56025b16c240 0x56025b16e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:42.444 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.439+0000 7f24997fa640 10 client.25733 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:42.445 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.439+0000 7f24b5913640 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56025b16c240 0x56025b16e660 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca30a609570b002 server_cookie=accd9477a668cf7c in_seq=0 out_seq=0 2024-03-16T10:18:42.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24997fa640 1 -- 172.21.15.77:0/2364889049 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56025b05c2d0 con 0x56025b16c240 2024-03-16T10:18:42.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24997fa640 10 client.25733 handle_command_reply: tid=0 2024-03-16T10:18:42.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24ba765740 1 client.25733 shutdown 2024-03-16T10:18:42.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24ba765740 1 -- 172.21.15.77:0/2364889049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0efd30 msgr2=0x56025b0fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24ba765740 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0efd30 0x56025b0fea20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x56025b107bc0 tx=0x7f24a0004730 comp rx=0 tx=0).stop 2024-03-16T10:18:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24ba765740 1 -- 172.21.15.77:0/2364889049 shutdown_connections 2024-03-16T10:18:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24ba765740 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56025b0f4ea0 0x56025b0ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24ba765740 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56025b0efd30 0x56025b0fea20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24ba765740 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56025b16c240 0x56025b16e660 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24ba765740 1 --2- 172.21.15.77:0/2364889049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56025b0ef340 0x56025b0fe4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24997fa640 0 client.25733 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24ba765740 1 -- 172.21.15.77:0/2364889049 >> 172.21.15.77:0/2364889049 conn(0x56025b0ea8b0 msgr2=0x56025b0eac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24ba765740 1 -- 172.21.15.77:0/2364889049 shutdown_connections 2024-03-16T10:18:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.448+0000 7f24ba765740 1 -- 172.21.15.77:0/2364889049 wait complete. 2024-03-16T10:18:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24ba765740 20 client.25733 trim_cache size 0 max 16384 2024-03-16T10:18:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:18:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:18:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:18:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:18:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24b7b77640 1 -- 172.21.15.77:0/3844624957 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f249c0a28a0 msgr2=0x7f249c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24b7b77640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f249c0a28a0 0x7f249c0a4d60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f24b0069740 tx=0x7f24a0005f90 comp rx=0 tx=0).stop 2024-03-16T10:18:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24b7b77640 1 -- 172.21.15.77:0/3844624957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24b0100700 msgr2=0x7f24b00699d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24b7b77640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24b0100700 0x7f24b00699d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00e9c0 tx=0x7f24ac00ee80 comp rx=0 tx=0).stop 2024-03-16T10:18:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24b7b77640 1 -- 172.21.15.77:0/3844624957 shutdown_connections 2024-03-16T10:18:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24b7b77640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f249c0a28a0 0x7f249c0a4d60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24b7b77640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24b01022b0 0x7f24b019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24b7b77640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24b01018f0 0x7f24b019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24b7b77640 1 --2- 172.21.15.77:0/3844624957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24b0100700 0x7f24b00699d0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24b7b77640 1 -- 172.21.15.77:0/3844624957 >> 172.21.15.77:0/3844624957 conn(0x7f24b00fbc70 msgr2=0x7f24b0104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24b7b77640 1 -- 172.21.15.77:0/3844624957 shutdown_connections 2024-03-16T10:18:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:42.449+0000 7f24b7b77640 1 -- 172.21.15.77:0/3844624957 wait complete. 2024-03-16T10:18:42.496 INFO:tasks.workunit.client.0.smithi077.stdout:checking for strftime... yes 2024-03-16T10:18:42.573 INFO:tasks.workunit.client.0.smithi077.stdout:checking whether lstat dereferences a symlink specified with a trailing slash... yes 2024-03-16T10:18:42.641 INFO:tasks.workunit.client.0.smithi077.stdout:checking whether stat accepts an empty string... no 2024-03-16T10:18:42.695 INFO:tasks.workunit.client.0.smithi077.stdout:checking for vprintf... yes 2024-03-16T10:18:42.760 INFO:tasks.workunit.client.0.smithi077.stdout:checking for _doprnt... no 2024-03-16T10:18:42.785 INFO:tasks.workunit.client.0.smithi077.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-16T10:18:42.797 INFO:tasks.workunit.client.0.smithi077.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-16T10:18:42.861 INFO:tasks.workunit.client.0.smithi077.stdout:checking for the pthreads library -lpthreads... no 2024-03-16T10:18:42.914 INFO:tasks.workunit.client.0.smithi077.stdout:checking whether pthreads work without any flags... yes 2024-03-16T10:18:42.959 INFO:tasks.workunit.client.0.smithi077.stdout:checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE 2024-03-16T10:18:42.959 INFO:tasks.workunit.client.0.smithi077.stdout:checking if more special flags are required for pthreads... no 2024-03-16T10:18:42.959 INFO:tasks.workunit.client.0.smithi077.stdout:checking for cc_r... gcc 2024-03-16T10:18:43.041 INFO:tasks.workunit.client.0.smithi077.stdout:checking for setrlimit... yes 2024-03-16T10:18:43.104 INFO:tasks.workunit.client.0.smithi077.stdout:checking for setproctitle... no 2024-03-16T10:18:43.163 INFO:tasks.workunit.client.0.smithi077.stdout:checking for getopt_long... yes 2024-03-16T10:18:43.222 INFO:tasks.workunit.client.0.smithi077.stdout:checking for strtoull... yes 2024-03-16T10:18:43.287 INFO:tasks.workunit.client.0.smithi077.stdout:checking for strtoq... yes 2024-03-16T10:18:43.351 INFO:tasks.workunit.client.0.smithi077.stdout:checking for snprintf... yes 2024-03-16T10:18:43.415 INFO:tasks.workunit.client.0.smithi077.stdout:checking for vsnprintf... yes 2024-03-16T10:18:43.483 INFO:tasks.workunit.client.0.smithi077.stdout:checking for vfprintf... yes 2024-03-16T10:18:43.548 INFO:tasks.workunit.client.0.smithi077.stdout:checking for gethostname... yes 2024-03-16T10:18:43.609 INFO:tasks.workunit.client.0.smithi077.stdout:checking for setlocale... yes 2024-03-16T10:18:43.678 INFO:tasks.workunit.client.0.smithi077.stdout:checking for sync... yes 2024-03-16T10:18:43.749 INFO:tasks.workunit.client.0.smithi077.stdout:checking whether snprintf is C99 conformant... done 2024-03-16T10:18:43.761 INFO:tasks.workunit.client.0.smithi077.stdout:configure: creating ./config.status 2024-03-16T10:18:43.935 INFO:tasks.workunit.client.0.smithi077.stdout:config.status: creating Makefile 2024-03-16T10:18:43.955 INFO:tasks.workunit.client.0.smithi077.stdout:config.status: creating src/Makefile 2024-03-16T10:18:43.976 INFO:tasks.workunit.client.0.smithi077.stdout:config.status: creating man/Makefile 2024-03-16T10:18:43.989 INFO:tasks.workunit.client.0.smithi077.stdout:config.status: creating config.h 2024-03-16T10:18:44.047 INFO:tasks.workunit.client.0.smithi077.stdout:config.status: executing depfiles commands 2024-03-16T10:18:44.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:43 smithi192 ceph-mon[36354]: pgmap v399: 97 pgs: 97 active+clean; 3.1 MiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 239 KiB/s wr, 35 op/s 2024-03-16T10:18:44.186 INFO:tasks.workunit.client.0.smithi077.stderr:+ make 2024-03-16T10:18:44.195 INFO:tasks.workunit.client.0.smithi077.stdout:make all-recursive 2024-03-16T10:18:44.198 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-16T10:18:44.200 INFO:tasks.workunit.client.0.smithi077.stdout:Making all in src 2024-03-16T10:18:44.204 INFO:tasks.workunit.client.0.smithi077.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-16T10:18:44.204 INFO:tasks.workunit.client.0.smithi077.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT blogbench.o -MD -MP -MF ".deps/blogbench.Tpo" -c -o blogbench.o blogbench.c; \ 2024-03-16T10:18:44.204 INFO:tasks.workunit.client.0.smithi077.stdout:then mv -f ".deps/blogbench.Tpo" ".deps/blogbench.Po"; else rm -f ".deps/blogbench.Tpo"; exit 1; fi 2024-03-16T10:18:44.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:43 smithi077 ceph-mon[32400]: pgmap v399: 97 pgs: 97 active+clean; 3.1 MiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 239 KiB/s wr, 35 op/s 2024-03-16T10:18:44.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:43 smithi193 ceph-mon[36311]: pgmap v399: 97 pgs: 97 active+clean; 3.1 MiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 239 KiB/s wr, 35 op/s 2024-03-16T10:18:44.301 INFO:tasks.workunit.client.0.smithi077.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT process.o -MD -MP -MF ".deps/process.Tpo" -c -o process.o process.c; \ 2024-03-16T10:18:44.301 INFO:tasks.workunit.client.0.smithi077.stdout:then mv -f ".deps/process.Tpo" ".deps/process.Po"; else rm -f ".deps/process.Tpo"; exit 1; fi 2024-03-16T10:18:44.394 INFO:tasks.workunit.client.0.smithi077.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT writer.o -MD -MP -MF ".deps/writer.Tpo" -c -o writer.o writer.c; \ 2024-03-16T10:18:44.394 INFO:tasks.workunit.client.0.smithi077.stdout:then mv -f ".deps/writer.Tpo" ".deps/writer.Po"; else rm -f ".deps/writer.Tpo"; exit 1; fi 2024-03-16T10:18:44.469 INFO:tasks.workunit.client.0.smithi077.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT rewriter.o -MD -MP -MF ".deps/rewriter.Tpo" -c -o rewriter.o rewriter.c; \ 2024-03-16T10:18:44.469 INFO:tasks.workunit.client.0.smithi077.stdout:then mv -f ".deps/rewriter.Tpo" ".deps/rewriter.Po"; else rm -f ".deps/rewriter.Tpo"; exit 1; fi 2024-03-16T10:18:44.550 INFO:tasks.workunit.client.0.smithi077.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT reader.o -MD -MP -MF ".deps/reader.Tpo" -c -o reader.o reader.c; \ 2024-03-16T10:18:44.550 INFO:tasks.workunit.client.0.smithi077.stdout:then mv -f ".deps/reader.Tpo" ".deps/reader.Po"; else rm -f ".deps/reader.Tpo"; exit 1; fi 2024-03-16T10:18:44.621 INFO:tasks.workunit.client.0.smithi077.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT commenter.o -MD -MP -MF ".deps/commenter.Tpo" -c -o commenter.o commenter.c; \ 2024-03-16T10:18:44.622 INFO:tasks.workunit.client.0.smithi077.stdout:then mv -f ".deps/commenter.Tpo" ".deps/commenter.Po"; else rm -f ".deps/commenter.Tpo"; exit 1; fi 2024-03-16T10:18:44.698 INFO:tasks.workunit.client.0.smithi077.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT helpers.o -MD -MP -MF ".deps/helpers.Tpo" -c -o helpers.o helpers.c; \ 2024-03-16T10:18:44.698 INFO:tasks.workunit.client.0.smithi077.stdout:then mv -f ".deps/helpers.Tpo" ".deps/helpers.Po"; else rm -f ".deps/helpers.Tpo"; exit 1; fi 2024-03-16T10:18:44.802 INFO:tasks.workunit.client.0.smithi077.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT bsd-getopt_long.o -MD -MP -MF ".deps/bsd-getopt_long.Tpo" -c -o bsd-getopt_long.o bsd-getopt_long.c; \ 2024-03-16T10:18:44.802 INFO:tasks.workunit.client.0.smithi077.stdout:then mv -f ".deps/bsd-getopt_long.Tpo" ".deps/bsd-getopt_long.Po"; else rm -f ".deps/bsd-getopt_long.Tpo"; exit 1; fi 2024-03-16T10:18:44.864 INFO:tasks.workunit.client.0.smithi077.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT fakesnprintf.o -MD -MP -MF ".deps/fakesnprintf.Tpo" -c -o fakesnprintf.o fakesnprintf.c; \ 2024-03-16T10:18:44.864 INFO:tasks.workunit.client.0.smithi077.stdout:then mv -f ".deps/fakesnprintf.Tpo" ".deps/fakesnprintf.Po"; else rm -f ".deps/fakesnprintf.Tpo"; exit 1; fi 2024-03-16T10:18:44.922 INFO:tasks.workunit.client.0.smithi077.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT mysnprintf.o -MD -MP -MF ".deps/mysnprintf.Tpo" -c -o mysnprintf.o mysnprintf.c; \ 2024-03-16T10:18:44.923 INFO:tasks.workunit.client.0.smithi077.stdout:then mv -f ".deps/mysnprintf.Tpo" ".deps/mysnprintf.Po"; else rm -f ".deps/mysnprintf.Tpo"; exit 1; fi 2024-03-16T10:18:44.974 INFO:tasks.workunit.client.0.smithi077.stdout:gcc -g -O2 -o blogbench blogbench.o process.o writer.o rewriter.o reader.o commenter.o helpers.o bsd-getopt_long.o fakesnprintf.o mysnprintf.o 2024-03-16T10:18:44.993 INFO:tasks.workunit.client.0.smithi077.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-16T10:18:44.993 INFO:tasks.workunit.client.0.smithi077.stdout:Making all in man 2024-03-16T10:18:44.996 INFO:tasks.workunit.client.0.smithi077.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-16T10:18:44.996 INFO:tasks.workunit.client.0.smithi077.stdout:make[2]: Nothing to be done for 'all'. 2024-03-16T10:18:44.996 INFO:tasks.workunit.client.0.smithi077.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-16T10:18:44.997 INFO:tasks.workunit.client.0.smithi077.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-16T10:18:44.998 INFO:tasks.workunit.client.0.smithi077.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-16T10:18:44.998 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-16T10:18:44.999 INFO:tasks.workunit.client.0.smithi077.stderr:+ cd src 2024-03-16T10:18:45.003 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir blogtest_in 2024-03-16T10:18:45.007 INFO:tasks.workunit.client.0.smithi077.stdout:running blogbench 2024-03-16T10:18:45.007 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 'running blogbench' 2024-03-16T10:18:45.007 INFO:tasks.workunit.client.0.smithi077.stderr:+ ./blogbench -d blogtest_in 2024-03-16T10:18:45.008 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T10:18:45.008 INFO:tasks.workunit.client.0.smithi077.stdout:Frequency = 10 secs 2024-03-16T10:18:45.008 INFO:tasks.workunit.client.0.smithi077.stdout:Scratch dir = [blogtest_in] 2024-03-16T10:18:46.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:45 smithi192 ceph-mon[36354]: pgmap v400: 97 pgs: 97 active+clean; 3.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 254 KiB/s wr, 41 op/s 2024-03-16T10:18:46.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:45 smithi077 ceph-mon[32400]: pgmap v400: 97 pgs: 97 active+clean; 3.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 254 KiB/s wr, 41 op/s 2024-03-16T10:18:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:45 smithi193 ceph-mon[36311]: pgmap v400: 97 pgs: 97 active+clean; 3.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 254 KiB/s wr, 41 op/s 2024-03-16T10:18:46.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:18:46.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:18:46.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.579+0000 7f5bb2962640 1 -- 172.21.15.77:0/61670050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bac102290 msgr2=0x7f5bac108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.579+0000 7f5bb2962640 1 --2- 172.21.15.77:0/61670050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bac102290 0x7f5bac108b20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009ca0 tx=0x7f5ba402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.579+0000 7f5bb2962640 1 -- 172.21.15.77:0/61670050 shutdown_connections 2024-03-16T10:18:46.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.579+0000 7f5bb2962640 1 --2- 172.21.15.77:0/61670050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bac102290 0x7f5bac108b20 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.579+0000 7f5bb2962640 1 --2- 172.21.15.77:0/61670050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5bac1018d0 0x7f5bac101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.579+0000 7f5bb2962640 1 --2- 172.21.15.77:0/61670050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bac1006e0 0x7f5bac100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.579+0000 7f5bb2962640 1 -- 172.21.15.77:0/61670050 >> 172.21.15.77:0/61670050 conn(0x7f5bac0fbc70 msgr2=0x7f5bac0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:46.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.579+0000 7f5bb2962640 1 -- 172.21.15.77:0/61670050 shutdown_connections 2024-03-16T10:18:46.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.579+0000 7f5bb2962640 1 -- 172.21.15.77:0/61670050 wait complete. 2024-03-16T10:18:46.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.580+0000 7f5bb2962640 1 Processor -- start 2024-03-16T10:18:46.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.580+0000 7f5bb2962640 1 -- start start 2024-03-16T10:18:46.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.580+0000 7f5bb2962640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bac1006e0 0x7f5bac197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.580+0000 7f5bb2962640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5bac1018d0 0x7f5bac1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.580+0000 7f5bb2962640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bac102290 0x7f5bac19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.580+0000 7f5bb2962640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac10c360 con 0x7f5bac1006e0 2024-03-16T10:18:46.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.580+0000 7f5bb2962640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac10c1e0 con 0x7f5bac102290 2024-03-16T10:18:46.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.580+0000 7f5bb2962640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac10c4e0 con 0x7f5bac1018d0 2024-03-16T10:18:46.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5babfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bac1006e0 0x7f5bac197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5babfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bac1006e0 0x7f5bac197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60784/0 (socket says 172.21.15.77:60784) 2024-03-16T10:18:46.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5babfff640 1 -- 172.21.15.77:0/3197223415 learned_addr learned my addr 172.21.15.77:0/3197223415 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:46.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5bb0eff640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bac102290 0x7f5bac19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.584 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5bab7fe640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5bac1018d0 0x7f5bac1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5babfff640 1 -- 172.21.15.77:0/3197223415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5bac1018d0 msgr2=0x7f5bac1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5babfff640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5bac1018d0 0x7f5bac1980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5babfff640 1 -- 172.21.15.77:0/3197223415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bac102290 msgr2=0x7f5bac19f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5babfff640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bac102290 0x7f5bac19f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5babfff640 1 -- 172.21.15.77:0/3197223415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bac19f730 con 0x7f5bac1006e0 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5bb0eff640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bac102290 0x7f5bac19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5babfff640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bac1006e0 0x7f5bac197b60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00e9c0 tx=0x7f5b9c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5ba97fa640 1 -- 172.21.15.77:0/3197223415 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c00cd80 con 0x7f5bac1006e0 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5bb2962640 1 -- 172.21.15.77:0/3197223415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bac19fa20 con 0x7f5bac1006e0 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5ba97fa640 1 -- 172.21.15.77:0/3197223415 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b9c0035f0 con 0x7f5bac1006e0 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5ba97fa640 1 -- 172.21.15.77:0/3197223415 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c010690 con 0x7f5bac1006e0 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.581+0000 7f5bb2962640 1 -- 172.21.15.77:0/3197223415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bac19ff70 con 0x7f5bac1006e0 2024-03-16T10:18:46.585 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.583+0000 7f5ba97fa640 1 -- 172.21.15.77:0/3197223415 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f5b9c0037a0 con 0x7f5bac1006e0 2024-03-16T10:18:46.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bb562f740 1 Processor -- start 2024-03-16T10:18:46.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5ba97fa640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5b940a2800 0x7f5b940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bb562f740 1 -- start start 2024-03-16T10:18:46.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bb562f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17a8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bb562f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17ad660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bb562f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17b0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bb562f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5589e170d500 con 0x5589e17a8170 2024-03-16T10:18:46.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bb562f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5589e16b5be0 con 0x5589e17adcd0 2024-03-16T10:18:46.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bb562f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5589e17095a0 con 0x5589e17a8b60 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5ba97fa640 1 -- 172.21.15.77:0/3197223415 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f5b9c0ced80 con 0x7f5bac1006e0 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bb0eff640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5b940a2800 0x7f5b940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5babfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bab7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17ad660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5babfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17a8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60788/0 (socket says 172.21.15.77:60788) 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5babfff640 1 -- 172.21.15.77:0/4036204067 learned_addr learned my addr 172.21.15.77:0/4036204067 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bb0eff640 1 --2- 172.21.15.77:0/4036204067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5babfff640 1 -- 172.21.15.77:0/4036204067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 msgr2=0x5589e17ad660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5babfff640 1 --2- 172.21.15.77:0/4036204067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17ad660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5babfff640 1 -- 172.21.15.77:0/4036204067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 msgr2=0x5589e17b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5babfff640 1 --2- 172.21.15.77:0/4036204067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5babfff640 1 -- 172.21.15.77:0/4036204067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589e16c94b0 con 0x5589e17a8170 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bab7fe640 1 --2- 172.21.15.77:0/4036204067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17ad660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:46.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.584+0000 7f5bb0eff640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5b940a2800 0x7f5b940a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4005e30 tx=0x7f5ba402ce60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.585+0000 7f5babfff640 1 --2- 172.21.15.77:0/4036204067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17a8590 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0a3a20 tx=0x7f5b9c097b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1d72ce3c45aa5e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.585+0000 7f5b927fc640 1 -- 172.21.15.77:0/4036204067 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c0b4070 con 0x5589e17a8170 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.585+0000 7f5b927fc640 1 -- 172.21.15.77:0/4036204067 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b9c09f900 con 0x5589e17a8170 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.585+0000 7f5b927fc640 1 -- 172.21.15.77:0/4036204067 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c0861e0 con 0x5589e17a8170 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.585+0000 7f5bb562f740 1 -- 172.21.15.77:0/4036204067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 msgr2=0x5589e17a8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.585+0000 7f5bb562f740 1 --2- 172.21.15.77:0/4036204067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17a8590 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0a3a20 tx=0x7f5b9c097b80 comp rx=0 tx=0).stop 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.585+0000 7f5bb562f740 1 -- 172.21.15.77:0/4036204067 shutdown_connections 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.585+0000 7f5bb562f740 1 --2- 172.21.15.77:0/4036204067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.585+0000 7f5bb562f740 1 --2- 172.21.15.77:0/4036204067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17ad660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.585+0000 7f5bb562f740 1 --2- 172.21.15.77:0/4036204067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17a8590 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.585+0000 7f5bb562f740 1 -- 172.21.15.77:0/4036204067 >> 172.21.15.77:0/4036204067 conn(0x5589e17a36e0 msgr2=0x5589e17a3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5bb562f740 1 -- 172.21.15.77:0/4036204067 shutdown_connections 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5bb562f740 1 -- 172.21.15.77:0/4036204067 wait complete. 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5bb562f740 1 -- start start 2024-03-16T10:18:46.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5bb562f740 1 Processor -- start 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5bb562f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5bb562f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5bb562f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17b9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5bb562f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5589e17b0c80 con 0x5589e17a8170 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5bb562f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5589e16c9160 con 0x5589e17adcd0 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5bb562f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5589e17b0e00 con 0x5589e17a8b60 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5bb562f740 10 client.0 _ll_register_callbacks cb 0x5589e16ad150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5babfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5babfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60794/0 (socket says 172.21.15.77:60794) 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5babfff640 1 -- 172.21.15.77:0/3104393564 learned_addr learned my addr 172.21.15.77:0/3104393564 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.586+0000 7f5bab7fe640 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5bb0eff640 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17b9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5babfff640 1 -- 172.21.15.77:0/3104393564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 msgr2=0x5589e17b2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5babfff640 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17b2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5babfff640 1 -- 172.21.15.77:0/3104393564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 msgr2=0x5589e17b9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5babfff640 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17b9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5babfff640 1 -- 172.21.15.77:0/3104393564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589e14eb7d0 con 0x5589e17a8170 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5babfff640 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17b2740 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c098180 tx=0x7f5b9c003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b320872bec955ba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5b8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5b8b7fe640 1 -- 172.21.15.77:0/3104393564 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c07f060 con 0x5589e17a8170 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5b8b7fe640 1 -- 172.21.15.77:0/3104393564 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f5b9c099850 con 0x5589e17a8170 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5bb562f740 1 -- 172.21.15.77:0/3104393564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b9c09f900 con 0x5589e17a8170 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5b8b7fe640 1 -- 172.21.15.77:0/3104393564 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5b9c0b8030 con 0x5589e17a8170 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5bb562f740 10 client.15957 fetch_fsmap learned FSMap version 28 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5bb562f740 1 -- 172.21.15.77:0/3104393564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5589e14213f0 con 0x5589e17a8170 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5b8b7fe640 1 -- 172.21.15.77:0/3104393564 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b9c09f900 con 0x5589e17a8170 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.587+0000 7f5b8b7fe640 1 -- 172.21.15.77:0/3104393564 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f5b9c0975e0 con 0x5589e17a8170 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.588+0000 7f5bb562f740 10 client.15957 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.588+0000 7f5bb562f740 10 client.15957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:46.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.588+0000 7f5bb562f740 20 client.15957 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.588+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5589e17bde20 0x5589e17c0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.588+0000 7f5bb562f740 4 client.15957 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.588+0000 7f5bb562f740 1 -- 172.21.15.77:0/3104393564 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589e171aa40 con 0x5589e17bde20 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.588+0000 7f5bab7fe640 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5589e17bde20 0x5589e17c0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.588+0000 7f5b8b7fe640 10 client.15957 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.588+0000 7f5bab7fe640 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5589e17bde20 0x5589e17c0210 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f16ed88182f9fa1 server_cookie=d0e07d49c1047c23 in_seq=0 out_seq=0 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5b8b7fe640 1 -- 172.21.15.77:0/3104393564 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5589e171aa40 con 0x5589e17bde20 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5b8b7fe640 10 client.15957 handle_command_reply: tid=0 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5bb562f740 1 client.15957 shutdown 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5bb562f740 1 -- 172.21.15.77:0/3104393564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 msgr2=0x5589e17b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17b2740 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c098180 tx=0x7f5b9c003e30 comp rx=0 tx=0).stop 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5bb562f740 1 -- 172.21.15.77:0/3104393564 shutdown_connections 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17b9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5589e17bde20 0x5589e17c0210 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3104393564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17b2740 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5b8b7fe640 0 client.15957 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:46.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5bb562f740 1 -- 172.21.15.77:0/3104393564 >> 172.21.15.77:0/3104393564 conn(0x5589e17a36e0 msgr2=0x5589e17a2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:46.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5bb562f740 1 -- 172.21.15.77:0/3104393564 shutdown_connections 2024-03-16T10:18:46.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5bb562f740 1 -- 172.21.15.77:0/3104393564 wait complete. 2024-03-16T10:18:46.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.589+0000 7f5bb562f740 20 client.15957 trim_cache size 0 max 16384 2024-03-16T10:18:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.606+0000 7f5bb562f740 1 Processor -- start 2024-03-16T10:18:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.606+0000 7f5bb562f740 1 -- start start 2024-03-16T10:18:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5bb562f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5bb562f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17b9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5bb562f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5bb562f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1a8c50 con 0x5589e17a8170 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5bb562f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1aa7c0 con 0x5589e17adcd0 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5bb562f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1a9320 con 0x5589e17a8b60 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5babfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5babfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17b2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60806/0 (socket says 172.21.15.77:60806) 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5babfff640 1 -- 172.21.15.77:0/3463409178 learned_addr learned my addr 172.21.15.77:0/3463409178 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5bb0eff640 1 --2- 172.21.15.77:0/3463409178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5bab7fe640 1 --2- 172.21.15.77:0/3463409178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17b9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5babfff640 1 -- 172.21.15.77:0/3463409178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 msgr2=0x5589e17b9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5babfff640 1 --2- 172.21.15.77:0/3463409178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17b9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5babfff640 1 -- 172.21.15.77:0/3463409178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 msgr2=0x5589e17a3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5babfff640 1 --2- 172.21.15.77:0/3463409178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17a3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5babfff640 1 -- 172.21.15.77:0/3463409178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589e13e10f0 con 0x5589e17a8170 2024-03-16T10:18:46.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5bb0eff640 1 --2- 172.21.15.77:0/3463409178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5babfff640 1 --2- 172.21.15.77:0/3463409178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17b2be0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00c6a0 tx=0x7f5b9c003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35132ed0a880f382 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5b91ffb640 1 -- 172.21.15.77:0/3463409178 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c0b4040 con 0x5589e17a8170 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5b91ffb640 1 -- 172.21.15.77:0/3463409178 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b9c09f900 con 0x5589e17a8170 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.607+0000 7f5b91ffb640 1 -- 172.21.15.77:0/3463409178 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c0afda0 con 0x5589e17a8170 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.608+0000 7f5bb562f740 1 -- 172.21.15.77:0/3463409178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 msgr2=0x5589e17b2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.608+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3463409178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17b2be0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00c6a0 tx=0x7f5b9c003ea0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.608+0000 7f5bb562f740 1 -- 172.21.15.77:0/3463409178 shutdown_connections 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.608+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3463409178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17adcd0 0x5589e17a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.608+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3463409178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8b60 0x5589e17b9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.608+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3463409178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17a8170 0x5589e17b2be0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.608+0000 7f5bb562f740 1 -- 172.21.15.77:0/3463409178 >> 172.21.15.77:0/3463409178 conn(0x5589e17a36e0 msgr2=0x5589e17b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.608+0000 7f5bb562f740 1 -- 172.21.15.77:0/3463409178 shutdown_connections 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.608+0000 7f5bb562f740 1 -- 172.21.15.77:0/3463409178 wait complete. 2024-03-16T10:18:46.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.608+0000 7f5bb562f740 1 -- start start 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bb562f740 1 Processor -- start 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bb562f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8170 0x5589e17b6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bb562f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17a8b60 0x5589e17b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bb562f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17adcd0 0x5589e17b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bb562f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1aa0c0 con 0x5589e17adcd0 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bb562f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1a99f0 con 0x5589e17a8b60 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bb562f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1a6c30 con 0x5589e17a8170 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bb562f740 10 client.0 _ll_register_callbacks cb 0x7f5b9c0b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bb0eff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17adcd0 0x5589e17b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5babfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8170 0x5589e17b6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bb0eff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17adcd0 0x5589e17b81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60820/0 (socket says 172.21.15.77:60820) 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bb0eff640 1 -- 172.21.15.77:0/3225188346 learned_addr learned my addr 172.21.15.77:0/3225188346 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bab7fe640 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17a8b60 0x5589e17b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5babfff640 1 -- 172.21.15.77:0/3225188346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17a8b60 msgr2=0x5589e17b71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5babfff640 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17a8b60 0x5589e17b71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5babfff640 1 -- 172.21.15.77:0/3225188346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17adcd0 msgr2=0x5589e17b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5babfff640 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17adcd0 0x5589e17b81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5babfff640 1 -- 172.21.15.77:0/3225188346 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589e16d49d0 con 0x5589e17a8170 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bab7fe640 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17a8b60 0x5589e17b71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5bb0eff640 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17adcd0 0x5589e17b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5babfff640 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8170 0x5589e17b6ca0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0058c0 tx=0x7f5b9c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b63048295f0c1cd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.609+0000 7f5b8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5b8b7fe640 1 -- 172.21.15.77:0/3225188346 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c0b4040 con 0x5589e17a8170 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5b8b7fe640 1 -- 172.21.15.77:0/3225188346 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f5b9c097710 con 0x5589e17a8170 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5bb562f740 1 -- 172.21.15.77:0/3225188346 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b9c09f900 con 0x5589e17a8170 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5b8b7fe640 1 -- 172.21.15.77:0/3225188346 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5b9c09b580 con 0x5589e17a8170 2024-03-16T10:18:46.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5bb562f740 10 client.25735 fetch_fsmap learned FSMap version 28 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5bb562f740 1 -- 172.21.15.77:0/3225188346 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5589e16d49d0 con 0x5589e17a8170 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5b8b7fe640 1 -- 172.21.15.77:0/3225188346 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b9c09f900 con 0x5589e17a8170 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5b8b7fe640 1 -- 172.21.15.77:0/3225188346 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f5b9c0a08b0 con 0x5589e17a8170 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5bb562f740 10 client.25735 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5bb562f740 10 client.25735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5bb562f740 20 client.25735 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5589e1824040 0x5589e1826430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5bb562f740 4 client.25735 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5bb562f740 1 -- 172.21.15.77:0/3225188346 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5589e1714d90 con 0x5589e1824040 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.610+0000 7f5bab7fe640 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5589e1824040 0x5589e1826430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5b8b7fe640 10 client.25735 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bab7fe640 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5589e1824040 0x5589e1826430 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82cc0d9439f0fa7d server_cookie=33c864f50397463c in_seq=0 out_seq=0 2024-03-16T10:18:46.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5b8b7fe640 1 -- 172.21.15.77:0/3225188346 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5589e1714d90 con 0x5589e1824040 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5b8b7fe640 10 client.25735 handle_command_reply: tid=0 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bb562f740 1 client.25735 shutdown 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bb562f740 1 -- 172.21.15.77:0/3225188346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8170 msgr2=0x5589e17b6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8170 0x5589e17b6ca0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0058c0 tx=0x7f5b9c0046c0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bb562f740 1 -- 172.21.15.77:0/3225188346 shutdown_connections 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5589e1824040 0x5589e1826430 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589e17adcd0 0x5589e17b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589e17a8b60 0x5589e17b71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5b8b7fe640 0 client.25735 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bb562f740 1 --2- 172.21.15.77:0/3225188346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589e17a8170 0x5589e17b6ca0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bb562f740 1 -- 172.21.15.77:0/3225188346 >> 172.21.15.77:0/3225188346 conn(0x5589e17a36e0 msgr2=0x5589e17b2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bb562f740 1 -- 172.21.15.77:0/3225188346 shutdown_connections 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bb562f740 1 -- 172.21.15.77:0/3225188346 wait complete. 2024-03-16T10:18:46.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.611+0000 7f5bb562f740 20 client.25735 trim_cache size 0 max 16384 2024-03-16T10:18:46.620 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:18:46.620 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T10:18:46.620 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T10:18:46.620 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:18:46.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.612+0000 7f5bb2962640 1 -- 172.21.15.77:0/3197223415 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5b940a2800 msgr2=0x7f5b940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.612+0000 7f5bb2962640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5b940a2800 0x7f5b940a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4005e30 tx=0x7f5ba402ce60 comp rx=0 tx=0).stop 2024-03-16T10:18:46.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.612+0000 7f5bb2962640 1 -- 172.21.15.77:0/3197223415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bac1006e0 msgr2=0x7f5bac197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.612+0000 7f5bb2962640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bac1006e0 0x7f5bac197b60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00e9c0 tx=0x7f5b9c00ee80 comp rx=0 tx=0).stop 2024-03-16T10:18:46.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.612+0000 7f5bb2962640 1 -- 172.21.15.77:0/3197223415 shutdown_connections 2024-03-16T10:18:46.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.612+0000 7f5bb2962640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bac102290 0x7f5bac19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.612+0000 7f5bb2962640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f5b940a2800 0x7f5b940a4cc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.612+0000 7f5bb2962640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5bac1018d0 0x7f5bac1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.612+0000 7f5bb2962640 1 --2- 172.21.15.77:0/3197223415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bac1006e0 0x7f5bac197b60 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.612+0000 7f5bb2962640 1 -- 172.21.15.77:0/3197223415 >> 172.21.15.77:0/3197223415 conn(0x7f5bac0fbc70 msgr2=0x7f5bac0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:46.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.612+0000 7f5bb2962640 1 -- 172.21.15.77:0/3197223415 shutdown_connections 2024-03-16T10:18:46.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.612+0000 7f5bb2962640 1 -- 172.21.15.77:0/3197223415 wait complete. 2024-03-16T10:18:46.623 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T10:18:46.623 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.726+0000 7f8f1d07e640 1 -- 172.21.15.77:0/4236037307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f18100700 msgr2=0x7f8f18100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.726+0000 7f8f1d07e640 1 --2- 172.21.15.77:0/4236037307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f18100700 0x7f8f18100b20 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8f08009900 tx=0x7f8f0802bc30 comp rx=0 tx=0).stop 2024-03-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.726+0000 7f8f1d07e640 1 -- 172.21.15.77:0/4236037307 shutdown_connections 2024-03-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.726+0000 7f8f1d07e640 1 --2- 172.21.15.77:0/4236037307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f181022b0 0x7f8f18108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.726+0000 7f8f1d07e640 1 --2- 172.21.15.77:0/4236037307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f181018f0 0x7f8f18101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.726+0000 7f8f1d07e640 1 --2- 172.21.15.77:0/4236037307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f18100700 0x7f8f18100b20 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.726+0000 7f8f1d07e640 1 -- 172.21.15.77:0/4236037307 >> 172.21.15.77:0/4236037307 conn(0x7f8f180fbc70 msgr2=0x7f8f180fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.726+0000 7f8f1d07e640 1 -- 172.21.15.77:0/4236037307 shutdown_connections 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.726+0000 7f8f1d07e640 1 -- 172.21.15.77:0/4236037307 wait complete. 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f1d07e640 1 Processor -- start 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f1d07e640 1 -- start start 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f1d07e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f18100700 0x7f8f18196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f1d07e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f181018f0 0x7f8f18196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f1d07e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f181022b0 0x7f8f1819dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f1d07e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1810c380 con 0x7f8f18100700 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f1d07e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1810c200 con 0x7f8f181018f0 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f1d07e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1810c500 con 0x7f8f181022b0 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f16d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f18100700 0x7f8f18196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f1759e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f181022b0 0x7f8f1819dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f16d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f18100700 0x7f8f18196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60826/0 (socket says 172.21.15.77:60826) 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f1759e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f181022b0 0x7f8f1819dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:40158/0 (socket says 172.21.15.77:40158) 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f16d9d640 1 -- 172.21.15.77:0/351101964 learned_addr learned my addr 172.21.15.77:0/351101964 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f1659c640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f181018f0 0x7f8f18196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f16d9d640 1 -- 172.21.15.77:0/351101964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f181022b0 msgr2=0x7f8f1819dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f16d9d640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f181022b0 0x7f8f1819dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f16d9d640 1 -- 172.21.15.77:0/351101964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f181018f0 msgr2=0x7f8f18196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f16d9d640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f181018f0 0x7f8f18196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.727+0000 7f8f16d9d640 1 -- 172.21.15.77:0/351101964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1819e640 con 0x7f8f18100700 2024-03-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.728+0000 7f8f16d9d640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f18100700 0x7f8f18196a20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8f08000c00 tx=0x7f8f0802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.728+0000 7f8efffff640 1 -- 172.21.15.77:0/351101964 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f08004450 con 0x7f8f18100700 2024-03-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.728+0000 7f8f1d07e640 1 -- 172.21.15.77:0/351101964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f18068940 con 0x7f8f18100700 2024-03-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.728+0000 7f8efffff640 1 -- 172.21.15.77:0/351101964 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f080054e0 con 0x7f8f18100700 2024-03-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.728+0000 7f8efffff640 1 -- 172.21.15.77:0/351101964 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f0803b440 con 0x7f8f18100700 2024-03-16T10:18:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.728+0000 7f8f1d07e640 1 -- 172.21.15.77:0/351101964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f18068e80 con 0x7f8f18100700 2024-03-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.729+0000 7f8f1d07e640 1 -- 172.21.15.77:0/351101964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f180685d0 con 0x7f8f18100700 2024-03-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.731+0000 7f8efffff640 1 -- 172.21.15.77:0/351101964 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f8f08003780 con 0x7f8f18100700 2024-03-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.731+0000 7f8efffff640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8f000a2800 0x7f8f000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:46.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.731+0000 7f8f1659c640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8f000a2800 0x7f8f000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.732+0000 7f8efffff640 1 -- 172.21.15.77:0/351101964 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8f080f2870 con 0x7f8f18100700 2024-03-16T10:18:46.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.732+0000 7f8f1659c640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8f000a2800 0x7f8f000a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8f18197f30 tx=0x7f8f0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:46.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.738+0000 7f8efffff640 1 -- 172.21.15.77:0/351101964 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f080bf990 con 0x7f8f18100700 2024-03-16T10:18:46.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.961+0000 7f8f1d07e640 1 -- 172.21.15.77:0/351101964 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f18107490 con 0x7f8f18100700 2024-03-16T10:18:46.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.962+0000 7f8efffff640 1 -- 172.21.15.77:0/351101964 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11945 (secure 0 0 0) 0x7f8f080bf290 con 0x7f8f18100700 2024-03-16T10:18:46.963 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:18:46.963 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:18:46.041217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24454,24475,24482,24455,14658]},"id":1}]} 2024-03-16T10:18:46.963 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 29 2024-03-16T10:18:46.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.964+0000 7f8f1d07e640 1 -- 172.21.15.77:0/351101964 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8f000a2800 msgr2=0x7f8f000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.964+0000 7f8f1d07e640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8f000a2800 0x7f8f000a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8f18197f30 tx=0x7f8f0c005f90 comp rx=0 tx=0).stop 2024-03-16T10:18:46.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.964+0000 7f8f1d07e640 1 -- 172.21.15.77:0/351101964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f18100700 msgr2=0x7f8f18196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:46.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.964+0000 7f8f1d07e640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f18100700 0x7f8f18196a20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8f08000c00 tx=0x7f8f0802c640 comp rx=0 tx=0).stop 2024-03-16T10:18:46.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.964+0000 7f8f1d07e640 1 -- 172.21.15.77:0/351101964 shutdown_connections 2024-03-16T10:18:46.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.964+0000 7f8f1d07e640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8f000a2800 0x7f8f000a4cc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.964+0000 7f8f1d07e640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f181022b0 0x7f8f1819dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.964+0000 7f8f1d07e640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f181018f0 0x7f8f18196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.964+0000 7f8f1d07e640 1 --2- 172.21.15.77:0/351101964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f18100700 0x7f8f18196a20 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:46.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.964+0000 7f8f1d07e640 1 -- 172.21.15.77:0/351101964 >> 172.21.15.77:0/351101964 conn(0x7f8f180fbc70 msgr2=0x7f8f18104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:46.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.964+0000 7f8f1d07e640 1 -- 172.21.15.77:0/351101964 shutdown_connections 2024-03-16T10:18:46.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:46.964+0000 7f8f1d07e640 1 -- 172.21.15.77:0/351101964 wait complete. 2024-03-16T10:18:46.976 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-16T10:18:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.086+0000 7fb55892f640 1 -- 172.21.15.77:0/4039508728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5541018d0 msgr2=0x7fb554101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.086+0000 7fb55892f640 1 --2- 172.21.15.77:0/4039508728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5541018d0 0x7fb554101d50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb548009900 tx=0x7fb54802bc30 comp rx=0 tx=0).stop 2024-03-16T10:18:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.086+0000 7fb55892f640 1 -- 172.21.15.77:0/4039508728 shutdown_connections 2024-03-16T10:18:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.086+0000 7fb55892f640 1 --2- 172.21.15.77:0/4039508728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb554102290 0x7fb554108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.086+0000 7fb55892f640 1 --2- 172.21.15.77:0/4039508728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5541018d0 0x7fb554101d50 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.086+0000 7fb55892f640 1 --2- 172.21.15.77:0/4039508728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5541006e0 0x7fb554100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.086+0000 7fb55892f640 1 -- 172.21.15.77:0/4039508728 >> 172.21.15.77:0/4039508728 conn(0x7fb5540fbc70 msgr2=0x7fb5540fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.086+0000 7fb55892f640 1 -- 172.21.15.77:0/4039508728 shutdown_connections 2024-03-16T10:18:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.086+0000 7fb55892f640 1 -- 172.21.15.77:0/4039508728 wait complete. 2024-03-16T10:18:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.086+0000 7fb55892f640 1 Processor -- start 2024-03-16T10:18:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.086+0000 7fb55892f640 1 -- start start 2024-03-16T10:18:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55892f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5541006e0 0x7fb5540694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55892f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5541018d0 0x7fb55419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55892f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb554102290 0x7fb55419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55892f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb55410c360 con 0x7fb5541006e0 2024-03-16T10:18:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55892f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb55410c1e0 con 0x7fb5541018d0 2024-03-16T10:18:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55892f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb55410c4e0 con 0x7fb554102290 2024-03-16T10:18:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5541006e0 0x7fb5540694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5541006e0 0x7fb5540694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60844/0 (socket says 172.21.15.77:60844) 2024-03-16T10:18:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55259c640 1 -- 172.21.15.77:0/2762386058 learned_addr learned my addr 172.21.15.77:0/2762386058 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb551d9b640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5541018d0 0x7fb55419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb552d9d640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb554102290 0x7fb55419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55259c640 1 -- 172.21.15.77:0/2762386058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb554102290 msgr2=0x7fb55419ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55259c640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb554102290 0x7fb55419ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55259c640 1 -- 172.21.15.77:0/2762386058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5541018d0 msgr2=0x7fb55419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55259c640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5541018d0 0x7fb55419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55259c640 1 -- 172.21.15.77:0/2762386058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb55419f7a0 con 0x7fb5541006e0 2024-03-16T10:18:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55259c640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5541006e0 0x7fb5540694b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb54400e9c0 tx=0x7fb54400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb5437fe640 1 -- 172.21.15.77:0/2762386058 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54400cd80 con 0x7fb5541006e0 2024-03-16T10:18:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb5437fe640 1 -- 172.21.15.77:0/2762386058 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5440035f0 con 0x7fb5541006e0 2024-03-16T10:18:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55892f640 1 -- 172.21.15.77:0/2762386058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb55419fa30 con 0x7fb5541006e0 2024-03-16T10:18:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb5437fe640 1 -- 172.21.15.77:0/2762386058 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb544010690 con 0x7fb5541006e0 2024-03-16T10:18:47.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.087+0000 7fb55892f640 1 -- 172.21.15.77:0/2762386058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb55419ff00 con 0x7fb5541006e0 2024-03-16T10:18:47.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.090+0000 7fb5437fe640 1 -- 172.21.15.77:0/2762386058 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fb5440037a0 con 0x7fb5541006e0 2024-03-16T10:18:47.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.090+0000 7fb5437fe640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb53c0a2800 0x7fb53c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.090+0000 7fb551d9b640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb53c0a2800 0x7fb53c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.090+0000 7fb5437fe640 1 -- 172.21.15.77:0/2762386058 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb5440ced30 con 0x7fb5541006e0 2024-03-16T10:18:47.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.090+0000 7fb55b5f0740 1 Processor -- start 2024-03-16T10:18:47.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55b5f0740 1 -- start start 2024-03-16T10:18:47.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb551d9b640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb53c0a2800 0x7fb53c0a4cc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb54802c7b0 tx=0x7fb548005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55b5f0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 0x55a86ccfd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55b5f0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfd650 0x55a86cd02150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55b5f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86cd027c0 0x55a86cd04c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55b5f0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a86cc5d490 con 0x55a86ccfcc60 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55b5f0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a86cc05c90 con 0x55a86ccfd650 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55b5f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a86cc59530 con 0x55a86cd027c0 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 0x55a86ccfd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 0x55a86ccfd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60858/0 (socket says 172.21.15.77:60858) 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55259c640 1 -- 172.21.15.77:0/1672212680 learned_addr learned my addr 172.21.15.77:0/1672212680 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb552d9d640 1 --2- 172.21.15.77:0/1672212680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86cd027c0 0x55a86cd04c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb551d9b640 1 --2- 172.21.15.77:0/1672212680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfd650 0x55a86cd02150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55259c640 1 -- 172.21.15.77:0/1672212680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86cd027c0 msgr2=0x55a86cd04c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55259c640 1 --2- 172.21.15.77:0/1672212680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86cd027c0 0x55a86cd04c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55259c640 1 -- 172.21.15.77:0/1672212680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfd650 msgr2=0x55a86cd02150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55259c640 1 --2- 172.21.15.77:0/1672212680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfd650 0x55a86cd02150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.091+0000 7fb55259c640 1 -- 172.21.15.77:0/1672212680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86ccc0500 con 0x55a86ccfcc60 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.092+0000 7fb55259c640 1 --2- 172.21.15.77:0/1672212680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 0x55a86ccfd080 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb544097bc0 tx=0x7fb54409b0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308e096137dc0e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.092+0000 7fb540ff9640 1 -- 172.21.15.77:0/1672212680 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5440b4070 con 0x55a86ccfcc60 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.092+0000 7fb540ff9640 1 -- 172.21.15.77:0/1672212680 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54409f8b0 con 0x55a86ccfcc60 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.092+0000 7fb540ff9640 1 -- 172.21.15.77:0/1672212680 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5440a3a70 con 0x55a86ccfcc60 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.093+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1672212680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 msgr2=0x55a86ccfd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.093+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/1672212680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 0x55a86ccfd080 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb544097bc0 tx=0x7fb54409b0c0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.093+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1672212680 shutdown_connections 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.093+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/1672212680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86cd027c0 0x55a86cd04c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.093+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/1672212680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfd650 0x55a86cd02150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.093+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/1672212680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 0x55a86ccfd080 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.093+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1672212680 >> 172.21.15.77:0/1672212680 conn(0x55a86ccf3dc0 msgr2=0x55a86ccf41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.093+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1672212680 shutdown_connections 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.093+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1672212680 wait complete. 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.093+0000 7fb55b5f0740 1 -- start start 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.093+0000 7fb55b5f0740 1 Processor -- start 2024-03-16T10:18:47.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55b5f0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 0x55a86cd071b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55b5f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfd650 0x55a86cd076f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55b5f0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86cd027c0 0x55a86cd0e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55b5f0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a86cd05770 con 0x55a86ccfcc60 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55b5f0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a86cc19230 con 0x55a86cd027c0 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55b5f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a86cd058f0 con 0x55a86ccfd650 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 0x55a86cd071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55b5f0740 10 client.0 _ll_register_callbacks cb 0x55a86cc55010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 0x55a86cd071b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60860/0 (socket says 172.21.15.77:60860) 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55259c640 1 -- 172.21.15.77:0/1156122424 learned_addr learned my addr 172.21.15.77:0/1156122424 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb551d9b640 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfd650 0x55a86cd076f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb552d9d640 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86cd027c0 0x55a86cd0e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55259c640 1 -- 172.21.15.77:0/1156122424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfd650 msgr2=0x55a86cd076f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55259c640 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfd650 0x55a86cd076f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55259c640 1 -- 172.21.15.77:0/1156122424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86cd027c0 msgr2=0x55a86cd0e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55259c640 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86cd027c0 0x55a86cd0e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55259c640 1 -- 172.21.15.77:0/1156122424 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a86cc28100 con 0x55a86ccfcc60 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55259c640 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 0x55a86cd071b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb54400c6a0 tx=0x7fb544005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de2743f04f444b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb539ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb539ffb640 1 -- 172.21.15.77:0/1156122424 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5440b8070 con 0x55a86ccfcc60 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb539ffb640 1 -- 172.21.15.77:0/1156122424 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb544097dd0 con 0x55a86ccfcc60 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1156122424 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb54409f8b0 con 0x55a86ccfcc60 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb539ffb640 1 -- 172.21.15.77:0/1156122424 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb5440b4040 con 0x55a86ccfcc60 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55b5f0740 10 client.15981 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.094+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1156122424 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a86c9713f0 con 0x55a86ccfcc60 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.095+0000 7fb539ffb640 1 -- 172.21.15.77:0/1156122424 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54409f8b0 con 0x55a86ccfcc60 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.095+0000 7fb539ffb640 1 -- 172.21.15.77:0/1156122424 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb5440ccce0 con 0x55a86ccfcc60 2024-03-16T10:18:47.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.095+0000 7fb55b5f0740 10 client.15981 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.095+0000 7fb55b5f0740 10 client.15981 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.095+0000 7fb55b5f0740 20 client.15981 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.095+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a86cd12c20 0x55a86cd15010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.095+0000 7fb55b5f0740 4 client.15981 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.095+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1156122424 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a86cc6a9d0 con 0x55a86cd12c20 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.095+0000 7fb551d9b640 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a86cd12c20 0x55a86cd15010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.095+0000 7fb539ffb640 10 client.15981 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.095+0000 7fb551d9b640 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a86cd12c20 0x55a86cd15010 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1c1b37aa8d5ab544 server_cookie=28207e3539fa46c7 in_seq=0 out_seq=0 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb539ffb640 1 -- 172.21.15.77:0/1156122424 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a86cc6a9d0 con 0x55a86cd12c20 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb539ffb640 10 client.15981 handle_command_reply: tid=0 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb55b5f0740 1 client.15981 shutdown 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1156122424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 msgr2=0x55a86cd071b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 0x55a86cd071b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb54400c6a0 tx=0x7fb544005710 comp rx=0 tx=0).stop 2024-03-16T10:18:47.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1156122424 shutdown_connections 2024-03-16T10:18:47.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86cd027c0 0x55a86cd0e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a86cd12c20 0x55a86cd15010 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfd650 0x55a86cd076f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/1156122424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86ccfcc60 0x55a86cd071b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb539ffb640 0 client.15981 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:18:47.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1156122424 >> 172.21.15.77:0/1156122424 conn(0x55a86ccf3dc0 msgr2=0x55a86ccf2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1156122424 shutdown_connections 2024-03-16T10:18:47.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb55b5f0740 1 -- 172.21.15.77:0/1156122424 wait complete. 2024-03-16T10:18:47.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.096+0000 7fb55b5f0740 20 client.15981 trim_cache size 0 max 16384 2024-03-16T10:18:47.105 INFO:tasks.workunit.client.0.smithi077.stdout:Spawning 3 writers... 2024-03-16T10:18:47.105 INFO:tasks.workunit.client.0.smithi077.stdout:Spawning 1 rewriters... 2024-03-16T10:18:47.105 INFO:tasks.workunit.client.0.smithi077.stdout:Spawning 5 commenters... 2024-03-16T10:18:47.105 INFO:tasks.workunit.client.0.smithi077.stdout:Spawning 100 readers... 2024-03-16T10:18:47.105 INFO:tasks.workunit.client.0.smithi077.stdout:Benchmarking for 30 iterations. 2024-03-16T10:18:47.105 INFO:tasks.workunit.client.0.smithi077.stdout:The test will run during 5 minutes. 2024-03-16T10:18:47.105 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T10:18:47.106 INFO:tasks.workunit.client.0.smithi077.stdout: Nb blogs R articles W articles R pictures W pictures R comments W comments 2024-03-16T10:18:47.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb55b5f0740 1 Processor -- start 2024-03-16T10:18:47.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb55b5f0740 1 -- start start 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb55b5f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfcc60 0x55a86cd07650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb55b5f0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfd650 0x55a86cd0e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb55b5f0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 0x55a86ccf7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb55b5f0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb5541a69a0 con 0x55a86cd027c0 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb55b5f0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb5541a85a0 con 0x55a86ccfd650 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb55b5f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5541a7070 con 0x55a86ccfcc60 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb552d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 0x55a86ccf7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb551d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfd650 0x55a86cd0e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb552d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 0x55a86ccf7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60876/0 (socket says 172.21.15.77:60876) 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb552d9d640 1 -- 172.21.15.77:0/2328518576 learned_addr learned my addr 172.21.15.77:0/2328518576 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb55259c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfcc60 0x55a86cd07650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb552d9d640 1 -- 172.21.15.77:0/2328518576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfcc60 msgr2=0x55a86cd07650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb552d9d640 1 --2- 172.21.15.77:0/2328518576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfcc60 0x55a86cd07650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb552d9d640 1 -- 172.21.15.77:0/2328518576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfd650 msgr2=0x55a86cd0e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb552d9d640 1 --2- 172.21.15.77:0/2328518576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfd650 0x55a86cd0e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.112+0000 7fb552d9d640 1 -- 172.21.15.77:0/2328518576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86ca3b7d0 con 0x55a86cd027c0 2024-03-16T10:18:47.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.113+0000 7fb552d9d640 1 --2- 172.21.15.77:0/2328518576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 0x55a86ccf7730 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb54c003cc0 tx=0x7fb54c004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b046cdee2fdb12cd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.113+0000 7fb53bfff640 1 -- 172.21.15.77:0/2328518576 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c010430 con 0x55a86cd027c0 2024-03-16T10:18:47.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.113+0000 7fb53bfff640 1 -- 172.21.15.77:0/2328518576 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54c010ab0 con 0x55a86cd027c0 2024-03-16T10:18:47.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.113+0000 7fb53bfff640 1 -- 172.21.15.77:0/2328518576 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c01baa0 con 0x55a86cd027c0 2024-03-16T10:18:47.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.113+0000 7fb55b5f0740 1 -- 172.21.15.77:0/2328518576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 msgr2=0x55a86ccf7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.113+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/2328518576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 0x55a86ccf7730 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb54c003cc0 tx=0x7fb54c004410 comp rx=0 tx=0).stop 2024-03-16T10:18:47.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.113+0000 7fb55b5f0740 1 -- 172.21.15.77:0/2328518576 shutdown_connections 2024-03-16T10:18:47.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.113+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/2328518576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 0x55a86ccf7730 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/2328518576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfd650 0x55a86cd0e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/2328518576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfcc60 0x55a86cd07650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 -- 172.21.15.77:0/2328518576 >> 172.21.15.77:0/2328518576 conn(0x55a86ccf3dc0 msgr2=0x55a86ccf7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 -- 172.21.15.77:0/2328518576 shutdown_connections 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 -- 172.21.15.77:0/2328518576 wait complete. 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 -- start start 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 Processor -- start 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfcc60 0x55a86cd0b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfd650 0x55a86cd0be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 0x55a86cd0ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb5541a7e10 con 0x55a86cd027c0 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb5541a7740 con 0x55a86ccfcc60 2024-03-16T10:18:47.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5541a4980 con 0x55a86ccfd650 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55b5f0740 10 client.0 _ll_register_callbacks cb 0x7fb5440b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb552d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 0x55a86cd0ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb551d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfd650 0x55a86cd0be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfcc60 0x55a86cd0b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb551d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfd650 0x55a86cd0be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:40230/0 (socket says 172.21.15.77:40230) 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb552d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 0x55a86cd0ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60880/0 (socket says 172.21.15.77:60880) 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb55259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfcc60 0x55a86cd0b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:56038/0 (socket says 172.21.15.77:56038) 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.114+0000 7fb551d9b640 1 -- 172.21.15.77:0/3363738306 learned_addr learned my addr 172.21.15.77:0/3363738306 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb551d9b640 1 -- 172.21.15.77:0/3363738306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfcc60 msgr2=0x55a86cd0b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb551d9b640 1 --2- 172.21.15.77:0/3363738306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfcc60 0x55a86cd0b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb551d9b640 1 -- 172.21.15.77:0/3363738306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 msgr2=0x55a86cd0ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb551d9b640 1 --2- 172.21.15.77:0/3363738306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 0x55a86cd0ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb551d9b640 1 -- 172.21.15.77:0/3363738306 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a86cd0d410 con 0x55a86ccfd650 2024-03-16T10:18:47.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb552d9d640 1 --2- 172.21.15.77:0/3363738306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 0x55a86cd0ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb551d9b640 1 --2- 172.21.15.77:0/3363738306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfd650 0x55a86cd0be60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb54802c710 tx=0x7fb548009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6c0a62ce8bcdf40 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb539ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:18:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb539ffb640 1 -- 172.21.15.77:0/3363738306 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb548058070 con 0x55a86ccfd650 2024-03-16T10:18:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb55b5f0740 1 -- 172.21.15.77:0/3363738306 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb54c010ab0 con 0x55a86ccfd650 2024-03-16T10:18:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb539ffb640 1 -- 172.21.15.77:0/3363738306 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb548054410 con 0x55a86ccfd650 2024-03-16T10:18:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb539ffb640 1 -- 172.21.15.77:0/3363738306 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb548064050 con 0x55a86ccfd650 2024-03-16T10:18:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb55b5f0740 10 client.25759 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb55b5f0740 1 -- 172.21.15.77:0/3363738306 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a86cd0e7b0 con 0x55a86ccfd650 2024-03-16T10:18:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb539ffb640 1 -- 172.21.15.77:0/3363738306 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54c010ab0 con 0x55a86ccfd650 2024-03-16T10:18:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.115+0000 7fb539ffb640 1 -- 172.21.15.77:0/3363738306 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb5480047c0 con 0x55a86ccfd650 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 10 client.25759 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 10 client.25759 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 20 client.25759 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/3363738306 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a86cd79250 0x55a86cd7b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 4 client.25759 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 1 -- 172.21.15.77:0/3363738306 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a86cc64d20 con 0x55a86cd79250 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55259c640 1 --2- 172.21.15.77:0/3363738306 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a86cd79250 0x55a86cd7b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb539ffb640 10 client.25759 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55259c640 1 --2- 172.21.15.77:0/3363738306 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a86cd79250 0x55a86cd7b670 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=82fc3f357144b638 server_cookie=2f4337118e9c9356 in_seq=0 out_seq=0 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb539ffb640 1 -- 172.21.15.77:0/3363738306 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a86cc64d20 con 0x55a86cd79250 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb539ffb640 10 client.25759 handle_command_reply: tid=0 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 1 client.25759 shutdown 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 1 -- 172.21.15.77:0/3363738306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfd650 msgr2=0x55a86cd0be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/3363738306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfd650 0x55a86cd0be60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb54802c710 tx=0x7fb548009260 comp rx=0 tx=0).stop 2024-03-16T10:18:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 1 -- 172.21.15.77:0/3363738306 shutdown_connections 2024-03-16T10:18:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/3363738306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86cd027c0 0x55a86cd0ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/3363738306 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55a86cd79250 0x55a86cd7b670 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.116+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/3363738306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a86ccfd650 0x55a86cd0be60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55b5f0740 1 --2- 172.21.15.77:0/3363738306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a86ccfcc60 0x55a86cd0b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb539ffb640 0 client.25759 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:18:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55b5f0740 1 -- 172.21.15.77:0/3363738306 >> 172.21.15.77:0/3363738306 conn(0x55a86ccf3dc0 msgr2=0x55a86ccf78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55b5f0740 1 -- 172.21.15.77:0/3363738306 shutdown_connections 2024-03-16T10:18:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55b5f0740 1 -- 172.21.15.77:0/3363738306 wait complete. 2024-03-16T10:18:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55b5f0740 20 client.25759 trim_cache size 0 max 16384 2024-03-16T10:18:47.125 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:18:47.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55892f640 1 -- 172.21.15.77:0/2762386058 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb53c0a2800 msgr2=0x7fb53c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55892f640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb53c0a2800 0x7fb53c0a4cc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb54802c7b0 tx=0x7fb548005fd0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55892f640 1 -- 172.21.15.77:0/2762386058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5541006e0 msgr2=0x7fb5540694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55892f640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5541006e0 0x7fb5540694b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb54400e9c0 tx=0x7fb54400ee80 comp rx=0 tx=0).stop 2024-03-16T10:18:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55892f640 1 -- 172.21.15.77:0/2762386058 shutdown_connections 2024-03-16T10:18:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55892f640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb53c0a2800 0x7fb53c0a4cc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55892f640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb554102290 0x7fb55419ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55892f640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5541018d0 0x7fb55419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55892f640 1 --2- 172.21.15.77:0/2762386058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5541006e0 0x7fb5540694b0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55892f640 1 -- 172.21.15.77:0/2762386058 >> 172.21.15.77:0/2762386058 conn(0x7fb5540fbc70 msgr2=0x7fb554104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55892f640 1 -- 172.21.15.77:0/2762386058 shutdown_connections 2024-03-16T10:18:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.117+0000 7fb55892f640 1 -- 172.21.15.77:0/2762386058 wait complete. 2024-03-16T10:18:47.129 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-16T10:18:47.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.234+0000 7f26885f3640 1 -- 172.21.15.77:0/2100612803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26801006f0 msgr2=0x7f2680100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.234+0000 7f26885f3640 1 --2- 172.21.15.77:0/2100612803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26801006f0 0x7f2680100b10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f2670009900 tx=0x7f267002bc30 comp rx=0 tx=0).stop 2024-03-16T10:18:47.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.234+0000 7f26885f3640 1 -- 172.21.15.77:0/2100612803 shutdown_connections 2024-03-16T10:18:47.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.234+0000 7f26885f3640 1 --2- 172.21.15.77:0/2100612803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26801022a0 0x7f2680108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.234+0000 7f26885f3640 1 --2- 172.21.15.77:0/2100612803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26801018e0 0x7f2680101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.234+0000 7f26885f3640 1 --2- 172.21.15.77:0/2100612803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26801006f0 0x7f2680100b10 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.234+0000 7f26885f3640 1 -- 172.21.15.77:0/2100612803 >> 172.21.15.77:0/2100612803 conn(0x7f26800fbc80 msgr2=0x7f26800fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f26885f3640 1 -- 172.21.15.77:0/2100612803 shutdown_connections 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f26885f3640 1 -- 172.21.15.77:0/2100612803 wait complete. 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f26885f3640 1 Processor -- start 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f26885f3640 1 -- start start 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f26885f3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26801006f0 0x7f2680197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f26885f3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26801018e0 0x7f26801980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f26885f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26801022a0 0x7f268019f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f26885f3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f268010c370 con 0x7f26801022a0 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f26885f3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f268010c1f0 con 0x7f26801006f0 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f26885f3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f268010c4f0 con 0x7f26801018e0 2024-03-16T10:18:47.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f2686b90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26801022a0 0x7f268019f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f2686b90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26801022a0 0x7f268019f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60896/0 (socket says 172.21.15.77:60896) 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f268638f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26801006f0 0x7f2680197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f2686b90640 1 -- 172.21.15.77:0/2661488914 learned_addr learned my addr 172.21.15.77:0/2661488914 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.235+0000 7f2685b8e640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26801018e0 0x7f26801980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.236+0000 7f2686b90640 1 -- 172.21.15.77:0/2661488914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26801018e0 msgr2=0x7f26801980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.236+0000 7f2686b90640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26801018e0 0x7f26801980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.236+0000 7f2686b90640 1 -- 172.21.15.77:0/2661488914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26801006f0 msgr2=0x7f2680197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.236+0000 7f2686b90640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26801006f0 0x7f2680197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.236+0000 7f2686b90640 1 -- 172.21.15.77:0/2661488914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268019f780 con 0x7f26801022a0 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.236+0000 7f2686b90640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26801022a0 0x7f268019f110 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f267800eb10 tx=0x7f267800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.236+0000 7f26777fe640 1 -- 172.21.15.77:0/2661488914 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678004b20 con 0x7f26801022a0 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.236+0000 7f26885f3640 1 -- 172.21.15.77:0/2661488914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268019fa70 con 0x7f26801022a0 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.236+0000 7f26777fe640 1 -- 172.21.15.77:0/2661488914 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2678004cc0 con 0x7f26801022a0 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.236+0000 7f26777fe640 1 -- 172.21.15.77:0/2661488914 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678010760 con 0x7f26801022a0 2024-03-16T10:18:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.236+0000 7f26885f3640 1 -- 172.21.15.77:0/2661488914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26801a0010 con 0x7f26801022a0 2024-03-16T10:18:47.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.238+0000 7f26777fe640 1 -- 172.21.15.77:0/2661488914 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f2678005020 con 0x7f26801022a0 2024-03-16T10:18:47.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f268b1d7740 1 Processor -- start 2024-03-16T10:18:47.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f268b1d7740 1 -- start start 2024-03-16T10:18:47.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f268b1d7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65d3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f268b1d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65d8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f268b1d7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 0x5609d65dac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f268b1d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5609d6533490 con 0x5609d65d3650 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f268b1d7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5609d64dbc90 con 0x5609d65d2c60 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f268b1d7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5609d652f530 con 0x5609d65d87c0 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f26777fe640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f266c0a2800 0x7f266c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f268638f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65d8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f268638f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65d8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60900/0 (socket says 172.21.15.77:60900) 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f268638f640 1 -- 172.21.15.77:0/3535199699 learned_addr learned my addr 172.21.15.77:0/3535199699 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f2686b90640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f266c0a2800 0x7f266c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.239+0000 7f2685b8e640 1 --2- 172.21.15.77:0/3535199699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65d3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f2685b8e640 1 --2- 172.21.15.77:0/3535199699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 0x5609d65dac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f26777fe640 1 -- 172.21.15.77:0/2661488914 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f26780cf010 con 0x7f26801022a0 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f268638f640 1 -- 172.21.15.77:0/3535199699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 msgr2=0x5609d65dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f268638f640 1 --2- 172.21.15.77:0/3535199699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 0x5609d65dac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f268638f640 1 -- 172.21.15.77:0/3535199699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 msgr2=0x5609d65d3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f268638f640 1 --2- 172.21.15.77:0/3535199699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65d3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f268638f640 1 -- 172.21.15.77:0/3535199699 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609d6596500 con 0x5609d65d3650 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f2685b8e640 1 --2- 172.21.15.77:0/3535199699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65d3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f2686b90640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f266c0a2800 0x7f266c0a4cc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2678005b40 tx=0x7f2678066ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f268638f640 1 --2- 172.21.15.77:0/3535199699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65d8150 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2670009900 tx=0x7f26700095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce3f85161f9d81f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f2674bfe640 1 -- 172.21.15.77:0/3535199699 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267002cb80 con 0x5609d65d3650 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f2674bfe640 1 -- 172.21.15.77:0/3535199699 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2670005040 con 0x5609d65d3650 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.240+0000 7f2674bfe640 1 -- 172.21.15.77:0/3535199699 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2670004920 con 0x5609d65d3650 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.241+0000 7f268b1d7740 1 -- 172.21.15.77:0/3535199699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 msgr2=0x5609d65d8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.241+0000 7f268b1d7740 1 --2- 172.21.15.77:0/3535199699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65d8150 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2670009900 tx=0x7f26700095e0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.241+0000 7f268b1d7740 1 -- 172.21.15.77:0/3535199699 shutdown_connections 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.241+0000 7f268b1d7740 1 --2- 172.21.15.77:0/3535199699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 0x5609d65dac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.241+0000 7f268b1d7740 1 --2- 172.21.15.77:0/3535199699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65d8150 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.241+0000 7f268b1d7740 1 --2- 172.21.15.77:0/3535199699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65d3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.241+0000 7f268b1d7740 1 -- 172.21.15.77:0/3535199699 >> 172.21.15.77:0/3535199699 conn(0x5609d65c9dc0 msgr2=0x5609d65ca1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.241+0000 7f268b1d7740 1 -- 172.21.15.77:0/3535199699 shutdown_connections 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.241+0000 7f268b1d7740 1 -- 172.21.15.77:0/3535199699 wait complete. 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f268b1d7740 1 -- start start 2024-03-16T10:18:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f268b1d7740 1 Processor -- start 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f268b1d7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d2c60 0x5609d65dd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f268b1d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65dd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f268b1d7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d87c0 0x5609d65e4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f268b1d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5609d65db710 con 0x5609d65d3650 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f268b1d7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5609d64ef230 con 0x5609d65d87c0 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f268b1d7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5609d65db890 con 0x5609d65d2c60 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f268b1d7740 10 client.0 _ll_register_callbacks cb 0x5609d652b010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f268638f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65dd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f268638f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65dd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60910/0 (socket says 172.21.15.77:60910) 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f2685b8e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d2c60 0x5609d65dd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f268638f640 1 -- 172.21.15.77:0/2738086624 learned_addr learned my addr 172.21.15.77:0/2738086624 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.242+0000 7f2685b8e640 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d87c0 0x5609d65e4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268638f640 1 -- 172.21.15.77:0/2738086624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d2c60 msgr2=0x5609d65dd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268638f640 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d2c60 0x5609d65dd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268638f640 1 -- 172.21.15.77:0/2738086624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d87c0 msgr2=0x5609d65e4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268638f640 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d87c0 0x5609d65e4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268638f640 1 -- 172.21.15.77:0/2738086624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609d64fe100 con 0x5609d65d3650 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f2685b8e640 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d2c60 0x5609d65dd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268638f640 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65dd650 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f267002c7b0 tx=0x7f26700054a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f069ce33a31dbe20 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f2669ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f2669ffb640 1 -- 172.21.15.77:0/2738086624 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2670004ad0 con 0x5609d65d3650 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f2669ffb640 1 -- 172.21.15.77:0/2738086624 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2670004c70 con 0x5609d65d3650 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268b1d7740 1 -- 172.21.15.77:0/2738086624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2670005040 con 0x5609d65d3650 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f2669ffb640 1 -- 172.21.15.77:0/2738086624 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2670056050 con 0x5609d65d3650 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268b1d7740 10 client.15999 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268b1d7740 1 -- 172.21.15.77:0/2738086624 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5609d62473f0 con 0x5609d65d3650 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f2669ffb640 1 -- 172.21.15.77:0/2738086624 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2670005040 con 0x5609d65d3650 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f2669ffb640 1 -- 172.21.15.77:0/2738086624 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f267004d070 con 0x5609d65d3650 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268b1d7740 10 client.15999 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268b1d7740 10 client.15999 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:18:47.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268b1d7740 20 client.15999 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268b1d7740 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5609d65e8b60 0x5609d65eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268b1d7740 4 client.15999 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.243+0000 7f268b1d7740 1 -- 172.21.15.77:0/2738086624 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609d65409d0 con 0x5609d65e8b60 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.244+0000 7f2685b8e640 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5609d65e8b60 0x5609d65eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.244+0000 7f2669ffb640 10 client.15999 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.244+0000 7f2685b8e640 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5609d65e8b60 0x5609d65eaf50 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f4bfc06cdeec05e server_cookie=4294eb357369ea9 in_seq=0 out_seq=0 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f2669ffb640 1 -- 172.21.15.77:0/2738086624 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5609d65409d0 con 0x5609d65e8b60 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f2669ffb640 10 client.15999 handle_command_reply: tid=0 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f268b1d7740 1 client.15999 shutdown 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f268b1d7740 1 -- 172.21.15.77:0/2738086624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 msgr2=0x5609d65dd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f268b1d7740 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65dd650 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f267002c7b0 tx=0x7f26700054a0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f268b1d7740 1 -- 172.21.15.77:0/2738086624 shutdown_connections 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f268b1d7740 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5609d65e8b60 0x5609d65eaf50 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f268b1d7740 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d87c0 0x5609d65e4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f268b1d7740 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65dd650 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f268b1d7740 1 --2- 172.21.15.77:0/2738086624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d2c60 0x5609d65dd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f268b1d7740 1 -- 172.21.15.77:0/2738086624 >> 172.21.15.77:0/2738086624 conn(0x5609d65c9dc0 msgr2=0x5609d65c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f2669ffb640 0 client.15999 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:18:47.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f268b1d7740 1 -- 172.21.15.77:0/2738086624 shutdown_connections 2024-03-16T10:18:47.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f268b1d7740 1 -- 172.21.15.77:0/2738086624 wait complete. 2024-03-16T10:18:47.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.245+0000 7f268b1d7740 20 client.15999 trim_cache size 0 max 16384 2024-03-16T10:18:47.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268b1d7740 1 Processor -- start 2024-03-16T10:18:47.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268b1d7740 1 -- start start 2024-03-16T10:18:47.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268b1d7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65dd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268b1d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65dd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268b1d7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 0x5609d65e4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268b1d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f26801a6940 con 0x5609d65d3650 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268b1d7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f26801a8540 con 0x5609d65d2c60 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268b1d7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f26801a7010 con 0x5609d65d87c0 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268638f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65dd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268638f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65dd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60924/0 (socket says 172.21.15.77:60924) 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268638f640 1 -- 172.21.15.77:0/2559368577 learned_addr learned my addr 172.21.15.77:0/2559368577 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f2685b8e640 1 --2- 172.21.15.77:0/2559368577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65dd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f2685b8e640 1 --2- 172.21.15.77:0/2559368577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 0x5609d65e4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268638f640 1 -- 172.21.15.77:0/2559368577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 msgr2=0x5609d65e4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268638f640 1 --2- 172.21.15.77:0/2559368577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 0x5609d65e4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268638f640 1 -- 172.21.15.77:0/2559368577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 msgr2=0x5609d65dd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268638f640 1 --2- 172.21.15.77:0/2559368577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65dd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268638f640 1 -- 172.21.15.77:0/2559368577 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609d65dc230 con 0x5609d65d3650 2024-03-16T10:18:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.260+0000 7f268638f640 1 --2- 172.21.15.77:0/2559368577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65dd650 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2670009900 tx=0x7f2670035fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723fab831b2d5a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.261+0000 7f266bfff640 1 -- 172.21.15.77:0/2559368577 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267005f070 con 0x5609d65d3650 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.261+0000 7f266bfff640 1 -- 172.21.15.77:0/2559368577 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2670045dd0 con 0x5609d65d3650 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.261+0000 7f266bfff640 1 -- 172.21.15.77:0/2559368577 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2670004b10 con 0x5609d65d3650 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.261+0000 7f268b1d7740 1 -- 172.21.15.77:0/2559368577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 msgr2=0x5609d65dd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.261+0000 7f268b1d7740 1 --2- 172.21.15.77:0/2559368577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65dd650 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2670009900 tx=0x7f2670035fe0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.261+0000 7f268b1d7740 1 -- 172.21.15.77:0/2559368577 shutdown_connections 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.261+0000 7f268b1d7740 1 --2- 172.21.15.77:0/2559368577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 0x5609d65e4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.261+0000 7f268b1d7740 1 --2- 172.21.15.77:0/2559368577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65dd650 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.261+0000 7f268b1d7740 1 --2- 172.21.15.77:0/2559368577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65dd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.261+0000 7f268b1d7740 1 -- 172.21.15.77:0/2559368577 >> 172.21.15.77:0/2559368577 conn(0x5609d65c9dc0 msgr2=0x5609d65ca1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268b1d7740 1 -- 172.21.15.77:0/2559368577 shutdown_connections 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268b1d7740 1 -- 172.21.15.77:0/2559368577 wait complete. 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268b1d7740 1 -- start start 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268b1d7740 1 Processor -- start 2024-03-16T10:18:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268b1d7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65e1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268b1d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65e1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268b1d7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 0x5609d664cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268b1d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f26801a7db0 con 0x5609d65d3650 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268b1d7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f26801a76e0 con 0x5609d65d2c60 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268b1d7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f26801a4920 con 0x5609d65d87c0 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268b1d7740 10 client.0 _ll_register_callbacks cb 0x7f267c00bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268638f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65e1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268638f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65e1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60930/0 (socket says 172.21.15.77:60930) 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f2685b8e640 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 0x5609d664cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268638f640 1 -- 172.21.15.77:0/4237227360 learned_addr learned my addr 172.21.15.77:0/4237227360 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f2685b8e640 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65e1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268638f640 1 -- 172.21.15.77:0/4237227360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 msgr2=0x5609d664cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268638f640 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 0x5609d664cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268638f640 1 -- 172.21.15.77:0/4237227360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 msgr2=0x5609d65e1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268638f640 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65e1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.262+0000 7f268638f640 1 -- 172.21.15.77:0/4237227360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609d63117d0 con 0x5609d65d3650 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f268638f640 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65e1980 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f267005d040 tx=0x7f26700056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f2bb1125e5830a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f2669ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f2669ffb640 1 -- 172.21.15.77:0/4237227360 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267005f070 con 0x5609d65d3650 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f2669ffb640 1 -- 172.21.15.77:0/4237227360 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2670004040 con 0x5609d65d3650 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f268b1d7740 1 -- 172.21.15.77:0/4237227360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2670045dd0 con 0x5609d65d3650 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f2669ffb640 1 -- 172.21.15.77:0/4237227360 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2670009520 con 0x5609d65d3650 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f268b1d7740 10 client.16005 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f268b1d7740 1 -- 172.21.15.77:0/4237227360 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5609d664e190 con 0x5609d65d3650 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f2669ffb640 1 -- 172.21.15.77:0/4237227360 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2670045dd0 con 0x5609d65d3650 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f2669ffb640 1 -- 172.21.15.77:0/4237227360 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f267004d070 con 0x5609d65d3650 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f268b1d7740 10 client.16005 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f268b1d7740 10 client.16005 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:18:47.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f268b1d7740 20 client.16005 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f268b1d7740 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5609d664e920 0x5609d6650d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f268b1d7740 4 client.16005 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f268b1d7740 1 -- 172.21.15.77:0/4237227360 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5609d653ad20 con 0x5609d664e920 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.263+0000 7f2685b8e640 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5609d664e920 0x5609d6650d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f2669ffb640 10 client.16005 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f2685b8e640 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5609d664e920 0x5609d6650d40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b981860f670d7a74 server_cookie=1d59c3da6b2bcc6b in_seq=0 out_seq=0 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f2669ffb640 1 -- 172.21.15.77:0/4237227360 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5609d653ad20 con 0x5609d664e920 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f2669ffb640 10 client.16005 handle_command_reply: tid=0 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f268b1d7740 1 client.16005 shutdown 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f268b1d7740 1 -- 172.21.15.77:0/4237227360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 msgr2=0x5609d65e1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f268b1d7740 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65e1980 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f267005d040 tx=0x7f26700056b0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f268b1d7740 1 -- 172.21.15.77:0/4237227360 shutdown_connections 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f268b1d7740 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5609d664e920 0x5609d6650d40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f268b1d7740 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609d65d87c0 0x5609d664cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f268b1d7740 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5609d65d3650 0x5609d65e1980 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f268b1d7740 1 --2- 172.21.15.77:0/4237227360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609d65d2c60 0x5609d65e1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f2669ffb640 0 client.16005 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:18:47.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f268b1d7740 1 -- 172.21.15.77:0/4237227360 >> 172.21.15.77:0/4237227360 conn(0x5609d65c9dc0 msgr2=0x5609d65dd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f268b1d7740 1 -- 172.21.15.77:0/4237227360 shutdown_connections 2024-03-16T10:18:47.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f268b1d7740 1 -- 172.21.15.77:0/4237227360 wait complete. 2024-03-16T10:18:47.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.264+0000 7f268b1d7740 20 client.16005 trim_cache size 0 max 16384 2024-03-16T10:18:47.272 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:18:47.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.265+0000 7f26885f3640 1 -- 172.21.15.77:0/2661488914 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f266c0a2800 msgr2=0x7f266c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.265+0000 7f26885f3640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f266c0a2800 0x7f266c0a4cc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2678005b40 tx=0x7f2678066ca0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.265+0000 7f26885f3640 1 -- 172.21.15.77:0/2661488914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26801022a0 msgr2=0x7f268019f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.265+0000 7f26885f3640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26801022a0 0x7f268019f110 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f267800eb10 tx=0x7f267800c2d0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.265+0000 7f26885f3640 1 -- 172.21.15.77:0/2661488914 shutdown_connections 2024-03-16T10:18:47.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.265+0000 7f26885f3640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26801022a0 0x7f268019f110 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.265+0000 7f26885f3640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f266c0a2800 0x7f266c0a4cc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.265+0000 7f26885f3640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f26801018e0 0x7f26801980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.265+0000 7f26885f3640 1 --2- 172.21.15.77:0/2661488914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26801006f0 0x7f2680197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.265+0000 7f26885f3640 1 -- 172.21.15.77:0/2661488914 >> 172.21.15.77:0/2661488914 conn(0x7f26800fbc80 msgr2=0x7f2680069d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.265+0000 7f26885f3640 1 -- 172.21.15.77:0/2661488914 shutdown_connections 2024-03-16T10:18:47.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.265+0000 7f26885f3640 1 -- 172.21.15.77:0/2661488914 wait complete. 2024-03-16T10:18:47.276 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T10:18:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.382+0000 7f28e0990640 1 -- 172.21.15.77:0/3147781970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28dc102290 msgr2=0x7f28dc108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.382+0000 7f28e0990640 1 --2- 172.21.15.77:0/3147781970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28dc102290 0x7f28dc108b20 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009ca0 tx=0x7f28d402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.382+0000 7f28e0990640 1 -- 172.21.15.77:0/3147781970 shutdown_connections 2024-03-16T10:18:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.382+0000 7f28e0990640 1 --2- 172.21.15.77:0/3147781970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28dc102290 0x7f28dc108b20 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.382+0000 7f28e0990640 1 --2- 172.21.15.77:0/3147781970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28dc1018d0 0x7f28dc101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.382+0000 7f28e0990640 1 --2- 172.21.15.77:0/3147781970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28dc1006e0 0x7f28dc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.382+0000 7f28e0990640 1 -- 172.21.15.77:0/3147781970 >> 172.21.15.77:0/3147781970 conn(0x7f28dc0fbc70 msgr2=0x7f28dc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.382+0000 7f28e0990640 1 -- 172.21.15.77:0/3147781970 shutdown_connections 2024-03-16T10:18:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.382+0000 7f28e0990640 1 -- 172.21.15.77:0/3147781970 wait complete. 2024-03-16T10:18:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28e0990640 1 Processor -- start 2024-03-16T10:18:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28e0990640 1 -- start start 2024-03-16T10:18:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28e0990640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28dc1006e0 0x7f28dc0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28e0990640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28dc1018d0 0x7f28dc19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28e0990640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28dc102290 0x7f28dc19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28e0990640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc10c340 con 0x7f28dc1006e0 2024-03-16T10:18:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28e0990640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc10c1c0 con 0x7f28dc1018d0 2024-03-16T10:18:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28e0990640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc10c4c0 con 0x7f28dc102290 2024-03-16T10:18:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28da59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28dc1006e0 0x7f28dc0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28da59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28dc1006e0 0x7f28dc0694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60956/0 (socket says 172.21.15.77:60956) 2024-03-16T10:18:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28da59c640 1 -- 172.21.15.77:0/2427420541 learned_addr learned my addr 172.21.15.77:0/2427420541 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28d3fff640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28dc1018d0 0x7f28dc19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28dad9d640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28dc102290 0x7f28dc19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28da59c640 1 -- 172.21.15.77:0/2427420541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28dc102290 msgr2=0x7f28dc19ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28da59c640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28dc102290 0x7f28dc19ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28da59c640 1 -- 172.21.15.77:0/2427420541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28dc1018d0 msgr2=0x7f28dc19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28da59c640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28dc1018d0 0x7f28dc19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.383+0000 7f28da59c640 1 -- 172.21.15.77:0/2427420541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28dc19f760 con 0x7f28dc1006e0 2024-03-16T10:18:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.384+0000 7f28da59c640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28dc1006e0 0x7f28dc0694b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f28cc00e9c0 tx=0x7f28cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.384+0000 7f28d37fe640 1 -- 172.21.15.77:0/2427420541 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc00cd80 con 0x7f28dc1006e0 2024-03-16T10:18:47.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.384+0000 7f28d37fe640 1 -- 172.21.15.77:0/2427420541 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28cc0035f0 con 0x7f28dc1006e0 2024-03-16T10:18:47.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.384+0000 7f28e0990640 1 -- 172.21.15.77:0/2427420541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28dc19f9f0 con 0x7f28dc1006e0 2024-03-16T10:18:47.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.384+0000 7f28d37fe640 1 -- 172.21.15.77:0/2427420541 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc010690 con 0x7f28dc1006e0 2024-03-16T10:18:47.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.384+0000 7f28e0990640 1 -- 172.21.15.77:0/2427420541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28dc19ff40 con 0x7f28dc1006e0 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.386+0000 7f28e3577740 1 Processor -- start 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28e3577740 1 -- start start 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28e3577740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7bab1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28e3577740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7bab6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28e3577740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bab9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28e3577740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561c7ba16490 con 0x561c7bab73b0 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28e3577740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561c7b9bec90 con 0x561c7bab2240 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28e3577740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561c7ba12530 con 0x561c7bab1850 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28d37fe640 1 -- 172.21.15.77:0/2427420541 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f28cc0037a0 con 0x7f28dc1006e0 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28d3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bab9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28dad9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7bab1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28d3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bab9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60960/0 (socket says 172.21.15.77:60960) 2024-03-16T10:18:47.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28d3fff640 1 -- 172.21.15.77:0/2594851207 learned_addr learned my addr 172.21.15.77:0/2594851207 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28da59c640 1 --2- 172.21.15.77:0/2594851207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7bab6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28dad9d640 1 -- 172.21.15.77:0/2594851207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 msgr2=0x561c7bab6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28dad9d640 1 --2- 172.21.15.77:0/2594851207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7bab6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28dad9d640 1 -- 172.21.15.77:0/2594851207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 msgr2=0x561c7bab9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28dad9d640 1 --2- 172.21.15.77:0/2594851207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bab9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28dad9d640 1 -- 172.21.15.77:0/2594851207 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c7ba79500 con 0x561c7bab1850 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.387+0000 7f28d3fff640 1 --2- 172.21.15.77:0/2594851207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bab9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.388+0000 7f28dad9d640 1 --2- 172.21.15.77:0/2594851207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7bab1c70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f28d402c630 tx=0x7f28d40323c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9fbc5f91919b92e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.388+0000 7f28d37fe640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f28c80a2800 0x7f28c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.388+0000 7f28d0ff9640 1 -- 172.21.15.77:0/2594851207 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d4043070 con 0x561c7bab1850 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.388+0000 7f28d0ff9640 1 -- 172.21.15.77:0/2594851207 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28d4032c90 con 0x561c7bab1850 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.388+0000 7f28d0ff9640 1 -- 172.21.15.77:0/2594851207 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d4039040 con 0x561c7bab1850 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.388+0000 7f28da59c640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f28c80a2800 0x7f28c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.388+0000 7f28d37fe640 1 -- 172.21.15.77:0/2427420541 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f28cc0cedd0 con 0x7f28dc1006e0 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 -- 172.21.15.77:0/2594851207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 msgr2=0x561c7bab1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28da59c640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f28c80a2800 0x7f28c80a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x561c7bab2640 tx=0x7f28cc0d2420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 --2- 172.21.15.77:0/2594851207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7bab1c70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f28d402c630 tx=0x7f28d40323c0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 -- 172.21.15.77:0/2594851207 shutdown_connections 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 --2- 172.21.15.77:0/2594851207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bab9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 --2- 172.21.15.77:0/2594851207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7bab6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 --2- 172.21.15.77:0/2594851207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7bab1c70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 -- 172.21.15.77:0/2594851207 >> 172.21.15.77:0/2594851207 conn(0x561c7baacdc0 msgr2=0x561c7baad1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 -- 172.21.15.77:0/2594851207 shutdown_connections 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 -- 172.21.15.77:0/2594851207 wait complete. 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 -- start start 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 Processor -- start 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7babbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7babc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bac3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561c7baba300 con 0x561c7bab73b0 2024-03-16T10:18:47.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561c7b9d2230 con 0x561c7bab2240 2024-03-16T10:18:47.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561c7baba480 con 0x561c7bab1850 2024-03-16T10:18:47.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28e3577740 10 client.0 _ll_register_callbacks cb 0x561c7ba0e010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:47.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28dad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bac3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28d3fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7babc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28dad9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7babbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28d3fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7babc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:56124/0 (socket says 172.21.15.77:56124) 2024-03-16T10:18:47.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.389+0000 7f28d3fff640 1 -- 172.21.15.77:0/2724218910 learned_addr learned my addr 172.21.15.77:0/2724218910 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28d3fff640 1 -- 172.21.15.77:0/2724218910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 msgr2=0x561c7babbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28d3fff640 1 --2- 172.21.15.77:0/2724218910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7babbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28d3fff640 1 -- 172.21.15.77:0/2724218910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 msgr2=0x561c7bac3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28d3fff640 1 --2- 172.21.15.77:0/2724218910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bac3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28d3fff640 1 -- 172.21.15.77:0/2724218910 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c7b9e1100 con 0x561c7bab2240 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28dad9d640 1 --2- 172.21.15.77:0/2724218910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7babbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28d3fff640 1 --2- 172.21.15.77:0/2724218910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7babc290 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f28c4012700 tx=0x7f28c4012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7af69cbf3d81143b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28c1ffb640 1 -- 172.21.15.77:0/2724218910 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c400f920 con 0x561c7bab2240 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28c1ffb640 1 -- 172.21.15.77:0/2724218910 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f28c400fac0 con 0x561c7bab2240 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28e3577740 1 -- 172.21.15.77:0/2724218910 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28d4032c90 con 0x561c7bab2240 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28c1ffb640 1 -- 172.21.15.77:0/2724218910 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f28c400fcc0 con 0x561c7bab2240 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28e3577740 10 client.25805 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.390+0000 7f28e3577740 1 -- 172.21.15.77:0/2724218910 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561c7b72a3f0 con 0x561c7bab2240 2024-03-16T10:18:47.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.391+0000 7f28c1ffb640 1 -- 172.21.15.77:0/2724218910 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28d4032c90 con 0x561c7bab2240 2024-03-16T10:18:47.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.391+0000 7f28c1ffb640 1 -- 172.21.15.77:0/2724218910 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f28c4025050 con 0x561c7bab2240 2024-03-16T10:18:47.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.391+0000 7f28e3577740 10 client.25805 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:47.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.391+0000 7f28e3577740 10 client.25805 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:18:47.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.391+0000 7f28e3577740 20 client.25805 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:47.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.391+0000 7f28e3577740 1 --2- 172.21.15.77:0/2724218910 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x561c7bac7b70 0x561c7bac9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.391+0000 7f28e3577740 4 client.25805 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:47.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.391+0000 7f28e3577740 1 -- 172.21.15.77:0/2724218910 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c7ba239d0 con 0x561c7bac7b70 2024-03-16T10:18:47.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.391+0000 7f28dad9d640 1 --2- 172.21.15.77:0/2724218910 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x561c7bac7b70 0x561c7bac9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.391+0000 7f28c1ffb640 10 client.25805 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:18:47.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.391+0000 7f28dad9d640 1 --2- 172.21.15.77:0/2724218910 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x561c7bac7b70 0x561c7bac9fb0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fcf97a8bc9d6e618 server_cookie=98e33c7579a202d5 in_seq=0 out_seq=0 2024-03-16T10:18:47.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28c1ffb640 1 -- 172.21.15.77:0/2724218910 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561c7ba239d0 con 0x561c7bac7b70 2024-03-16T10:18:47.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28c1ffb640 10 client.25805 handle_command_reply: tid=0 2024-03-16T10:18:47.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28e3577740 1 client.25805 shutdown 2024-03-16T10:18:47.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28e3577740 1 -- 172.21.15.77:0/2724218910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 msgr2=0x561c7babc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28e3577740 1 --2- 172.21.15.77:0/2724218910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7babc290 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f28c4012700 tx=0x7f28c4012bc0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28e3577740 1 -- 172.21.15.77:0/2724218910 shutdown_connections 2024-03-16T10:18:47.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28e3577740 1 --2- 172.21.15.77:0/2724218910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bac3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28e3577740 1 --2- 172.21.15.77:0/2724218910 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x561c7bac7b70 0x561c7bac9fb0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28e3577740 1 --2- 172.21.15.77:0/2724218910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7babc290 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28e3577740 1 --2- 172.21.15.77:0/2724218910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7babbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28e3577740 1 -- 172.21.15.77:0/2724218910 >> 172.21.15.77:0/2724218910 conn(0x561c7baacdc0 msgr2=0x561c7baabfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28c1ffb640 0 client.25805 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:18:47.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28e3577740 1 -- 172.21.15.77:0/2724218910 shutdown_connections 2024-03-16T10:18:47.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28e3577740 1 -- 172.21.15.77:0/2724218910 wait complete. 2024-03-16T10:18:47.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.396+0000 7f28e3577740 20 client.25805 trim_cache size 0 max 16384 2024-03-16T10:18:47.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28e3577740 1 Processor -- start 2024-03-16T10:18:47.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28e3577740 1 -- start start 2024-03-16T10:18:47.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28e3577740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7baba820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28e3577740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab2240 0x561c7babad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28e3577740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab73b0 0x561c7bac3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28e3577740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc1a8c60 con 0x561c7bab2240 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28e3577740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc1aa860 con 0x561c7bab73b0 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28e3577740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc1a9330 con 0x561c7bab1850 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28d3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab2240 0x561c7babad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28dad9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7baba820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28d3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab2240 0x561c7babad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60966/0 (socket says 172.21.15.77:60966) 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28d3fff640 1 -- 172.21.15.77:0/3273952548 learned_addr learned my addr 172.21.15.77:0/3273952548 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.415+0000 7f28dad9d640 1 --2- 172.21.15.77:0/3273952548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab73b0 0x561c7bac3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.416+0000 7f28d3fff640 1 -- 172.21.15.77:0/3273952548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 msgr2=0x561c7baba820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.416+0000 7f28d3fff640 1 --2- 172.21.15.77:0/3273952548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7baba820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.416+0000 7f28d3fff640 1 -- 172.21.15.77:0/3273952548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab73b0 msgr2=0x561c7bac3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.416+0000 7f28d3fff640 1 --2- 172.21.15.77:0/3273952548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab73b0 0x561c7bac3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.424 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.416+0000 7f28d3fff640 1 -- 172.21.15.77:0/3273952548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c7babb8f0 con 0x561c7bab2240 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.416+0000 7f28dad9d640 1 --2- 172.21.15.77:0/3273952548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7baba820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.416+0000 7f28d3fff640 1 --2- 172.21.15.77:0/3273952548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab2240 0x561c7babad60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f28c401c4a0 tx=0x7f28c40093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a0339592b4bd72 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.416+0000 7f28c3fff640 1 -- 172.21.15.77:0/3273952548 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c400f920 con 0x561c7bab2240 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.416+0000 7f28c3fff640 1 -- 172.21.15.77:0/3273952548 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c400fac0 con 0x561c7bab2240 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.416+0000 7f28c3fff640 1 -- 172.21.15.77:0/3273952548 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4004660 con 0x561c7bab2240 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 -- 172.21.15.77:0/3273952548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab2240 msgr2=0x561c7babad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 --2- 172.21.15.77:0/3273952548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab2240 0x561c7babad60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f28c401c4a0 tx=0x7f28c40093f0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 -- 172.21.15.77:0/3273952548 shutdown_connections 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 --2- 172.21.15.77:0/3273952548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab73b0 0x561c7bac3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 --2- 172.21.15.77:0/3273952548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab2240 0x561c7babad60 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 --2- 172.21.15.77:0/3273952548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7baba820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 -- 172.21.15.77:0/3273952548 >> 172.21.15.77:0/3273952548 conn(0x561c7baacdc0 msgr2=0x561c7babbd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 -- 172.21.15.77:0/3273952548 shutdown_connections 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 -- 172.21.15.77:0/3273952548 wait complete. 2024-03-16T10:18:47.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 -- start start 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 Processor -- start 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7bac0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7bac0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bb2bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc1aa0d0 con 0x561c7bab73b0 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc1a9a00 con 0x561c7bab2240 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc1a6c40 con 0x561c7bab1850 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28e3577740 10 client.0 _ll_register_callbacks cb 0x7f28c4009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28dad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bb2bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28dad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bb2bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60974/0 (socket says 172.21.15.77:60974) 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28dad9d640 1 -- 172.21.15.77:0/3422128443 learned_addr learned my addr 172.21.15.77:0/3422128443 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28d3fff640 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7bac0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.417+0000 7f28dad9d640 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7bac0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28dad9d640 1 -- 172.21.15.77:0/3422128443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 msgr2=0x561c7bac0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28dad9d640 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7bac0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28dad9d640 1 -- 172.21.15.77:0/3422128443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 msgr2=0x561c7bac0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28dad9d640 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7bac0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28dad9d640 1 -- 172.21.15.77:0/3422128443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c7b7f47d0 con 0x561c7bab73b0 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28dad9d640 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bb2bea0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f28d4032420 tx=0x7f28d4036040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc8ab66bfb85790 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28c1ffb640 1 -- 172.21.15.77:0/3422128443 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d4039040 con 0x561c7bab73b0 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28c1ffb640 1 -- 172.21.15.77:0/3422128443 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f28d40579c0 con 0x561c7bab73b0 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28e3577740 1 -- 172.21.15.77:0/3422128443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28c400fac0 con 0x561c7bab73b0 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28c1ffb640 1 -- 172.21.15.77:0/3422128443 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f28d4043040 con 0x561c7bab73b0 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28e3577740 10 client.16023 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28e3577740 1 -- 172.21.15.77:0/3422128443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561c7bb2d480 con 0x561c7bab73b0 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28c1ffb640 1 -- 172.21.15.77:0/3422128443 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c400fac0 con 0x561c7bab73b0 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28c1ffb640 1 -- 172.21.15.77:0/3422128443 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f28d4069470 con 0x561c7bab73b0 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28e3577740 10 client.16023 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:47.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28e3577740 10 client.16023 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:18:47.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28e3577740 20 client.16023 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:47.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28e3577740 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x561c7bb2d9d0 0x561c7bb2fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28e3577740 4 client.16023 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:18:47.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.418+0000 7f28e3577740 1 -- 172.21.15.77:0/3422128443 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561c7ba1dd20 con 0x561c7bb2d9d0 2024-03-16T10:18:47.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.419+0000 7f28d3fff640 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x561c7bb2d9d0 0x561c7bb2fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.419+0000 7f28c1ffb640 10 client.16023 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:18:47.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.419+0000 7f28d3fff640 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x561c7bb2d9d0 0x561c7bb2fdc0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=33e8a1780bdfcd3b server_cookie=93445f3d0e82213c in_seq=0 out_seq=0 2024-03-16T10:18:47.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.425+0000 7f28c1ffb640 1 -- 172.21.15.77:0/3422128443 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561c7ba1dd20 con 0x561c7bb2d9d0 2024-03-16T10:18:47.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.425+0000 7f28c1ffb640 10 client.16023 handle_command_reply: tid=0 2024-03-16T10:18:47.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.425+0000 7f28e3577740 1 client.16023 shutdown 2024-03-16T10:18:47.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e3577740 1 -- 172.21.15.77:0/3422128443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 msgr2=0x561c7bb2bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e3577740 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bb2bea0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f28d4032420 tx=0x7f28d4036040 comp rx=0 tx=0).stop 2024-03-16T10:18:47.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e3577740 1 -- 172.21.15.77:0/3422128443 shutdown_connections 2024-03-16T10:18:47.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e3577740 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c7bab73b0 0x561c7bb2bea0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e3577740 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x561c7bb2d9d0 0x561c7bb2fdc0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e3577740 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561c7bab2240 0x561c7bac0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e3577740 1 --2- 172.21.15.77:0/3422128443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c7bab1850 0x561c7bac0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e3577740 1 -- 172.21.15.77:0/3422128443 >> 172.21.15.77:0/3422128443 conn(0x561c7baacdc0 msgr2=0x561c7babbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28c1ffb640 0 client.16023 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:18:47.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e3577740 1 -- 172.21.15.77:0/3422128443 shutdown_connections 2024-03-16T10:18:47.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e3577740 1 -- 172.21.15.77:0/3422128443 wait complete. 2024-03-16T10:18:47.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e3577740 20 client.16023 trim_cache size 0 max 16384 2024-03-16T10:18:47.430 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:18:47.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e0990640 1 -- 172.21.15.77:0/2427420541 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f28c80a2800 msgr2=0x7f28c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e0990640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f28c80a2800 0x7f28c80a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x561c7bab2640 tx=0x7f28cc0d2420 comp rx=0 tx=0).stop 2024-03-16T10:18:47.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e0990640 1 -- 172.21.15.77:0/2427420541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28dc1006e0 msgr2=0x7f28dc0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e0990640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28dc1006e0 0x7f28dc0694b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f28cc00e9c0 tx=0x7f28cc00ee80 comp rx=0 tx=0).stop 2024-03-16T10:18:47.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e0990640 1 -- 172.21.15.77:0/2427420541 shutdown_connections 2024-03-16T10:18:47.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e0990640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f28c80a2800 0x7f28c80a4cc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e0990640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28dc102290 0x7f28dc19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e0990640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28dc1018d0 0x7f28dc19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e0990640 1 --2- 172.21.15.77:0/2427420541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28dc1006e0 0x7f28dc0694b0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e0990640 1 -- 172.21.15.77:0/2427420541 >> 172.21.15.77:0/2427420541 conn(0x7f28dc0fbc70 msgr2=0x7f28dc104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e0990640 1 -- 172.21.15.77:0/2427420541 shutdown_connections 2024-03-16T10:18:47.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.426+0000 7f28e0990640 1 -- 172.21.15.77:0/2427420541 wait complete. 2024-03-16T10:18:47.438 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T10:18:47.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.550+0000 7f35bdb5b640 1 -- 172.21.15.77:0/3184316828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b81006e0 msgr2=0x7f35b8100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.550+0000 7f35bdb5b640 1 --2- 172.21.15.77:0/3184316828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b81006e0 0x7f35b8100b00 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f35a8009900 tx=0x7f35a802bc30 comp rx=0 tx=0).stop 2024-03-16T10:18:47.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.550+0000 7f35bdb5b640 1 -- 172.21.15.77:0/3184316828 shutdown_connections 2024-03-16T10:18:47.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.550+0000 7f35bdb5b640 1 --2- 172.21.15.77:0/3184316828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35b8102290 0x7f35b8108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.550+0000 7f35bdb5b640 1 --2- 172.21.15.77:0/3184316828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35b81018d0 0x7f35b8101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.550+0000 7f35bdb5b640 1 --2- 172.21.15.77:0/3184316828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b81006e0 0x7f35b8100b00 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.550+0000 7f35bdb5b640 1 -- 172.21.15.77:0/3184316828 >> 172.21.15.77:0/3184316828 conn(0x7f35b80fbc70 msgr2=0x7f35b80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.550+0000 7f35bdb5b640 1 -- 172.21.15.77:0/3184316828 shutdown_connections 2024-03-16T10:18:47.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.550+0000 7f35bdb5b640 1 -- 172.21.15.77:0/3184316828 wait complete. 2024-03-16T10:18:47.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35bdb5b640 1 Processor -- start 2024-03-16T10:18:47.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35bdb5b640 1 -- start start 2024-03-16T10:18:47.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35bdb5b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35b81006e0 0x7f35b80694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35bdb5b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b81018d0 0x7f35b819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35bdb5b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35b8102290 0x7f35b819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35bdb5b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f35b810c360 con 0x7f35b81018d0 2024-03-16T10:18:47.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35bdb5b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f35b810c1e0 con 0x7f35b81006e0 2024-03-16T10:18:47.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35bdb5b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35b810c4e0 con 0x7f35b8102290 2024-03-16T10:18:47.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35b6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b81018d0 0x7f35b819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35b6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b81018d0 0x7f35b819c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60998/0 (socket says 172.21.15.77:60998) 2024-03-16T10:18:47.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35b6ffd640 1 -- 172.21.15.77:0/1127837446 learned_addr learned my addr 172.21.15.77:0/1127837446 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35b7fff640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35b8102290 0x7f35b819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35b77fe640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35b81006e0 0x7f35b80694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35b6ffd640 1 -- 172.21.15.77:0/1127837446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35b8102290 msgr2=0x7f35b819ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35b8102290 0x7f35b819ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35b6ffd640 1 -- 172.21.15.77:0/1127837446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35b81006e0 msgr2=0x7f35b80694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.551+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35b81006e0 0x7f35b80694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.552+0000 7f35b6ffd640 1 -- 172.21.15.77:0/1127837446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b819f750 con 0x7f35b81018d0 2024-03-16T10:18:47.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.552+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b81018d0 0x7f35b819c850 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f35ac00d950 tx=0x7f35ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.552+0000 7f35b4ff9640 1 -- 172.21.15.77:0/1127837446 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac017030 con 0x7f35b81018d0 2024-03-16T10:18:47.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.552+0000 7f35b4ff9640 1 -- 172.21.15.77:0/1127837446 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35ac004020 con 0x7f35b81018d0 2024-03-16T10:18:47.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.552+0000 7f35bdb5b640 1 -- 172.21.15.77:0/1127837446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b819fa40 con 0x7f35b81018d0 2024-03-16T10:18:47.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.552+0000 7f35b4ff9640 1 -- 172.21.15.77:0/1127837446 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac0047a0 con 0x7f35b81018d0 2024-03-16T10:18:47.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.552+0000 7f35bdb5b640 1 -- 172.21.15.77:0/1127837446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b819ff90 con 0x7f35b81018d0 2024-03-16T10:18:47.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.554+0000 7f35b4ff9640 1 -- 172.21.15.77:0/1127837446 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f35ac0041d0 con 0x7f35b81018d0 2024-03-16T10:18:47.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.555+0000 7f35b4ff9640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f35a00a2800 0x7f35a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.555+0000 7f35b77fe640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f35a00a2800 0x7f35a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.555+0000 7f35b4ff9640 1 -- 172.21.15.77:0/1127837446 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f35ac0ce370 con 0x7f35b81018d0 2024-03-16T10:18:47.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.555+0000 7f35b77fe640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f35a00a2800 0x7f35a00a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f35a8000c00 tx=0x7f35a8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.555+0000 7f35c0829740 1 Processor -- start 2024-03-16T10:18:47.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.555+0000 7f35c0829740 1 -- start start 2024-03-16T10:18:47.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.555+0000 7f35c0829740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 0x55c25b1cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.555+0000 7f35c0829740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1ce240 0x55c25b1d2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.555+0000 7f35c0829740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1d5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.555+0000 7f35c0829740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c25b132490 con 0x55c25b1cd850 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.555+0000 7f35c0829740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c25b0dac90 con 0x55c25b1d33b0 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.555+0000 7f35c0829740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c25b12e530 con 0x55c25b1ce240 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f35b77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 0x55c25b1cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f35b77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 0x55c25b1cdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32768/0 (socket says 172.21.15.77:32768) 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f35b77fe640 1 -- 172.21.15.77:0/630996332 learned_addr learned my addr 172.21.15.77:0/630996332 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f35b7fff640 1 --2- 172.21.15.77:0/630996332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1d5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/630996332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1ce240 0x55c25b1d2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f35b77fe640 1 -- 172.21.15.77:0/630996332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1ce240 msgr2=0x55c25b1d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f35b77fe640 1 --2- 172.21.15.77:0/630996332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1ce240 0x55c25b1d2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f35b77fe640 1 -- 172.21.15.77:0/630996332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 msgr2=0x55c25b1d5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f35b77fe640 1 --2- 172.21.15.77:0/630996332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1d5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f35b77fe640 1 -- 172.21.15.77:0/630996332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c25b195500 con 0x55c25b1cd850 2024-03-16T10:18:47.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f35b7fff640 1 --2- 172.21.15.77:0/630996332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1d5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f35b77fe640 1 --2- 172.21.15.77:0/630996332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 0x55c25b1cdc70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f35a8005ec0 tx=0x7f35a803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d0a994011a240 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f359dffb640 1 -- 172.21.15.77:0/630996332 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a804c030 con 0x55c25b1cd850 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f359dffb640 1 -- 172.21.15.77:0/630996332 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35a8005530 con 0x55c25b1cd850 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.556+0000 7f359dffb640 1 -- 172.21.15.77:0/630996332 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a8004d70 con 0x55c25b1cd850 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.557+0000 7f35c0829740 1 -- 172.21.15.77:0/630996332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 msgr2=0x55c25b1cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.557+0000 7f35c0829740 1 --2- 172.21.15.77:0/630996332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 0x55c25b1cdc70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f35a8005ec0 tx=0x7f35a803e000 comp rx=0 tx=0).stop 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.557+0000 7f35c0829740 1 -- 172.21.15.77:0/630996332 shutdown_connections 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.557+0000 7f35c0829740 1 --2- 172.21.15.77:0/630996332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1d5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.557+0000 7f35c0829740 1 --2- 172.21.15.77:0/630996332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1ce240 0x55c25b1d2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.557+0000 7f35c0829740 1 --2- 172.21.15.77:0/630996332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 0x55c25b1cdc70 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.557+0000 7f35c0829740 1 -- 172.21.15.77:0/630996332 >> 172.21.15.77:0/630996332 conn(0x55c25b1c8dc0 msgr2=0x55c25b1c91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.557+0000 7f35c0829740 1 -- 172.21.15.77:0/630996332 shutdown_connections 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.557+0000 7f35c0829740 1 -- 172.21.15.77:0/630996332 wait complete. 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.557+0000 7f35c0829740 1 -- start start 2024-03-16T10:18:47.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35c0829740 1 Processor -- start 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35c0829740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 0x55c25b1d7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35c0829740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1ce240 0x55c25b1d8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35c0829740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1df260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35c0829740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c25b1d6300 con 0x55c25b1cd850 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35c0829740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c25b0ee230 con 0x55c25b1d33b0 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35c0829740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c25b1d6480 con 0x55c25b1ce240 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35b77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 0x55c25b1d7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35c0829740 10 client.0 _ll_register_callbacks cb 0x55c25b12a010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35b77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 0x55c25b1d7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32772/0 (socket says 172.21.15.77:32772) 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35b77fe640 1 -- 172.21.15.77:0/4271643122 learned_addr learned my addr 172.21.15.77:0/4271643122 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1ce240 0x55c25b1d8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35b7fff640 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1df260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35b77fe640 1 -- 172.21.15.77:0/4271643122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1ce240 msgr2=0x55c25b1d8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35b77fe640 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1ce240 0x55c25b1d8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35b77fe640 1 -- 172.21.15.77:0/4271643122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 msgr2=0x55c25b1df260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35b77fe640 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1df260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35b77fe640 1 -- 172.21.15.77:0/4271643122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c25b0fd100 con 0x55c25b1cd850 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f35b77fe640 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 0x55c25b1d7cf0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f35a802c170 tx=0x7f35a8005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67cdcc86d127a519 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.558+0000 7f3596ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f3596ffd640 1 -- 172.21.15.77:0/4271643122 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a804c040 con 0x55c25b1cd850 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f3596ffd640 1 -- 172.21.15.77:0/4271643122 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f35a8004cf0 con 0x55c25b1cd850 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f35c0829740 1 -- 172.21.15.77:0/4271643122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35a8005530 con 0x55c25b1cd850 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f3596ffd640 1 -- 172.21.15.77:0/4271643122 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f35a8004750 con 0x55c25b1cd850 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f35c0829740 10 client.16035 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f35c0829740 1 -- 172.21.15.77:0/4271643122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c25ae463f0 con 0x55c25b1cd850 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f3596ffd640 1 -- 172.21.15.77:0/4271643122 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35a8005530 con 0x55c25b1cd850 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f3596ffd640 1 -- 172.21.15.77:0/4271643122 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f35a8048540 con 0x55c25b1cd850 2024-03-16T10:18:47.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f35c0829740 10 client.16035 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f35c0829740 10 client.16035 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f35c0829740 20 client.16035 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f35c0829740 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55c25b1e3750 0x55c25b1e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f35c0829740 4 client.16035 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f35c0829740 1 -- 172.21.15.77:0/4271643122 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c25b13f9d0 con 0x55c25b1e3750 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.559+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55c25b1e3750 0x55c25b1e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.560+0000 7f3596ffd640 10 client.16035 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.560+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55c25b1e3750 0x55c25b1e5b40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f895fb9f9b212bba server_cookie=7548c8c3a5c7d38f in_seq=0 out_seq=0 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f3596ffd640 1 -- 172.21.15.77:0/4271643122 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c25b13f9d0 con 0x55c25b1e3750 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f3596ffd640 10 client.16035 handle_command_reply: tid=0 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f35c0829740 1 client.16035 shutdown 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f35c0829740 1 -- 172.21.15.77:0/4271643122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 msgr2=0x55c25b1d7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f35c0829740 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 0x55c25b1d7cf0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f35a802c170 tx=0x7f35a8005280 comp rx=0 tx=0).stop 2024-03-16T10:18:47.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f35c0829740 1 -- 172.21.15.77:0/4271643122 shutdown_connections 2024-03-16T10:18:47.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f35c0829740 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55c25b1e3750 0x55c25b1e5b40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f35c0829740 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1df260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f35c0829740 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1ce240 0x55c25b1d8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f3596ffd640 0 client.16035 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:18:47.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f35c0829740 1 --2- 172.21.15.77:0/4271643122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1cd850 0x55c25b1d7cf0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f35c0829740 1 -- 172.21.15.77:0/4271643122 >> 172.21.15.77:0/4271643122 conn(0x55c25b1c8dc0 msgr2=0x55c25b1c7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f35c0829740 1 -- 172.21.15.77:0/4271643122 shutdown_connections 2024-03-16T10:18:47.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f35c0829740 1 -- 172.21.15.77:0/4271643122 wait complete. 2024-03-16T10:18:47.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.561+0000 7f35c0829740 20 client.16035 trim_cache size 0 max 16384 2024-03-16T10:18:47.586 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.580+0000 7f35c0829740 1 Processor -- start 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35c0829740 1 -- start start 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35c0829740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1cd850 0x55c25b1d8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35c0829740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 0x55c25b1df220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35c0829740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1d33b0 0x55c25b1cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35c0829740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f35b81a8be0 con 0x55c25b1ce240 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35c0829740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f35b81aa7e0 con 0x55c25b1cd850 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35c0829740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35b81a92b0 con 0x55c25b1d33b0 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35b6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 0x55c25b1df220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35b6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 0x55c25b1df220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32776/0 (socket says 172.21.15.77:32776) 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35b6ffd640 1 -- 172.21.15.77:0/454773420 learned_addr learned my addr 172.21.15.77:0/454773420 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35b77fe640 1 --2- 172.21.15.77:0/454773420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1cd850 0x55c25b1d8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35b7fff640 1 --2- 172.21.15.77:0/454773420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1d33b0 0x55c25b1cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35b6ffd640 1 -- 172.21.15.77:0/454773420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1d33b0 msgr2=0x55c25b1cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/454773420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1d33b0 0x55c25b1cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35b6ffd640 1 -- 172.21.15.77:0/454773420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1cd850 msgr2=0x55c25b1d8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/454773420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1cd850 0x55c25b1d8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35b6ffd640 1 -- 172.21.15.77:0/454773420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c25af107d0 con 0x55c25b1ce240 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35b7fff640 1 --2- 172.21.15.77:0/454773420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1d33b0 0x55c25b1cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/454773420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 0x55c25b1df220 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f35ac097200 tx=0x7f35ac00df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c030f8529d42b7a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f359d7fa640 1 -- 172.21.15.77:0/454773420 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac07e070 con 0x55c25b1ce240 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f359d7fa640 1 -- 172.21.15.77:0/454773420 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35ac0b9070 con 0x55c25b1ce240 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.581+0000 7f359d7fa640 1 -- 172.21.15.77:0/454773420 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac0b3e60 con 0x55c25b1ce240 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.582+0000 7f35c0829740 1 -- 172.21.15.77:0/454773420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 msgr2=0x55c25b1df220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.582+0000 7f35c0829740 1 --2- 172.21.15.77:0/454773420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 0x55c25b1df220 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f35ac097200 tx=0x7f35ac00df80 comp rx=0 tx=0).stop 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.582+0000 7f35c0829740 1 -- 172.21.15.77:0/454773420 shutdown_connections 2024-03-16T10:18:47.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.582+0000 7f35c0829740 1 --2- 172.21.15.77:0/454773420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1d33b0 0x55c25b1cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.582+0000 7f35c0829740 1 --2- 172.21.15.77:0/454773420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 0x55c25b1df220 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.582+0000 7f35c0829740 1 --2- 172.21.15.77:0/454773420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1cd850 0x55c25b1d8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.582+0000 7f35c0829740 1 -- 172.21.15.77:0/454773420 >> 172.21.15.77:0/454773420 conn(0x55c25b1c8dc0 msgr2=0x55c25b1cc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.582+0000 7f35c0829740 1 -- 172.21.15.77:0/454773420 shutdown_connections 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.582+0000 7f35c0829740 1 -- 172.21.15.77:0/454773420 wait complete. 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35c0829740 1 -- start start 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35c0829740 1 Processor -- start 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35c0829740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1cd850 0x55c25b1dc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35c0829740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 0x55c25b1dcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35c0829740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1ddb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35c0829740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f35b81aa050 con 0x55c25b1ce240 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35c0829740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f35b81a9980 con 0x55c25b1d33b0 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35c0829740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35b81a6c10 con 0x55c25b1cd850 2024-03-16T10:18:47.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35c0829740 10 client.0 _ll_register_callbacks cb 0x7f35b0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35b6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 0x55c25b1dcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35b77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1cd850 0x55c25b1dc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35b77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1cd850 0x55c25b1dc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:40388/0 (socket says 172.21.15.77:40388) 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35b6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 0x55c25b1dcb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32784/0 (socket says 172.21.15.77:32784) 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35b77fe640 1 -- 172.21.15.77:0/1598563786 learned_addr learned my addr 172.21.15.77:0/1598563786 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35b7fff640 1 --2- 172.21.15.77:0/1598563786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1ddb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35b6ffd640 1 -- 172.21.15.77:0/1598563786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1cd850 msgr2=0x55c25b1dc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/1598563786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1cd850 0x55c25b1dc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35b6ffd640 1 -- 172.21.15.77:0/1598563786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 msgr2=0x55c25b1ddb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/1598563786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1ddb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.583+0000 7f35b6ffd640 1 -- 172.21.15.77:0/1598563786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c25b1de130 con 0x55c25b1ce240 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f35b6ffd640 1 --2- 172.21.15.77:0/1598563786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 0x55c25b1dcb30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f35ac097200 tx=0x7f35ac005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2681a6cb7f1ed server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f3596ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f3596ffd640 1 -- 172.21.15.77:0/1598563786 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac07e070 con 0x55c25b1ce240 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f3596ffd640 1 -- 172.21.15.77:0/1598563786 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f35ac09a1f0 con 0x55c25b1ce240 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f35c0829740 1 -- 172.21.15.77:0/1598563786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35ac0b9070 con 0x55c25b1ce240 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f3596ffd640 1 -- 172.21.15.77:0/1598563786 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f35ac0aa050 con 0x55c25b1ce240 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f35c0829740 10 client.16041 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f35c0829740 1 -- 172.21.15.77:0/1598563786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c25b1df460 con 0x55c25b1ce240 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f3596ffd640 1 -- 172.21.15.77:0/1598563786 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35ac0b9070 con 0x55c25b1ce240 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f3596ffd640 1 -- 172.21.15.77:0/1598563786 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f35ac09fe10 con 0x55c25b1ce240 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f35c0829740 10 client.16041 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f35c0829740 10 client.16041 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f35c0829740 20 client.16041 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f35c0829740 1 --2- 172.21.15.77:0/1598563786 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55c25b249e80 0x55c25b24c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f35c0829740 4 client.16041 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.584+0000 7f35c0829740 1 -- 172.21.15.77:0/1598563786 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c25b139d20 con 0x55c25b249e80 2024-03-16T10:18:47.591 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35b77fe640 1 --2- 172.21.15.77:0/1598563786 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55c25b249e80 0x55c25b24c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35b77fe640 1 --2- 172.21.15.77:0/1598563786 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55c25b249e80 0x55c25b24c2a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=837e4c2fa383373 server_cookie=efc01ad39b45c3b2 in_seq=0 out_seq=0 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f3596ffd640 10 client.16041 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f3596ffd640 1 -- 172.21.15.77:0/1598563786 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c25b139d20 con 0x55c25b249e80 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f3596ffd640 10 client.16041 handle_command_reply: tid=0 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35c0829740 1 client.16041 shutdown 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35c0829740 1 -- 172.21.15.77:0/1598563786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 msgr2=0x55c25b1dcb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35c0829740 1 --2- 172.21.15.77:0/1598563786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 0x55c25b1dcb30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f35ac097200 tx=0x7f35ac005bf0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35c0829740 1 -- 172.21.15.77:0/1598563786 shutdown_connections 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35c0829740 1 --2- 172.21.15.77:0/1598563786 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55c25b249e80 0x55c25b24c2a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35c0829740 1 --2- 172.21.15.77:0/1598563786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c25b1d33b0 0x55c25b1ddb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35c0829740 1 --2- 172.21.15.77:0/1598563786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c25b1ce240 0x55c25b1dcb30 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f3596ffd640 0 client.16041 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35c0829740 1 --2- 172.21.15.77:0/1598563786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c25b1cd850 0x55c25b1dc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35c0829740 1 -- 172.21.15.77:0/1598563786 >> 172.21.15.77:0/1598563786 conn(0x55c25b1c8dc0 msgr2=0x55c25b1d7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35c0829740 1 -- 172.21.15.77:0/1598563786 shutdown_connections 2024-03-16T10:18:47.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.585+0000 7f35c0829740 1 -- 172.21.15.77:0/1598563786 wait complete. 2024-03-16T10:18:47.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35c0829740 20 client.16041 trim_cache size 0 max 16384 2024-03-16T10:18:47.593 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:18:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35bdb5b640 1 -- 172.21.15.77:0/1127837446 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f35a00a2800 msgr2=0x7f35a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35bdb5b640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f35a00a2800 0x7f35a00a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f35a8000c00 tx=0x7f35a8005fd0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35bdb5b640 1 -- 172.21.15.77:0/1127837446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b81018d0 msgr2=0x7f35b819c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35bdb5b640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b81018d0 0x7f35b819c850 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f35ac00d950 tx=0x7f35ac00de10 comp rx=0 tx=0).stop 2024-03-16T10:18:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35bdb5b640 1 -- 172.21.15.77:0/1127837446 shutdown_connections 2024-03-16T10:18:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35bdb5b640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f35a00a2800 0x7f35a00a4cc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35bdb5b640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35b8102290 0x7f35b819ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35bdb5b640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35b81018d0 0x7f35b819c850 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35bdb5b640 1 --2- 172.21.15.77:0/1127837446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35b81006e0 0x7f35b80694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35bdb5b640 1 -- 172.21.15.77:0/1127837446 >> 172.21.15.77:0/1127837446 conn(0x7f35b80fbc70 msgr2=0x7f35b8104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35bdb5b640 1 -- 172.21.15.77:0/1127837446 shutdown_connections 2024-03-16T10:18:47.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.586+0000 7f35bdb5b640 1 -- 172.21.15.77:0/1127837446 wait complete. 2024-03-16T10:18:47.598 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T10:18:47.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.702+0000 7f2cc2036640 1 -- 172.21.15.77:0/1835632579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cbc1022b0 msgr2=0x7f2cbc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.702+0000 7f2cc2036640 1 --2- 172.21.15.77:0/1835632579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cbc1022b0 0x7f2cbc108b40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4009900 tx=0x7f2cb402bc30 comp rx=0 tx=0).stop 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.702+0000 7f2cc2036640 1 -- 172.21.15.77:0/1835632579 shutdown_connections 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.702+0000 7f2cc2036640 1 --2- 172.21.15.77:0/1835632579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cbc1022b0 0x7f2cbc108b40 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.702+0000 7f2cc2036640 1 --2- 172.21.15.77:0/1835632579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cbc1018f0 0x7f2cbc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.702+0000 7f2cc2036640 1 --2- 172.21.15.77:0/1835632579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cbc100700 0x7f2cbc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.702+0000 7f2cc2036640 1 -- 172.21.15.77:0/1835632579 >> 172.21.15.77:0/1835632579 conn(0x7f2cbc0fbc90 msgr2=0x7f2cbc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.703+0000 7f2cc2036640 1 -- 172.21.15.77:0/1835632579 shutdown_connections 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.703+0000 7f2cc2036640 1 -- 172.21.15.77:0/1835632579 wait complete. 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.703+0000 7f2cc2036640 1 Processor -- start 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.703+0000 7f2cc2036640 1 -- start start 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.703+0000 7f2cc2036640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cbc100700 0x7f2cbc06a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.703+0000 7f2cc2036640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cbc1018f0 0x7f2cbc19c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.703+0000 7f2cc2036640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cbc1022b0 0x7f2cbc19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.703+0000 7f2cc2036640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2cbc10c360 con 0x7f2cbc100700 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.703+0000 7f2cc2036640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2cbc10c1e0 con 0x7f2cbc1022b0 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.703+0000 7f2cc2036640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2cbc10c4e0 con 0x7f2cbc1018f0 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cbb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cbc100700 0x7f2cbc06a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cbb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cbc100700 0x7f2cbc06a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32800/0 (socket says 172.21.15.77:32800) 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cbb7fe640 1 -- 172.21.15.77:0/1093000833 learned_addr learned my addr 172.21.15.77:0/1093000833 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cbbfff640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cbc1022b0 0x7f2cbc19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cbaffd640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cbc1018f0 0x7f2cbc19c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cbb7fe640 1 -- 172.21.15.77:0/1093000833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cbc1018f0 msgr2=0x7f2cbc19c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cbc1018f0 0x7f2cbc19c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cbb7fe640 1 -- 172.21.15.77:0/1093000833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cbc1022b0 msgr2=0x7f2cbc19ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cbc1022b0 0x7f2cbc19ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cbb7fe640 1 -- 172.21.15.77:0/1093000833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cbc19f6a0 con 0x7f2cbc100700 2024-03-16T10:18:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cbc100700 0x7f2cbc06a940 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00ebc0 tx=0x7f2cac00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cb8ff9640 1 -- 172.21.15.77:0/1093000833 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac003b70 con 0x7f2cbc100700 2024-03-16T10:18:47.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cc2036640 1 -- 172.21.15.77:0/1093000833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cbc19f990 con 0x7f2cbc100700 2024-03-16T10:18:47.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cb8ff9640 1 -- 172.21.15.77:0/1093000833 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cac003d10 con 0x7f2cbc100700 2024-03-16T10:18:47.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cb8ff9640 1 -- 172.21.15.77:0/1093000833 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac019980 con 0x7f2cbc100700 2024-03-16T10:18:47.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.704+0000 7f2cc2036640 1 -- 172.21.15.77:0/1093000833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cbc19fee0 con 0x7f2cbc100700 2024-03-16T10:18:47.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cb8ff9640 1 -- 172.21.15.77:0/1093000833 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f2cac019b20 con 0x7f2cbc100700 2024-03-16T10:18:47.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cc4c24740 1 Processor -- start 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cc4c24740 1 -- start start 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cc4c24740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 0x55dba0657c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cc4c24740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0658240 0x55dba065cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cc4c24740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba065f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cc4c24740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55dba05bc490 con 0x55dba065d3b0 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cc4c24740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55dba0564c90 con 0x55dba0657850 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cc4c24740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dba05b8530 con 0x55dba0658240 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cbaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba065f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 0x55dba0657c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cbb7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0658240 0x55dba065cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cb8ff9640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ca40a2800 0x7f2ca40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 0x55dba0657c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:56208/0 (socket says 172.21.15.77:56208) 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.707+0000 7f2cbbfff640 1 -- 172.21.15.77:0/1207290054 learned_addr learned my addr 172.21.15.77:0/1207290054 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2cbbfff640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ca40a2800 0x7f2ca40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2cbaffd640 1 -- 172.21.15.77:0/1207290054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0658240 msgr2=0x55dba065cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2cbaffd640 1 --2- 172.21.15.77:0/1207290054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0658240 0x55dba065cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2cbaffd640 1 -- 172.21.15.77:0/1207290054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 msgr2=0x55dba0657c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2cb8ff9640 1 -- 172.21.15.77:0/1093000833 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2cac0cf1d0 con 0x7f2cbc100700 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2cbaffd640 1 --2- 172.21.15.77:0/1207290054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 0x55dba0657c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2cbaffd640 1 -- 172.21.15.77:0/1207290054 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dba061f500 con 0x55dba065d3b0 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/1207290054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0658240 0x55dba065cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2cbaffd640 1 --2- 172.21.15.77:0/1207290054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba065f840 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009bf0 tx=0x7f2cb0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3de362c142d3f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2cbbfff640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ca40a2800 0x7f2ca40a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4005c40 tx=0x7f2cb402bc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2ca1ffb640 1 -- 172.21.15.77:0/1207290054 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0004050 con 0x55dba065d3b0 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2ca1ffb640 1 -- 172.21.15.77:0/1207290054 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cb0005500 con 0x55dba065d3b0 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.708+0000 7f2ca1ffb640 1 -- 172.21.15.77:0/1207290054 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0004e20 con 0x55dba065d3b0 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.709+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1207290054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 msgr2=0x55dba065f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.709+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/1207290054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba065f840 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009bf0 tx=0x7f2cb0011810 comp rx=0 tx=0).stop 2024-03-16T10:18:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.709+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1207290054 shutdown_connections 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.709+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/1207290054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba065f840 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.709+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/1207290054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0658240 0x55dba065cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.709+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/1207290054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 0x55dba0657c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.709+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1207290054 >> 172.21.15.77:0/1207290054 conn(0x55dba0652dc0 msgr2=0x55dba06531c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.709+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1207290054 shutdown_connections 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.709+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1207290054 wait complete. 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.709+0000 7f2cc4c24740 1 -- start start 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.709+0000 7f2cc4c24740 1 Processor -- start 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cc4c24740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0657850 0x55dba0661d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cc4c24740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0658240 0x55dba0662260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cc4c24740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba0669260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cc4c24740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55dba0660360 con 0x55dba065d3b0 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cc4c24740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55dba0578230 con 0x55dba0658240 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cc4c24740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dba06604e0 con 0x55dba0657850 2024-03-16T10:18:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cc4c24740 10 client.0 _ll_register_callbacks cb 0x55dba05b4010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba0669260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cbaffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0658240 0x55dba0662260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba0669260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32812/0 (socket says 172.21.15.77:32812) 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cbbfff640 1 -- 172.21.15.77:0/3558497557 learned_addr learned my addr 172.21.15.77:0/3558497557 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0657850 0x55dba0661d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cbbfff640 1 -- 172.21.15.77:0/3558497557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0657850 msgr2=0x55dba0661d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cbbfff640 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0657850 0x55dba0661d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cbbfff640 1 -- 172.21.15.77:0/3558497557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0658240 msgr2=0x55dba0662260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cbbfff640 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0658240 0x55dba0662260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cbbfff640 1 -- 172.21.15.77:0/3558497557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dba0587100 con 0x55dba065d3b0 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cbaffd640 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0658240 0x55dba0662260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cbbfff640 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba0669260 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4005ab0 tx=0x7f2cb4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6800fe0c955698 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2c9affd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2c9affd640 1 -- 172.21.15.77:0/3558497557 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4052060 con 0x55dba065d3b0 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2c9affd640 1 -- 172.21.15.77:0/3558497557 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2cb4005300 con 0x55dba065d3b0 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.710+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3558497557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cb0005500 con 0x55dba065d3b0 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.711+0000 7f2c9affd640 1 -- 172.21.15.77:0/3558497557 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2cb4004d10 con 0x55dba065d3b0 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.711+0000 7f2cc4c24740 10 client.16053 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.711+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3558497557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dba02d03f0 con 0x55dba065d3b0 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.711+0000 7f2c9affd640 1 -- 172.21.15.77:0/3558497557 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cb0005500 con 0x55dba065d3b0 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.711+0000 7f2c9affd640 1 -- 172.21.15.77:0/3558497557 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2cb4042440 con 0x55dba065d3b0 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.711+0000 7f2cc4c24740 10 client.16053 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.711+0000 7f2cc4c24740 10 client.16053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.711+0000 7f2cc4c24740 20 client.16053 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.711+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dba066db90 0x55dba0670030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.711+0000 7f2cc4c24740 4 client.16053 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.711+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3558497557 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dba05c99d0 con 0x55dba066db90 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.711+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dba066db90 0x55dba0670030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.712+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dba066db90 0x55dba0670030 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=315333e5d59c6861 server_cookie=dbad76a543de6e2a in_seq=0 out_seq=0 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.712+0000 7f2c9affd640 10 client.16053 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.712+0000 7f2c9affd640 1 -- 172.21.15.77:0/3558497557 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dba05c99d0 con 0x55dba066db90 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.712+0000 7f2c9affd640 10 client.16053 handle_command_reply: tid=0 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.712+0000 7f2cc4c24740 1 client.16053 shutdown 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.712+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3558497557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 msgr2=0x55dba0669260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.712+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba0669260 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4005ab0 tx=0x7f2cb4009260 comp rx=0 tx=0).stop 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.712+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3558497557 shutdown_connections 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.712+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dba066db90 0x55dba0670030 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.713+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba0669260 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.713+0000 7f2c9affd640 0 client.16053 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.713+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0658240 0x55dba0662260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.713+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/3558497557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0657850 0x55dba0661d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.713+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3558497557 >> 172.21.15.77:0/3558497557 conn(0x55dba0652dc0 msgr2=0x55dba0651fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.713+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3558497557 shutdown_connections 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.713+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3558497557 wait complete. 2024-03-16T10:18:47.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.713+0000 7f2cc4c24740 20 client.16053 trim_cache size 0 max 16384 2024-03-16T10:18:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cc4c24740 1 Processor -- start 2024-03-16T10:18:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cc4c24740 1 -- start start 2024-03-16T10:18:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cc4c24740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba0657850 0x55dba06596a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cc4c24740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0658240 0x55dba0659be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cc4c24740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba065d3b0 0x55dba0669130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cc4c24740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2cbc1a8c00 con 0x55dba0657850 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cc4c24740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2cbc1aa800 con 0x55dba0658240 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cc4c24740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2cbc1a92d0 con 0x55dba065d3b0 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cbb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba0657850 0x55dba06596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cbb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba0657850 0x55dba06596a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32822/0 (socket says 172.21.15.77:32822) 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cbb7fe640 1 -- 172.21.15.77:0/3909295662 learned_addr learned my addr 172.21.15.77:0/3909295662 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cbaffd640 1 --2- 172.21.15.77:0/3909295662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0658240 0x55dba0659be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cbbfff640 1 --2- 172.21.15.77:0/3909295662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba065d3b0 0x55dba0669130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cbb7fe640 1 -- 172.21.15.77:0/3909295662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba065d3b0 msgr2=0x55dba0669130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/3909295662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba065d3b0 0x55dba0669130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cbb7fe640 1 -- 172.21.15.77:0/3909295662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0658240 msgr2=0x55dba0659be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/3909295662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0658240 0x55dba0659be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cbb7fe640 1 -- 172.21.15.77:0/3909295662 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dba039a7d0 con 0x55dba0657850 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.726+0000 7f2cbbfff640 1 --2- 172.21.15.77:0/3909295662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba065d3b0 0x55dba0669130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/3909295662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba0657850 0x55dba06596a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00cba0 tx=0x7f2cac00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96a1a00c3fd8734 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2ca17fa640 1 -- 172.21.15.77:0/3909295662 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac086040 con 0x55dba0657850 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2ca17fa640 1 -- 172.21.15.77:0/3909295662 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cac0a4380 con 0x55dba0657850 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2ca17fa640 1 -- 172.21.15.77:0/3909295662 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac098250 con 0x55dba0657850 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3909295662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba0657850 msgr2=0x55dba06596a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/3909295662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba0657850 0x55dba06596a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00cba0 tx=0x7f2cac00cf90 comp rx=0 tx=0).stop 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3909295662 shutdown_connections 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/3909295662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba065d3b0 0x55dba0669130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/3909295662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0658240 0x55dba0659be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/3909295662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba0657850 0x55dba06596a0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3909295662 >> 172.21.15.77:0/3909295662 conn(0x55dba0652dc0 msgr2=0x55dba0661d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3909295662 shutdown_connections 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.727+0000 7f2cc4c24740 1 -- 172.21.15.77:0/3909295662 wait complete. 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cc4c24740 1 -- start start 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cc4c24740 1 Processor -- start 2024-03-16T10:18:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cc4c24740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 0x55dba0665eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cc4c24740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0658240 0x55dba06663f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cc4c24740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba06d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cc4c24740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2cbc1aa070 con 0x55dba065d3b0 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cc4c24740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2cbc1a99a0 con 0x55dba0657850 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cc4c24740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2cbc1a6be0 con 0x55dba0658240 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cc4c24740 10 client.0 _ll_register_callbacks cb 0x7f2cb4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba06d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cbb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 0x55dba0665eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba06d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32836/0 (socket says 172.21.15.77:32836) 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cbb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 0x55dba0665eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:56226/0 (socket says 172.21.15.77:56226) 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cbbfff640 1 -- 172.21.15.77:0/1015363627 learned_addr learned my addr 172.21.15.77:0/1015363627 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cbaffd640 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0658240 0x55dba06663f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cbbfff640 1 -- 172.21.15.77:0/1015363627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0658240 msgr2=0x55dba06663f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cbbfff640 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0658240 0x55dba06663f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cbbfff640 1 -- 172.21.15.77:0/1015363627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 msgr2=0x55dba0665eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cbbfff640 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 0x55dba0665eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cbbfff640 1 -- 172.21.15.77:0/1015363627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dba0666a60 con 0x55dba065d3b0 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.728+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 0x55dba0665eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2cbbfff640 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba06d1f20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2cb402ce70 tx=0x7f2cb4042cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5014eb325a682cd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2c9affd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2c9affd640 1 -- 172.21.15.77:0/1015363627 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4004360 con 0x55dba065d3b0 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2c9affd640 1 -- 172.21.15.77:0/1015363627 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2cb4005510 con 0x55dba065d3b0 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1015363627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cac0a4380 con 0x55dba065d3b0 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2c9affd640 1 -- 172.21.15.77:0/1015363627 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2cb40035f0 con 0x55dba065d3b0 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2cc4c24740 10 client.16059 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1015363627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dba06d35c0 con 0x55dba065d3b0 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2c9affd640 1 -- 172.21.15.77:0/1015363627 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cac0a4380 con 0x55dba065d3b0 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2c9affd640 1 -- 172.21.15.77:0/1015363627 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2cb4034790 con 0x55dba065d3b0 2024-03-16T10:18:47.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2cc4c24740 10 client.16059 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2cc4c24740 10 client.16059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2cc4c24740 20 client.16059 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dba066db90 0x55dba06d60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2cc4c24740 4 client.16059 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.729+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1015363627 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dba05c3d20 con 0x55dba066db90 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dba066db90 0x55dba06d60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2c9affd640 10 client.16059 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cbb7fe640 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dba066db90 0x55dba06d60a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec4d264948e8309f server_cookie=6ba2f829821aaf26 in_seq=0 out_seq=0 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2c9affd640 1 -- 172.21.15.77:0/1015363627 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dba05c3d20 con 0x55dba066db90 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2c9affd640 10 client.16059 handle_command_reply: tid=0 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cc4c24740 1 client.16059 shutdown 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1015363627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 msgr2=0x55dba06d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba06d1f20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2cb402ce70 tx=0x7f2cb4042cb0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1015363627 shutdown_connections 2024-03-16T10:18:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55dba066db90 0x55dba06d60a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dba065d3b0 0x55dba06d1f20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dba0658240 0x55dba06663f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2c9affd640 0 client.16059 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:47.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cc4c24740 1 --2- 172.21.15.77:0/1015363627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dba0657850 0x55dba0665eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1015363627 >> 172.21.15.77:0/1015363627 conn(0x55dba0652dc0 msgr2=0x55dba06596a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1015363627 shutdown_connections 2024-03-16T10:18:47.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cc4c24740 1 -- 172.21.15.77:0/1015363627 wait complete. 2024-03-16T10:18:47.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.730+0000 7f2cc4c24740 20 client.16059 trim_cache size 0 max 16384 2024-03-16T10:18:47.738 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:18:47.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.731+0000 7f2cc2036640 1 -- 172.21.15.77:0/1093000833 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ca40a2800 msgr2=0x7f2ca40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.731+0000 7f2cc2036640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ca40a2800 0x7f2ca40a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4005c40 tx=0x7f2cb402bc30 comp rx=0 tx=0).stop 2024-03-16T10:18:47.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.731+0000 7f2cc2036640 1 -- 172.21.15.77:0/1093000833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cbc100700 msgr2=0x7f2cbc06a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:47.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.731+0000 7f2cc2036640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cbc100700 0x7f2cbc06a940 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00ebc0 tx=0x7f2cac00c6a0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.731+0000 7f2cc2036640 1 -- 172.21.15.77:0/1093000833 shutdown_connections 2024-03-16T10:18:47.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.731+0000 7f2cc2036640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cbc1022b0 0x7f2cbc19ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.731+0000 7f2cc2036640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2ca40a2800 0x7f2ca40a4cc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.731+0000 7f2cc2036640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cbc1018f0 0x7f2cbc19c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.731+0000 7f2cc2036640 1 --2- 172.21.15.77:0/1093000833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cbc100700 0x7f2cbc06a940 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:47.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.731+0000 7f2cc2036640 1 -- 172.21.15.77:0/1093000833 >> 172.21.15.77:0/1093000833 conn(0x7f2cbc0fbc90 msgr2=0x7f2cbc104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:47.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.731+0000 7f2cc2036640 1 -- 172.21.15.77:0/1093000833 shutdown_connections 2024-03-16T10:18:47.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:47.731+0000 7f2cc2036640 1 -- 172.21.15.77:0/1093000833 wait complete. 2024-03-16T10:18:48.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:47 smithi192 ceph-mon[36354]: pgmap v401: 97 pgs: 97 active+clean; 3.4 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 265 KiB/s wr, 41 op/s 2024-03-16T10:18:48.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:47 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:18:48.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:47 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/351101964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:18:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:47 smithi077 ceph-mon[32400]: pgmap v401: 97 pgs: 97 active+clean; 3.4 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 265 KiB/s wr, 41 op/s 2024-03-16T10:18:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:47 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:18:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:47 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/351101964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:18:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:47 smithi193 ceph-mon[36311]: pgmap v401: 97 pgs: 97 active+clean; 3.4 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 265 KiB/s wr, 41 op/s 2024-03-16T10:18:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:47 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:18:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:47 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/351101964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:18:48.743 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T10:18:48.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:18:48.744 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T10:18:48.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.851+0000 7f0824522640 1 -- 172.21.15.77:0/3145758397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f081c107560 msgr2=0x7f081c109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:48.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.851+0000 7f0824522640 1 --2- 172.21.15.77:0/3145758397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f081c107560 0x7f081c109950 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f0814009ca0 tx=0x7f081402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.851+0000 7f0824522640 1 -- 172.21.15.77:0/3145758397 shutdown_connections 2024-03-16T10:18:48.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.851+0000 7f0824522640 1 --2- 172.21.15.77:0/3145758397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f081c107560 0x7f081c109950 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.851+0000 7f0824522640 1 --2- 172.21.15.77:0/3145758397 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f081c104c30 0x7f081c107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.851+0000 7f0824522640 1 --2- 172.21.15.77:0/3145758397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f081c102300 0x7f081c1046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.851+0000 7f0824522640 1 -- 172.21.15.77:0/3145758397 >> 172.21.15.77:0/3145758397 conn(0x7f081c0fbc70 msgr2=0x7f081c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:48.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.851+0000 7f0824522640 1 -- 172.21.15.77:0/3145758397 shutdown_connections 2024-03-16T10:18:48.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.851+0000 7f0824522640 1 -- 172.21.15.77:0/3145758397 wait complete. 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.851+0000 7f0824522640 1 Processor -- start 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.851+0000 7f0824522640 1 -- start start 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0824522640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f081c102300 0x7f081c101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0824522640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f081c104c30 0x7f081c1002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0824522640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f081c107560 0x7f081c1007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0824522640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f081c10d460 con 0x7f081c107560 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0824522640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f081c10d2e0 con 0x7f081c102300 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0824522640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f081c10d5e0 con 0x7f081c104c30 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0822abf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f081c107560 0x7f081c1007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0821abd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f081c104c30 0x7f081c1002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f08222be640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f081c102300 0x7f081c101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0822abf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f081c107560 0x7f081c1007f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32850/0 (socket says 172.21.15.77:32850) 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0822abf640 1 -- 172.21.15.77:0/3343337888 learned_addr learned my addr 172.21.15.77:0/3343337888 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:48.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0822abf640 1 -- 172.21.15.77:0/3343337888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f081c104c30 msgr2=0x7f081c1002b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:48.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0822abf640 1 --2- 172.21.15.77:0/3343337888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f081c104c30 0x7f081c1002b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0822abf640 1 -- 172.21.15.77:0/3343337888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f081c102300 msgr2=0x7f081c101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:48.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0822abf640 1 --2- 172.21.15.77:0/3343337888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f081c102300 0x7f081c101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0822abf640 1 -- 172.21.15.77:0/3343337888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f081c101080 con 0x7f081c107560 2024-03-16T10:18:48.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.852+0000 7f0822abf640 1 --2- 172.21.15.77:0/3343337888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f081c107560 0x7f081c1007f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0814037040 tx=0x7f0814032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.853+0000 7f080f7fe640 1 -- 172.21.15.77:0/3343337888 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0814041020 con 0x7f081c107560 2024-03-16T10:18:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.853+0000 7f080f7fe640 1 -- 172.21.15.77:0/3343337888 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0814034cd0 con 0x7f081c107560 2024-03-16T10:18:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.853+0000 7f0824522640 1 -- 172.21.15.77:0/3343337888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f081c1a1bc0 con 0x7f081c107560 2024-03-16T10:18:48.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.853+0000 7f080f7fe640 1 -- 172.21.15.77:0/3343337888 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0814039040 con 0x7f081c107560 2024-03-16T10:18:48.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.853+0000 7f0824522640 1 -- 172.21.15.77:0/3343337888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f081c1a2110 con 0x7f081c107560 2024-03-16T10:18:48.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.855+0000 7f080f7fe640 1 -- 172.21.15.77:0/3343337888 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f0814044020 con 0x7f081c107560 2024-03-16T10:18:48.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.856+0000 7f08271ec740 1 Processor -- start 2024-03-16T10:18:48.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.856+0000 7f08271ec740 1 -- start start 2024-03-16T10:18:48.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.856+0000 7f08271ec740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01d5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.856+0000 7f08271ec740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01dae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.856+0000 7f08271ec740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01dd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.856+0000 7f08271ec740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560cf013a0d0 con 0x560cf01db4f0 2024-03-16T10:18:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.856+0000 7f08271ec740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x560cf00e2c70 con 0x560cf01d6380 2024-03-16T10:18:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.856+0000 7f08271ec740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560cf0136170 con 0x560cf01d5990 2024-03-16T10:18:48.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.856+0000 7f080f7fe640 1 --2- 172.21.15.77:0/3343337888 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f08080a2800 0x7f08080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.856+0000 7f08222be640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01dae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f0821abd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01d5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f08222be640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01dae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:56258/0 (socket says 172.21.15.77:56258) 2024-03-16T10:18:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f08222be640 1 -- 172.21.15.77:0/3452203709 learned_addr learned my addr 172.21.15.77:0/3452203709 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f0821abd640 1 --2- 172.21.15.77:0/3452203709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01d5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:40500/0 (socket says 172.21.15.77:40500) 2024-03-16T10:18:48.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f0821abd640 1 --2- 172.21.15.77:0/3452203709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01dd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f0822abf640 1 --2- 172.21.15.77:0/3343337888 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f08080a2800 0x7f08080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f0821abd640 1 -- 172.21.15.77:0/3452203709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 msgr2=0x560cf01dae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f0821abd640 1 --2- 172.21.15.77:0/3452203709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01dae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f0821abd640 1 -- 172.21.15.77:0/3452203709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 msgr2=0x560cf01dd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f0821abd640 1 --2- 172.21.15.77:0/3452203709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01dd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f0821abd640 1 -- 172.21.15.77:0/3452203709 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ceff167e0 con 0x560cf01d5990 2024-03-16T10:18:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f08222be640 1 --2- 172.21.15.77:0/3452203709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01dae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f080f7fe640 1 -- 172.21.15.77:0/3343337888 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f08140f0890 con 0x7f081c107560 2024-03-16T10:18:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f0821abd640 1 --2- 172.21.15.77:0/3452203709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01d5db0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0818009730 tx=0x7f08180096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc9eb8fd3e8e0072 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f0822abf640 1 --2- 172.21.15.77:0/3343337888 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f08080a2800 0x7f08080a4cc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f081404c000 tx=0x7f08140a7e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f080cbfe640 1 -- 172.21.15.77:0/3452203709 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0818004050 con 0x560cf01d5990 2024-03-16T10:18:48.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f080cbfe640 1 -- 172.21.15.77:0/3452203709 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0818005500 con 0x560cf01d5990 2024-03-16T10:18:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.857+0000 7f080cbfe640 1 -- 172.21.15.77:0/3452203709 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0818004d90 con 0x560cf01d5990 2024-03-16T10:18:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.858+0000 7f08271ec740 1 -- 172.21.15.77:0/3452203709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 msgr2=0x560cf01d5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.858+0000 7f08271ec740 1 --2- 172.21.15.77:0/3452203709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01d5db0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0818009730 tx=0x7f08180096c0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.858+0000 7f08271ec740 1 -- 172.21.15.77:0/3452203709 shutdown_connections 2024-03-16T10:18:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.858+0000 7f08271ec740 1 --2- 172.21.15.77:0/3452203709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01dd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.858+0000 7f08271ec740 1 --2- 172.21.15.77:0/3452203709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01dae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.858+0000 7f08271ec740 1 --2- 172.21.15.77:0/3452203709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01d5db0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.858+0000 7f08271ec740 1 -- 172.21.15.77:0/3452203709 >> 172.21.15.77:0/3452203709 conn(0x560cf019dc50 msgr2=0x560cf01d1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.858+0000 7f08271ec740 1 -- 172.21.15.77:0/3452203709 shutdown_connections 2024-03-16T10:18:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.858+0000 7f08271ec740 1 -- 172.21.15.77:0/3452203709 wait complete. 2024-03-16T10:18:48.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.858+0000 7f08271ec740 1 -- start start 2024-03-16T10:18:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f08271ec740 1 Processor -- start 2024-03-16T10:18:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f08271ec740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01dff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f08271ec740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f08271ec740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01e7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f08271ec740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560cf01de4a0 con 0x560cf01db4f0 2024-03-16T10:18:48.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f08271ec740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x560cf00f60d0 con 0x560cf01d6380 2024-03-16T10:18:48.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f08271ec740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560cf01de620 con 0x560cf01d5990 2024-03-16T10:18:48.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f08271ec740 10 client.0 _ll_register_callbacks cb 0x560cf00bf760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:48.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f0821abd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01e7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f0821abd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01dff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f08222be640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f0821abd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01dff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:40516/0 (socket says 172.21.15.77:40516) 2024-03-16T10:18:48.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f0821abd640 1 -- 172.21.15.77:0/922714537 learned_addr learned my addr 172.21.15.77:0/922714537 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:48.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f0821abd640 1 -- 172.21.15.77:0/922714537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 msgr2=0x560cf01dff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:48.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f0821abd640 1 --2- 172.21.15.77:0/922714537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01dff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f0821abd640 1 -- 172.21.15.77:0/922714537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 msgr2=0x560cf01e0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:48.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f0821abd640 1 --2- 172.21.15.77:0/922714537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f0821abd640 1 -- 172.21.15.77:0/922714537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ceff76f70 con 0x560cf01db4f0 2024-03-16T10:18:48.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f08222be640 1 --2- 172.21.15.77:0/922714537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:48.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f0821abd640 1 --2- 172.21.15.77:0/922714537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01e7490 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f08180097c0 tx=0x7f0818005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b40dcbd61b4e33bd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:48.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.859+0000 7f0805ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:48.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f0805ffb640 1 -- 172.21.15.77:0/922714537 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0818028040 con 0x560cf01db4f0 2024-03-16T10:18:48.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f0805ffb640 1 -- 172.21.15.77:0/922714537 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f0818004250 con 0x560cf01db4f0 2024-03-16T10:18:48.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f08271ec740 1 -- 172.21.15.77:0/922714537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0818005500 con 0x560cf01db4f0 2024-03-16T10:18:48.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f0805ffb640 1 -- 172.21.15.77:0/922714537 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0818034050 con 0x560cf01db4f0 2024-03-16T10:18:48.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f08271ec740 10 client.16071 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:48.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f08271ec740 1 -- 172.21.15.77:0/922714537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560cefe0e170 con 0x560cf01db4f0 2024-03-16T10:18:48.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f0805ffb640 1 -- 172.21.15.77:0/922714537 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0818005500 con 0x560cf01db4f0 2024-03-16T10:18:48.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f0805ffb640 1 -- 172.21.15.77:0/922714537 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0818003780 con 0x560cf01db4f0 2024-03-16T10:18:48.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f08271ec740 10 client.16071 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:48.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f08271ec740 10 client.16071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:48.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f08271ec740 20 client.16071 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:48.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f08271ec740 1 --2- 172.21.15.77:0/922714537 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560cf01eb920 0x560cf01edd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f08271ec740 4 client.16071 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:48.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f08271ec740 1 -- 172.21.15.77:0/922714537 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cf0147610 con 0x560cf01eb920 2024-03-16T10:18:48.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.860+0000 7f08222be640 1 --2- 172.21.15.77:0/922714537 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560cf01eb920 0x560cf01edd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.861+0000 7f0805ffb640 10 client.16071 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:48.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.861+0000 7f08222be640 1 --2- 172.21.15.77:0/922714537 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560cf01eb920 0x560cf01edd60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f9c5b2765dc2d4 server_cookie=34aa9bc487074d9f in_seq=0 out_seq=0 2024-03-16T10:18:48.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.861+0000 7f0805ffb640 1 -- 172.21.15.77:0/922714537 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560cf0147610 con 0x560cf01eb920 2024-03-16T10:18:48.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.861+0000 7f0805ffb640 10 client.16071 handle_command_reply: tid=0 2024-03-16T10:18:48.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.861+0000 7f08271ec740 1 client.16071 shutdown 2024-03-16T10:18:48.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.862+0000 7f08271ec740 1 -- 172.21.15.77:0/922714537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 msgr2=0x560cf01e7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:48.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.862+0000 7f08271ec740 1 --2- 172.21.15.77:0/922714537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01e7490 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f08180097c0 tx=0x7f0818005850 comp rx=0 tx=0).stop 2024-03-16T10:18:48.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.862+0000 7f08271ec740 1 -- 172.21.15.77:0/922714537 shutdown_connections 2024-03-16T10:18:48.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.862+0000 7f08271ec740 1 --2- 172.21.15.77:0/922714537 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560cf01eb920 0x560cf01edd60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.862+0000 7f08271ec740 1 --2- 172.21.15.77:0/922714537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01e7490 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.862+0000 7f0805ffb640 0 client.16071 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:48.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.862+0000 7f08271ec740 1 --2- 172.21.15.77:0/922714537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.862+0000 7f08271ec740 1 --2- 172.21.15.77:0/922714537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01dff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.862+0000 7f08271ec740 1 -- 172.21.15.77:0/922714537 >> 172.21.15.77:0/922714537 conn(0x560cf019dc50 msgr2=0x560cf01d04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:48.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.862+0000 7f08271ec740 1 -- 172.21.15.77:0/922714537 shutdown_connections 2024-03-16T10:18:48.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.862+0000 7f08271ec740 1 -- 172.21.15.77:0/922714537 wait complete. 2024-03-16T10:18:48.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.862+0000 7f08271ec740 20 client.16071 trim_cache size 0 max 16384 2024-03-16T10:18:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f08271ec740 1 Processor -- start 2024-03-16T10:18:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f08271ec740 1 -- start start 2024-03-16T10:18:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f08271ec740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01d4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f08271ec740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f08271ec740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01f3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f08271ec740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1aad90 con 0x560cf01db4f0 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f08271ec740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1ac990 con 0x560cf01d6380 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f08271ec740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1ab460 con 0x560cf01d5990 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f0821abd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f0821abd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01f3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32874/0 (socket says 172.21.15.77:32874) 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f0821abd640 1 -- 172.21.15.77:0/956599606 learned_addr learned my addr 172.21.15.77:0/956599606 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f08222be640 1 --2- 172.21.15.77:0/956599606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f0821abd640 1 --2- 172.21.15.77:0/956599606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01d4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f0821abd640 1 -- 172.21.15.77:0/956599606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 msgr2=0x560cf01d4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f0821abd640 1 --2- 172.21.15.77:0/956599606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01d4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f0821abd640 1 -- 172.21.15.77:0/956599606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 msgr2=0x560cf01e7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f0821abd640 1 --2- 172.21.15.77:0/956599606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.880+0000 7f0821abd640 1 -- 172.21.15.77:0/956599606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cf01d7f20 con 0x560cf01db4f0 2024-03-16T10:18:48.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f0821abd640 1 --2- 172.21.15.77:0/956599606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01f3630 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0818006010 tx=0x7f0818005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de46b9aaf177a0eb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f0807fff640 1 -- 172.21.15.77:0/956599606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08180099f0 con 0x560cf01db4f0 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f0807fff640 1 -- 172.21.15.77:0/956599606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f081800fd90 con 0x560cf01db4f0 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f0807fff640 1 -- 172.21.15.77:0/956599606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081800f9f0 con 0x560cf01db4f0 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f08271ec740 1 -- 172.21.15.77:0/956599606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 msgr2=0x560cf01f3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f08271ec740 1 --2- 172.21.15.77:0/956599606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01f3630 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0818006010 tx=0x7f0818005b90 comp rx=0 tx=0).stop 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f08271ec740 1 -- 172.21.15.77:0/956599606 shutdown_connections 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f08271ec740 1 --2- 172.21.15.77:0/956599606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01f3630 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f08271ec740 1 --2- 172.21.15.77:0/956599606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f08271ec740 1 --2- 172.21.15.77:0/956599606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01d4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f08271ec740 1 -- 172.21.15.77:0/956599606 >> 172.21.15.77:0/956599606 conn(0x560cf019dc50 msgr2=0x560cf01d4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f08271ec740 1 -- 172.21.15.77:0/956599606 shutdown_connections 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.881+0000 7f08271ec740 1 -- 172.21.15.77:0/956599606 wait complete. 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08271ec740 1 -- start start 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08271ec740 1 Processor -- start 2024-03-16T10:18:48.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08271ec740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01e4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08271ec740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08271ec740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01e6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08271ec740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1ac200 con 0x560cf01db4f0 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08271ec740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1abb30 con 0x560cf01d6380 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08271ec740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1a8d70 con 0x560cf01d5990 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08271ec740 10 client.0 _ll_register_callbacks cb 0x7f0810009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f0821abd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01e6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08222be640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08222be640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:56294/0 (socket says 172.21.15.77:56294) 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f0821abd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01e6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32888/0 (socket says 172.21.15.77:32888) 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08222be640 1 -- 172.21.15.77:0/3020824401 learned_addr learned my addr 172.21.15.77:0/3020824401 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f0821abd640 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01e4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f0821abd640 1 -- 172.21.15.77:0/3020824401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 msgr2=0x560cf01e4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:48.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f0821abd640 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01e4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f0821abd640 1 -- 172.21.15.77:0/3020824401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 msgr2=0x560cf01e51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f0821abd640 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f0821abd640 1 -- 172.21.15.77:0/3020824401 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cf01e67f0 con 0x560cf01db4f0 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.882+0000 7f08222be640 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f0821abd640 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01e6200 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f08180097c0 tx=0x7f0818005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a7ba270ebf70b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f0805ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f0805ffb640 1 -- 172.21.15.77:0/3020824401 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08180099f0 con 0x560cf01db4f0 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f0805ffb640 1 -- 172.21.15.77:0/3020824401 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f08180035f0 con 0x560cf01db4f0 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f08271ec740 1 -- 172.21.15.77:0/3020824401 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f081800fd90 con 0x560cf01db4f0 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f0805ffb640 1 -- 172.21.15.77:0/3020824401 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0818004850 con 0x560cf01db4f0 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f08271ec740 10 client.16077 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f08271ec740 1 -- 172.21.15.77:0/3020824401 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560cf0251db0 con 0x560cf01db4f0 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f0805ffb640 1 -- 172.21.15.77:0/3020824401 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f081800fd90 con 0x560cf01db4f0 2024-03-16T10:18:48.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f0805ffb640 1 -- 172.21.15.77:0/3020824401 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0818004070 con 0x560cf01db4f0 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f08271ec740 10 client.16077 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f08271ec740 10 client.16077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f08271ec740 20 client.16077 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f08271ec740 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560cf0252680 0x560cf0254aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f08271ec740 4 client.16077 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f08271ec740 1 -- 172.21.15.77:0/3020824401 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560cf0141960 con 0x560cf0252680 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.883+0000 7f08222be640 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560cf0252680 0x560cf0254aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.884+0000 7f08222be640 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560cf0252680 0x560cf0254aa0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ae8e18df2bf477d server_cookie=5740c4445d292966 in_seq=0 out_seq=0 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.884+0000 7f0805ffb640 10 client.16077 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f0805ffb640 1 -- 172.21.15.77:0/3020824401 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560cf0141960 con 0x560cf0252680 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f0805ffb640 10 client.16077 handle_command_reply: tid=0 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f08271ec740 1 client.16077 shutdown 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f08271ec740 1 -- 172.21.15.77:0/3020824401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 msgr2=0x560cf01e6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f08271ec740 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01e6200 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f08180097c0 tx=0x7f0818005970 comp rx=0 tx=0).stop 2024-03-16T10:18:48.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f08271ec740 1 -- 172.21.15.77:0/3020824401 shutdown_connections 2024-03-16T10:18:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f08271ec740 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560cf0252680 0x560cf0254aa0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f08271ec740 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560cf01db4f0 0x560cf01e6200 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f08271ec740 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560cf01d6380 0x560cf01e51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f0805ffb640 0 client.16077 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f08271ec740 1 --2- 172.21.15.77:0/3020824401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cf01d5990 0x560cf01e4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f08271ec740 1 -- 172.21.15.77:0/3020824401 >> 172.21.15.77:0/3020824401 conn(0x560cf019dc50 msgr2=0x560cf01f2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f08271ec740 1 -- 172.21.15.77:0/3020824401 shutdown_connections 2024-03-16T10:18:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f08271ec740 1 -- 172.21.15.77:0/3020824401 wait complete. 2024-03-16T10:18:48.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f08271ec740 20 client.16077 trim_cache size 0 max 16384 2024-03-16T10:18:48.896 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:18:48.897 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T10:18:48.897 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "bf9f313c-ca13-4923-a775-539cd1f60df2", 2024-03-16T10:18:48.897 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T10:18:48.897 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:18:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f0824522640 1 -- 172.21.15.77:0/3343337888 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f08080a2800 msgr2=0x7f08080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f0824522640 1 --2- 172.21.15.77:0/3343337888 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f08080a2800 0x7f08080a4cc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f081404c000 tx=0x7f08140a7e30 comp rx=0 tx=0).stop 2024-03-16T10:18:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f0824522640 1 -- 172.21.15.77:0/3343337888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f081c107560 msgr2=0x7f081c1007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.885+0000 7f0824522640 1 --2- 172.21.15.77:0/3343337888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f081c107560 0x7f081c1007f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0814037040 tx=0x7f0814032f40 comp rx=0 tx=0).stop 2024-03-16T10:18:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.886+0000 7f0824522640 1 -- 172.21.15.77:0/3343337888 shutdown_connections 2024-03-16T10:18:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.886+0000 7f0824522640 1 --2- 172.21.15.77:0/3343337888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f081c107560 0x7f081c1007f0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.886+0000 7f0824522640 1 --2- 172.21.15.77:0/3343337888 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f08080a2800 0x7f08080a4cc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.886+0000 7f0824522640 1 --2- 172.21.15.77:0/3343337888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f081c104c30 0x7f081c1002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.886+0000 7f0824522640 1 --2- 172.21.15.77:0/3343337888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f081c102300 0x7f081c101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.886+0000 7f0824522640 1 -- 172.21.15.77:0/3343337888 >> 172.21.15.77:0/3343337888 conn(0x7f081c0fbc70 msgr2=0x7f081c102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.886+0000 7f0824522640 1 -- 172.21.15.77:0/3343337888 shutdown_connections 2024-03-16T10:18:48.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:48.886+0000 7f0824522640 1 -- 172.21.15.77:0/3343337888 wait complete. 2024-03-16T10:18:48.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:18:48.899 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:18:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.007+0000 7f18a4068640 1 -- 172.21.15.77:0/1847023802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f189c102290 msgr2=0x7f189c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.007+0000 7f18a4068640 1 --2- 172.21.15.77:0/1847023802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f189c102290 0x7f189c108b20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1894009ca0 tx=0x7f189402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.007+0000 7f18a4068640 1 -- 172.21.15.77:0/1847023802 shutdown_connections 2024-03-16T10:18:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.007+0000 7f18a4068640 1 --2- 172.21.15.77:0/1847023802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f189c102290 0x7f189c108b20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.007+0000 7f18a4068640 1 --2- 172.21.15.77:0/1847023802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f189c1018d0 0x7f189c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.007+0000 7f18a4068640 1 --2- 172.21.15.77:0/1847023802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f189c1006e0 0x7f189c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.007+0000 7f18a4068640 1 -- 172.21.15.77:0/1847023802 >> 172.21.15.77:0/1847023802 conn(0x7f189c0fbc70 msgr2=0x7f189c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.007+0000 7f18a4068640 1 -- 172.21.15.77:0/1847023802 shutdown_connections 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.007+0000 7f18a4068640 1 -- 172.21.15.77:0/1847023802 wait complete. 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.007+0000 7f18a4068640 1 Processor -- start 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a4068640 1 -- start start 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a4068640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f189c1006e0 0x7f189c1969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a4068640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f189c1018d0 0x7f189c196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a4068640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f189c102290 0x7f189c19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a4068640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f189c10c360 con 0x7f189c102290 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a4068640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f189c10c1e0 con 0x7f189c1006e0 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a4068640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f189c10c4e0 con 0x7f189c1018d0 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a2605640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f189c102290 0x7f189c19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a2605640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f189c102290 0x7f189c19df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32904/0 (socket says 172.21.15.77:32904) 2024-03-16T10:18:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a2605640 1 -- 172.21.15.77:0/3381854468 learned_addr learned my addr 172.21.15.77:0/3381854468 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a1e04640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f189c1006e0 0x7f189c1969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a1603640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f189c1018d0 0x7f189c196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a2605640 1 -- 172.21.15.77:0/3381854468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f189c1018d0 msgr2=0x7f189c196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a2605640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f189c1018d0 0x7f189c196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a2605640 1 -- 172.21.15.77:0/3381854468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f189c1006e0 msgr2=0x7f189c1969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a2605640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f189c1006e0 0x7f189c1969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a2605640 1 -- 172.21.15.77:0/3381854468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f189c19e5d0 con 0x7f189c102290 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.008+0000 7f18a2605640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f189c102290 0x7f189c19df60 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f189402c700 tx=0x7f1894032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.009+0000 7f1892ffd640 1 -- 172.21.15.77:0/3381854468 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1894039040 con 0x7f189c102290 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.009+0000 7f18a4068640 1 -- 172.21.15.77:0/3381854468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f189c19e860 con 0x7f189c102290 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.009+0000 7f1892ffd640 1 -- 172.21.15.77:0/3381854468 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1894044070 con 0x7f189c102290 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.009+0000 7f1892ffd640 1 -- 172.21.15.77:0/3381854468 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18940042a0 con 0x7f189c102290 2024-03-16T10:18:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.009+0000 7f18a4068640 1 -- 172.21.15.77:0/3381854468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f189c19eda0 con 0x7f189c102290 2024-03-16T10:18:49.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.011+0000 7f1892ffd640 1 -- 172.21.15.77:0/3381854468 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f1894003970 con 0x7f189c102290 2024-03-16T10:18:49.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.011+0000 7f1892ffd640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f18880a2800 0x7f18880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a1e04640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f18880a2800 0x7f18880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f1892ffd640 1 -- 172.21.15.77:0/3381854468 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f18940f0e50 con 0x7f189c102290 2024-03-16T10:18:49.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a6c4e740 1 Processor -- start 2024-03-16T10:18:49.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a6c4e740 1 -- start start 2024-03-16T10:18:49.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a1e04640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f18880a2800 0x7f18880a4cc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f189c101730 tx=0x7f189800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:49.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a6c4e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd89170 0x561cecd89590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a6c4e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89b60 0x561cecd8e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a6c4e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd8ecd0 0x561cecd91160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a6c4e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ceccee500 con 0x561cecd89b60 2024-03-16T10:18:49.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a6c4e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561cecc96be0 con 0x561cecd89170 2024-03-16T10:18:49.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a6c4e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561ceccea5a0 con 0x561cecd8ecd0 2024-03-16T10:18:49.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a1603640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89b60 0x561cecd8e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a1e04640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd89170 0x561cecd89590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a1603640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89b60 0x561cecd8e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32910/0 (socket says 172.21.15.77:32910) 2024-03-16T10:18:49.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a1603640 1 -- 172.21.15.77:0/2332155957 learned_addr learned my addr 172.21.15.77:0/2332155957 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:49.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.012+0000 7f18a2605640 1 --2- 172.21.15.77:0/2332155957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd8ecd0 0x561cecd91160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.013+0000 7f18a1603640 1 -- 172.21.15.77:0/2332155957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd8ecd0 msgr2=0x561cecd91160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:49.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.013+0000 7f18a1603640 1 --2- 172.21.15.77:0/2332155957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd8ecd0 0x561cecd91160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.013+0000 7f18a1603640 1 -- 172.21.15.77:0/2332155957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd89170 msgr2=0x561cecd89590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:49.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.013+0000 7f18a1603640 1 --2- 172.21.15.77:0/2332155957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd89170 0x561cecd89590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.013+0000 7f18a1603640 1 -- 172.21.15.77:0/2332155957 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ceccaa4b0 con 0x561cecd89b60 2024-03-16T10:18:49.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.013+0000 7f18a1e04640 1 --2- 172.21.15.77:0/2332155957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd89170 0x561cecd89590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:49.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.013+0000 7f18a2605640 1 --2- 172.21.15.77:0/2332155957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd8ecd0 0x561cecd91160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:49.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.013+0000 7f18a1603640 1 --2- 172.21.15.77:0/2332155957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89b60 0x561cecd8e660 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f188c00abe0 tx=0x7f188c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4928a2eef15691f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:49.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.013+0000 7f1887fff640 1 -- 172.21.15.77:0/2332155957 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c01d070 con 0x561cecd89b60 2024-03-16T10:18:49.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.013+0000 7f1887fff640 1 -- 172.21.15.77:0/2332155957 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f188c018b20 con 0x561cecd89b60 2024-03-16T10:18:49.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.013+0000 7f1887fff640 1 -- 172.21.15.77:0/2332155957 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c010b60 con 0x561cecd89b60 2024-03-16T10:18:49.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.014+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2332155957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89b60 msgr2=0x561cecd8e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:49.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.014+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/2332155957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89b60 0x561cecd8e660 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f188c00abe0 tx=0x7f188c00f790 comp rx=0 tx=0).stop 2024-03-16T10:18:49.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.014+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2332155957 shutdown_connections 2024-03-16T10:18:49.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.014+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/2332155957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd8ecd0 0x561cecd91160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.014+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/2332155957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89b60 0x561cecd8e660 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.014+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/2332155957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd89170 0x561cecd89590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.014+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2332155957 >> 172.21.15.77:0/2332155957 conn(0x561cecd846e0 msgr2=0x561cecd84ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:49.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.014+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2332155957 shutdown_connections 2024-03-16T10:18:49.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.014+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2332155957 wait complete. 2024-03-16T10:18:49.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.014+0000 7f18a6c4e740 1 -- start start 2024-03-16T10:18:49.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.014+0000 7f18a6c4e740 1 Processor -- start 2024-03-16T10:18:49.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.014+0000 7f18a6c4e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd95890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a6c4e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd89b60 0x561cecd95dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a6c4e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd8ecd0 0x561cecd9ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a6c4e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561cecd91c80 con 0x561cecd89170 2024-03-16T10:18:49.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a6c4e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561ceccaa160 con 0x561cecd89b60 2024-03-16T10:18:49.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a6c4e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561cecd91e00 con 0x561cecd8ecd0 2024-03-16T10:18:49.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a6c4e740 10 client.0 _ll_register_callbacks cb 0x561cecc8e150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:49.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a1e04640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd95890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a1e04640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd95890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32920/0 (socket says 172.21.15.77:32920) 2024-03-16T10:18:49.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a1e04640 1 -- 172.21.15.77:0/2638441987 learned_addr learned my addr 172.21.15.77:0/2638441987 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:49.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a1603640 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd89b60 0x561cecd95dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a2605640 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd8ecd0 0x561cecd9ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a1e04640 1 -- 172.21.15.77:0/2638441987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd8ecd0 msgr2=0x561cecd9ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a1e04640 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd8ecd0 0x561cecd9ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a1e04640 1 -- 172.21.15.77:0/2638441987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd89b60 msgr2=0x561cecd95dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a1e04640 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd89b60 0x561cecd95dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a1e04640 1 -- 172.21.15.77:0/2638441987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cecacc7d0 con 0x561cecd89170 2024-03-16T10:18:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a1603640 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd89b60 0x561cecd95dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a1e04640 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd95890 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1898011770 tx=0x7f1898004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4649aa8e039345be server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18857fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18857fa640 1 -- 172.21.15.77:0/2638441987 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189801fa00 con 0x561cecd89170 2024-03-16T10:18:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18857fa640 1 -- 172.21.15.77:0/2638441987 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f189801fba0 con 0x561cecd89170 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2638441987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f188c018b20 con 0x561cecd89170 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.015+0000 7f18857fa640 1 -- 172.21.15.77:0/2638441987 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f189801fda0 con 0x561cecd89170 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.016+0000 7f18a6c4e740 10 client.16089 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.016+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2638441987 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561ceca023f0 con 0x561cecd89170 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.016+0000 7f18857fa640 1 -- 172.21.15.77:0/2638441987 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f188c018b20 con 0x561cecd89170 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.016+0000 7f18857fa640 1 -- 172.21.15.77:0/2638441987 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1898031050 con 0x561cecd89170 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.016+0000 7f18a6c4e740 10 client.16089 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.016+0000 7f18a6c4e740 10 client.16089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.016+0000 7f18a6c4e740 20 client.16089 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.016+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cecda1400 0x561cecda38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.016+0000 7f18a6c4e740 4 client.16089 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.016+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2638441987 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ceccfba40 con 0x561cecda1400 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.016+0000 7f18a1603640 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cecda1400 0x561cecda38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18857fa640 10 client.16089 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18a1603640 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cecda1400 0x561cecda38a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7913c7339ea8f579 server_cookie=7a19055f98fd733d in_seq=0 out_seq=0 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18857fa640 1 -- 172.21.15.77:0/2638441987 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561ceccfba40 con 0x561cecda1400 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18857fa640 10 client.16089 handle_command_reply: tid=0 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18a6c4e740 1 client.16089 shutdown 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2638441987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 msgr2=0x561cecd95890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd95890 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1898011770 tx=0x7f1898004730 comp rx=0 tx=0).stop 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2638441987 shutdown_connections 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd8ecd0 0x561cecd9ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd89b60 0x561cecd95dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cecda1400 0x561cecda38a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/2638441987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd95890 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2638441987 >> 172.21.15.77:0/2638441987 conn(0x561cecd846e0 msgr2=0x561cecd83900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.017+0000 7f18857fa640 0 client.16089 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.018+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2638441987 shutdown_connections 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.018+0000 7f18a6c4e740 1 -- 172.21.15.77:0/2638441987 wait complete. 2024-03-16T10:18:49.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.018+0000 7f18a6c4e740 20 client.16089 trim_cache size 0 max 16384 2024-03-16T10:18:49.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a6c4e740 1 Processor -- start 2024-03-16T10:18:49.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a6c4e740 1 -- start start 2024-03-16T10:18:49.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a6c4e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd83070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a6c4e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd89b60 0x561cecd9cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a6c4e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd8ecd0 0x561cecd835b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a6c4e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f189c1a6980 con 0x561cecd89170 2024-03-16T10:18:49.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a6c4e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f189c1a8580 con 0x561cecd8ecd0 2024-03-16T10:18:49.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a6c4e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f189c1a7050 con 0x561cecd89b60 2024-03-16T10:18:49.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a1e04640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd83070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a1e04640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd83070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32936/0 (socket says 172.21.15.77:32936) 2024-03-16T10:18:49.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a1e04640 1 -- 172.21.15.77:0/354571920 learned_addr learned my addr 172.21.15.77:0/354571920 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:49.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a2605640 1 --2- 172.21.15.77:0/354571920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd8ecd0 0x561cecd835b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.035+0000 7f18a1603640 1 --2- 172.21.15.77:0/354571920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd89b60 0x561cecd9cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.036+0000 7f18a1e04640 1 -- 172.21.15.77:0/354571920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd89b60 msgr2=0x561cecd9cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.036+0000 7f18a1e04640 1 --2- 172.21.15.77:0/354571920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd89b60 0x561cecd9cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.036+0000 7f18a1e04640 1 -- 172.21.15.77:0/354571920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd8ecd0 msgr2=0x561cecd835b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.036+0000 7f18a1e04640 1 --2- 172.21.15.77:0/354571920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd8ecd0 0x561cecd835b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.036+0000 7f18a1e04640 1 -- 172.21.15.77:0/354571920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cec9c20f0 con 0x561cecd89170 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.036+0000 7f18a1e04640 1 --2- 172.21.15.77:0/354571920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd83070 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1898011740 tx=0x7f1898011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69d8ebb6997fd35 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.036+0000 7f18877fe640 1 -- 172.21.15.77:0/354571920 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898032980 con 0x561cecd89170 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.036+0000 7f18877fe640 1 -- 172.21.15.77:0/354571920 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1898037070 con 0x561cecd89170 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.036+0000 7f18877fe640 1 -- 172.21.15.77:0/354571920 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18980043a0 con 0x561cecd89170 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 -- 172.21.15.77:0/354571920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 msgr2=0x561cecd83070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/354571920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd83070 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1898011740 tx=0x7f1898011440 comp rx=0 tx=0).stop 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 -- 172.21.15.77:0/354571920 shutdown_connections 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/354571920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd8ecd0 0x561cecd835b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/354571920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd89b60 0x561cecd9cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/354571920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd83070 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 -- 172.21.15.77:0/354571920 >> 172.21.15.77:0/354571920 conn(0x561cecd846e0 msgr2=0x561cecd954a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 -- 172.21.15.77:0/354571920 shutdown_connections 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 -- 172.21.15.77:0/354571920 wait complete. 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 -- start start 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 Processor -- start 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd9c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd89b60 0x561cecd9aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd8ecd0 0x561cecd9b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f189c1a7df0 con 0x561cecd89170 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f189c1a7720 con 0x561cecd8ecd0 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f189c1a4960 con 0x561cecd89b60 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a6c4e740 10 client.0 _ll_register_callbacks cb 0x7f1898010c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:49.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.037+0000 7f18a1e04640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd9c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a1e04640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd9c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32944/0 (socket says 172.21.15.77:32944) 2024-03-16T10:18:49.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a1e04640 1 -- 172.21.15.77:0/685647754 learned_addr learned my addr 172.21.15.77:0/685647754 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:49.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a1603640 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd89b60 0x561cecd9aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a2605640 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd8ecd0 0x561cecd9b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a1e04640 1 -- 172.21.15.77:0/685647754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd89b60 msgr2=0x561cecd9aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:49.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a1e04640 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd89b60 0x561cecd9aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a1e04640 1 -- 172.21.15.77:0/685647754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd8ecd0 msgr2=0x561cecd9b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:49.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a1e04640 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd8ecd0 0x561cecd9b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a1e04640 1 -- 172.21.15.77:0/685647754 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ceccb59d0 con 0x561cecd89170 2024-03-16T10:18:49.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a1e04640 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd9c920 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1898032710 tx=0x7f18980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c91fd753ee679913 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:49.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18857fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:49.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18857fa640 1 -- 172.21.15.77:0/685647754 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898032b00 con 0x561cecd89170 2024-03-16T10:18:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18857fa640 1 -- 172.21.15.77:0/685647754 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f1898032ca0 con 0x561cecd89170 2024-03-16T10:18:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a6c4e740 1 -- 172.21.15.77:0/685647754 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1898037070 con 0x561cecd89170 2024-03-16T10:18:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18857fa640 1 -- 172.21.15.77:0/685647754 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1898004490 con 0x561cecd89170 2024-03-16T10:18:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a6c4e740 10 client.16095 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18a6c4e740 1 -- 172.21.15.77:0/685647754 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561cece082d0 con 0x561cecd89170 2024-03-16T10:18:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.038+0000 7f18857fa640 1 -- 172.21.15.77:0/685647754 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1898037070 con 0x561cecd89170 2024-03-16T10:18:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18857fa640 1 -- 172.21.15.77:0/685647754 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1898031040 con 0x561cecd89170 2024-03-16T10:18:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18a6c4e740 10 client.16095 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18a6c4e740 10 client.16095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18a6c4e740 20 client.16095 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cece08950 0x561cece0ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:49.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18a6c4e740 4 client.16095 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18a6c4e740 1 -- 172.21.15.77:0/685647754 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561ceccf5d90 con 0x561cece08950 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18a1603640 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cece08950 0x561cece0ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18a1603640 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cece08950 0x561cece0ad70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af3bb1acb32f624c server_cookie=747f372bc6033676 in_seq=0 out_seq=0 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18857fa640 10 client.16095 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18857fa640 1 -- 172.21.15.77:0/685647754 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x561ceccf5d90 con 0x561cece08950 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18857fa640 10 client.16095 handle_command_reply: tid=0 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18a6c4e740 1 client.16095 shutdown 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18a6c4e740 1 -- 172.21.15.77:0/685647754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 msgr2=0x561cecd9c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.039+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd9c920 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1898032710 tx=0x7f18980057f0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a6c4e740 1 -- 172.21.15.77:0/685647754 shutdown_connections 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cecd8ecd0 0x561cecd9b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cecd89b60 0x561cecd9aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cece08950 0x561cece0ad70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a6c4e740 1 --2- 172.21.15.77:0/685647754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cecd89170 0x561cecd9c920 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a6c4e740 1 -- 172.21.15.77:0/685647754 >> 172.21.15.77:0/685647754 conn(0x561cecd846e0 msgr2=0x561cecd87940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:49.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18857fa640 0 client.16095 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:49.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a6c4e740 1 -- 172.21.15.77:0/685647754 shutdown_connections 2024-03-16T10:18:49.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a6c4e740 1 -- 172.21.15.77:0/685647754 wait complete. 2024-03-16T10:18:49.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a6c4e740 20 client.16095 trim_cache size 0 max 16384 2024-03-16T10:18:49.048 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:18:49.048 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-16T10:18:49.048 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:18:49.048 INFO:teuthology.orchestra.run.smithi077.stdout: "bf9f313c-ca13-4923-a775-539cd1f60df2": { 2024-03-16T10:18:49.048 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:18:49.048 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "bf9f313c-ca13-4923-a775-539cd1f60df2", 2024-03-16T10:18:49.049 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:18:49.049 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:18:49.049 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:18:49.049 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:18:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a4068640 1 -- 172.21.15.77:0/3381854468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f18880a2800 msgr2=0x7f18880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a4068640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f18880a2800 0x7f18880a4cc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f189c101730 tx=0x7f189800b390 comp rx=0 tx=0).stop 2024-03-16T10:18:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a4068640 1 -- 172.21.15.77:0/3381854468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f189c102290 msgr2=0x7f189c19df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a4068640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f189c102290 0x7f189c19df60 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f189402c700 tx=0x7f1894032f20 comp rx=0 tx=0).stop 2024-03-16T10:18:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a4068640 1 -- 172.21.15.77:0/3381854468 shutdown_connections 2024-03-16T10:18:49.050 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a4068640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f189c102290 0x7f189c19df60 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a4068640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f18880a2800 0x7f18880a4cc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a4068640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f189c1018d0 0x7f189c196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a4068640 1 --2- 172.21.15.77:0/3381854468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f189c1006e0 0x7f189c1969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a4068640 1 -- 172.21.15.77:0/3381854468 >> 172.21.15.77:0/3381854468 conn(0x7f189c0fbc70 msgr2=0x7f189c104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a4068640 1 -- 172.21.15.77:0/3381854468 shutdown_connections 2024-03-16T10:18:49.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:18:49.040+0000 7f18a4068640 1 -- 172.21.15.77:0/3381854468 wait complete. 2024-03-16T10:18:49.055 INFO:tasks.cephfs.filesystem:scrub status for tag:bf9f313c-ca13-4923-a775-539cd1f60df2 - {'path': '/', 'tag': 'bf9f313c-ca13-4923-a775-539cd1f60df2', 'options': 'recursive,force'} 2024-03-16T10:18:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:49 smithi192 ceph-mon[36354]: pgmap v402: 97 pgs: 97 active+clean; 4.9 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 457 KiB/s wr, 66 op/s 2024-03-16T10:18:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:49 smithi192 ceph-mon[36354]: scrub queued for path: / 2024-03-16T10:18:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:49 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:49 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:18:50.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:49 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:18:50.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:49 smithi077 ceph-mon[32400]: pgmap v402: 97 pgs: 97 active+clean; 4.9 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 457 KiB/s wr, 66 op/s 2024-03-16T10:18:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:49 smithi077 ceph-mon[32400]: scrub queued for path: / 2024-03-16T10:18:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:49 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:49 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:18:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:49 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:18:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:49 smithi193 ceph-mon[36311]: pgmap v402: 97 pgs: 97 active+clean; 4.9 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 457 KiB/s wr, 66 op/s 2024-03-16T10:18:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:49 smithi193 ceph-mon[36311]: scrub queued for path: / 2024-03-16T10:18:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:49 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:49 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:18:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:49 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:18:51.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:50 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:51.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:50 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:51.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:50 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:52.147 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:51 smithi077 ceph-mon[32400]: pgmap v403: 97 pgs: 97 active+clean; 4.9 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 379 KiB/s wr, 65 op/s 2024-03-16T10:18:52.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:51 smithi192 ceph-mon[36354]: pgmap v403: 97 pgs: 97 active+clean; 4.9 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 379 KiB/s wr, 65 op/s 2024-03-16T10:18:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:51 smithi193 ceph-mon[36311]: pgmap v403: 97 pgs: 97 active+clean; 4.9 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 379 KiB/s wr, 65 op/s 2024-03-16T10:18:53.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:53 smithi077 ceph-mon[32400]: pgmap v404: 97 pgs: 97 active+clean; 11 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 943 KiB/s wr, 75 op/s 2024-03-16T10:18:53.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:53 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:53 smithi192 ceph-mon[36354]: pgmap v404: 97 pgs: 97 active+clean; 11 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 943 KiB/s wr, 75 op/s 2024-03-16T10:18:54.170 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:53 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:53 smithi193 ceph-mon[36311]: pgmap v404: 97 pgs: 97 active+clean; 11 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 943 KiB/s wr, 75 op/s 2024-03-16T10:18:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:53 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:18:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.538+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2452255490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f981013c0 msgr2=0x7f9f98108c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.538+0000 7f9f9f84b640 1 --2- 172.21.15.77:0/2452255490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f981013c0 0x7f9f98108c60 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f9f90009ca0 tx=0x7f9f9002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.538+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2452255490 shutdown_connections 2024-03-16T10:18:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.538+0000 7f9f9f84b640 1 --2- 172.21.15.77:0/2452255490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f981013c0 0x7f9f98108c60 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.538+0000 7f9f9f84b640 1 --2- 172.21.15.77:0/2452255490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f98100a00 0x7f9f98100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.538+0000 7f9f9f84b640 1 --2- 172.21.15.77:0/2452255490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f98101cd0 0x7f9f981020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.538+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2452255490 >> 172.21.15.77:0/2452255490 conn(0x7f9f980fb410 msgr2=0x7f9f980fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.538+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2452255490 shutdown_connections 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.538+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2452255490 wait complete. 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9f84b640 1 Processor -- start 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9f84b640 1 -- start start 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9f84b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f98100a00 0x7f9f98195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9f84b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f981013c0 0x7f9f98195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9f84b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f98101cd0 0x7f9f9819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9f84b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9810b2e0 con 0x7f9f98101cd0 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9f84b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9810b160 con 0x7f9f981013c0 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9f84b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9810b460 con 0x7f9f98100a00 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9dde8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f98101cd0 0x7f9f9819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9dde8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f98101cd0 0x7f9f9819cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55020/0 (socket says 172.21.15.77:55020) 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9cde6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f981013c0 0x7f9f98195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9dde8640 1 -- 172.21.15.77:0/2095269209 learned_addr learned my addr 172.21.15.77:0/2095269209 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f98100a00 0x7f9f98195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9dde8640 1 -- 172.21.15.77:0/2095269209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f98100a00 msgr2=0x7f9f98195810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9dde8640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f98100a00 0x7f9f98195810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9dde8640 1 -- 172.21.15.77:0/2095269209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f981013c0 msgr2=0x7f9f98195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9dde8640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f981013c0 0x7f9f98195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9dde8640 1 -- 172.21.15.77:0/2095269209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f9819d430 con 0x7f9f98101cd0 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9cde6640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f981013c0 0x7f9f98195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.539+0000 7f9f9dde8640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f98101cd0 0x7f9f9819cdc0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9f90037040 tx=0x7f9f900346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.540+0000 7f9f8a7fc640 1 -- 172.21.15.77:0/2095269209 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90041020 con 0x7f9f98101cd0 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.540+0000 7f9f8a7fc640 1 -- 172.21.15.77:0/2095269209 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f90032890 con 0x7f9f98101cd0 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.540+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2095269209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f9819d6c0 con 0x7f9f98101cd0 2024-03-16T10:18:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.540+0000 7f9f8a7fc640 1 -- 172.21.15.77:0/2095269209 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90039040 con 0x7f9f98101cd0 2024-03-16T10:18:55.545 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.540+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2095269209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f9819dc00 con 0x7f9f98101cd0 2024-03-16T10:18:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.542+0000 7f9f8a7fc640 1 -- 172.21.15.77:0/2095269209 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f9f90044020 con 0x7f9f98101cd0 2024-03-16T10:18:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.542+0000 7f9fa243b740 1 Processor -- start 2024-03-16T10:18:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9fa243b740 1 -- start start 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9fa243b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806daf340 0x559806daf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9fa243b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 0x559806db4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9fa243b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 0x559806db7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9fa243b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559806d14a40 con 0x559806dafd30 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9fa243b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559806cbd920 con 0x559806db4ea0 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9fa243b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559806d10ae0 con 0x559806daf340 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f8a7fc640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9f840a2920 0x7f9f840a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f9cde6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806daf340 0x559806daf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f9d5e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 0x559806db4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f9dde8640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9f840a2920 0x7f9f840a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f9cde6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806daf340 0x559806daf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:36132/0 (socket says 172.21.15.77:36132) 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f9cde6640 1 -- 172.21.15.77:0/1817715835 learned_addr learned my addr 172.21.15.77:0/1817715835 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f9cde6640 1 --2- 172.21.15.77:0/1817715835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 0x559806db7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f8a7fc640 1 -- 172.21.15.77:0/2095269209 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f9f900f0770 con 0x7f9f98101cd0 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f9d5e7640 1 -- 172.21.15.77:0/1817715835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806daf340 msgr2=0x559806daf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/1817715835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806daf340 0x559806daf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f9d5e7640 1 -- 172.21.15.77:0/1817715835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 msgr2=0x559806db7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/1817715835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 0x559806db7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f9d5e7640 1 -- 172.21.15.77:0/1817715835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559806cd5460 con 0x559806dafd30 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.543+0000 7f9f9cde6640 1 --2- 172.21.15.77:0/1817715835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806daf340 0x559806daf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.544+0000 7f9f9dde8640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9f840a2920 0x7f9f840a4de0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9f900b9c00 tx=0x7f9f900914e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.544+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/1817715835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 0x559806db4830 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c010b90 tx=0x7f9f8c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a999ca2eb8f5f1d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.544+0000 7f9f837fe640 1 -- 172.21.15.77:0/1817715835 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c004230 con 0x559806dafd30 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.544+0000 7f9f837fe640 1 -- 172.21.15.77:0/1817715835 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f8c0043d0 con 0x559806dafd30 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.544+0000 7f9f837fe640 1 -- 172.21.15.77:0/1817715835 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c01b7b0 con 0x559806dafd30 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.544+0000 7f9fa243b740 1 -- 172.21.15.77:0/1817715835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 msgr2=0x559806db4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.544+0000 7f9fa243b740 1 --2- 172.21.15.77:0/1817715835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 0x559806db4830 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c010b90 tx=0x7f9f8c005f90 comp rx=0 tx=0).stop 2024-03-16T10:18:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.544+0000 7f9fa243b740 1 -- 172.21.15.77:0/1817715835 shutdown_connections 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.544+0000 7f9fa243b740 1 --2- 172.21.15.77:0/1817715835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 0x559806db7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.544+0000 7f9fa243b740 1 --2- 172.21.15.77:0/1817715835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 0x559806db4830 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.544+0000 7f9fa243b740 1 --2- 172.21.15.77:0/1817715835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806daf340 0x559806daf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.544+0000 7f9fa243b740 1 -- 172.21.15.77:0/1817715835 >> 172.21.15.77:0/1817715835 conn(0x559806daa8b0 msgr2=0x559806daacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9fa243b740 1 -- 172.21.15.77:0/1817715835 shutdown_connections 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9fa243b740 1 -- 172.21.15.77:0/1817715835 wait complete. 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9fa243b740 1 -- start start 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9fa243b740 1 Processor -- start 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9fa243b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 0x559806db9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9fa243b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806dafd30 0x559806db9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9fa243b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806db4ea0 0x559806dc0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9fa243b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559806db7e50 con 0x559806daf340 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9fa243b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559806cd0d70 con 0x559806dafd30 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9fa243b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559806db7fd0 con 0x559806db4ea0 2024-03-16T10:18:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9fa243b740 10 client.0 _ll_register_callbacks cb 0x559806a28a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9f9cde6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 0x559806db9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9f9cde6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 0x559806db9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55026/0 (socket says 172.21.15.77:55026) 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9f9cde6640 1 -- 172.21.15.77:0/3501594333 learned_addr learned my addr 172.21.15.77:0/3501594333 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.545+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806dafd30 0x559806db9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9f9cde6640 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806db4ea0 0x559806dc0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9f9cde6640 1 -- 172.21.15.77:0/3501594333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806db4ea0 msgr2=0x559806dc0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9f9cde6640 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806db4ea0 0x559806dc0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9f9cde6640 1 -- 172.21.15.77:0/3501594333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806dafd30 msgr2=0x559806db9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9f9cde6640 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806dafd30 0x559806db9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9f9cde6640 1 -- 172.21.15.77:0/3501594333 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559806a29210 con 0x559806daf340 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9f9cde6640 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 0x559806db9640 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9f94011410 tx=0x7f9f94005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a68da9bca2e686e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9f80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9f80ff9640 1 -- 172.21.15.77:0/3501594333 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94004460 con 0x559806daf340 2024-03-16T10:18:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9f80ff9640 1 -- 172.21.15.77:0/3501594333 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f9f94010930 con 0x559806daf340 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9fa243b740 1 -- 172.21.15.77:0/3501594333 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f8c0043d0 con 0x559806daf340 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9f80ff9640 1 -- 172.21.15.77:0/3501594333 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f94010b30 con 0x559806daf340 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9fa243b740 10 client.16107 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9fa243b740 1 -- 172.21.15.77:0/3501594333 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559806af5620 con 0x559806daf340 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.546+0000 7f9f80ff9640 1 -- 172.21.15.77:0/3501594333 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f8c0043d0 con 0x559806daf340 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.547+0000 7f9f80ff9640 1 -- 172.21.15.77:0/3501594333 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9f940203f0 con 0x559806daf340 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.547+0000 7f9fa243b740 10 client.16107 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.547+0000 7f9fa243b740 10 client.16107 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.547+0000 7f9fa243b740 20 client.16107 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.547+0000 7f9fa243b740 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559806dc5510 0x559806dc7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.547+0000 7f9fa243b740 4 client.16107 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.547+0000 7f9fa243b740 1 -- 172.21.15.77:0/3501594333 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559806d21f80 con 0x559806dc5510 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.547+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559806dc5510 0x559806dc7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.547+0000 7f9f80ff9640 10 client.16107 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:55.560 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.547+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559806dc5510 0x559806dc7970 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c91ea63ad20b4ee server_cookie=732d18fbded3eb3b in_seq=0 out_seq=0 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.548+0000 7f9f80ff9640 1 -- 172.21.15.77:0/3501594333 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559806d21f80 con 0x559806dc5510 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.548+0000 7f9f80ff9640 10 client.16107 handle_command_reply: tid=0 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.548+0000 7f9fa243b740 1 client.16107 shutdown 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.548+0000 7f9fa243b740 1 -- 172.21.15.77:0/3501594333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 msgr2=0x559806db9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.548+0000 7f9fa243b740 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 0x559806db9640 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9f94011410 tx=0x7f9f94005f90 comp rx=0 tx=0).stop 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.548+0000 7f9fa243b740 1 -- 172.21.15.77:0/3501594333 shutdown_connections 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.548+0000 7f9fa243b740 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806db4ea0 0x559806dc0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.548+0000 7f9fa243b740 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806dafd30 0x559806db9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.548+0000 7f9fa243b740 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559806dc5510 0x559806dc7970 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.548+0000 7f9fa243b740 1 --2- 172.21.15.77:0/3501594333 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 0x559806db9640 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.548+0000 7f9fa243b740 1 -- 172.21.15.77:0/3501594333 >> 172.21.15.77:0/3501594333 conn(0x559806daa8b0 msgr2=0x559806da9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.548+0000 7f9f80ff9640 0 client.16107 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.549+0000 7f9fa243b740 1 -- 172.21.15.77:0/3501594333 shutdown_connections 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.549+0000 7f9fa243b740 1 -- 172.21.15.77:0/3501594333 wait complete. 2024-03-16T10:18:55.561 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.549+0000 7f9fa243b740 20 client.16107 trim_cache size 0 max 16384 2024-03-16T10:18:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9fa243b740 1 Processor -- start 2024-03-16T10:18:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9fa243b740 1 -- start start 2024-03-16T10:18:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9fa243b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806daf340 0x559806db8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9fa243b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 0x559806dc0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9fa243b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 0x559806da9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9fa243b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9f981a6870 con 0x559806dafd30 2024-03-16T10:18:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9fa243b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f9f981a8470 con 0x559806db4ea0 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9fa243b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9f981a6f40 con 0x559806daf340 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f9d5e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 0x559806dc0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f9cde6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806daf340 0x559806db8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f9d5e7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 0x559806dc0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55040/0 (socket says 172.21.15.77:55040) 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f9d5e7640 1 -- 172.21.15.77:0/4066325176 learned_addr learned my addr 172.21.15.77:0/4066325176 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f9cde6640 1 --2- 172.21.15.77:0/4066325176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 0x559806da9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f9d5e7640 1 -- 172.21.15.77:0/4066325176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806daf340 msgr2=0x559806db8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/4066325176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806daf340 0x559806db8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f9d5e7640 1 -- 172.21.15.77:0/4066325176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 msgr2=0x559806da9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/4066325176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 0x559806da9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f9d5e7640 1 -- 172.21.15.77:0/4066325176 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559806da6310 con 0x559806dafd30 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/4066325176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 0x559806dc0b80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c010b90 tx=0x7f9f8c010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32661de0d303cd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f82ffd640 1 -- 172.21.15.77:0/4066325176 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c030b20 con 0x559806dafd30 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f82ffd640 1 -- 172.21.15.77:0/4066325176 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f8c030cc0 con 0x559806dafd30 2024-03-16T10:18:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.564+0000 7f9f82ffd640 1 -- 172.21.15.77:0/4066325176 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c0052a0 con 0x559806dafd30 2024-03-16T10:18:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.565+0000 7f9fa243b740 1 -- 172.21.15.77:0/4066325176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 msgr2=0x559806dc0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.565+0000 7f9fa243b740 1 --2- 172.21.15.77:0/4066325176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 0x559806dc0b80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c010b90 tx=0x7f9f8c010c80 comp rx=0 tx=0).stop 2024-03-16T10:18:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.565+0000 7f9fa243b740 1 -- 172.21.15.77:0/4066325176 shutdown_connections 2024-03-16T10:18:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.565+0000 7f9fa243b740 1 --2- 172.21.15.77:0/4066325176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 0x559806da9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.565+0000 7f9fa243b740 1 --2- 172.21.15.77:0/4066325176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806dafd30 0x559806dc0b80 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.565+0000 7f9fa243b740 1 --2- 172.21.15.77:0/4066325176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806daf340 0x559806db8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.565+0000 7f9fa243b740 1 -- 172.21.15.77:0/4066325176 >> 172.21.15.77:0/4066325176 conn(0x559806daa8b0 msgr2=0x559806db98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.565+0000 7f9fa243b740 1 -- 172.21.15.77:0/4066325176 shutdown_connections 2024-03-16T10:18:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.565+0000 7f9fa243b740 1 -- 172.21.15.77:0/4066325176 wait complete. 2024-03-16T10:18:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.565+0000 7f9fa243b740 1 -- start start 2024-03-16T10:18:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.565+0000 7f9fa243b740 1 Processor -- start 2024-03-16T10:18:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9fa243b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 0x559806dbe820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9fa243b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806dafd30 0x559806dbed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9fa243b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 0x559806dbfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9fa243b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9f981a7ce0 con 0x559806daf340 2024-03-16T10:18:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9fa243b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f9f981a7610 con 0x559806db4ea0 2024-03-16T10:18:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9fa243b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9f981a4850 con 0x559806dafd30 2024-03-16T10:18:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9fa243b740 10 client.0 _ll_register_callbacks cb 0x7f9f94011230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:18:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f9cde6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 0x559806dbe820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f9cde6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 0x559806dbe820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55052/0 (socket says 172.21.15.77:55052) 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f9cde6640 1 -- 172.21.15.77:0/1013650814 learned_addr learned my addr 172.21.15.77:0/1013650814 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806dafd30 0x559806dbed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f9cde6640 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 0x559806dbfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f9cde6640 1 -- 172.21.15.77:0/1013650814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806dafd30 msgr2=0x559806dbed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f9cde6640 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806dafd30 0x559806dbed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f9cde6640 1 -- 172.21.15.77:0/1013650814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 msgr2=0x559806dbfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f9cde6640 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 0x559806dbfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f9cde6640 1 -- 172.21.15.77:0/1013650814 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559806dc0360 con 0x559806daf340 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f9cde6640 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 0x559806dbe820 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9f94020a00 tx=0x7f9f9402d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78b6f5bc84cbbff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f80ff9640 1 -- 172.21.15.77:0/1013650814 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94020620 con 0x559806daf340 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9fa243b740 1 -- 172.21.15.77:0/1013650814 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f8c030cc0 con 0x559806daf340 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f80ff9640 1 -- 172.21.15.77:0/1013650814 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f9f94020a80 con 0x559806daf340 2024-03-16T10:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.566+0000 7f9f80ff9640 1 -- 172.21.15.77:0/1013650814 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f9403f050 con 0x559806daf340 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9fa243b740 10 client.16113 fetch_fsmap learned FSMap version 29 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9fa243b740 1 -- 172.21.15.77:0/1013650814 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559806e2be60 con 0x559806daf340 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9f80ff9640 1 -- 172.21.15.77:0/1013650814 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f8c030cc0 con 0x559806daf340 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9f80ff9640 1 -- 172.21.15.77:0/1013650814 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9f9402b470 con 0x559806daf340 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9fa243b740 10 client.16113 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9fa243b740 10 client.16113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9fa243b740 20 client.16113 populate_metadata read hostname 'smithi077' 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9fa243b740 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559806e2c950 0x559806e2ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9fa243b740 4 client.16113 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9fa243b740 1 -- 172.21.15.77:0/1013650814 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559806d1c2d0 con 0x559806e2c950 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559806e2c950 0x559806e2ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9f80ff9640 10 client.16113 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.567+0000 7f9f9d5e7640 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559806e2c950 0x559806e2ed70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ce6bc01ace926f1 server_cookie=12f9c67accc4a5ac in_seq=0 out_seq=0 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.575+0000 7f9f80ff9640 1 -- 172.21.15.77:0/1013650814 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559806d1c2d0 con 0x559806e2c950 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.575+0000 7f9f80ff9640 10 client.16113 handle_command_reply: tid=0 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.575+0000 7f9fa243b740 1 client.16113 shutdown 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.575+0000 7f9fa243b740 1 -- 172.21.15.77:0/1013650814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 msgr2=0x559806dbe820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.575+0000 7f9fa243b740 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 0x559806dbe820 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9f94020a00 tx=0x7f9f9402d040 comp rx=0 tx=0).stop 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.575+0000 7f9fa243b740 1 -- 172.21.15.77:0/1013650814 shutdown_connections 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.575+0000 7f9fa243b740 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559806db4ea0 0x559806dbfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.575+0000 7f9fa243b740 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559806dafd30 0x559806dbed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.575+0000 7f9fa243b740 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559806e2c950 0x559806e2ed70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.575+0000 7f9fa243b740 1 --2- 172.21.15.77:0/1013650814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806daf340 0x559806dbe820 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.575+0000 7f9f80ff9640 0 client.16113 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.575+0000 7f9fa243b740 1 -- 172.21.15.77:0/1013650814 >> 172.21.15.77:0/1013650814 conn(0x559806daa8b0 msgr2=0x559806dadb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9fa243b740 1 -- 172.21.15.77:0/1013650814 shutdown_connections 2024-03-16T10:18:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9fa243b740 1 -- 172.21.15.77:0/1013650814 wait complete. 2024-03-16T10:18:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9fa243b740 20 client.16113 trim_cache size 0 max 16384 2024-03-16T10:18:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:18:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:18:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:18:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:18:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2095269209 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9f840a2920 msgr2=0x7f9f840a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9f9f84b640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9f840a2920 0x7f9f840a4de0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9f900b9c00 tx=0x7f9f900914e0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2095269209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f98101cd0 msgr2=0x7f9f9819cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:18:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9f9f84b640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f98101cd0 0x7f9f9819cdc0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9f90037040 tx=0x7f9f900346f0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2095269209 shutdown_connections 2024-03-16T10:18:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9f9f84b640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f98101cd0 0x7f9f9819cdc0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9f9f84b640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f981013c0 0x7f9f98195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9f9f84b640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f9f840a2920 0x7f9f840a4de0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9f9f84b640 1 --2- 172.21.15.77:0/2095269209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f98100a00 0x7f9f98195810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:18:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2095269209 >> 172.21.15.77:0/2095269209 conn(0x7f9f980fb410 msgr2=0x7f9f98104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:18:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2095269209 shutdown_connections 2024-03-16T10:18:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:18:55.576+0000 7f9f9f84b640 1 -- 172.21.15.77:0/2095269209 wait complete. 2024-03-16T10:18:56.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:55 smithi192 ceph-mon[36354]: pgmap v405: 97 pgs: 97 active+clean; 18 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 1.4 MiB/s wr, 95 op/s 2024-03-16T10:18:56.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:55 smithi077 ceph-mon[32400]: pgmap v405: 97 pgs: 97 active+clean; 18 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 1.4 MiB/s wr, 95 op/s 2024-03-16T10:18:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:55 smithi193 ceph-mon[36311]: pgmap v405: 97 pgs: 97 active+clean; 18 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 1.4 MiB/s wr, 95 op/s 2024-03-16T10:18:57.102 INFO:tasks.workunit.client.0.smithi077.stdout: 12 174709 726 123106 798 27183 732 2024-03-16T10:18:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:57 smithi192 ceph-mon[36354]: pgmap v406: 97 pgs: 97 active+clean; 26 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 2.1 MiB/s wr, 124 op/s 2024-03-16T10:18:58.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:57 smithi077 ceph-mon[32400]: pgmap v406: 97 pgs: 97 active+clean; 26 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 2.1 MiB/s wr, 124 op/s 2024-03-16T10:18:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:57 smithi193 ceph-mon[36311]: pgmap v406: 97 pgs: 97 active+clean; 26 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 2.1 MiB/s wr, 124 op/s 2024-03-16T10:18:59.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:18:59 smithi192 ceph-mon[36354]: pgmap v407: 97 pgs: 97 active+clean; 46 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 191 op/s 2024-03-16T10:18:59.744 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:18:59 smithi077 ceph-mon[32400]: pgmap v407: 97 pgs: 97 active+clean; 46 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 191 op/s 2024-03-16T10:18:59.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:18:59 smithi193 ceph-mon[36311]: pgmap v407: 97 pgs: 97 active+clean; 46 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 191 op/s 2024-03-16T10:19:02.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:01 smithi192 ceph-mon[36354]: pgmap v408: 97 pgs: 97 active+clean; 46 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 167 op/s 2024-03-16T10:19:02.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:01 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:19:02.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:01 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:19:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:01 smithi077 ceph-mon[32400]: pgmap v408: 97 pgs: 97 active+clean; 46 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 167 op/s 2024-03-16T10:19:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:01 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:19:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:01 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:19:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:01 smithi193 ceph-mon[36311]: pgmap v408: 97 pgs: 97 active+clean; 46 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 167 op/s 2024-03-16T10:19:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:01 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:19:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:01 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:19:04.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:03 smithi192 ceph-mon[36354]: pgmap v409: 97 pgs: 97 active+clean; 48 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 174 op/s 2024-03-16T10:19:04.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:03 smithi077 ceph-mon[32400]: pgmap v409: 97 pgs: 97 active+clean; 48 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 174 op/s 2024-03-16T10:19:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:03 smithi193 ceph-mon[36311]: pgmap v409: 97 pgs: 97 active+clean; 48 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 174 op/s 2024-03-16T10:19:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:05 smithi192 ceph-mon[36354]: pgmap v410: 97 pgs: 97 active+clean; 51 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 170 op/s 2024-03-16T10:19:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:05 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:19:06.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:05 smithi077 ceph-mon[32400]: pgmap v410: 97 pgs: 97 active+clean; 51 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 170 op/s 2024-03-16T10:19:06.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:05 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:19:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:05 smithi193 ceph-mon[36311]: pgmap v410: 97 pgs: 97 active+clean; 51 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 170 op/s 2024-03-16T10:19:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:05 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:19:07.102 INFO:tasks.workunit.client.0.smithi077.stdout: 22 561958 621 365163 422 191561 1194 2024-03-16T10:19:08.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:07 smithi192 ceph-mon[36354]: pgmap v411: 97 pgs: 97 active+clean; 51 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 137 op/s 2024-03-16T10:19:08.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:07 smithi077 ceph-mon[32400]: pgmap v411: 97 pgs: 97 active+clean; 51 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 137 op/s 2024-03-16T10:19:08.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:07 smithi193 ceph-mon[36311]: pgmap v411: 97 pgs: 97 active+clean; 51 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 137 op/s 2024-03-16T10:19:08.672 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.671+0000 7fba896dc640 1 -- 172.21.15.77:0/3499652187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba841033e0 msgr2=0x7fba84109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:08.672 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.671+0000 7fba896dc640 1 --2- 172.21.15.77:0/3499652187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba841033e0 0x7fba84109c70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009be0 tx=0x7fba7c02bef0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.672 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.671+0000 7fba896dc640 1 -- 172.21.15.77:0/3499652187 shutdown_connections 2024-03-16T10:19:08.672 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.671+0000 7fba896dc640 1 --2- 172.21.15.77:0/3499652187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba841033e0 0x7fba84109c70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.672 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.671+0000 7fba896dc640 1 --2- 172.21.15.77:0/3499652187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba84102a20 0x7fba84102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.672 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.671+0000 7fba896dc640 1 --2- 172.21.15.77:0/3499652187 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba84101830 0x7fba84101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.672 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.671+0000 7fba896dc640 1 -- 172.21.15.77:0/3499652187 >> 172.21.15.77:0/3499652187 conn(0x7fba840fcda0 msgr2=0x7fba840ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:08.672 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.671+0000 7fba896dc640 1 -- 172.21.15.77:0/3499652187 shutdown_connections 2024-03-16T10:19:08.672 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.671+0000 7fba896dc640 1 -- 172.21.15.77:0/3499652187 wait complete. 2024-03-16T10:19:08.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba896dc640 1 Processor -- start 2024-03-16T10:19:08.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba896dc640 1 -- start start 2024-03-16T10:19:08.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba896dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba84101830 0x7fba84075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba896dc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba84102a20 0x7fba84073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba896dc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba841033e0 0x7fba84074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba896dc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fba8410d4b0 con 0x7fba84101830 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba896dc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fba8410d330 con 0x7fba841033e0 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba896dc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba8410d630 con 0x7fba84102a20 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba84101830 0x7fba84075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba84102a20 0x7fba84073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba84102a20 0x7fba84073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:38816/0 (socket says 172.21.15.77:38816) 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba7bfff640 1 -- 172.21.15.77:0/2280352130 learned_addr learned my addr 172.21.15.77:0/2280352130 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba837fe640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba841033e0 0x7fba84074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba7bfff640 1 -- 172.21.15.77:0/2280352130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba841033e0 msgr2=0x7fba84074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba7bfff640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba841033e0 0x7fba84074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba7bfff640 1 -- 172.21.15.77:0/2280352130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba84101830 msgr2=0x7fba84075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:08.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba7bfff640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba84101830 0x7fba84075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.672+0000 7fba7bfff640 1 -- 172.21.15.77:0/2280352130 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba840749a0 con 0x7fba84102a20 2024-03-16T10:19:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.673+0000 7fba7bfff640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba84102a20 0x7fba84073bd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fba6c00ea60 tx=0x7fba6c00ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.673+0000 7fba80ff9640 1 -- 172.21.15.77:0/2280352130 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c00cd20 con 0x7fba84102a20 2024-03-16T10:19:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.673+0000 7fba80ff9640 1 -- 172.21.15.77:0/2280352130 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba6c0035f0 con 0x7fba84102a20 2024-03-16T10:19:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.673+0000 7fba80ff9640 1 -- 172.21.15.77:0/2280352130 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c010690 con 0x7fba84102a20 2024-03-16T10:19:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.673+0000 7fba896dc640 1 -- 172.21.15.77:0/2280352130 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba841a1b50 con 0x7fba84102a20 2024-03-16T10:19:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.673+0000 7fba896dc640 1 -- 172.21.15.77:0/2280352130 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba841a20f0 con 0x7fba84102a20 2024-03-16T10:19:08.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.675+0000 7fba80ff9640 1 -- 172.21.15.77:0/2280352130 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fba6c0037a0 con 0x7fba84102a20 2024-03-16T10:19:08.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.676+0000 7fba80ff9640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fba700a2920 0x7fba700a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.676+0000 7fba82ffd640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fba700a2920 0x7fba700a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.676+0000 7fba80ff9640 1 -- 172.21.15.77:0/2280352130 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fba6c0cecd0 con 0x7fba84102a20 2024-03-16T10:19:08.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba82ffd640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fba700a2920 0x7fba700a4de0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fba84102880 tx=0x7fba74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:08.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba8c3a2740 1 Processor -- start 2024-03-16T10:19:08.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba8c3a2740 1 -- start start 2024-03-16T10:19:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba8c3a2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8ef760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba8c3a2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8efd30 0x55a80b8f4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba8c3a2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8f4ea0 0x55a80b8f7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba8c3a2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a80b854a40 con 0x55a80b8f4ea0 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba8c3a2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a80b7fd920 con 0x55a80b8efd30 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba8c3a2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a80b850ae0 con 0x55a80b8ef340 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8ef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba837fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8f4ea0 0x55a80b8f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8ef760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:38818/0 (socket says 172.21.15.77:38818) 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba82ffd640 1 -- 172.21.15.77:0/232786881 learned_addr learned my addr 172.21.15.77:0/232786881 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.677+0000 7fba7bfff640 1 --2- 172.21.15.77:0/232786881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8efd30 0x55a80b8f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.678+0000 7fba82ffd640 1 -- 172.21.15.77:0/232786881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8efd30 msgr2=0x55a80b8f4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.678+0000 7fba82ffd640 1 --2- 172.21.15.77:0/232786881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8efd30 0x55a80b8f4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.678+0000 7fba82ffd640 1 -- 172.21.15.77:0/232786881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8f4ea0 msgr2=0x55a80b8f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.678+0000 7fba82ffd640 1 --2- 172.21.15.77:0/232786881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8f4ea0 0x55a80b8f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.678+0000 7fba82ffd640 1 -- 172.21.15.77:0/232786881 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a80b815460 con 0x55a80b8ef340 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.678+0000 7fba82ffd640 1 --2- 172.21.15.77:0/232786881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8ef760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fba7400e5e0 tx=0x7fba74014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fcb7ffe5354aa85 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.678+0000 7fba797fa640 1 -- 172.21.15.77:0/232786881 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74005020 con 0x55a80b8ef340 2024-03-16T10:19:08.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.678+0000 7fba797fa640 1 -- 172.21.15.77:0/232786881 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba740047a0 con 0x55a80b8ef340 2024-03-16T10:19:08.681 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.678+0000 7fba797fa640 1 -- 172.21.15.77:0/232786881 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74026790 con 0x55a80b8ef340 2024-03-16T10:19:08.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.679+0000 7fba8c3a2740 1 -- 172.21.15.77:0/232786881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 msgr2=0x55a80b8ef760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:08.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.679+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/232786881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8ef760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fba7400e5e0 tx=0x7fba74014000 comp rx=0 tx=0).stop 2024-03-16T10:19:08.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.679+0000 7fba8c3a2740 1 -- 172.21.15.77:0/232786881 shutdown_connections 2024-03-16T10:19:08.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.679+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/232786881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8f4ea0 0x55a80b8f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.679+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/232786881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8efd30 0x55a80b8f4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.685 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.679+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/232786881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8ef760 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.679+0000 7fba8c3a2740 1 -- 172.21.15.77:0/232786881 >> 172.21.15.77:0/232786881 conn(0x55a80b8ea8b0 msgr2=0x55a80b8eacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.679+0000 7fba8c3a2740 1 -- 172.21.15.77:0/232786881 shutdown_connections 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.679+0000 7fba8c3a2740 1 -- 172.21.15.77:0/232786881 wait complete. 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.680+0000 7fba8c3a2740 1 -- start start 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.680+0000 7fba8c3a2740 1 Processor -- start 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.680+0000 7fba8c3a2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8f9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.680+0000 7fba8c3a2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 0x55a80b8f9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.680+0000 7fba8c3a2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8f4ea0 0x55a80b900c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.680+0000 7fba8c3a2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a80b8f7e50 con 0x55a80b8efd30 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.680+0000 7fba8c3a2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a80b810d70 con 0x55a80b8f4ea0 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.680+0000 7fba8c3a2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a80b8f7fd0 con 0x55a80b8ef340 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.680+0000 7fba8c3a2740 10 client.0 _ll_register_callbacks cb 0x55a80b568a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 0x55a80b8f9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 0x55a80b8f9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55032/0 (socket says 172.21.15.77:55032) 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba7bfff640 1 -- 172.21.15.77:0/1523792581 learned_addr learned my addr 172.21.15.77:0/1523792581 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba82ffd640 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8f9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba837fe640 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8f4ea0 0x55a80b900c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba7bfff640 1 -- 172.21.15.77:0/1523792581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 msgr2=0x55a80b8f9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba7bfff640 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8f9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba7bfff640 1 -- 172.21.15.77:0/1523792581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8f4ea0 msgr2=0x55a80b900c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba7bfff640 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8f4ea0 0x55a80b900c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba7bfff640 1 -- 172.21.15.77:0/1523792581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a80b569210 con 0x55a80b8efd30 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba82ffd640 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8f9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba7bfff640 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 0x55a80b8f9bc0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0980b0 tx=0x7fba6c09b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c704752ecf440792 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba6a7fc640 1 -- 172.21.15.77:0/1523792581 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c0b3020 con 0x55a80b8efd30 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba6a7fc640 1 -- 172.21.15.77:0/1523792581 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fba6c0999e0 con 0x55a80b8efd30 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1523792581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba740047a0 con 0x55a80b8efd30 2024-03-16T10:19:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba6a7fc640 1 -- 172.21.15.77:0/1523792581 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fba6c09f830 con 0x55a80b8efd30 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba8c3a2740 10 client.16125 fetch_fsmap learned FSMap version 30 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.681+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1523792581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a80b635620 con 0x55a80b8efd30 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.682+0000 7fba6a7fc640 1 -- 172.21.15.77:0/1523792581 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba740047a0 con 0x55a80b8efd30 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.682+0000 7fba6a7fc640 1 -- 172.21.15.77:0/1523792581 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fba6c09ab30 con 0x55a80b8efd30 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.682+0000 7fba8c3a2740 10 client.16125 fetch_fsmap finished waiting for FSMap version 30 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.682+0000 7fba8c3a2740 10 client.16125 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.682+0000 7fba8c3a2740 20 client.16125 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.682+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a80b905500 0x55a80b907980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.682+0000 7fba8c3a2740 4 client.16125 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.682+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1523792581 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a80b861f80 con 0x55a80b905500 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.682+0000 7fba82ffd640 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a80b905500 0x55a80b907980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.682+0000 7fba6a7fc640 10 client.16125 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.682+0000 7fba82ffd640 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a80b905500 0x55a80b907980 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dadbc89dbb36d93 server_cookie=a7b34840b7ce9d5a in_seq=0 out_seq=0 2024-03-16T10:19:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.683+0000 7fba6a7fc640 1 -- 172.21.15.77:0/1523792581 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a80b861f80 con 0x55a80b905500 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.683+0000 7fba6a7fc640 10 client.16125 handle_command_reply: tid=0 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.683+0000 7fba8c3a2740 1 client.16125 shutdown 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.684+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1523792581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 msgr2=0x55a80b8f9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.684+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 0x55a80b8f9bc0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0980b0 tx=0x7fba6c09b040 comp rx=0 tx=0).stop 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.684+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1523792581 shutdown_connections 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.684+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8f4ea0 0x55a80b900c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.684+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a80b905500 0x55a80b907980 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.684+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 0x55a80b8f9bc0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.684+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1523792581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8f9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.684+0000 7fba6a7fc640 0 client.16125 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.684+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1523792581 >> 172.21.15.77:0/1523792581 conn(0x55a80b8ea8b0 msgr2=0x55a80b8e9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.684+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1523792581 shutdown_connections 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.684+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1523792581 wait complete. 2024-03-16T10:19:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.684+0000 7fba8c3a2740 20 client.16125 trim_cache size 0 max 16384 2024-03-16T10:19:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.709+0000 7fba8c3a2740 1 Processor -- start 2024-03-16T10:19:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.709+0000 7fba8c3a2740 1 -- start start 2024-03-16T10:19:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.709+0000 7fba8c3a2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8f1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.709+0000 7fba8c3a2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8efd30 0x55a80b8f15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.709+0000 7fba8c3a2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8f4ea0 0x55a80b900bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.709+0000 7fba8c3a2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fba841aada0 con 0x55a80b8f4ea0 2024-03-16T10:19:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.709+0000 7fba8c3a2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fba841ac9d0 con 0x55a80b8efd30 2024-03-16T10:19:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.709+0000 7fba8c3a2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba841ab470 con 0x55a80b8ef340 2024-03-16T10:19:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba837fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8f4ea0 0x55a80b900bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8efd30 0x55a80b8f15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8f1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba837fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8f4ea0 0x55a80b900bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55038/0 (socket says 172.21.15.77:55038) 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba837fe640 1 -- 172.21.15.77:0/1317901315 learned_addr learned my addr 172.21.15.77:0/1317901315 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba82ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8f1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:38840/0 (socket says 172.21.15.77:38840) 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba7bfff640 1 --2- 172.21.15.77:0/1317901315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8efd30 0x55a80b8f15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:46724/0 (socket says 172.21.15.77:46724) 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba82ffd640 1 -- 172.21.15.77:0/1317901315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8efd30 msgr2=0x55a80b8f15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba82ffd640 1 --2- 172.21.15.77:0/1317901315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8efd30 0x55a80b8f15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba82ffd640 1 -- 172.21.15.77:0/1317901315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8f4ea0 msgr2=0x55a80b900bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba82ffd640 1 --2- 172.21.15.77:0/1317901315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8f4ea0 0x55a80b900bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba82ffd640 1 -- 172.21.15.77:0/1317901315 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a80b8eb300 con 0x55a80b8ef340 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba82ffd640 1 --2- 172.21.15.77:0/1317901315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8f1090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fba740099f0 tx=0x7fba74009600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3686f9f6d4ee61a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba78ff9640 1 -- 172.21.15.77:0/1317901315 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74026a90 con 0x55a80b8ef340 2024-03-16T10:19:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba78ff9640 1 -- 172.21.15.77:0/1317901315 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba74026c30 con 0x55a80b8ef340 2024-03-16T10:19:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.710+0000 7fba78ff9640 1 -- 172.21.15.77:0/1317901315 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba740344e0 con 0x55a80b8ef340 2024-03-16T10:19:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.711+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1317901315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 msgr2=0x55a80b8f1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.711+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1317901315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8f1090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fba740099f0 tx=0x7fba74009600 comp rx=0 tx=0).stop 2024-03-16T10:19:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.711+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1317901315 shutdown_connections 2024-03-16T10:19:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.711+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1317901315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8f4ea0 0x55a80b900bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.711+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1317901315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8efd30 0x55a80b8f15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.711+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1317901315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8ef340 0x55a80b8f1090 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.711+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1317901315 >> 172.21.15.77:0/1317901315 conn(0x55a80b8ea8b0 msgr2=0x55a80b8f9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.711+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1317901315 shutdown_connections 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.711+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1317901315 wait complete. 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba8c3a2740 1 -- start start 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba8c3a2740 1 Processor -- start 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba8c3a2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8ef340 0x55a80b8fdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba8c3a2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 0x55a80b8fe530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba8c3a2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8f4ea0 0x55a80b8ff540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba8c3a2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fba841ac240 con 0x55a80b8efd30 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba8c3a2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fba841abb70 con 0x55a80b8ef340 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba8c3a2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba841a8da0 con 0x55a80b8f4ea0 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba8c3a2740 10 client.0 _ll_register_callbacks cb 0x7fba6c00e9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 0x55a80b8fe530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 0x55a80b8fe530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55044/0 (socket says 172.21.15.77:55044) 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba7bfff640 1 -- 172.21.15.77:0/1522309370 learned_addr learned my addr 172.21.15.77:0/1522309370 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba82ffd640 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8ef340 0x55a80b8fdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba837fe640 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8f4ea0 0x55a80b8ff540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba7bfff640 1 -- 172.21.15.77:0/1522309370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8f4ea0 msgr2=0x55a80b8ff540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba7bfff640 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8f4ea0 0x55a80b8ff540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba7bfff640 1 -- 172.21.15.77:0/1522309370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8ef340 msgr2=0x55a80b8fdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba7bfff640 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8ef340 0x55a80b8fdff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba7bfff640 1 -- 172.21.15.77:0/1522309370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a80b8ffb30 con 0x55a80b8efd30 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba7bfff640 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 0x55a80b8fe530 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0b4340 tx=0x7fba6c005ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc2bbaaec538107 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.712+0000 7fba6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba6a7fc640 1 -- 172.21.15.77:0/1522309370 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c0b3040 con 0x55a80b8efd30 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba6a7fc640 1 -- 172.21.15.77:0/1522309370 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fba6c003b30 con 0x55a80b8efd30 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1522309370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba74026c30 con 0x55a80b8efd30 2024-03-16T10:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba6a7fc640 1 -- 172.21.15.77:0/1522309370 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fba6c09bde0 con 0x55a80b8efd30 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba8c3a2740 10 client.16131 fetch_fsmap learned FSMap version 30 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1522309370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a80b900eb0 con 0x55a80b8efd30 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba6a7fc640 1 -- 172.21.15.77:0/1522309370 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba74026c30 con 0x55a80b8efd30 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba6a7fc640 1 -- 172.21.15.77:0/1522309370 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fba6c0ab070 con 0x55a80b8efd30 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba8c3a2740 10 client.16131 fetch_fsmap finished waiting for FSMap version 30 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba8c3a2740 10 client.16131 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba8c3a2740 20 client.16131 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a80b96bf10 0x55a80b96e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba8c3a2740 4 client.16131 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1522309370 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a80b85c2d0 con 0x55a80b96bf10 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.713+0000 7fba82ffd640 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a80b96bf10 0x55a80b96e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.714+0000 7fba6a7fc640 10 client.16131 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.714+0000 7fba82ffd640 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a80b96bf10 0x55a80b96e330 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1210ebf461a2a65c server_cookie=3334e309dd7876b7 in_seq=0 out_seq=0 2024-03-16T10:19:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.760+0000 7fba6a7fc640 1 -- 172.21.15.77:0/1522309370 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a80b85c2d0 con 0x55a80b96bf10 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.760+0000 7fba6a7fc640 10 client.16131 handle_command_reply: tid=0 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.760+0000 7fba8c3a2740 1 client.16131 shutdown 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.760+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1522309370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 msgr2=0x55a80b8fe530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.760+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 0x55a80b8fe530 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0b4340 tx=0x7fba6c005ab0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.760+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1522309370 shutdown_connections 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.760+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a80b8f4ea0 0x55a80b8ff540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.760+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a80b96bf10 0x55a80b96e330 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.760+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a80b8efd30 0x55a80b8fe530 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.760+0000 7fba6a7fc640 0 client.16131 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.760+0000 7fba8c3a2740 1 --2- 172.21.15.77:0/1522309370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a80b8ef340 0x55a80b8fdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.760+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1522309370 >> 172.21.15.77:0/1522309370 conn(0x55a80b8ea8b0 msgr2=0x55a80b8f9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1522309370 shutdown_connections 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba8c3a2740 1 -- 172.21.15.77:0/1522309370 wait complete. 2024-03-16T10:19:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba8c3a2740 20 client.16131 trim_cache size 0 max 16384 2024-03-16T10:19:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:19:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:19:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:19:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:19:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba896dc640 1 -- 172.21.15.77:0/2280352130 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fba700a2920 msgr2=0x7fba700a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba896dc640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fba700a2920 0x7fba700a4de0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fba84102880 tx=0x7fba74009260 comp rx=0 tx=0).stop 2024-03-16T10:19:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba896dc640 1 -- 172.21.15.77:0/2280352130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba84102a20 msgr2=0x7fba84073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba896dc640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba84102a20 0x7fba84073bd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fba6c00ea60 tx=0x7fba6c00ee60 comp rx=0 tx=0).stop 2024-03-16T10:19:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba896dc640 1 -- 172.21.15.77:0/2280352130 shutdown_connections 2024-03-16T10:19:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba896dc640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba841033e0 0x7fba84074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba896dc640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fba700a2920 0x7fba700a4de0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba896dc640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba84102a20 0x7fba84073bd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba896dc640 1 --2- 172.21.15.77:0/2280352130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba84101830 0x7fba84075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba896dc640 1 -- 172.21.15.77:0/2280352130 >> 172.21.15.77:0/2280352130 conn(0x7fba840fcda0 msgr2=0x7fba84105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba896dc640 1 -- 172.21.15.77:0/2280352130 shutdown_connections 2024-03-16T10:19:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:08.761+0000 7fba896dc640 1 -- 172.21.15.77:0/2280352130 wait complete. 2024-03-16T10:19:10.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:09 smithi192 ceph-mon[36354]: pgmap v412: 97 pgs: 97 active+clean; 57 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 115 op/s 2024-03-16T10:19:10.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:09 smithi077 ceph-mon[32400]: pgmap v412: 97 pgs: 97 active+clean; 57 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 115 op/s 2024-03-16T10:19:10.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:09 smithi193 ceph-mon[36311]: pgmap v412: 97 pgs: 97 active+clean; 57 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 115 op/s 2024-03-16T10:19:12.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:11 smithi192 ceph-mon[36354]: pgmap v413: 97 pgs: 97 active+clean; 57 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 49 op/s 2024-03-16T10:19:12.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:11 smithi077 ceph-mon[32400]: pgmap v413: 97 pgs: 97 active+clean; 57 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 49 op/s 2024-03-16T10:19:12.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:11 smithi193 ceph-mon[36311]: pgmap v413: 97 pgs: 97 active+clean; 57 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 49 op/s 2024-03-16T10:19:14.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:13 smithi192 ceph-mon[36354]: pgmap v414: 97 pgs: 97 active+clean; 63 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 70 op/s 2024-03-16T10:19:14.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:13 smithi077 ceph-mon[32400]: pgmap v414: 97 pgs: 97 active+clean; 63 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 70 op/s 2024-03-16T10:19:14.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:13 smithi193 ceph-mon[36311]: pgmap v414: 97 pgs: 97 active+clean; 63 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 70 op/s 2024-03-16T10:19:16.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:15 smithi192 ceph-mon[36354]: pgmap v415: 97 pgs: 97 active+clean; 75 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 135 op/s 2024-03-16T10:19:16.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:15 smithi077 ceph-mon[32400]: pgmap v415: 97 pgs: 97 active+clean; 75 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 135 op/s 2024-03-16T10:19:16.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:15 smithi193 ceph-mon[36311]: pgmap v415: 97 pgs: 97 active+clean; 75 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 135 op/s 2024-03-16T10:19:17.102 INFO:tasks.workunit.client.0.smithi077.stdout: 25 356219 253 222521 199 147965 719 2024-03-16T10:19:18.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:17 smithi192 ceph-mon[36354]: pgmap v416: 97 pgs: 97 active+clean; 99 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 247 op/s 2024-03-16T10:19:18.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:17 smithi077 ceph-mon[32400]: pgmap v416: 97 pgs: 97 active+clean; 99 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 247 op/s 2024-03-16T10:19:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:17 smithi193 ceph-mon[36311]: pgmap v416: 97 pgs: 97 active+clean; 99 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 247 op/s 2024-03-16T10:19:19.055 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:19:19.055 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:19:19.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.188+0000 7f1b3a636640 1 -- 172.21.15.77:0/644830098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b34102a00 msgr2=0x7f1b34102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.188+0000 7f1b3a636640 1 --2- 172.21.15.77:0/644830098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b34102a00 0x7f1b34102e80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009900 tx=0x7f1b2802bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:19.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.188+0000 7f1b3a636640 1 -- 172.21.15.77:0/644830098 shutdown_connections 2024-03-16T10:19:19.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.188+0000 7f1b3a636640 1 --2- 172.21.15.77:0/644830098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b341033c0 0x7f1b34109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.188+0000 7f1b3a636640 1 --2- 172.21.15.77:0/644830098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b34102a00 0x7f1b34102e80 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.188+0000 7f1b3a636640 1 --2- 172.21.15.77:0/644830098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b34101810 0x7f1b34101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.188+0000 7f1b3a636640 1 -- 172.21.15.77:0/644830098 >> 172.21.15.77:0/644830098 conn(0x7f1b340fcd80 msgr2=0x7f1b340ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.188+0000 7f1b3a636640 1 -- 172.21.15.77:0/644830098 shutdown_connections 2024-03-16T10:19:19.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.188+0000 7f1b3a636640 1 -- 172.21.15.77:0/644830098 wait complete. 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.188+0000 7f1b3a636640 1 Processor -- start 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.188+0000 7f1b3a636640 1 -- start start 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b3a636640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b34101810 0x7f1b34197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b3a636640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b34102a00 0x7f1b341980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b3a636640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b341033c0 0x7f1b3419f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b3a636640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3410d490 con 0x7f1b341033c0 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b3a636640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3410d310 con 0x7f1b34102a00 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b3a636640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3410d610 con 0x7f1b34101810 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b38bd3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b341033c0 0x7f1b3419f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b337fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b34102a00 0x7f1b341980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b38bd3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b341033c0 0x7f1b3419f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60636/0 (socket says 172.21.15.77:60636) 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b33fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b34101810 0x7f1b34197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b38bd3640 1 -- 172.21.15.77:0/868209665 learned_addr learned my addr 172.21.15.77:0/868209665 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b33fff640 1 -- 172.21.15.77:0/868209665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b34102a00 msgr2=0x7f1b341980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b33fff640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b34102a00 0x7f1b341980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b33fff640 1 -- 172.21.15.77:0/868209665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b341033c0 msgr2=0x7f1b3419f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b33fff640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b341033c0 0x7f1b3419f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b33fff640 1 -- 172.21.15.77:0/868209665 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3419f780 con 0x7f1b34101810 2024-03-16T10:19:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b38bd3640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b341033c0 0x7f1b3419f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b337fe640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b34102a00 0x7f1b341980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.189+0000 7f1b33fff640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b34101810 0x7f1b34197b60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400e9c0 tx=0x7f1b2400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.190+0000 7f1b317fa640 1 -- 172.21.15.77:0/868209665 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2400ccc0 con 0x7f1b34101810 2024-03-16T10:19:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.190+0000 7f1b3a636640 1 -- 172.21.15.77:0/868209665 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b3419fa70 con 0x7f1b34101810 2024-03-16T10:19:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.190+0000 7f1b317fa640 1 -- 172.21.15.77:0/868209665 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b240035f0 con 0x7f1b34101810 2024-03-16T10:19:19.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.190+0000 7f1b317fa640 1 -- 172.21.15.77:0/868209665 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2400ce60 con 0x7f1b34101810 2024-03-16T10:19:19.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.190+0000 7f1b3a636640 1 -- 172.21.15.77:0/868209665 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b3419ff40 con 0x7f1b34101810 2024-03-16T10:19:19.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.192+0000 7f1b317fa640 1 -- 172.21.15.77:0/868209665 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f1b240037a0 con 0x7f1b34101810 2024-03-16T10:19:19.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.192+0000 7f1b3d224740 1 Processor -- start 2024-03-16T10:19:19.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.192+0000 7f1b3d224740 1 -- start start 2024-03-16T10:19:19.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.192+0000 7f1b3d224740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7170 0x55d4199c7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.192+0000 7f1b3d224740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7b60 0x55d4199cc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.192+0000 7f1b3d224740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199cccd0 0x55d4199cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.192+0000 7f1b3d224740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d41992c500 con 0x55d4199c7b60 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.192+0000 7f1b3d224740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d4198d4be0 con 0x55d4199c7170 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.192+0000 7f1b3d224740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d4199285a0 con 0x55d4199cccd0 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.192+0000 7f1b317fa640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1b1c0a2800 0x7f1b1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b33fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7b60 0x55d4199cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b38bd3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7170 0x55d4199c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b33fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7b60 0x55d4199cc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60640/0 (socket says 172.21.15.77:60640) 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b33fff640 1 -- 172.21.15.77:0/40352205 learned_addr learned my addr 172.21.15.77:0/40352205 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b337fe640 1 --2- 172.21.15.77:0/40352205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199cccd0 0x55d4199cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b317fa640 1 -- 172.21.15.77:0/868209665 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f1b240cef80 con 0x7f1b34101810 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b38bd3640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1b1c0a2800 0x7f1b1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b33fff640 1 -- 172.21.15.77:0/40352205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199cccd0 msgr2=0x55d4199cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b33fff640 1 --2- 172.21.15.77:0/40352205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199cccd0 0x55d4199cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b33fff640 1 -- 172.21.15.77:0/40352205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7170 msgr2=0x55d4199c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b33fff640 1 --2- 172.21.15.77:0/40352205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7170 0x55d4199c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b33fff640 1 -- 172.21.15.77:0/40352205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4198e84b0 con 0x55d4199c7b60 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b337fe640 1 --2- 172.21.15.77:0/40352205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199cccd0 0x55d4199cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b33fff640 1 --2- 172.21.15.77:0/40352205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7b60 0x55d4199cc660 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1b2406e610 tx=0x7f1b2406e640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80df78559b5a713 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b38bd3640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1b1c0a2800 0x7f1b1c0a4cc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x55d4199c83c0 tx=0x7f1b2c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b1a7fc640 1 -- 172.21.15.77:0/40352205 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2409b910 con 0x55d4199c7b60 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b1a7fc640 1 -- 172.21.15.77:0/40352205 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b2409fa60 con 0x55d4199c7b60 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.193+0000 7f1b1a7fc640 1 -- 172.21.15.77:0/40352205 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b24097f80 con 0x55d4199c7b60 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.194+0000 7f1b3d224740 1 -- 172.21.15.77:0/40352205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7b60 msgr2=0x55d4199cc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.194+0000 7f1b3d224740 1 --2- 172.21.15.77:0/40352205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7b60 0x55d4199cc660 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1b2406e610 tx=0x7f1b2406e640 comp rx=0 tx=0).stop 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.194+0000 7f1b3d224740 1 -- 172.21.15.77:0/40352205 shutdown_connections 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.194+0000 7f1b3d224740 1 --2- 172.21.15.77:0/40352205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199cccd0 0x55d4199cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.194+0000 7f1b3d224740 1 --2- 172.21.15.77:0/40352205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7b60 0x55d4199cc660 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.194+0000 7f1b3d224740 1 --2- 172.21.15.77:0/40352205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7170 0x55d4199c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.194+0000 7f1b3d224740 1 -- 172.21.15.77:0/40352205 >> 172.21.15.77:0/40352205 conn(0x55d4199c26e0 msgr2=0x55d4199c2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.194+0000 7f1b3d224740 1 -- 172.21.15.77:0/40352205 shutdown_connections 2024-03-16T10:19:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.194+0000 7f1b3d224740 1 -- 172.21.15.77:0/40352205 wait complete. 2024-03-16T10:19:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.194+0000 7f1b3d224740 1 -- start start 2024-03-16T10:19:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b3d224740 1 Processor -- start 2024-03-16T10:19:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b3d224740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 0x55d4199d1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b3d224740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199c7b60 0x55d4199d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b3d224740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199cccd0 0x55d4199d8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b3d224740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d4199cfc20 con 0x55d4199c7170 2024-03-16T10:19:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b3d224740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d4198e8160 con 0x55d4199cccd0 2024-03-16T10:19:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b3d224740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d4199cfda0 con 0x55d4199c7b60 2024-03-16T10:19:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b3d224740 10 client.0 _ll_register_callbacks cb 0x55d4198cc150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b33fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 0x55d4199d1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b33fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 0x55d4199d1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60646/0 (socket says 172.21.15.77:60646) 2024-03-16T10:19:19.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b33fff640 1 -- 172.21.15.77:0/2276025871 learned_addr learned my addr 172.21.15.77:0/2276025871 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b337fe640 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199c7b60 0x55d4199d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b38bd3640 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199cccd0 0x55d4199d8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b33fff640 1 -- 172.21.15.77:0/2276025871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199c7b60 msgr2=0x55d4199d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b33fff640 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199c7b60 0x55d4199d1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b33fff640 1 -- 172.21.15.77:0/2276025871 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199cccd0 msgr2=0x55d4199d8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b33fff640 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199cccd0 0x55d4199d8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b33fff640 1 -- 172.21.15.77:0/2276025871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d41970a7d0 con 0x55d4199c7170 2024-03-16T10:19:19.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b33fff640 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 0x55d4199d1610 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1b2409b8a0 tx=0x7f1b240a0a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b781a993e5448a34 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.195+0000 7f1b137fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:19.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b137fe640 1 -- 172.21.15.77:0/2276025871 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b240a40b0 con 0x55d4199c7170 2024-03-16T10:19:19.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b137fe640 1 -- 172.21.15.77:0/2276025871 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f1b240a3c20 con 0x55d4199c7170 2024-03-16T10:19:19.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b3d224740 1 -- 172.21.15.77:0/2276025871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b2409fa60 con 0x55d4199c7170 2024-03-16T10:19:19.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b137fe640 1 -- 172.21.15.77:0/2276025871 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1b2409b910 con 0x55d4199c7170 2024-03-16T10:19:19.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b3d224740 10 client.16143 fetch_fsmap learned FSMap version 30 2024-03-16T10:19:19.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b3d224740 1 -- 172.21.15.77:0/2276025871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d4196403f0 con 0x55d4199c7170 2024-03-16T10:19:19.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b137fe640 1 -- 172.21.15.77:0/2276025871 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b2409fa60 con 0x55d4199c7170 2024-03-16T10:19:19.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b137fe640 1 -- 172.21.15.77:0/2276025871 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f1b24097660 con 0x55d4199c7170 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b3d224740 10 client.16143 fetch_fsmap finished waiting for FSMap version 30 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b3d224740 10 client.16143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b3d224740 20 client.16143 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b3d224740 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d4199dccf0 0x55d4199df0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b3d224740 4 client.16143 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b3d224740 1 -- 172.21.15.77:0/2276025871 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d419939a40 con 0x55d4199dccf0 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.196+0000 7f1b337fe640 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d4199dccf0 0x55d4199df0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.197+0000 7f1b337fe640 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d4199dccf0 0x55d4199df0e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a67697644dc2a6d8 server_cookie=c3edfd57eb725b41 in_seq=0 out_seq=0 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.197+0000 7f1b137fe640 10 client.16143 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b137fe640 1 -- 172.21.15.77:0/2276025871 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d419939a40 con 0x55d4199dccf0 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b137fe640 10 client.16143 handle_command_reply: tid=0 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b3d224740 1 client.16143 shutdown 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b3d224740 1 -- 172.21.15.77:0/2276025871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 msgr2=0x55d4199d1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b3d224740 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 0x55d4199d1610 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1b2409b8a0 tx=0x7f1b240a0a80 comp rx=0 tx=0).stop 2024-03-16T10:19:19.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b3d224740 1 -- 172.21.15.77:0/2276025871 shutdown_connections 2024-03-16T10:19:19.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b3d224740 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199cccd0 0x55d4199d8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b3d224740 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199c7b60 0x55d4199d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b3d224740 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d4199dccf0 0x55d4199df0e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b3d224740 1 --2- 172.21.15.77:0/2276025871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 0x55d4199d1610 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b3d224740 1 -- 172.21.15.77:0/2276025871 >> 172.21.15.77:0/2276025871 conn(0x55d4199c26e0 msgr2=0x55d4199c1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b137fe640 0 client.16143 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:19.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b3d224740 1 -- 172.21.15.77:0/2276025871 shutdown_connections 2024-03-16T10:19:19.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b3d224740 1 -- 172.21.15.77:0/2276025871 wait complete. 2024-03-16T10:19:19.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.198+0000 7f1b3d224740 20 client.16143 trim_cache size 0 max 16384 2024-03-16T10:19:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.230+0000 7f1b3d224740 1 Processor -- start 2024-03-16T10:19:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.230+0000 7f1b3d224740 1 -- start start 2024-03-16T10:19:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.230+0000 7f1b3d224740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7170 0x55d4199d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.230+0000 7f1b3d224740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199c7b60 0x55d4199d8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.230+0000 7f1b3d224740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199cccd0 0x55d4199c1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.230+0000 7f1b3d224740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1b341a8b40 con 0x55d4199cccd0 2024-03-16T10:19:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.230+0000 7f1b3d224740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1b341aa770 con 0x55d4199c7170 2024-03-16T10:19:19.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.230+0000 7f1b3d224740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1b341a9210 con 0x55d4199c7b60 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b38bd3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199cccd0 0x55d4199c1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b337fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199c7b60 0x55d4199d8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b33fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7170 0x55d4199d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b38bd3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199cccd0 0x55d4199c1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60648/0 (socket says 172.21.15.77:60648) 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b38bd3640 1 -- 172.21.15.77:0/5823177 learned_addr learned my addr 172.21.15.77:0/5823177 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b337fe640 1 -- 172.21.15.77:0/5823177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7170 msgr2=0x55d4199d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b337fe640 1 --2- 172.21.15.77:0/5823177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7170 0x55d4199d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b337fe640 1 -- 172.21.15.77:0/5823177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199cccd0 msgr2=0x55d4199c1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b337fe640 1 --2- 172.21.15.77:0/5823177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199cccd0 0x55d4199c1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b337fe640 1 -- 172.21.15.77:0/5823177 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4196000f0 con 0x55d4199c7b60 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b337fe640 1 --2- 172.21.15.77:0/5823177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199c7b60 0x55d4199d8b50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009230 tx=0x7f1b2802e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6066c0d4729a0d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b19ffb640 1 -- 172.21.15.77:0/5823177 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28046ca0 con 0x55d4199c7b60 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b19ffb640 1 -- 172.21.15.77:0/5823177 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b28046e40 con 0x55d4199c7b60 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.231+0000 7f1b19ffb640 1 -- 172.21.15.77:0/5823177 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b280054d0 con 0x55d4199c7b60 2024-03-16T10:19:19.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.232+0000 7f1b3d224740 1 -- 172.21.15.77:0/5823177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199c7b60 msgr2=0x55d4199d8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.232+0000 7f1b3d224740 1 --2- 172.21.15.77:0/5823177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199c7b60 0x55d4199d8b50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009230 tx=0x7f1b2802e040 comp rx=0 tx=0).stop 2024-03-16T10:19:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.232+0000 7f1b3d224740 1 -- 172.21.15.77:0/5823177 shutdown_connections 2024-03-16T10:19:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.232+0000 7f1b3d224740 1 --2- 172.21.15.77:0/5823177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199cccd0 0x55d4199c1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.232+0000 7f1b3d224740 1 --2- 172.21.15.77:0/5823177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199c7b60 0x55d4199d8b50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.232+0000 7f1b3d224740 1 --2- 172.21.15.77:0/5823177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7170 0x55d4199d1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.232+0000 7f1b3d224740 1 -- 172.21.15.77:0/5823177 >> 172.21.15.77:0/5823177 conn(0x55d4199c26e0 msgr2=0x55d4199d1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.232+0000 7f1b3d224740 1 -- 172.21.15.77:0/5823177 shutdown_connections 2024-03-16T10:19:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.232+0000 7f1b3d224740 1 -- 172.21.15.77:0/5823177 wait complete. 2024-03-16T10:19:19.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b3d224740 1 -- start start 2024-03-16T10:19:19.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b3d224740 1 Processor -- start 2024-03-16T10:19:19.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b3d224740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 0x55d4199d5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b3d224740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7b60 0x55d4199d64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b3d224740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199cccd0 0x55d4199d74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b3d224740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1b341a9fe0 con 0x55d4199c7170 2024-03-16T10:19:19.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b3d224740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1b341a9910 con 0x55d4199c7b60 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b3d224740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1b341a6b20 con 0x55d4199cccd0 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b3d224740 10 client.0 _ll_register_callbacks cb 0x7f1b2c013890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b33fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 0x55d4199d5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b33fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 0x55d4199d5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60650/0 (socket says 172.21.15.77:60650) 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b33fff640 1 -- 172.21.15.77:0/1525603563 learned_addr learned my addr 172.21.15.77:0/1525603563 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b337fe640 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7b60 0x55d4199d64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.233+0000 7f1b38bd3640 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199cccd0 0x55d4199d74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b337fe640 1 -- 172.21.15.77:0/1525603563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199cccd0 msgr2=0x55d4199d74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b337fe640 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199cccd0 0x55d4199d74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b337fe640 1 -- 172.21.15.77:0/1525603563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 msgr2=0x55d4199d5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b337fe640 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 0x55d4199d5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b337fe640 1 -- 172.21.15.77:0/1525603563 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4198f39d0 con 0x55d4199c7b60 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b33fff640 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 0x55d4199d5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b337fe640 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7b60 0x55d4199d64d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009850 tx=0x7f1b2802c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35f983fec8ac5f60 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b137fe640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b137fe640 1 -- 172.21.15.77:0/1525603563 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28054070 con 0x55d4199c7b60 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b137fe640 1 -- 172.21.15.77:0/1525603563 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f1b280042c0 con 0x55d4199c7b60 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b3d224740 1 -- 172.21.15.77:0/1525603563 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d419933d90 con 0x55d4199c7b60 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b137fe640 1 -- 172.21.15.77:0/1525603563 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1b28060050 con 0x55d4199c7b60 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b3d224740 10 client.25934 fetch_fsmap learned FSMap version 30 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.234+0000 7f1b3d224740 1 -- 172.21.15.77:0/1525603563 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d4199d8c40 con 0x55d4199c7b60 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.235+0000 7f1b137fe640 1 -- 172.21.15.77:0/1525603563 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d419933d90 con 0x55d4199c7b60 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.235+0000 7f1b137fe640 1 -- 172.21.15.77:0/1525603563 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f1b280039a0 con 0x55d4199c7b60 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.235+0000 7f1b3d224740 10 client.25934 fetch_fsmap finished waiting for FSMap version 30 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.235+0000 7f1b3d224740 10 client.25934 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.235+0000 7f1b3d224740 20 client.25934 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.235+0000 7f1b3d224740 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d419a433c0 0x55d419a457e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.235+0000 7f1b3d224740 4 client.25934 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.235+0000 7f1b3d224740 1 -- 172.21.15.77:0/1525603563 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d4198b1540 con 0x55d419a433c0 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.235+0000 7f1b33fff640 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d419a433c0 0x55d419a457e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.235+0000 7f1b137fe640 10 client.25934 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.235+0000 7f1b33fff640 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d419a433c0 0x55d419a457e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e31efa3280ffe534 server_cookie=1c210e428bda853a in_seq=0 out_seq=0 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b137fe640 1 -- 172.21.15.77:0/1525603563 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d4198b1540 con 0x55d419a433c0 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b137fe640 10 client.25934 handle_command_reply: tid=0 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b3d224740 1 client.25934 shutdown 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b3d224740 1 -- 172.21.15.77:0/1525603563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7b60 msgr2=0x55d4199d64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b3d224740 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7b60 0x55d4199d64d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009850 tx=0x7f1b2802c330 comp rx=0 tx=0).stop 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b3d224740 1 -- 172.21.15.77:0/1525603563 shutdown_connections 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b3d224740 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d4199cccd0 0x55d4199d74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b3d224740 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4199c7b60 0x55d4199d64d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b3d224740 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d419a433c0 0x55d419a457e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b3d224740 1 --2- 172.21.15.77:0/1525603563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d4199c7170 0x55d4199d5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b137fe640 0 client.25934 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:19.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b3d224740 1 -- 172.21.15.77:0/1525603563 >> 172.21.15.77:0/1525603563 conn(0x55d4199c26e0 msgr2=0x55d4199c5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b3d224740 1 -- 172.21.15.77:0/1525603563 shutdown_connections 2024-03-16T10:19:19.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b3d224740 1 -- 172.21.15.77:0/1525603563 wait complete. 2024-03-16T10:19:19.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.236+0000 7f1b3d224740 20 client.25934 trim_cache size 0 max 16384 2024-03-16T10:19:19.243 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:19:19.243 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T10:19:19.243 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T10:19:19.243 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:19:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.237+0000 7f1b3a636640 1 -- 172.21.15.77:0/868209665 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1b1c0a2800 msgr2=0x7f1b1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.237+0000 7f1b3a636640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1b1c0a2800 0x7f1b1c0a4cc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x55d4199c83c0 tx=0x7f1b2c009500 comp rx=0 tx=0).stop 2024-03-16T10:19:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.237+0000 7f1b3a636640 1 -- 172.21.15.77:0/868209665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b34101810 msgr2=0x7f1b34197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.237+0000 7f1b3a636640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b34101810 0x7f1b34197b60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400e9c0 tx=0x7f1b2400ee80 comp rx=0 tx=0).stop 2024-03-16T10:19:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.237+0000 7f1b3a636640 1 -- 172.21.15.77:0/868209665 shutdown_connections 2024-03-16T10:19:19.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.237+0000 7f1b3a636640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b341033c0 0x7f1b3419f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.237+0000 7f1b3a636640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b34102a00 0x7f1b341980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.237+0000 7f1b3a636640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1b1c0a2800 0x7f1b1c0a4cc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.237+0000 7f1b3a636640 1 --2- 172.21.15.77:0/868209665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b34101810 0x7f1b34197b60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.237+0000 7f1b3a636640 1 -- 172.21.15.77:0/868209665 >> 172.21.15.77:0/868209665 conn(0x7f1b340fcd80 msgr2=0x7f1b34105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.237+0000 7f1b3a636640 1 -- 172.21.15.77:0/868209665 shutdown_connections 2024-03-16T10:19:19.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.237+0000 7f1b3a636640 1 -- 172.21.15.77:0/868209665 wait complete. 2024-03-16T10:19:19.252 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T10:19:19.253 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:19:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.372+0000 7f4d62a7e640 1 -- 172.21.15.77:0/419180298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d5c1018f0 msgr2=0x7f4d5c101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.372+0000 7f4d62a7e640 1 --2- 172.21.15.77:0/419180298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d5c1018f0 0x7f4d5c101d70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009900 tx=0x7f4d4802bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.372+0000 7f4d62a7e640 1 -- 172.21.15.77:0/419180298 shutdown_connections 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.372+0000 7f4d62a7e640 1 --2- 172.21.15.77:0/419180298 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d5c1022b0 0x7f4d5c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.372+0000 7f4d62a7e640 1 --2- 172.21.15.77:0/419180298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d5c1018f0 0x7f4d5c101d70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.372+0000 7f4d62a7e640 1 --2- 172.21.15.77:0/419180298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d5c100700 0x7f4d5c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.372+0000 7f4d62a7e640 1 -- 172.21.15.77:0/419180298 >> 172.21.15.77:0/419180298 conn(0x7f4d5c0fbc70 msgr2=0x7f4d5c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.372+0000 7f4d62a7e640 1 -- 172.21.15.77:0/419180298 shutdown_connections 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.372+0000 7f4d62a7e640 1 -- 172.21.15.77:0/419180298 wait complete. 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d62a7e640 1 Processor -- start 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d62a7e640 1 -- start start 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d62a7e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d5c100700 0x7f4d5c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d62a7e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d5c1018f0 0x7f4d5c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d62a7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d5c1022b0 0x7f4d5c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d62a7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c10c380 con 0x7f4d5c1022b0 2024-03-16T10:19:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d62a7e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c10c200 con 0x7f4d5c1018f0 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d62a7e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c10c500 con 0x7f4d5c100700 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d6081a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d5c100700 0x7f4d5c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d6101b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d5c1022b0 0x7f4d5c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d6081a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d5c100700 0x7f4d5c197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50584/0 (socket says 172.21.15.77:50584) 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d6081a640 1 -- 172.21.15.77:0/1228461198 learned_addr learned my addr 172.21.15.77:0/1228461198 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.373+0000 7f4d5bfff640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d5c1018f0 0x7f4d5c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d6101b640 1 -- 172.21.15.77:0/1228461198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d5c100700 msgr2=0x7f4d5c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d6101b640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d5c100700 0x7f4d5c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d6101b640 1 -- 172.21.15.77:0/1228461198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d5c1018f0 msgr2=0x7f4d5c198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d6101b640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d5c1018f0 0x7f4d5c198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d6101b640 1 -- 172.21.15.77:0/1228461198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5c19f740 con 0x7f4d5c1022b0 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d6081a640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d5c100700 0x7f4d5c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d5bfff640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d5c1018f0 0x7f4d5c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d6101b640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d5c1022b0 0x7f4d5c19f0d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f4d54009fd0 tx=0x7f4d5400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d59ffb640 1 -- 172.21.15.77:0/1228461198 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5400ce20 con 0x7f4d5c1022b0 2024-03-16T10:19:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d62a7e640 1 -- 172.21.15.77:0/1228461198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5c19fa30 con 0x7f4d5c1022b0 2024-03-16T10:19:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d59ffb640 1 -- 172.21.15.77:0/1228461198 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d54005020 con 0x7f4d5c1022b0 2024-03-16T10:19:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d59ffb640 1 -- 172.21.15.77:0/1228461198 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d54010720 con 0x7f4d5c1022b0 2024-03-16T10:19:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.374+0000 7f4d62a7e640 1 -- 172.21.15.77:0/1228461198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d5c19ffa0 con 0x7f4d5c1022b0 2024-03-16T10:19:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.375+0000 7f4d62a7e640 1 -- 172.21.15.77:0/1228461198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d5c1a0290 con 0x7f4d5c1022b0 2024-03-16T10:19:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.376+0000 7f4d59ffb640 1 -- 172.21.15.77:0/1228461198 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f4d54005540 con 0x7f4d5c1022b0 2024-03-16T10:19:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.377+0000 7f4d59ffb640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d4c0a2800 0x7f4d4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.377+0000 7f4d6081a640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d4c0a2800 0x7f4d4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.377+0000 7f4d59ffb640 1 -- 172.21.15.77:0/1228461198 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f4d540cefe0 con 0x7f4d5c1022b0 2024-03-16T10:19:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.377+0000 7f4d6081a640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d4c0a2800 0x7f4d4c0a4cc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4d500058c0 tx=0x7f4d50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.383+0000 7f4d59ffb640 1 -- 172.21.15.77:0/1228461198 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4d5409c110 con 0x7f4d5c1022b0 2024-03-16T10:19:19.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.645+0000 7f4d62a7e640 1 -- 172.21.15.77:0/1228461198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d5c1a0460 con 0x7f4d5c1022b0 2024-03-16T10:19:19.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.646+0000 7f4d59ffb640 1 -- 172.21.15.77:0/1228461198 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11946 (secure 0 0 0) 0x7f4d5409ba10 con 0x7f4d5c1022b0 2024-03-16T10:19:19.647 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:19:19.647 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:19:04.662498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[14658,24455,24482,24454,24475]},"id":1}]} 2024-03-16T10:19:19.647 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 30 2024-03-16T10:19:19.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.650+0000 7f4d62a7e640 1 -- 172.21.15.77:0/1228461198 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d4c0a2800 msgr2=0x7f4d4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.650+0000 7f4d62a7e640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d4c0a2800 0x7f4d4c0a4cc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4d500058c0 tx=0x7f4d50005f90 comp rx=0 tx=0).stop 2024-03-16T10:19:19.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.650+0000 7f4d62a7e640 1 -- 172.21.15.77:0/1228461198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d5c1022b0 msgr2=0x7f4d5c19f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.650+0000 7f4d62a7e640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d5c1022b0 0x7f4d5c19f0d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f4d54009fd0 tx=0x7f4d5400ef10 comp rx=0 tx=0).stop 2024-03-16T10:19:19.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.650+0000 7f4d62a7e640 1 -- 172.21.15.77:0/1228461198 shutdown_connections 2024-03-16T10:19:19.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.650+0000 7f4d62a7e640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d5c1022b0 0x7f4d5c19f0d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.650+0000 7f4d62a7e640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d5c1018f0 0x7f4d5c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.650+0000 7f4d62a7e640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d4c0a2800 0x7f4d4c0a4cc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.650+0000 7f4d62a7e640 1 --2- 172.21.15.77:0/1228461198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d5c100700 0x7f4d5c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.650+0000 7f4d62a7e640 1 -- 172.21.15.77:0/1228461198 >> 172.21.15.77:0/1228461198 conn(0x7f4d5c0fbc70 msgr2=0x7f4d5c068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.650+0000 7f4d62a7e640 1 -- 172.21.15.77:0/1228461198 shutdown_connections 2024-03-16T10:19:19.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.650+0000 7f4d62a7e640 1 -- 172.21.15.77:0/1228461198 wait complete. 2024-03-16T10:19:19.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-16T10:19:19.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.790+0000 7fea2ee85640 1 -- 172.21.15.77:0/846578613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea281017f0 msgr2=0x7fea28101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.790+0000 7fea2ee85640 1 --2- 172.21.15.77:0/846578613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea281017f0 0x7fea28101c10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009900 tx=0x7fea1c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:19.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.790+0000 7fea2ee85640 1 -- 172.21.15.77:0/846578613 shutdown_connections 2024-03-16T10:19:19.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.790+0000 7fea2ee85640 1 --2- 172.21.15.77:0/846578613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea281033a0 0x7fea28109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.790+0000 7fea2ee85640 1 --2- 172.21.15.77:0/846578613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea281029e0 0x7fea28102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.790+0000 7fea2ee85640 1 --2- 172.21.15.77:0/846578613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea281017f0 0x7fea28101c10 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.790+0000 7fea2ee85640 1 -- 172.21.15.77:0/846578613 >> 172.21.15.77:0/846578613 conn(0x7fea280fcd40 msgr2=0x7fea280ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.790+0000 7fea2ee85640 1 -- 172.21.15.77:0/846578613 shutdown_connections 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.790+0000 7fea2ee85640 1 -- 172.21.15.77:0/846578613 wait complete. 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2ee85640 1 Processor -- start 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2ee85640 1 -- start start 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2ee85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea281017f0 0x7fea28197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2ee85640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea281029e0 0x7fea28198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2ee85640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea281033a0 0x7fea2819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2ee85640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fea2810d440 con 0x7fea281017f0 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2ee85640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fea2810d2c0 con 0x7fea281033a0 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2ee85640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea2810d5c0 con 0x7fea281029e0 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2cc21640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea281017f0 0x7fea28197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2cc21640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea281017f0 0x7fea28197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60686/0 (socket says 172.21.15.77:60686) 2024-03-16T10:19:19.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2cc21640 1 -- 172.21.15.77:0/1849547647 learned_addr learned my addr 172.21.15.77:0/1849547647 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea27fff640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea281029e0 0x7fea28198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2d422640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea281033a0 0x7fea2819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2cc21640 1 -- 172.21.15.77:0/1849547647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea281029e0 msgr2=0x7fea28198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2cc21640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea281029e0 0x7fea28198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2cc21640 1 -- 172.21.15.77:0/1849547647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea281033a0 msgr2=0x7fea2819f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2cc21640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea281033a0 0x7fea2819f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.791+0000 7fea2cc21640 1 -- 172.21.15.77:0/1849547647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2819f730 con 0x7fea281017f0 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.792+0000 7fea2cc21640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea281017f0 0x7fea28197b10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fea1c000c00 tx=0x7fea1c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.792+0000 7fea25ffb640 1 -- 172.21.15.77:0/1849547647 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c041070 con 0x7fea281017f0 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.792+0000 7fea2ee85640 1 -- 172.21.15.77:0/1849547647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2819f9c0 con 0x7fea281017f0 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.792+0000 7fea25ffb640 1 -- 172.21.15.77:0/1849547647 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea1c004450 con 0x7fea281017f0 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.792+0000 7fea25ffb640 1 -- 172.21.15.77:0/1849547647 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c034440 con 0x7fea281017f0 2024-03-16T10:19:19.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.792+0000 7fea2ee85640 1 -- 172.21.15.77:0/1849547647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea2819ff00 con 0x7fea281017f0 2024-03-16T10:19:19.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.794+0000 7fea25ffb640 1 -- 172.21.15.77:0/1849547647 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fea1c044070 con 0x7fea281017f0 2024-03-16T10:19:19.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.794+0000 7fea25ffb640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fea180a2800 0x7fea180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea27fff640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fea180a2800 0x7fea180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea25ffb640 1 -- 172.21.15.77:0/1849547647 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fea1c0f0490 con 0x7fea281017f0 2024-03-16T10:19:19.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea31a7b740 1 Processor -- start 2024-03-16T10:19:19.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea31a7b740 1 -- start start 2024-03-16T10:19:19.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea31a7b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a96bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea31a7b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a96c240 0x56055a970d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea31a7b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a9713b0 0x56055a973840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea31a7b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56055a8d0490 con 0x56055a96c240 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea31a7b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56055a878c90 con 0x56055a96b850 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea31a7b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56055a8cc530 con 0x56055a9713b0 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea27fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a96c240 0x56055a970d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea27fff640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fea180a2800 0x7fea180a4cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fea28199020 tx=0x7fea14005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea2cc21640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a96bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea27fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a96c240 0x56055a970d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60700/0 (socket says 172.21.15.77:60700) 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea27fff640 1 -- 172.21.15.77:0/1359563386 learned_addr learned my addr 172.21.15.77:0/1359563386 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea2d422640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a9713b0 0x56055a973840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea27fff640 1 -- 172.21.15.77:0/1359563386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a9713b0 msgr2=0x56055a973840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea27fff640 1 --2- 172.21.15.77:0/1359563386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a9713b0 0x56055a973840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea27fff640 1 -- 172.21.15.77:0/1359563386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 msgr2=0x56055a96bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea27fff640 1 --2- 172.21.15.77:0/1359563386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a96bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.795+0000 7fea27fff640 1 -- 172.21.15.77:0/1359563386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055a933500 con 0x56055a96c240 2024-03-16T10:19:19.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.796+0000 7fea2cc21640 1 --2- 172.21.15.77:0/1359563386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a96bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.796+0000 7fea27fff640 1 --2- 172.21.15.77:0/1359563386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a96c240 0x56055a970d40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fea1400e610 tx=0x7fea14014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719e7a2af0550e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.796+0000 7fea12ffd640 1 -- 172.21.15.77:0/1359563386 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea14005020 con 0x56055a96c240 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.796+0000 7fea12ffd640 1 -- 172.21.15.77:0/1359563386 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea140047a0 con 0x56055a96c240 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.796+0000 7fea12ffd640 1 -- 172.21.15.77:0/1359563386 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea14026860 con 0x56055a96c240 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.797+0000 7fea31a7b740 1 -- 172.21.15.77:0/1359563386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a96c240 msgr2=0x56055a970d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.797+0000 7fea31a7b740 1 --2- 172.21.15.77:0/1359563386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a96c240 0x56055a970d40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fea1400e610 tx=0x7fea14014000 comp rx=0 tx=0).stop 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.797+0000 7fea31a7b740 1 -- 172.21.15.77:0/1359563386 shutdown_connections 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.797+0000 7fea31a7b740 1 --2- 172.21.15.77:0/1359563386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a9713b0 0x56055a973840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.797+0000 7fea31a7b740 1 --2- 172.21.15.77:0/1359563386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a96c240 0x56055a970d40 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.797+0000 7fea31a7b740 1 --2- 172.21.15.77:0/1359563386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a96bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.797+0000 7fea31a7b740 1 -- 172.21.15.77:0/1359563386 >> 172.21.15.77:0/1359563386 conn(0x56055a966dc0 msgr2=0x56055a9671c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.797+0000 7fea31a7b740 1 -- 172.21.15.77:0/1359563386 shutdown_connections 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.797+0000 7fea31a7b740 1 -- 172.21.15.77:0/1359563386 wait complete. 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.797+0000 7fea31a7b740 1 -- start start 2024-03-16T10:19:19.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.797+0000 7fea31a7b740 1 Processor -- start 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea31a7b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96b850 0x56055a975da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea31a7b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a96c240 0x56055a9762e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea31a7b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a9713b0 0x56055a97d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea31a7b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56055a974360 con 0x56055a96c240 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea31a7b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56055a88c230 con 0x56055a9713b0 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea31a7b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56055a9744e0 con 0x56055a96b850 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea2d422640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a9713b0 0x56055a97d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea2cc21640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96b850 0x56055a975da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea2d422640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a9713b0 0x56055a97d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50954/0 (socket says 172.21.15.77:50954) 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea2d422640 1 -- 172.21.15.77:0/2268353400 learned_addr learned my addr 172.21.15.77:0/2268353400 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea27fff640 1 --2- 172.21.15.77:0/2268353400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a96c240 0x56055a9762e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea31a7b740 10 client.0 _ll_register_callbacks cb 0x56055a8c8010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea2d422640 1 -- 172.21.15.77:0/2268353400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96b850 msgr2=0x56055a975da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea2d422640 1 --2- 172.21.15.77:0/2268353400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96b850 0x56055a975da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea2d422640 1 -- 172.21.15.77:0/2268353400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a96c240 msgr2=0x56055a9762e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea2d422640 1 --2- 172.21.15.77:0/2268353400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a96c240 0x56055a9762e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.798+0000 7fea2d422640 1 -- 172.21.15.77:0/2268353400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56055a89b100 con 0x56055a9713b0 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.799+0000 7fea2d422640 1 --2- 172.21.15.77:0/2268353400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a9713b0 0x56055a97d320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fea20011d10 tx=0x7fea2000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd38c969417f0ae0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.799+0000 7fea07fff640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.799+0000 7fea07fff640 1 -- 172.21.15.77:0/2268353400 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2000fe60 con 0x56055a9713b0 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.799+0000 7fea07fff640 1 -- 172.21.15.77:0/2268353400 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fea20004b20 con 0x56055a9713b0 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.799+0000 7fea31a7b740 1 -- 172.21.15.77:0/2268353400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea140047a0 con 0x56055a9713b0 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.799+0000 7fea07fff640 1 -- 172.21.15.77:0/2268353400 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fea20004d20 con 0x56055a9713b0 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.799+0000 7fea31a7b740 10 client.25952 fetch_fsmap learned FSMap version 30 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.799+0000 7fea31a7b740 1 -- 172.21.15.77:0/2268353400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56055a5e43f0 con 0x56055a9713b0 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.799+0000 7fea07fff640 1 -- 172.21.15.77:0/2268353400 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea140047a0 con 0x56055a9713b0 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.800+0000 7fea07fff640 1 -- 172.21.15.77:0/2268353400 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fea20019030 con 0x56055a9713b0 2024-03-16T10:19:19.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.800+0000 7fea31a7b740 10 client.25952 fetch_fsmap finished waiting for FSMap version 30 2024-03-16T10:19:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.800+0000 7fea31a7b740 10 client.25952 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:19:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.800+0000 7fea31a7b740 20 client.25952 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.800+0000 7fea31a7b740 1 --2- 172.21.15.77:0/2268353400 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56055a985f30 0x56055a9883c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.800+0000 7fea31a7b740 4 client.25952 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.800+0000 7fea31a7b740 1 -- 172.21.15.77:0/2268353400 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56055a8dd9d0 con 0x56055a985f30 2024-03-16T10:19:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.800+0000 7fea2cc21640 1 --2- 172.21.15.77:0/2268353400 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56055a985f30 0x56055a9883c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.800+0000 7fea07fff640 10 client.25952 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:19:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.800+0000 7fea2cc21640 1 --2- 172.21.15.77:0/2268353400 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56055a985f30 0x56055a9883c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=21ada87d196b50fa server_cookie=eae9d7e091140b7e in_seq=0 out_seq=0 2024-03-16T10:19:19.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.801+0000 7fea07fff640 1 -- 172.21.15.77:0/2268353400 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56055a8dd9d0 con 0x56055a985f30 2024-03-16T10:19:19.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.801+0000 7fea07fff640 10 client.25952 handle_command_reply: tid=0 2024-03-16T10:19:19.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.801+0000 7fea31a7b740 1 client.25952 shutdown 2024-03-16T10:19:19.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.801+0000 7fea31a7b740 1 -- 172.21.15.77:0/2268353400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a9713b0 msgr2=0x56055a97d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.801+0000 7fea31a7b740 1 --2- 172.21.15.77:0/2268353400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a9713b0 0x56055a97d320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fea20011d10 tx=0x7fea2000f790 comp rx=0 tx=0).stop 2024-03-16T10:19:19.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.801+0000 7fea31a7b740 1 -- 172.21.15.77:0/2268353400 shutdown_connections 2024-03-16T10:19:19.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.801+0000 7fea31a7b740 1 --2- 172.21.15.77:0/2268353400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a9713b0 0x56055a97d320 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.801+0000 7fea31a7b740 1 --2- 172.21.15.77:0/2268353400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a96c240 0x56055a9762e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.801+0000 7fea31a7b740 1 --2- 172.21.15.77:0/2268353400 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56055a985f30 0x56055a9883c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.801+0000 7fea31a7b740 1 --2- 172.21.15.77:0/2268353400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96b850 0x56055a975da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.801+0000 7fea31a7b740 1 -- 172.21.15.77:0/2268353400 >> 172.21.15.77:0/2268353400 conn(0x56055a966dc0 msgr2=0x56055a965fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.801+0000 7fea07fff640 0 client.25952 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:19:19.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.802+0000 7fea31a7b740 1 -- 172.21.15.77:0/2268353400 shutdown_connections 2024-03-16T10:19:19.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.802+0000 7fea31a7b740 1 -- 172.21.15.77:0/2268353400 wait complete. 2024-03-16T10:19:19.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.802+0000 7fea31a7b740 20 client.25952 trim_cache size 0 max 16384 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea31a7b740 1 Processor -- start 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea31a7b740 1 -- start start 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea31a7b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a96a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea31a7b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 0x56055a97d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea31a7b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a9713b0 0x56055a98cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea31a7b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fea281a69f0 con 0x56055a9713b0 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea31a7b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fea281a8620 con 0x56055a96b850 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea31a7b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea281a70c0 con 0x56055a96c240 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea2d422640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a9713b0 0x56055a98cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea2cc21640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a96a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea27fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 0x56055a97d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea2cc21640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a96a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50960/0 (socket says 172.21.15.77:50960) 2024-03-16T10:19:19.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea27fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 0x56055a97d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50632/0 (socket says 172.21.15.77:50632) 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea2cc21640 1 -- 172.21.15.77:0/221504062 learned_addr learned my addr 172.21.15.77:0/221504062 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea27fff640 1 -- 172.21.15.77:0/221504062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 msgr2=0x56055a96a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea27fff640 1 --2- 172.21.15.77:0/221504062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a96a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea27fff640 1 -- 172.21.15.77:0/221504062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a9713b0 msgr2=0x56055a98cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.820+0000 7fea27fff640 1 --2- 172.21.15.77:0/221504062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a9713b0 0x56055a98cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.821+0000 7fea27fff640 1 -- 172.21.15.77:0/221504062 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055a6ae7d0 con 0x56055a96c240 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.821+0000 7fea2cc21640 1 --2- 172.21.15.77:0/221504062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a96a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.821+0000 7fea27fff640 1 --2- 172.21.15.77:0/221504062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 0x56055a97d2e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fea1400ead0 tx=0x7fea1400eed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4472b4867bc171a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.821+0000 7fea127fc640 1 -- 172.21.15.77:0/221504062 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea14005030 con 0x56055a96c240 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.821+0000 7fea127fc640 1 -- 172.21.15.77:0/221504062 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea140046e0 con 0x56055a96c240 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.821+0000 7fea127fc640 1 -- 172.21.15.77:0/221504062 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1401d410 con 0x56055a96c240 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.822+0000 7fea31a7b740 1 -- 172.21.15.77:0/221504062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 msgr2=0x56055a97d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.822+0000 7fea31a7b740 1 --2- 172.21.15.77:0/221504062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 0x56055a97d2e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fea1400ead0 tx=0x7fea1400eed0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.822+0000 7fea31a7b740 1 -- 172.21.15.77:0/221504062 shutdown_connections 2024-03-16T10:19:19.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.822+0000 7fea31a7b740 1 --2- 172.21.15.77:0/221504062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a9713b0 0x56055a98cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.822+0000 7fea31a7b740 1 --2- 172.21.15.77:0/221504062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 0x56055a97d2e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.822+0000 7fea31a7b740 1 --2- 172.21.15.77:0/221504062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a96a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.822+0000 7fea31a7b740 1 -- 172.21.15.77:0/221504062 >> 172.21.15.77:0/221504062 conn(0x56055a966dc0 msgr2=0x56055a96a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.822+0000 7fea31a7b740 1 -- 172.21.15.77:0/221504062 shutdown_connections 2024-03-16T10:19:19.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.822+0000 7fea31a7b740 1 -- 172.21.15.77:0/221504062 wait complete. 2024-03-16T10:19:19.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.822+0000 7fea31a7b740 1 -- start start 2024-03-16T10:19:19.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.822+0000 7fea31a7b740 1 Processor -- start 2024-03-16T10:19:19.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea31a7b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a97ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea31a7b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 0x56055a97b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea31a7b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a9713b0 0x56055a97c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea31a7b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fea281a7e90 con 0x56055a9713b0 2024-03-16T10:19:19.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea31a7b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fea281a77c0 con 0x56055a96b850 2024-03-16T10:19:19.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea31a7b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea281a49d0 con 0x56055a96c240 2024-03-16T10:19:19.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea31a7b740 10 client.0 _ll_register_callbacks cb 0x7fea20009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:19.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea2d422640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a9713b0 0x56055a97c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea27fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 0x56055a97b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea27fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 0x56055a97b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50642/0 (socket says 172.21.15.77:50642) 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea27fff640 1 -- 172.21.15.77:0/3993549664 learned_addr learned my addr 172.21.15.77:0/3993549664 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea2cc21640 1 --2- 172.21.15.77:0/3993549664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a97ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea27fff640 1 -- 172.21.15.77:0/3993549664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 msgr2=0x56055a97ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea27fff640 1 --2- 172.21.15.77:0/3993549664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a97ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea27fff640 1 -- 172.21.15.77:0/3993549664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a9713b0 msgr2=0x56055a97c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea27fff640 1 --2- 172.21.15.77:0/3993549664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a9713b0 0x56055a97c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea27fff640 1 -- 172.21.15.77:0/3993549664 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56055a97c670 con 0x56055a96c240 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea27fff640 1 --2- 172.21.15.77:0/3993549664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 0x56055a97b070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea14009a90 tx=0x7fea1400edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2469e840b8a11cd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.823+0000 7fea07fff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea07fff640 1 -- 172.21.15.77:0/3993549664 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea14004aa0 con 0x56055a96c240 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea07fff640 1 -- 172.21.15.77:0/3993549664 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fea14004c40 con 0x56055a96c240 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea31a7b740 1 -- 172.21.15.77:0/3993549664 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea140046e0 con 0x56055a96c240 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea07fff640 1 -- 172.21.15.77:0/3993549664 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fea14005030 con 0x56055a96c240 2024-03-16T10:19:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea31a7b740 10 client.25945 fetch_fsmap learned FSMap version 30 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea31a7b740 1 -- 172.21.15.77:0/3993549664 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56055a9ec3d0 con 0x56055a96c240 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea07fff640 1 -- 172.21.15.77:0/3993549664 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea140046e0 con 0x56055a96c240 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea07fff640 1 -- 172.21.15.77:0/3993549664 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7fea14026a50 con 0x56055a96c240 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea31a7b740 10 client.25945 fetch_fsmap finished waiting for FSMap version 30 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea31a7b740 10 client.25945 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea31a7b740 20 client.25945 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea31a7b740 1 --2- 172.21.15.77:0/3993549664 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56055a9eca20 0x56055a9eee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea31a7b740 4 client.25945 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea31a7b740 1 -- 172.21.15.77:0/3993549664 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56055a8d7d20 con 0x56055a9eca20 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.824+0000 7fea2cc21640 1 --2- 172.21.15.77:0/3993549664 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56055a9eca20 0x56055a9eee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.825+0000 7fea07fff640 10 client.25945 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.825+0000 7fea2cc21640 1 --2- 172.21.15.77:0/3993549664 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56055a9eca20 0x56055a9eee40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ccdce445c0acef37 server_cookie=8431036acaeccf6c in_seq=0 out_seq=0 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea07fff640 1 -- 172.21.15.77:0/3993549664 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56055a8d7d20 con 0x56055a9eca20 2024-03-16T10:19:19.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea07fff640 10 client.25945 handle_command_reply: tid=0 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea31a7b740 1 client.25945 shutdown 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea31a7b740 1 -- 172.21.15.77:0/3993549664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 msgr2=0x56055a97b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea31a7b740 1 --2- 172.21.15.77:0/3993549664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 0x56055a97b070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea14009a90 tx=0x7fea1400edd0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea31a7b740 1 -- 172.21.15.77:0/3993549664 shutdown_connections 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea31a7b740 1 --2- 172.21.15.77:0/3993549664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055a9713b0 0x56055a97c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea31a7b740 1 --2- 172.21.15.77:0/3993549664 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56055a9eca20 0x56055a9eee40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea31a7b740 1 --2- 172.21.15.77:0/3993549664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56055a96c240 0x56055a97b070 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea31a7b740 1 --2- 172.21.15.77:0/3993549664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56055a96b850 0x56055a97ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea07fff640 0 client.25945 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea31a7b740 1 -- 172.21.15.77:0/3993549664 >> 172.21.15.77:0/3993549664 conn(0x56055a966dc0 msgr2=0x56055a96a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea31a7b740 1 -- 172.21.15.77:0/3993549664 shutdown_connections 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea31a7b740 1 -- 172.21.15.77:0/3993549664 wait complete. 2024-03-16T10:19:19.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.826+0000 7fea31a7b740 20 client.25945 trim_cache size 0 max 16384 2024-03-16T10:19:19.833 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:19:19.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.827+0000 7fea2ee85640 1 -- 172.21.15.77:0/1849547647 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fea180a2800 msgr2=0x7fea180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.827+0000 7fea2ee85640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fea180a2800 0x7fea180a4cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fea28199020 tx=0x7fea14005f90 comp rx=0 tx=0).stop 2024-03-16T10:19:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.827+0000 7fea2ee85640 1 -- 172.21.15.77:0/1849547647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea281017f0 msgr2=0x7fea28197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.827+0000 7fea2ee85640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea281017f0 0x7fea28197b10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fea1c000c00 tx=0x7fea1c02c640 comp rx=0 tx=0).stop 2024-03-16T10:19:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.827+0000 7fea2ee85640 1 -- 172.21.15.77:0/1849547647 shutdown_connections 2024-03-16T10:19:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.827+0000 7fea2ee85640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea281033a0 0x7fea2819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.827+0000 7fea2ee85640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fea180a2800 0x7fea180a4cc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.827+0000 7fea2ee85640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea281029e0 0x7fea28198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.827+0000 7fea2ee85640 1 --2- 172.21.15.77:0/1849547647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea281017f0 0x7fea28197b10 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.827+0000 7fea2ee85640 1 -- 172.21.15.77:0/1849547647 >> 172.21.15.77:0/1849547647 conn(0x7fea280fcd40 msgr2=0x7fea28105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.827+0000 7fea2ee85640 1 -- 172.21.15.77:0/1849547647 shutdown_connections 2024-03-16T10:19:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.827+0000 7fea2ee85640 1 -- 172.21.15.77:0/1849547647 wait complete. 2024-03-16T10:19:19.839 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-16T10:19:19.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.950+0000 7f02d04a1640 1 -- 172.21.15.77:0/1882431918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02c81018d0 msgr2=0x7f02c8101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.950+0000 7f02d04a1640 1 --2- 172.21.15.77:0/1882431918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02c81018d0 0x7f02c8101d50 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f02b8009900 tx=0x7f02b802bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:19.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.950+0000 7f02d04a1640 1 -- 172.21.15.77:0/1882431918 shutdown_connections 2024-03-16T10:19:19.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.950+0000 7f02d04a1640 1 --2- 172.21.15.77:0/1882431918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02c8102290 0x7f02c8108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.950+0000 7f02d04a1640 1 --2- 172.21.15.77:0/1882431918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02c81018d0 0x7f02c8101d50 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.950+0000 7f02d04a1640 1 --2- 172.21.15.77:0/1882431918 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81006e0 0x7f02c8100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.950+0000 7f02d04a1640 1 -- 172.21.15.77:0/1882431918 >> 172.21.15.77:0/1882431918 conn(0x7f02c80fbc70 msgr2=0x7f02c80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.950+0000 7f02d04a1640 1 -- 172.21.15.77:0/1882431918 shutdown_connections 2024-03-16T10:19:19.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.950+0000 7f02d04a1640 1 -- 172.21.15.77:0/1882431918 wait complete. 2024-03-16T10:19:19.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02d04a1640 1 Processor -- start 2024-03-16T10:19:19.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02d04a1640 1 -- start start 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02d04a1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02c81006e0 0x7f02c8197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02d04a1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81018d0 0x7f02c8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02d04a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02c8102290 0x7f02c819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02d04a1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f02c810d480 con 0x7f02c8102290 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02d04a1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f02c810d300 con 0x7f02c81018d0 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02d04a1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f02c810d600 con 0x7f02c81006e0 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02cda3c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81018d0 0x7f02c8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02cea3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02c8102290 0x7f02c819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02cda3c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81018d0 0x7f02c8198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50984/0 (socket says 172.21.15.77:50984) 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02cda3c640 1 -- 172.21.15.77:0/3385560670 learned_addr learned my addr 172.21.15.77:0/3385560670 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.951+0000 7f02ce23d640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02c81006e0 0x7f02c8197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02cea3e640 1 -- 172.21.15.77:0/3385560670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02c81006e0 msgr2=0x7f02c8197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02cea3e640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02c81006e0 0x7f02c8197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02cea3e640 1 -- 172.21.15.77:0/3385560670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81018d0 msgr2=0x7f02c8198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02cea3e640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81018d0 0x7f02c8198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02cea3e640 1 -- 172.21.15.77:0/3385560670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c819f730 con 0x7f02c8102290 2024-03-16T10:19:19.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02ce23d640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02c81006e0 0x7f02c8197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02cda3c640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81018d0 0x7f02c8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02cea3e640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02c8102290 0x7f02c819f0c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009fd0 tx=0x7f02c000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02bf7fe640 1 -- 172.21.15.77:0/3385560670 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c000ce20 con 0x7f02c8102290 2024-03-16T10:19:19.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02d04a1640 1 -- 172.21.15.77:0/3385560670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c819fa20 con 0x7f02c8102290 2024-03-16T10:19:19.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02bf7fe640 1 -- 172.21.15.77:0/3385560670 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c0005020 con 0x7f02c8102290 2024-03-16T10:19:19.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02bf7fe640 1 -- 172.21.15.77:0/3385560670 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0010720 con 0x7f02c8102290 2024-03-16T10:19:19.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.952+0000 7f02d04a1640 1 -- 172.21.15.77:0/3385560670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c819ff40 con 0x7f02c8102290 2024-03-16T10:19:19.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.954+0000 7f02bf7fe640 1 -- 172.21.15.77:0/3385560670 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f02c0005540 con 0x7f02c8102290 2024-03-16T10:19:19.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.955+0000 7f02d316f740 1 Processor -- start 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.955+0000 7f02d316f740 1 -- start start 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.955+0000 7f02bf7fe640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f02b40a2800 0x7f02b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.955+0000 7f02d316f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2edc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.955+0000 7f02d316f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2ee240 0x55890c2f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.955+0000 7f02d316f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 0x55890c2f5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.955+0000 7f02d316f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55890c252490 con 0x55890c2f33b0 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.955+0000 7f02d316f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55890c1fac90 con 0x55890c2ed850 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.955+0000 7f02d316f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55890c24e530 con 0x55890c2ee240 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02cda3c640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f02b40a2800 0x7f02b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02ce23d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2edc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02bf7fe640 1 -- 172.21.15.77:0/3385560670 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f02c00cf0d0 con 0x7f02c8102290 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02cea3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 0x55890c2f5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02ce23d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2edc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50998/0 (socket says 172.21.15.77:50998) 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02ce23d640 1 -- 172.21.15.77:0/3742749564 learned_addr learned my addr 172.21.15.77:0/3742749564 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02cda3c640 1 --2- 172.21.15.77:0/3742749564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2ee240 0x55890c2f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02ce23d640 1 -- 172.21.15.77:0/3742749564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2ee240 msgr2=0x55890c2f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02ce23d640 1 --2- 172.21.15.77:0/3742749564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2ee240 0x55890c2f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02ce23d640 1 -- 172.21.15.77:0/3742749564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 msgr2=0x55890c2f5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02ce23d640 1 --2- 172.21.15.77:0/3742749564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 0x55890c2f5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02ce23d640 1 -- 172.21.15.77:0/3742749564 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55890c2b5500 con 0x55890c2ed850 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02cea3e640 1 --2- 172.21.15.77:0/3742749564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 0x55890c2f5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02cda3c640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f02b40a2800 0x7f02b40a4cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f02b8000c00 tx=0x7f02b80042a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02ce23d640 1 --2- 172.21.15.77:0/3742749564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2edc70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02c40114c0 tx=0x7f02c4011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b4549209674cd3b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02bcbfe640 1 -- 172.21.15.77:0/3742749564 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4004020 con 0x55890c2ed850 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.956+0000 7f02bcbfe640 1 -- 172.21.15.77:0/3742749564 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c4005500 con 0x55890c2ed850 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.957+0000 7f02bcbfe640 1 -- 172.21.15.77:0/3742749564 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4004d40 con 0x55890c2ed850 2024-03-16T10:19:19.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.957+0000 7f02d316f740 1 -- 172.21.15.77:0/3742749564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 msgr2=0x55890c2edc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.957+0000 7f02d316f740 1 --2- 172.21.15.77:0/3742749564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2edc70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02c40114c0 tx=0x7f02c4011980 comp rx=0 tx=0).stop 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 -- 172.21.15.77:0/3742749564 shutdown_connections 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 --2- 172.21.15.77:0/3742749564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 0x55890c2f5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 --2- 172.21.15.77:0/3742749564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2ee240 0x55890c2f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 --2- 172.21.15.77:0/3742749564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2edc70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 -- 172.21.15.77:0/3742749564 >> 172.21.15.77:0/3742749564 conn(0x55890c2e8dc0 msgr2=0x55890c2e91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 -- 172.21.15.77:0/3742749564 shutdown_connections 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 -- 172.21.15.77:0/3742749564 wait complete. 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 -- start start 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 Processor -- start 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2ee240 0x55890c2f8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 0x55890c2ff270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55890c2f6360 con 0x55890c2f33b0 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55890c20e230 con 0x55890c2ed850 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55890c2f64e0 con 0x55890c2ee240 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02d316f740 10 client.0 _ll_register_callbacks cb 0x55890c24a010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02cea3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 0x55890c2ff270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02cea3e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 0x55890c2ff270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60750/0 (socket says 172.21.15.77:60750) 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02cea3e640 1 -- 172.21.15.77:0/2387690395 learned_addr learned my addr 172.21.15.77:0/2387690395 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02ce23d640 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.958+0000 7f02cda3c640 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2ee240 0x55890c2f8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02cea3e640 1 -- 172.21.15.77:0/2387690395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2ee240 msgr2=0x55890c2f8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02cea3e640 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2ee240 0x55890c2f8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02cea3e640 1 -- 172.21.15.77:0/2387690395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 msgr2=0x55890c2f7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02cea3e640 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2f7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02cea3e640 1 -- 172.21.15.77:0/2387690395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55890c21d100 con 0x55890c2f33b0 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02cda3c640 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2ee240 0x55890c2f8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02cea3e640 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 0x55890c2ff270 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f02c001b000 tx=0x7f02c009c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d31e08736520655 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02b1ffb640 1 -- 172.21.15.77:0/2387690395 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c009fc30 con 0x55890c2f33b0 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02b1ffb640 1 -- 172.21.15.77:0/2387690395 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f02c00051d0 con 0x55890c2f33b0 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02d316f740 1 -- 172.21.15.77:0/2387690395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02c4005500 con 0x55890c2f33b0 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02b1ffb640 1 -- 172.21.15.77:0/2387690395 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f02c00a4280 con 0x55890c2f33b0 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02d316f740 10 client.16185 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02d316f740 1 -- 172.21.15.77:0/2387690395 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55890bf663f0 con 0x55890c2f33b0 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02b1ffb640 1 -- 172.21.15.77:0/2387690395 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c4005500 con 0x55890c2f33b0 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02b1ffb640 1 -- 172.21.15.77:0/2387690395 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f02c00a3e80 con 0x55890c2f33b0 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02d316f740 10 client.16185 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02d316f740 10 client.16185 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02d316f740 20 client.16185 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02d316f740 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55890c303b80 0x55890c305fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02d316f740 4 client.16185 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:19.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.959+0000 7f02d316f740 1 -- 172.21.15.77:0/2387690395 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55890c25f9d0 con 0x55890c303b80 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.960+0000 7f02ce23d640 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55890c303b80 0x55890c305fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.960+0000 7f02b1ffb640 10 client.16185 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.960+0000 7f02ce23d640 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55890c303b80 0x55890c305fe0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=423c20abea1a7e9c server_cookie=a868997ab4c8c6d8 in_seq=0 out_seq=0 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02b1ffb640 1 -- 172.21.15.77:0/2387690395 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55890c25f9d0 con 0x55890c303b80 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02b1ffb640 10 client.16185 handle_command_reply: tid=0 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02d316f740 1 client.16185 shutdown 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02d316f740 1 -- 172.21.15.77:0/2387690395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 msgr2=0x55890c2ff270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02d316f740 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 0x55890c2ff270 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f02c001b000 tx=0x7f02c009c2f0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02d316f740 1 -- 172.21.15.77:0/2387690395 shutdown_connections 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02d316f740 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55890c303b80 0x55890c305fe0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02d316f740 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2f33b0 0x55890c2ff270 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02d316f740 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2ee240 0x55890c2f8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02d316f740 1 --2- 172.21.15.77:0/2387690395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2f7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02d316f740 1 -- 172.21.15.77:0/2387690395 >> 172.21.15.77:0/2387690395 conn(0x55890c2e8dc0 msgr2=0x55890c2e7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02b1ffb640 0 client.16185 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:19:19.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02d316f740 1 -- 172.21.15.77:0/2387690395 shutdown_connections 2024-03-16T10:19:19.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02d316f740 1 -- 172.21.15.77:0/2387690395 wait complete. 2024-03-16T10:19:19.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.961+0000 7f02d316f740 20 client.16185 trim_cache size 0 max 16384 2024-03-16T10:19:19.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02d316f740 1 Processor -- start 2024-03-16T10:19:19.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02d316f740 1 -- start start 2024-03-16T10:19:19.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02d316f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2e7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02d316f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ee240 0x55890c2ff230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02d316f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2f33b0 0x55890c2e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02d316f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81a8aa0 con 0x55890c2ee240 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02d316f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81aa700 con 0x55890c2ed850 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02d316f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81a9170 con 0x55890c2f33b0 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02ce23d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2e7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02cda3c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ee240 0x55890c2ff230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02cea3e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2f33b0 0x55890c2e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02ce23d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2e7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:51026/0 (socket says 172.21.15.77:51026) 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02cda3c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ee240 0x55890c2ff230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60752/0 (socket says 172.21.15.77:60752) 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.975+0000 7f02ce23d640 1 -- 172.21.15.77:0/1417141498 learned_addr learned my addr 172.21.15.77:0/1417141498 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.976+0000 7f02cea3e640 1 -- 172.21.15.77:0/1417141498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 msgr2=0x55890c2e7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.976+0000 7f02cea3e640 1 --2- 172.21.15.77:0/1417141498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2e7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.976+0000 7f02cea3e640 1 -- 172.21.15.77:0/1417141498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ee240 msgr2=0x55890c2ff230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.976+0000 7f02cea3e640 1 --2- 172.21.15.77:0/1417141498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ee240 0x55890c2ff230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.976+0000 7f02cea3e640 1 -- 172.21.15.77:0/1417141498 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55890c0307d0 con 0x55890c2f33b0 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.976+0000 7f02ce23d640 1 --2- 172.21.15.77:0/1417141498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2e7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.976+0000 7f02cea3e640 1 --2- 172.21.15.77:0/1417141498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2f33b0 0x55890c2e7c90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f02c0005fd0 tx=0x7f02c0097eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bed28d3be7255845 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.976+0000 7f02b3fff640 1 -- 172.21.15.77:0/1417141498 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c009b8b0 con 0x55890c2f33b0 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.976+0000 7f02b3fff640 1 -- 172.21.15.77:0/1417141498 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c009afd0 con 0x55890c2f33b0 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.976+0000 7f02b3fff640 1 -- 172.21.15.77:0/1417141498 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c009a9a0 con 0x55890c2f33b0 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.977+0000 7f02d316f740 1 -- 172.21.15.77:0/1417141498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2f33b0 msgr2=0x55890c2e7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.977+0000 7f02d316f740 1 --2- 172.21.15.77:0/1417141498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2f33b0 0x55890c2e7c90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f02c0005fd0 tx=0x7f02c0097eb0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.977+0000 7f02d316f740 1 -- 172.21.15.77:0/1417141498 shutdown_connections 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.977+0000 7f02d316f740 1 --2- 172.21.15.77:0/1417141498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2f33b0 0x55890c2e7c90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.977+0000 7f02d316f740 1 --2- 172.21.15.77:0/1417141498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ee240 0x55890c2ff230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.977+0000 7f02d316f740 1 --2- 172.21.15.77:0/1417141498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ed850 0x55890c2e7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.977+0000 7f02d316f740 1 -- 172.21.15.77:0/1417141498 >> 172.21.15.77:0/1417141498 conn(0x55890c2e8dc0 msgr2=0x55890c2ef6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.977+0000 7f02d316f740 1 -- 172.21.15.77:0/1417141498 shutdown_connections 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.977+0000 7f02d316f740 1 -- 172.21.15.77:0/1417141498 wait complete. 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.977+0000 7f02d316f740 1 -- start start 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.977+0000 7f02d316f740 1 Processor -- start 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02d316f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ed850 0x55890c2fc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02d316f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ee240 0x55890c2fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02d316f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2f33b0 0x55890c2fdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02d316f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81a9f70 con 0x55890c2ed850 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02d316f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81a98a0 con 0x55890c2ee240 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02d316f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81a6aa0 con 0x55890c2f33b0 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02d316f740 10 client.0 _ll_register_callbacks cb 0x7f02c009b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02ce23d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ed850 0x55890c2fc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02cea3e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2f33b0 0x55890c2fdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02cda3c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ee240 0x55890c2fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02ce23d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ed850 0x55890c2fc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60762/0 (socket says 172.21.15.77:60762) 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02ce23d640 1 -- 172.21.15.77:0/1939188001 learned_addr learned my addr 172.21.15.77:0/1939188001 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02cea3e640 1 -- 172.21.15.77:0/1939188001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ee240 msgr2=0x55890c2fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02cea3e640 1 --2- 172.21.15.77:0/1939188001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ee240 0x55890c2fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02cea3e640 1 -- 172.21.15.77:0/1939188001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ed850 msgr2=0x55890c2fc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02cea3e640 1 --2- 172.21.15.77:0/1939188001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ed850 0x55890c2fc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02cea3e640 1 -- 172.21.15.77:0/1939188001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55890c2fe1e0 con 0x55890c2f33b0 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02ce23d640 1 --2- 172.21.15.77:0/1939188001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ed850 0x55890c2fc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02cea3e640 1 --2- 172.21.15.77:0/1939188001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2f33b0 0x55890c2fdbf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f02c00984b0 tx=0x7f02c00ccc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a0f20eac63927e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.978+0000 7f02b1ffb640 1 -- 172.21.15.77:0/1939188001 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c009b8b0 con 0x55890c2f33b0 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02b1ffb640 1 -- 172.21.15.77:0/1939188001 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f02c00b4da0 con 0x55890c2f33b0 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02d316f740 1 -- 172.21.15.77:0/1939188001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02c009afd0 con 0x55890c2f33b0 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02b1ffb640 1 -- 172.21.15.77:0/1939188001 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f02c0097830 con 0x55890c2f33b0 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02d316f740 10 client.25963 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:19.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02d316f740 1 -- 172.21.15.77:0/1939188001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55890c2ff4e0 con 0x55890c2f33b0 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02b1ffb640 1 -- 172.21.15.77:0/1939188001 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c009afd0 con 0x55890c2f33b0 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02b1ffb640 1 -- 172.21.15.77:0/1939188001 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f02c0004a40 con 0x55890c2f33b0 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02d316f740 10 client.25963 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02d316f740 10 client.25963 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02d316f740 20 client.25963 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02d316f740 1 --2- 172.21.15.77:0/1939188001 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55890c36a310 0x55890c36c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02d316f740 4 client.25963 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02d316f740 1 -- 172.21.15.77:0/1939188001 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55890c259d20 con 0x55890c36a310 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.979+0000 7f02ce23d640 1 --2- 172.21.15.77:0/1939188001 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55890c36a310 0x55890c36c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02b1ffb640 10 client.25963 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02ce23d640 1 --2- 172.21.15.77:0/1939188001 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55890c36a310 0x55890c36c730 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1a6b247776a20dcb server_cookie=3078b64e482f9336 in_seq=0 out_seq=0 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02b1ffb640 1 -- 172.21.15.77:0/1939188001 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55890c259d20 con 0x55890c36a310 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02b1ffb640 10 client.25963 handle_command_reply: tid=0 2024-03-16T10:19:19.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02d316f740 1 client.25963 shutdown 2024-03-16T10:19:19.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02d316f740 1 -- 172.21.15.77:0/1939188001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2f33b0 msgr2=0x55890c2fdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02d316f740 1 --2- 172.21.15.77:0/1939188001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2f33b0 0x55890c2fdbf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f02c00984b0 tx=0x7f02c00ccc00 comp rx=0 tx=0).stop 2024-03-16T10:19:19.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02d316f740 1 -- 172.21.15.77:0/1939188001 shutdown_connections 2024-03-16T10:19:19.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02d316f740 1 --2- 172.21.15.77:0/1939188001 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55890c36a310 0x55890c36c730 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02d316f740 1 --2- 172.21.15.77:0/1939188001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55890c2f33b0 0x55890c2fdbf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02d316f740 1 --2- 172.21.15.77:0/1939188001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55890c2ee240 0x55890c2fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02b1ffb640 0 client.25963 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:19:19.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02d316f740 1 --2- 172.21.15.77:0/1939188001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55890c2ed850 0x55890c2fc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02d316f740 1 -- 172.21.15.77:0/1939188001 >> 172.21.15.77:0/1939188001 conn(0x55890c2e8dc0 msgr2=0x55890c2ec020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02d316f740 1 -- 172.21.15.77:0/1939188001 shutdown_connections 2024-03-16T10:19:19.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02d316f740 1 -- 172.21.15.77:0/1939188001 wait complete. 2024-03-16T10:19:19.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.980+0000 7f02d316f740 20 client.25963 trim_cache size 0 max 16384 2024-03-16T10:19:19.988 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:19:19.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.981+0000 7f02d04a1640 1 -- 172.21.15.77:0/3385560670 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f02b40a2800 msgr2=0x7f02b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.981+0000 7f02d04a1640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f02b40a2800 0x7f02b40a4cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f02b8000c00 tx=0x7f02b80042a0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.981+0000 7f02d04a1640 1 -- 172.21.15.77:0/3385560670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02c8102290 msgr2=0x7f02c819f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:19.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.981+0000 7f02d04a1640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02c8102290 0x7f02c819f0c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009fd0 tx=0x7f02c000ef10 comp rx=0 tx=0).stop 2024-03-16T10:19:19.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.981+0000 7f02d04a1640 1 -- 172.21.15.77:0/3385560670 shutdown_connections 2024-03-16T10:19:19.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.981+0000 7f02d04a1640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02c8102290 0x7f02c819f0c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.981+0000 7f02d04a1640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81018d0 0x7f02c8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.981+0000 7f02d04a1640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f02b40a2800 0x7f02b40a4cc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.981+0000 7f02d04a1640 1 --2- 172.21.15.77:0/3385560670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02c81006e0 0x7f02c8197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:19.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.981+0000 7f02d04a1640 1 -- 172.21.15.77:0/3385560670 >> 172.21.15.77:0/3385560670 conn(0x7f02c80fbc70 msgr2=0x7f02c8104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:19.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.981+0000 7f02d04a1640 1 -- 172.21.15.77:0/3385560670 shutdown_connections 2024-03-16T10:19:19.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:19.981+0000 7f02d04a1640 1 -- 172.21.15.77:0/3385560670 wait complete. 2024-03-16T10:19:19.991 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:19 smithi077 ceph-mon[32400]: pgmap v417: 97 pgs: 97 active+clean; 134 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.6 MiB/s wr, 471 op/s 2024-03-16T10:19:19.991 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:19 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1228461198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:19:19.992 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T10:19:20.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.102+0000 7f69d9e59640 1 -- 172.21.15.77:0/2467719688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d41033a0 msgr2=0x7f69d4109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.102+0000 7f69d9e59640 1 --2- 172.21.15.77:0/2467719688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d41033a0 0x7f69d4109c30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f69cc009ca0 tx=0x7f69cc02c0f0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.102+0000 7f69d9e59640 1 -- 172.21.15.77:0/2467719688 shutdown_connections 2024-03-16T10:19:20.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.102+0000 7f69d9e59640 1 --2- 172.21.15.77:0/2467719688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d41033a0 0x7f69d4109c30 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.102+0000 7f69d9e59640 1 --2- 172.21.15.77:0/2467719688 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69d41029e0 0x7f69d4102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.102+0000 7f69d9e59640 1 --2- 172.21.15.77:0/2467719688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69d41017f0 0x7f69d4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.102+0000 7f69d9e59640 1 -- 172.21.15.77:0/2467719688 >> 172.21.15.77:0/2467719688 conn(0x7f69d40fcd80 msgr2=0x7f69d40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.102+0000 7f69d9e59640 1 -- 172.21.15.77:0/2467719688 shutdown_connections 2024-03-16T10:19:20.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.102+0000 7f69d9e59640 1 -- 172.21.15.77:0/2467719688 wait complete. 2024-03-16T10:19:20.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.102+0000 7f69d9e59640 1 Processor -- start 2024-03-16T10:19:20.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d9e59640 1 -- start start 2024-03-16T10:19:20.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d9e59640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69d41017f0 0x7f69d4197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d9e59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d41029e0 0x7f69d4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d9e59640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69d41033a0 0x7f69d419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d9e59640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f69d410d470 con 0x7f69d41029e0 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d9e59640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f69d410d2f0 con 0x7f69d41017f0 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d9e59640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f69d410d5f0 con 0x7f69d41033a0 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d41029e0 0x7f69d4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d41029e0 0x7f69d4197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60792/0 (socket says 172.21.15.77:60792) 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d2ffd640 1 -- 172.21.15.77:0/3444965441 learned_addr learned my addr 172.21.15.77:0/3444965441 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69d41033a0 0x7f69d419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d37fe640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69d41017f0 0x7f69d4197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d3fff640 1 -- 172.21.15.77:0/3444965441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69d41017f0 msgr2=0x7f69d4197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d3fff640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69d41017f0 0x7f69d4197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d3fff640 1 -- 172.21.15.77:0/3444965441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d41029e0 msgr2=0x7f69d4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d3fff640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d41029e0 0x7f69d4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d3fff640 1 -- 172.21.15.77:0/3444965441 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d419f6d0 con 0x7f69d41033a0 2024-03-16T10:19:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.103+0000 7f69d3fff640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69d41033a0 0x7f69d419f060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f69cc02c630 tx=0x7f69cc032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.104+0000 7f69d0ff9640 1 -- 172.21.15.77:0/3444965441 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc041020 con 0x7f69d41033a0 2024-03-16T10:19:20.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.104+0000 7f69d9e59640 1 -- 172.21.15.77:0/3444965441 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d419f960 con 0x7f69d41033a0 2024-03-16T10:19:20.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.104+0000 7f69d0ff9640 1 -- 172.21.15.77:0/3444965441 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69cc034cd0 con 0x7f69d41033a0 2024-03-16T10:19:20.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.104+0000 7f69d0ff9640 1 -- 172.21.15.77:0/3444965441 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc039040 con 0x7f69d41033a0 2024-03-16T10:19:20.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.104+0000 7f69d9e59640 1 -- 172.21.15.77:0/3444965441 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d419fe70 con 0x7f69d41033a0 2024-03-16T10:19:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.106+0000 7f69d0ff9640 1 -- 172.21.15.77:0/3444965441 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f69cc044020 con 0x7f69d41033a0 2024-03-16T10:19:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.106+0000 7f69d0ff9640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f69bc0a2800 0x7f69bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.106+0000 7f69dca44740 1 Processor -- start 2024-03-16T10:19:20.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.106+0000 7f69dca44740 1 -- start start 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.106+0000 7f69d37fe640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f69bc0a2800 0x7f69bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.106+0000 7f69d0ff9640 1 -- 172.21.15.77:0/3444965441 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f69cc0f0700 con 0x7f69d41033a0 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69dca44740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd63850 0x5629acd63c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69dca44740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd68d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69dca44740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd693b0 0x5629acd6b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69dca44740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5629accc8490 con 0x5629acd693b0 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69dca44740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5629acc70c90 con 0x5629acd63850 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69dca44740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5629accc4530 con 0x5629acd64240 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d37fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd63850 0x5629acd63c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d2ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd68d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd693b0 0x5629acd6b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d37fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd63850 0x5629acd63c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:51066/0 (socket says 172.21.15.77:51066) 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d2ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd68d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50730/0 (socket says 172.21.15.77:50730) 2024-03-16T10:19:20.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd693b0 0x5629acd6b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60800/0 (socket says 172.21.15.77:60800) 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d37fe640 1 -- 172.21.15.77:0/207956710 learned_addr learned my addr 172.21.15.77:0/207956710 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d37fe640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f69bc0a2800 0x7f69bc0a4cc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f69d4102840 tx=0x7f69c4011440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d37fe640 1 -- 172.21.15.77:0/207956710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 msgr2=0x5629acd68d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d37fe640 1 --2- 172.21.15.77:0/207956710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd68d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d37fe640 1 -- 172.21.15.77:0/207956710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd693b0 msgr2=0x5629acd6b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d37fe640 1 --2- 172.21.15.77:0/207956710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd693b0 0x5629acd6b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d37fe640 1 -- 172.21.15.77:0/207956710 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629acd2b500 con 0x5629acd63850 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/207956710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd68d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.107+0000 7f69d37fe640 1 --2- 172.21.15.77:0/207956710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd63850 0x5629acd63c70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f69c401e790 tx=0x7f69c4020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=909d4bae4b348bd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.108+0000 7f69b9ffb640 1 -- 172.21.15.77:0/207956710 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c4022040 con 0x5629acd63850 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.108+0000 7f69b9ffb640 1 -- 172.21.15.77:0/207956710 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69c402b070 con 0x5629acd63850 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.108+0000 7f69b9ffb640 1 -- 172.21.15.77:0/207956710 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c4017040 con 0x5629acd63850 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.108+0000 7f69dca44740 1 -- 172.21.15.77:0/207956710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd63850 msgr2=0x5629acd63c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.108+0000 7f69dca44740 1 --2- 172.21.15.77:0/207956710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd63850 0x5629acd63c70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f69c401e790 tx=0x7f69c4020000 comp rx=0 tx=0).stop 2024-03-16T10:19:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.108+0000 7f69dca44740 1 -- 172.21.15.77:0/207956710 shutdown_connections 2024-03-16T10:19:20.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.108+0000 7f69dca44740 1 --2- 172.21.15.77:0/207956710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd693b0 0x5629acd6b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.108+0000 7f69dca44740 1 --2- 172.21.15.77:0/207956710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd68d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.108+0000 7f69dca44740 1 --2- 172.21.15.77:0/207956710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd63850 0x5629acd63c70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.108+0000 7f69dca44740 1 -- 172.21.15.77:0/207956710 >> 172.21.15.77:0/207956710 conn(0x5629acd5edc0 msgr2=0x5629acd5f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.108+0000 7f69dca44740 1 -- 172.21.15.77:0/207956710 shutdown_connections 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.108+0000 7f69dca44740 1 -- 172.21.15.77:0/207956710 wait complete. 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69dca44740 1 -- start start 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69dca44740 1 Processor -- start 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69dca44740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd63850 0x5629acd6dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69dca44740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd64240 0x5629acd6e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69dca44740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acd75290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69dca44740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5629acd6c300 con 0x5629acd64240 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69dca44740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5629acc84230 con 0x5629acd693b0 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69dca44740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5629acd6c480 con 0x5629acd63850 2024-03-16T10:19:20.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69dca44740 10 client.0 _ll_register_callbacks cb 0x5629accc0010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69d2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd64240 0x5629acd6e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69d37fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd63850 0x5629acd6dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69d3fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acd75290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69d2ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd64240 0x5629acd6e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60814/0 (socket says 172.21.15.77:60814) 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69d2ffd640 1 -- 172.21.15.77:0/2901161182 learned_addr learned my addr 172.21.15.77:0/2901161182 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69d37fe640 1 -- 172.21.15.77:0/2901161182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 msgr2=0x5629acd75290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69d37fe640 1 --2- 172.21.15.77:0/2901161182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acd75290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.109+0000 7f69d37fe640 1 -- 172.21.15.77:0/2901161182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd64240 msgr2=0x5629acd6e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.110+0000 7f69d37fe640 1 --2- 172.21.15.77:0/2901161182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd64240 0x5629acd6e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.110+0000 7f69d37fe640 1 -- 172.21.15.77:0/2901161182 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629acc93100 con 0x5629acd63850 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.110+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/2901161182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd64240 0x5629acd6e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.110+0000 7f69d37fe640 1 --2- 172.21.15.77:0/2901161182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd63850 0x5629acd6dd50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f69c401e760 tx=0x7f69c4027f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb2ede26373d09e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.110+0000 7f69b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:19:20.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.110+0000 7f69b2ffd640 1 -- 172.21.15.77:0/2901161182 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c4022050 con 0x5629acd63850 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.110+0000 7f69b2ffd640 1 -- 172.21.15.77:0/2901161182 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f69c4005040 con 0x5629acd63850 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.110+0000 7f69dca44740 1 -- 172.21.15.77:0/2901161182 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69c402b070 con 0x5629acd63850 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.110+0000 7f69b2ffd640 1 -- 172.21.15.77:0/2901161182 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f69c4017040 con 0x5629acd63850 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.110+0000 7f69dca44740 10 client.25975 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.110+0000 7f69dca44740 1 -- 172.21.15.77:0/2901161182 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5629ac9dc3f0 con 0x5629acd63850 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.110+0000 7f69b2ffd640 1 -- 172.21.15.77:0/2901161182 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69c402b070 con 0x5629acd63850 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.111+0000 7f69b2ffd640 1 -- 172.21.15.77:0/2901161182 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f69c4025050 con 0x5629acd63850 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.111+0000 7f69dca44740 10 client.25975 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.111+0000 7f69dca44740 10 client.25975 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.111+0000 7f69dca44740 20 client.25975 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.111+0000 7f69dca44740 1 --2- 172.21.15.77:0/2901161182 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5629acd79910 0x5629acd7bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.111+0000 7f69dca44740 4 client.25975 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.111+0000 7f69dca44740 1 -- 172.21.15.77:0/2901161182 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629accd59d0 con 0x5629acd79910 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.111+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/2901161182 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5629acd79910 0x5629acd7bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.112+0000 7f69b2ffd640 10 client.25975 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:19:20.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.112+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/2901161182 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5629acd79910 0x5629acd7bd00 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5beb3bdbf1a2467a server_cookie=bb562db5c05025f6 in_seq=0 out_seq=0 2024-03-16T10:19:20.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.122+0000 7f69b2ffd640 1 -- 172.21.15.77:0/2901161182 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5629accd59d0 con 0x5629acd79910 2024-03-16T10:19:20.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.122+0000 7f69b2ffd640 10 client.25975 handle_command_reply: tid=0 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.122+0000 7f69dca44740 1 client.25975 shutdown 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.123+0000 7f69dca44740 1 -- 172.21.15.77:0/2901161182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd63850 msgr2=0x5629acd6dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.123+0000 7f69dca44740 1 --2- 172.21.15.77:0/2901161182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd63850 0x5629acd6dd50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f69c401e760 tx=0x7f69c4027f00 comp rx=0 tx=0).stop 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.123+0000 7f69dca44740 1 -- 172.21.15.77:0/2901161182 shutdown_connections 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.123+0000 7f69dca44740 1 --2- 172.21.15.77:0/2901161182 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5629acd79910 0x5629acd7bd00 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.123+0000 7f69dca44740 1 --2- 172.21.15.77:0/2901161182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acd75290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.123+0000 7f69dca44740 1 --2- 172.21.15.77:0/2901161182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd64240 0x5629acd6e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.123+0000 7f69dca44740 1 --2- 172.21.15.77:0/2901161182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd63850 0x5629acd6dd50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.123+0000 7f69b2ffd640 0 client.25975 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.123+0000 7f69dca44740 1 -- 172.21.15.77:0/2901161182 >> 172.21.15.77:0/2901161182 conn(0x5629acd5edc0 msgr2=0x5629acd5dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.123+0000 7f69dca44740 1 -- 172.21.15.77:0/2901161182 shutdown_connections 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.123+0000 7f69dca44740 1 -- 172.21.15.77:0/2901161182 wait complete. 2024-03-16T10:19:20.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.123+0000 7f69dca44740 20 client.25975 trim_cache size 0 max 16384 2024-03-16T10:19:20.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69dca44740 1 Processor -- start 2024-03-16T10:19:20.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69dca44740 1 -- start start 2024-03-16T10:19:20.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69dca44740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 0x5629acd6dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69dca44740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd75250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69dca44740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acd6e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69dca44740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f69d41a69a0 con 0x5629acd63850 2024-03-16T10:19:20.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69dca44740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f69d41a8600 con 0x5629acd693b0 2024-03-16T10:19:20.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69dca44740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f69d41a7070 con 0x5629acd64240 2024-03-16T10:19:20.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 0x5629acd6dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 0x5629acd6dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60830/0 (socket says 172.21.15.77:60830) 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69d37fe640 1 -- 172.21.15.77:0/1235121533 learned_addr learned my addr 172.21.15.77:0/1235121533 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/1235121533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd75250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69d3fff640 1 --2- 172.21.15.77:0/1235121533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acd6e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69d37fe640 1 -- 172.21.15.77:0/1235121533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 msgr2=0x5629acd75250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69d37fe640 1 --2- 172.21.15.77:0/1235121533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd75250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69d37fe640 1 -- 172.21.15.77:0/1235121533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 msgr2=0x5629acd6e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69d37fe640 1 --2- 172.21.15.77:0/1235121533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acd6e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69d37fe640 1 -- 172.21.15.77:0/1235121533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629acd6cdf0 con 0x5629acd63850 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69d3fff640 1 --2- 172.21.15.77:0/1235121533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acd6e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.137+0000 7f69d37fe640 1 --2- 172.21.15.77:0/1235121533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 0x5629acd6dcf0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f69c401efd0 tx=0x7f69c4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b4d51a9b5255fdd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.138+0000 7f69b97fa640 1 -- 172.21.15.77:0/1235121533 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c4015070 con 0x5629acd63850 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.138+0000 7f69b97fa640 1 -- 172.21.15.77:0/1235121533 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69c402b070 con 0x5629acd63850 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.138+0000 7f69b97fa640 1 -- 172.21.15.77:0/1235121533 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c4022040 con 0x5629acd63850 2024-03-16T10:19:20.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.138+0000 7f69dca44740 1 -- 172.21.15.77:0/1235121533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 msgr2=0x5629acd6dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.138+0000 7f69dca44740 1 --2- 172.21.15.77:0/1235121533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 0x5629acd6dcf0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f69c401efd0 tx=0x7f69c4005fd0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.138+0000 7f69dca44740 1 -- 172.21.15.77:0/1235121533 shutdown_connections 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.138+0000 7f69dca44740 1 --2- 172.21.15.77:0/1235121533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acd6e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.138+0000 7f69dca44740 1 --2- 172.21.15.77:0/1235121533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd75250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.138+0000 7f69dca44740 1 --2- 172.21.15.77:0/1235121533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 0x5629acd6dcf0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.138+0000 7f69dca44740 1 -- 172.21.15.77:0/1235121533 >> 172.21.15.77:0/1235121533 conn(0x5629acd5edc0 msgr2=0x5629acd5f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.138+0000 7f69dca44740 1 -- 172.21.15.77:0/1235121533 shutdown_connections 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.138+0000 7f69dca44740 1 -- 172.21.15.77:0/1235121533 wait complete. 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69dca44740 1 -- start start 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69dca44740 1 Processor -- start 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69dca44740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 0x5629acd71dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69dca44740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd72310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69dca44740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acdddb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69dca44740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f69d41a7e70 con 0x5629acd63850 2024-03-16T10:19:20.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69dca44740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f69d41a77a0 con 0x5629acd693b0 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69dca44740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f69d41a4980 con 0x5629acd64240 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69dca44740 10 client.0 _ll_register_callbacks cb 0x7f69c8009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 0x5629acd71dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 0x5629acd71dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60838/0 (socket says 172.21.15.77:60838) 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69d37fe640 1 -- 172.21.15.77:0/1172423167 learned_addr learned my addr 172.21.15.77:0/1172423167 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd72310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69d3fff640 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acdddb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69d37fe640 1 -- 172.21.15.77:0/1172423167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 msgr2=0x5629acd72310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69d37fe640 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd72310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69d37fe640 1 -- 172.21.15.77:0/1172423167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 msgr2=0x5629acdddb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69d37fe640 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acdddb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.139+0000 7f69d37fe640 1 -- 172.21.15.77:0/1172423167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629acaa67d0 con 0x5629acd63850 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69d37fe640 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 0x5629acd71dd0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f69c4011940 tx=0x7f69c40042b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64992e787710c7c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:20.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69b2ffd640 1 -- 172.21.15.77:0/1172423167 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c4015070 con 0x5629acd63850 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69b2ffd640 1 -- 172.21.15.77:0/1172423167 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f69c40048f0 con 0x5629acd63850 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69dca44740 1 -- 172.21.15.77:0/1172423167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69c402b070 con 0x5629acd63850 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69b2ffd640 1 -- 172.21.15.77:0/1172423167 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f69c4022040 con 0x5629acd63850 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69dca44740 10 client.16209 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69dca44740 1 -- 172.21.15.77:0/1172423167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5629acdded50 con 0x5629acd63850 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69b2ffd640 1 -- 172.21.15.77:0/1172423167 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69c402b070 con 0x5629acd63850 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69b2ffd640 1 -- 172.21.15.77:0/1172423167 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f69c4025040 con 0x5629acd63850 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69dca44740 10 client.16209 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69dca44740 10 client.16209 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69dca44740 20 client.16209 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69dca44740 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5629acddf6a0 0x5629acde1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69dca44740 4 client.16209 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69dca44740 1 -- 172.21.15.77:0/1172423167 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5629acccfd20 con 0x5629acddf6a0 2024-03-16T10:19:20.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.140+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5629acddf6a0 0x5629acde1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.141+0000 7f69d2ffd640 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5629acddf6a0 0x5629acde1ac0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=46e1fd3521e69fa0 server_cookie=70492f21e847c140 in_seq=0 out_seq=0 2024-03-16T10:19:20.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.141+0000 7f69b2ffd640 10 client.16209 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:19:20.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69b2ffd640 1 -- 172.21.15.77:0/1172423167 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5629acccfd20 con 0x5629acddf6a0 2024-03-16T10:19:20.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69b2ffd640 10 client.16209 handle_command_reply: tid=0 2024-03-16T10:19:20.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69dca44740 1 client.16209 shutdown 2024-03-16T10:19:20.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69dca44740 1 -- 172.21.15.77:0/1172423167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 msgr2=0x5629acd71dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69dca44740 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 0x5629acd71dd0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f69c4011940 tx=0x7f69c40042b0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69dca44740 1 -- 172.21.15.77:0/1172423167 shutdown_connections 2024-03-16T10:19:20.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69dca44740 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5629acddf6a0 0x5629acde1ac0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69dca44740 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629acd693b0 0x5629acdddb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69dca44740 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629acd64240 0x5629acd72310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69b2ffd640 0 client.16209 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:19:20.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69dca44740 1 --2- 172.21.15.77:0/1172423167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5629acd63850 0x5629acd71dd0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69dca44740 1 -- 172.21.15.77:0/1172423167 >> 172.21.15.77:0/1172423167 conn(0x5629acd5edc0 msgr2=0x5629acd6dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69dca44740 1 -- 172.21.15.77:0/1172423167 shutdown_connections 2024-03-16T10:19:20.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69dca44740 1 -- 172.21.15.77:0/1172423167 wait complete. 2024-03-16T10:19:20.152 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.150+0000 7f69dca44740 20 client.16209 trim_cache size 0 max 16384 2024-03-16T10:19:20.152 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:19:20.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.151+0000 7f69d9e59640 1 -- 172.21.15.77:0/3444965441 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f69bc0a2800 msgr2=0x7f69bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.151+0000 7f69d9e59640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f69bc0a2800 0x7f69bc0a4cc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f69d4102840 tx=0x7f69c4011440 comp rx=0 tx=0).stop 2024-03-16T10:19:20.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.151+0000 7f69d9e59640 1 -- 172.21.15.77:0/3444965441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69d41033a0 msgr2=0x7f69d419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.151+0000 7f69d9e59640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69d41033a0 0x7f69d419f060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f69cc02c630 tx=0x7f69cc032f20 comp rx=0 tx=0).stop 2024-03-16T10:19:20.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.151+0000 7f69d9e59640 1 -- 172.21.15.77:0/3444965441 shutdown_connections 2024-03-16T10:19:20.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.151+0000 7f69d9e59640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f69bc0a2800 0x7f69bc0a4cc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.151+0000 7f69d9e59640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69d41033a0 0x7f69d419f060 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.151+0000 7f69d9e59640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69d41029e0 0x7f69d4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.151+0000 7f69d9e59640 1 --2- 172.21.15.77:0/3444965441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69d41017f0 0x7f69d4197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.151+0000 7f69d9e59640 1 -- 172.21.15.77:0/3444965441 >> 172.21.15.77:0/3444965441 conn(0x7f69d40fcd80 msgr2=0x7f69d4105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.151+0000 7f69d9e59640 1 -- 172.21.15.77:0/3444965441 shutdown_connections 2024-03-16T10:19:20.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.151+0000 7f69d9e59640 1 -- 172.21.15.77:0/3444965441 wait complete. 2024-03-16T10:19:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:19 smithi192 ceph-mon[36354]: pgmap v417: 97 pgs: 97 active+clean; 134 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.6 MiB/s wr, 471 op/s 2024-03-16T10:19:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:19 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1228461198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:19:20.170 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T10:19:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:19 smithi193 ceph-mon[36311]: pgmap v417: 97 pgs: 97 active+clean; 134 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.6 MiB/s wr, 471 op/s 2024-03-16T10:19:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:19 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1228461198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:19:20.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.283+0000 7fd728834640 1 -- 172.21.15.77:0/2981524967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd724104c30 msgr2=0x7fd724107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.283+0000 7fd728834640 1 --2- 172.21.15.77:0/2981524967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd724104c30 0x7fd724107020 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd718009900 tx=0x7fd71802bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:20.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.283+0000 7fd728834640 1 -- 172.21.15.77:0/2981524967 shutdown_connections 2024-03-16T10:19:20.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.283+0000 7fd728834640 1 --2- 172.21.15.77:0/2981524967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd724107560 0x7fd724109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.283+0000 7fd728834640 1 --2- 172.21.15.77:0/2981524967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd724104c30 0x7fd724107020 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.283+0000 7fd728834640 1 --2- 172.21.15.77:0/2981524967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd724102300 0x7fd7241046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.283+0000 7fd728834640 1 -- 172.21.15.77:0/2981524967 >> 172.21.15.77:0/2981524967 conn(0x7fd7240fbc90 msgr2=0x7fd7240fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.283+0000 7fd728834640 1 -- 172.21.15.77:0/2981524967 shutdown_connections 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.283+0000 7fd728834640 1 -- 172.21.15.77:0/2981524967 wait complete. 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd728834640 1 Processor -- start 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd728834640 1 -- start start 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd728834640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd724102300 0x7fd724197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd728834640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd724104c30 0x7fd724198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd728834640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd724107560 0x7fd72419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd728834640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd72410d470 con 0x7fd724104c30 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd728834640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd72410d2f0 con 0x7fd724102300 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd728834640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd72410d5f0 con 0x7fd724107560 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd721d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd724104c30 0x7fd724198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd721d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd724104c30 0x7fd724198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60868/0 (socket says 172.21.15.77:60868) 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd721d9b640 1 -- 172.21.15.77:0/3745418286 learned_addr learned my addr 172.21.15.77:0/3745418286 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd722d9d640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd724107560 0x7fd72419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd72259c640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd724102300 0x7fd724197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd721d9b640 1 -- 172.21.15.77:0/3745418286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd724107560 msgr2=0x7fd72419f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd721d9b640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd724107560 0x7fd72419f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd721d9b640 1 -- 172.21.15.77:0/3745418286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd724102300 msgr2=0x7fd724197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd721d9b640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd724102300 0x7fd724197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.284+0000 7fd721d9b640 1 -- 172.21.15.77:0/3745418286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72419f740 con 0x7fd724104c30 2024-03-16T10:19:20.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.285+0000 7fd721d9b640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd724104c30 0x7fd724198060 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd718000c00 tx=0x7fd71802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.285+0000 7fd7137fe640 1 -- 172.21.15.77:0/3745418286 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718041070 con 0x7fd724104c30 2024-03-16T10:19:20.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.285+0000 7fd7137fe640 1 -- 172.21.15.77:0/3745418286 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd718004450 con 0x7fd724104c30 2024-03-16T10:19:20.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.285+0000 7fd728834640 1 -- 172.21.15.77:0/3745418286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72419f9d0 con 0x7fd724104c30 2024-03-16T10:19:20.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.285+0000 7fd7137fe640 1 -- 172.21.15.77:0/3745418286 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718034440 con 0x7fd724104c30 2024-03-16T10:19:20.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.285+0000 7fd728834640 1 -- 172.21.15.77:0/3745418286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72419ff10 con 0x7fd724104c30 2024-03-16T10:19:20.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd72b41a740 1 Processor -- start 2024-03-16T10:19:20.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd7137fe640 1 -- 172.21.15.77:0/3745418286 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fd718044070 con 0x7fd724104c30 2024-03-16T10:19:20.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd72b41a740 1 -- start start 2024-03-16T10:19:20.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd72b41a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff298f8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd72b41a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff298fdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd72b41a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff29900840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd72b41a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ff2985d490 con 0x55ff298f8850 2024-03-16T10:19:20.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd72b41a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ff29805c90 con 0x55ff298f9240 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd72b41a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ff29859530 con 0x55ff298fe3b0 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd722d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff298f8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd722d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff298f8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60884/0 (socket says 172.21.15.77:60884) 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd722d9d640 1 -- 172.21.15.77:0/1498136425 learned_addr learned my addr 172.21.15.77:0/1498136425 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd72259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff298fdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd721d9b640 1 --2- 172.21.15.77:0/1498136425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff29900840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd722d9d640 1 -- 172.21.15.77:0/1498136425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 msgr2=0x55ff29900840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd722d9d640 1 --2- 172.21.15.77:0/1498136425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff29900840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd722d9d640 1 -- 172.21.15.77:0/1498136425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 msgr2=0x55ff298fdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd722d9d640 1 --2- 172.21.15.77:0/1498136425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff298fdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd722d9d640 1 -- 172.21.15.77:0/1498136425 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff298c0500 con 0x55ff298f8850 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.288+0000 7fd72259c640 1 --2- 172.21.15.77:0/1498136425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff298fdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.289+0000 7fd722d9d640 1 --2- 172.21.15.77:0/1498136425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff298f8c70 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd71c009fd0 tx=0x7fd71c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12c275eaeeb6f54 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.289+0000 7fd7137fe640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd70c0a2880 0x7fd70c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.289+0000 7fd72259c640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd70c0a2880 0x7fd70c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.289+0000 7fd710ff9640 1 -- 172.21.15.77:0/1498136425 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c004d20 con 0x55ff298f8850 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.289+0000 7fd7137fe640 1 -- 172.21.15.77:0/3745418286 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd7180f0430 con 0x7fd724104c30 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.289+0000 7fd710ff9640 1 -- 172.21.15.77:0/1498136425 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd71c004050 con 0x55ff298f8850 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.289+0000 7fd710ff9640 1 -- 172.21.15.77:0/1498136425 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c005240 con 0x55ff298f8850 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 -- 172.21.15.77:0/1498136425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 msgr2=0x55ff298f8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72259c640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd70c0a2880 0x7fd70c0a4d40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55ff298f9640 tx=0x7fd714005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 --2- 172.21.15.77:0/1498136425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff298f8c70 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd71c009fd0 tx=0x7fd71c0118d0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 -- 172.21.15.77:0/1498136425 shutdown_connections 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 --2- 172.21.15.77:0/1498136425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff29900840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 --2- 172.21.15.77:0/1498136425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff298fdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 --2- 172.21.15.77:0/1498136425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff298f8c70 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 -- 172.21.15.77:0/1498136425 >> 172.21.15.77:0/1498136425 conn(0x55ff298f3dc0 msgr2=0x55ff298f41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 -- 172.21.15.77:0/1498136425 shutdown_connections 2024-03-16T10:19:20.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 -- 172.21.15.77:0/1498136425 wait complete. 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 -- start start 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 Processor -- start 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f8850 0x55ff29902d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298f9240 0x55ff299032b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298fe3b0 0x55ff2990a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ff29901360 con 0x55ff298fe3b0 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ff29819230 con 0x55ff298f8850 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.290+0000 7fd72b41a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ff299014e0 con 0x55ff298f9240 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd72b41a740 10 client.0 _ll_register_callbacks cb 0x55ff29855010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd722d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298fe3b0 0x55ff2990a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd72259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f8850 0x55ff29902d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd722d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298fe3b0 0x55ff2990a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60892/0 (socket says 172.21.15.77:60892) 2024-03-16T10:19:20.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd722d9d640 1 -- 172.21.15.77:0/1750205575 learned_addr learned my addr 172.21.15.77:0/1750205575 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd721d9b640 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298f9240 0x55ff299032b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd722d9d640 1 -- 172.21.15.77:0/1750205575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298f9240 msgr2=0x55ff299032b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd722d9d640 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298f9240 0x55ff299032b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd722d9d640 1 -- 172.21.15.77:0/1750205575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f8850 msgr2=0x55ff29902d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd722d9d640 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f8850 0x55ff29902d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd722d9d640 1 -- 172.21.15.77:0/1750205575 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff29828100 con 0x55ff298fe3b0 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd72259c640 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f8850 0x55ff29902d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd722d9d640 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298fe3b0 0x55ff2990a2f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd71c01b040 tx=0x7fd71c0044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a1f017c6af1eaa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd709ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd709ffb640 1 -- 172.21.15.77:0/1750205575 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c028020 con 0x55ff298fe3b0 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd709ffb640 1 -- 172.21.15.77:0/1750205575 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd71c004c50 con 0x55ff298fe3b0 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd72b41a740 1 -- 172.21.15.77:0/1750205575 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd71c004050 con 0x55ff298fe3b0 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.291+0000 7fd709ffb640 1 -- 172.21.15.77:0/1750205575 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd71c033050 con 0x55ff298fe3b0 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.292+0000 7fd72b41a740 10 client.16221 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:20.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.292+0000 7fd72b41a740 1 -- 172.21.15.77:0/1750205575 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ff295713f0 con 0x55ff298fe3b0 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.292+0000 7fd709ffb640 1 -- 172.21.15.77:0/1750205575 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd71c004050 con 0x55ff298fe3b0 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.292+0000 7fd709ffb640 1 -- 172.21.15.77:0/1750205575 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fd71c005040 con 0x55ff298fe3b0 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.292+0000 7fd72b41a740 10 client.16221 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.292+0000 7fd72b41a740 10 client.16221 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.292+0000 7fd72b41a740 20 client.16221 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.292+0000 7fd72b41a740 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55ff2990e9b0 0x55ff29910e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.292+0000 7fd72b41a740 4 client.16221 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.292+0000 7fd72b41a740 1 -- 172.21.15.77:0/1750205575 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff2986a9d0 con 0x55ff2990e9b0 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.292+0000 7fd72259c640 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55ff2990e9b0 0x55ff29910e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd709ffb640 10 client.16221 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd72259c640 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55ff2990e9b0 0x55ff29910e10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fffe71f9b3e3b8fd server_cookie=9227aa778bb4ce81 in_seq=0 out_seq=0 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd709ffb640 1 -- 172.21.15.77:0/1750205575 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ff2986a9d0 con 0x55ff2990e9b0 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd709ffb640 10 client.16221 handle_command_reply: tid=0 2024-03-16T10:19:20.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd72b41a740 1 client.16221 shutdown 2024-03-16T10:19:20.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd72b41a740 1 -- 172.21.15.77:0/1750205575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298fe3b0 msgr2=0x55ff2990a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd72b41a740 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298fe3b0 0x55ff2990a2f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd71c01b040 tx=0x7fd71c0044b0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd72b41a740 1 -- 172.21.15.77:0/1750205575 shutdown_connections 2024-03-16T10:19:20.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd72b41a740 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55ff2990e9b0 0x55ff29910e10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd72b41a740 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298fe3b0 0x55ff2990a2f0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd709ffb640 0 client.16221 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:19:20.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd72b41a740 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298f9240 0x55ff299032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd72b41a740 1 --2- 172.21.15.77:0/1750205575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f8850 0x55ff29902d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd72b41a740 1 -- 172.21.15.77:0/1750205575 >> 172.21.15.77:0/1750205575 conn(0x55ff298f3dc0 msgr2=0x55ff298f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd72b41a740 1 -- 172.21.15.77:0/1750205575 shutdown_connections 2024-03-16T10:19:20.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.293+0000 7fd72b41a740 1 -- 172.21.15.77:0/1750205575 wait complete. 2024-03-16T10:19:20.301 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.294+0000 7fd72b41a740 20 client.16221 trim_cache size 0 max 16384 2024-03-16T10:19:20.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd72b41a740 1 Processor -- start 2024-03-16T10:19:20.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd72b41a740 1 -- start start 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd72b41a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff298f7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd72b41a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff2990a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd72b41a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff299156d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd72b41a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241a8b70 con 0x55ff298f8850 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd72b41a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241aa7d0 con 0x55ff298f9240 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd72b41a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241a9240 con 0x55ff298fe3b0 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd72259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff298f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd721d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff2990a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd722d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff299156d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd721d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff2990a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:51140/0 (socket says 172.21.15.77:51140) 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd72259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff298f7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60894/0 (socket says 172.21.15.77:60894) 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd721d9b640 1 -- 172.21.15.77:0/201347741 learned_addr learned my addr 172.21.15.77:0/201347741 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd722d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff299156d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50800/0 (socket says 172.21.15.77:50800) 2024-03-16T10:19:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd721d9b640 1 -- 172.21.15.77:0/201347741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 msgr2=0x55ff299156d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd721d9b640 1 --2- 172.21.15.77:0/201347741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff299156d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd721d9b640 1 -- 172.21.15.77:0/201347741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 msgr2=0x55ff298f7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd721d9b640 1 --2- 172.21.15.77:0/201347741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff298f7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd721d9b640 1 -- 172.21.15.77:0/201347741 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff2963b7d0 con 0x55ff298f9240 2024-03-16T10:19:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.308+0000 7fd72259c640 1 --2- 172.21.15.77:0/201347741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff298f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.309+0000 7fd721d9b640 1 --2- 172.21.15.77:0/201347741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff2990a2b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd7180b9970 tx=0x7fd7180f3f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=852f9883dacfb4ab server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.309+0000 7fd70bfff640 1 -- 172.21.15.77:0/201347741 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71804b040 con 0x55ff298f9240 2024-03-16T10:19:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.309+0000 7fd70bfff640 1 -- 172.21.15.77:0/201347741 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd718004c80 con 0x55ff298f9240 2024-03-16T10:19:20.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.309+0000 7fd70bfff640 1 -- 172.21.15.77:0/201347741 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7180c5600 con 0x55ff298f9240 2024-03-16T10:19:20.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.309+0000 7fd72b41a740 1 -- 172.21.15.77:0/201347741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 msgr2=0x55ff2990a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.309+0000 7fd72b41a740 1 --2- 172.21.15.77:0/201347741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff2990a2b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd7180b9970 tx=0x7fd7180f3f50 comp rx=0 tx=0).stop 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.309+0000 7fd72b41a740 1 -- 172.21.15.77:0/201347741 shutdown_connections 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.309+0000 7fd72b41a740 1 --2- 172.21.15.77:0/201347741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff299156d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.309+0000 7fd72b41a740 1 --2- 172.21.15.77:0/201347741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff2990a2b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.309+0000 7fd72b41a740 1 --2- 172.21.15.77:0/201347741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff298f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.309+0000 7fd72b41a740 1 -- 172.21.15.77:0/201347741 >> 172.21.15.77:0/201347741 conn(0x55ff298f3dc0 msgr2=0x55ff298f7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72b41a740 1 -- 172.21.15.77:0/201347741 shutdown_connections 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72b41a740 1 -- 172.21.15.77:0/201347741 wait complete. 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72b41a740 1 -- start start 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72b41a740 1 Processor -- start 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72b41a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff29907520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72b41a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff29907a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72b41a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff29908a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72b41a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241aa040 con 0x55ff298f8850 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72b41a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241a9970 con 0x55ff298f9240 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72b41a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241a6b50 con 0x55ff298fe3b0 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff29907520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72b41a740 10 client.0 _ll_register_callbacks cb 0x7fd714009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff29907520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60912/0 (socket says 172.21.15.77:60912) 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72259c640 1 -- 172.21.15.77:0/2577702920 learned_addr learned my addr 172.21.15.77:0/2577702920 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd722d9d640 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff29908a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd721d9b640 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff29907a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72259c640 1 -- 172.21.15.77:0/2577702920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 msgr2=0x55ff29908a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72259c640 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff29908a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72259c640 1 -- 172.21.15.77:0/2577702920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 msgr2=0x55ff29907a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72259c640 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff29907a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd72259c640 1 -- 172.21.15.77:0/2577702920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff29909060 con 0x55ff298f8850 2024-03-16T10:19:20.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.310+0000 7fd721d9b640 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff29907a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd72259c640 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff29907520 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd71401e8a0 tx=0x7fd7140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef18e822dc37ecf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd709ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd709ffb640 1 -- 172.21.15.77:0/2577702920 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71401e950 con 0x55ff298f8850 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd709ffb640 1 -- 172.21.15.77:0/2577702920 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd71401eaf0 con 0x55ff298f8850 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd72b41a740 1 -- 172.21.15.77:0/2577702920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd718004c80 con 0x55ff298f8850 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd709ffb640 1 -- 172.21.15.77:0/2577702920 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd71401ecf0 con 0x55ff298f8850 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd72b41a740 10 client.16227 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd72b41a740 1 -- 172.21.15.77:0/2577702920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ff2990a430 con 0x55ff298f8850 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd709ffb640 1 -- 172.21.15.77:0/2577702920 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd718004c80 con 0x55ff298f8850 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd709ffb640 1 -- 172.21.15.77:0/2577702920 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fd714004a50 con 0x55ff298f8850 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd72b41a740 10 client.16227 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd72b41a740 10 client.16227 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd72b41a740 20 client.16227 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd72b41a740 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55ff29975280 0x55ff29977670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd72b41a740 4 client.16227 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.311+0000 7fd72b41a740 1 -- 172.21.15.77:0/2577702920 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ff29864d20 con 0x55ff29975280 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd721d9b640 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55ff29975280 0x55ff29977670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd721d9b640 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55ff29975280 0x55ff29977670 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ecf6ceb3cb215f96 server_cookie=b686a02d26e45e12 in_seq=0 out_seq=0 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd709ffb640 10 client.16227 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd709ffb640 1 -- 172.21.15.77:0/2577702920 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ff29864d20 con 0x55ff29975280 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd709ffb640 10 client.16227 handle_command_reply: tid=0 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd72b41a740 1 client.16227 shutdown 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd72b41a740 1 -- 172.21.15.77:0/2577702920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 msgr2=0x55ff29907520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd72b41a740 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff29907520 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd71401e8a0 tx=0x7fd7140046c0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd72b41a740 1 -- 172.21.15.77:0/2577702920 shutdown_connections 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd72b41a740 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55ff29975280 0x55ff29977670 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd72b41a740 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff298fe3b0 0x55ff29908a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd72b41a740 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff298f9240 0x55ff29907a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd72b41a740 1 --2- 172.21.15.77:0/2577702920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ff298f8850 0x55ff29907520 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd72b41a740 1 -- 172.21.15.77:0/2577702920 >> 172.21.15.77:0/2577702920 conn(0x55ff298f3dc0 msgr2=0x55ff298f78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd709ffb640 0 client.16227 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:19:20.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd72b41a740 1 -- 172.21.15.77:0/2577702920 shutdown_connections 2024-03-16T10:19:20.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd72b41a740 1 -- 172.21.15.77:0/2577702920 wait complete. 2024-03-16T10:19:20.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.312+0000 7fd72b41a740 20 client.16227 trim_cache size 0 max 16384 2024-03-16T10:19:20.321 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:19:20.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.313+0000 7fd728834640 1 -- 172.21.15.77:0/3745418286 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd70c0a2880 msgr2=0x7fd70c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.313+0000 7fd728834640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd70c0a2880 0x7fd70c0a4d40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55ff298f9640 tx=0x7fd714005f90 comp rx=0 tx=0).stop 2024-03-16T10:19:20.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.313+0000 7fd728834640 1 -- 172.21.15.77:0/3745418286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd724104c30 msgr2=0x7fd724198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.313+0000 7fd728834640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd724104c30 0x7fd724198060 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd718000c00 tx=0x7fd71802c640 comp rx=0 tx=0).stop 2024-03-16T10:19:20.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.313+0000 7fd728834640 1 -- 172.21.15.77:0/3745418286 shutdown_connections 2024-03-16T10:19:20.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.313+0000 7fd728834640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd70c0a2880 0x7fd70c0a4d40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.313+0000 7fd728834640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd724107560 0x7fd72419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.313+0000 7fd728834640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd724104c30 0x7fd724198060 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.313+0000 7fd728834640 1 --2- 172.21.15.77:0/3745418286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd724102300 0x7fd724197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.313+0000 7fd728834640 1 -- 172.21.15.77:0/3745418286 >> 172.21.15.77:0/3745418286 conn(0x7fd7240fbc90 msgr2=0x7fd724102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.313+0000 7fd728834640 1 -- 172.21.15.77:0/3745418286 shutdown_connections 2024-03-16T10:19:20.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.313+0000 7fd728834640 1 -- 172.21.15.77:0/3745418286 wait complete. 2024-03-16T10:19:20.325 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T10:19:20.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.445+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3964467449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab581006e0 msgr2=0x7fab58100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.445+0000 7fab5f5f3640 1 --2- 172.21.15.77:0/3964467449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab581006e0 0x7fab58100b00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fab54009900 tx=0x7fab5402bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:20.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.445+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3964467449 shutdown_connections 2024-03-16T10:19:20.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.445+0000 7fab5f5f3640 1 --2- 172.21.15.77:0/3964467449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab58102290 0x7fab58108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.445+0000 7fab5f5f3640 1 --2- 172.21.15.77:0/3964467449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab581018d0 0x7fab58101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.445+0000 7fab5f5f3640 1 --2- 172.21.15.77:0/3964467449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab581006e0 0x7fab58100b00 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.445+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3964467449 >> 172.21.15.77:0/3964467449 conn(0x7fab580fbc70 msgr2=0x7fab580fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.445+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3964467449 shutdown_connections 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.445+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3964467449 wait complete. 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.445+0000 7fab5f5f3640 1 Processor -- start 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.445+0000 7fab5f5f3640 1 -- start start 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5f5f3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab581006e0 0x7fab58196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5f5f3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab581018d0 0x7fab58196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5f5f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab58102290 0x7fab5819dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5f5f3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fab5810c360 con 0x7fab58102290 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5f5f3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fab5810c1e0 con 0x7fab581018d0 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5f5f3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fab5810c4e0 con 0x7fab581006e0 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5db90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab58102290 0x7fab5819dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5db90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab58102290 0x7fab5819dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60918/0 (socket says 172.21.15.77:60918) 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5db90640 1 -- 172.21.15.77:0/3912087971 learned_addr learned my addr 172.21.15.77:0/3912087971 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5d38f640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab581006e0 0x7fab58196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5cb8e640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab581018d0 0x7fab58196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5db90640 1 -- 172.21.15.77:0/3912087971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab581006e0 msgr2=0x7fab58196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5db90640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab581006e0 0x7fab58196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5db90640 1 -- 172.21.15.77:0/3912087971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab581018d0 msgr2=0x7fab58196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5db90640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab581018d0 0x7fab58196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5db90640 1 -- 172.21.15.77:0/3912087971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5819e640 con 0x7fab58102290 2024-03-16T10:19:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5db90640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab58102290 0x7fab5819dfd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fab5000eb10 tx=0x7fab5000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab4e7fc640 1 -- 172.21.15.77:0/3912087971 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab50004b20 con 0x7fab58102290 2024-03-16T10:19:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3912087971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5819e930 con 0x7fab58102290 2024-03-16T10:19:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab4e7fc640 1 -- 172.21.15.77:0/3912087971 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab50004cc0 con 0x7fab58102290 2024-03-16T10:19:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab4e7fc640 1 -- 172.21.15.77:0/3912087971 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab50010760 con 0x7fab58102290 2024-03-16T10:19:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.446+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3912087971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab5819eed0 con 0x7fab58102290 2024-03-16T10:19:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.449+0000 7fab4e7fc640 1 -- 172.21.15.77:0/3912087971 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fab50005020 con 0x7fab58102290 2024-03-16T10:19:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.449+0000 7fab621d9740 1 Processor -- start 2024-03-16T10:19:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.449+0000 7fab621d9740 1 -- start start 2024-03-16T10:19:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab621d9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68d9bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab621d9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9c240 0x558f68da0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab621d9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68da13b0 0x558f68da3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab621d9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558f68d00490 con 0x558f68d9b850 2024-03-16T10:19:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab621d9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x558f68ca8c90 con 0x558f68da13b0 2024-03-16T10:19:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab621d9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558f68cfc530 con 0x558f68d9c240 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5cb8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68d9bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5cb8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68d9bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60920/0 (socket says 172.21.15.77:60920) 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5cb8e640 1 -- 172.21.15.77:0/2064067502 learned_addr learned my addr 172.21.15.77:0/2064067502 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5d38f640 1 --2- 172.21.15.77:0/2064067502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9c240 0x558f68da0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5cb8e640 1 --2- 172.21.15.77:0/2064067502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68da13b0 0x558f68da3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5cb8e640 1 -- 172.21.15.77:0/2064067502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9c240 msgr2=0x558f68da0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5cb8e640 1 --2- 172.21.15.77:0/2064067502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9c240 0x558f68da0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5cb8e640 1 -- 172.21.15.77:0/2064067502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68da13b0 msgr2=0x558f68da3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5cb8e640 1 --2- 172.21.15.77:0/2064067502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68da13b0 0x558f68da3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5cb8e640 1 -- 172.21.15.77:0/2064067502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f68d63500 con 0x558f68d9b850 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5d38f640 1 --2- 172.21.15.77:0/2064067502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9c240 0x558f68da0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab4e7fc640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fab440a2880 0x7fab440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5cb8e640 1 --2- 172.21.15.77:0/2064067502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68d9bc70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fab48013620 tx=0x7fab48009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa529f8e838ac697 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab5d38f640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fab440a2880 0x7fab440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab4e7fc640 1 -- 172.21.15.77:0/3912087971 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fab500cf020 con 0x7fab58102290 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.450+0000 7fab437fe640 1 -- 172.21.15.77:0/2064067502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab48004050 con 0x558f68d9b850 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.451+0000 7fab437fe640 1 -- 172.21.15.77:0/2064067502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab480041f0 con 0x558f68d9b850 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.451+0000 7fab437fe640 1 -- 172.21.15.77:0/2064067502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4800f7c0 con 0x558f68d9b850 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.451+0000 7fab621d9740 1 -- 172.21.15.77:0/2064067502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 msgr2=0x558f68d9bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.451+0000 7fab5d38f640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fab440a2880 0x7fab440a4d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fab54000c00 tx=0x7fab54005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.451+0000 7fab621d9740 1 --2- 172.21.15.77:0/2064067502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68d9bc70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fab48013620 tx=0x7fab48009850 comp rx=0 tx=0).stop 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.451+0000 7fab621d9740 1 -- 172.21.15.77:0/2064067502 shutdown_connections 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.451+0000 7fab621d9740 1 --2- 172.21.15.77:0/2064067502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68da13b0 0x558f68da3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.451+0000 7fab621d9740 1 --2- 172.21.15.77:0/2064067502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9c240 0x558f68da0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.451+0000 7fab621d9740 1 --2- 172.21.15.77:0/2064067502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68d9bc70 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.451+0000 7fab621d9740 1 -- 172.21.15.77:0/2064067502 >> 172.21.15.77:0/2064067502 conn(0x558f68d96dc0 msgr2=0x558f68d971c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.451+0000 7fab621d9740 1 -- 172.21.15.77:0/2064067502 shutdown_connections 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.451+0000 7fab621d9740 1 -- 172.21.15.77:0/2064067502 wait complete. 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab621d9740 1 -- start start 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab621d9740 1 Processor -- start 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab621d9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68da5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab621d9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9c240 0x558f68da6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab621d9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68da13b0 0x558f68dad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab621d9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558f68da4300 con 0x558f68d9b850 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab621d9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x558f68cbc230 con 0x558f68da13b0 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab621d9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558f68da4480 con 0x558f68d9c240 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab621d9740 10 client.0 _ll_register_callbacks cb 0x558f68cf8010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab5d38f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68da5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab5d38f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68da5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60934/0 (socket says 172.21.15.77:60934) 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab5d38f640 1 -- 172.21.15.77:0/1270816693 learned_addr learned my addr 172.21.15.77:0/1270816693 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab5cb8e640 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9c240 0x558f68da6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab5db90640 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68da13b0 0x558f68dad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab5d38f640 1 -- 172.21.15.77:0/1270816693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9c240 msgr2=0x558f68da6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab5d38f640 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9c240 0x558f68da6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab5d38f640 1 -- 172.21.15.77:0/1270816693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68da13b0 msgr2=0x558f68dad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab5d38f640 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68da13b0 0x558f68dad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.452+0000 7fab5d38f640 1 -- 172.21.15.77:0/1270816693 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f68ccb100 con 0x558f68d9b850 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab5d38f640 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68da5d20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fab58101730 tx=0x7fab54009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7a20a35a03d7df server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab40ff9640 1 -- 172.21.15.77:0/1270816693 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5404e070 con 0x558f68d9b850 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab40ff9640 1 -- 172.21.15.77:0/1270816693 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fab54034440 con 0x558f68d9b850 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab621d9740 1 -- 172.21.15.77:0/1270816693 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab480041f0 con 0x558f68d9b850 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab40ff9640 1 -- 172.21.15.77:0/1270816693 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fab5405a050 con 0x558f68d9b850 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab621d9740 10 client.16239 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab621d9740 1 -- 172.21.15.77:0/1270816693 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558f68a143f0 con 0x558f68d9b850 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab40ff9640 1 -- 172.21.15.77:0/1270816693 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab480041f0 con 0x558f68d9b850 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab40ff9640 1 -- 172.21.15.77:0/1270816693 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fab540485c0 con 0x558f68d9b850 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab621d9740 10 client.16239 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab621d9740 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab621d9740 20 client.16239 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab621d9740 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f68db1b60 0x558f68db3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab621d9740 4 client.16239 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.453+0000 7fab621d9740 1 -- 172.21.15.77:0/1270816693 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f68d0d9d0 con 0x558f68db1b60 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.454+0000 7fab5cb8e640 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f68db1b60 0x558f68db3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.454+0000 7fab40ff9640 10 client.16239 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.454+0000 7fab5cb8e640 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f68db1b60 0x558f68db3fe0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95299d55e98ce168 server_cookie=74c042b8a1b972d5 in_seq=0 out_seq=0 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab40ff9640 1 -- 172.21.15.77:0/1270816693 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558f68d0d9d0 con 0x558f68db1b60 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab40ff9640 10 client.16239 handle_command_reply: tid=0 2024-03-16T10:19:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab621d9740 1 client.16239 shutdown 2024-03-16T10:19:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab621d9740 1 -- 172.21.15.77:0/1270816693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 msgr2=0x558f68da5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab621d9740 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68da5d20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fab58101730 tx=0x7fab54009260 comp rx=0 tx=0).stop 2024-03-16T10:19:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab621d9740 1 -- 172.21.15.77:0/1270816693 shutdown_connections 2024-03-16T10:19:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab621d9740 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68da13b0 0x558f68dad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab621d9740 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9c240 0x558f68da6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab621d9740 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f68db1b60 0x558f68db3fe0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab621d9740 1 --2- 172.21.15.77:0/1270816693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68da5d20 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab40ff9640 0 client.16239 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab621d9740 1 -- 172.21.15.77:0/1270816693 >> 172.21.15.77:0/1270816693 conn(0x558f68d96dc0 msgr2=0x558f68d95fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab621d9740 1 -- 172.21.15.77:0/1270816693 shutdown_connections 2024-03-16T10:19:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.455+0000 7fab621d9740 1 -- 172.21.15.77:0/1270816693 wait complete. 2024-03-16T10:19:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.456+0000 7fab621d9740 20 client.16239 trim_cache size 0 max 16384 2024-03-16T10:19:20.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab621d9740 1 Processor -- start 2024-03-16T10:19:20.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab621d9740 1 -- start start 2024-03-16T10:19:20.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab621d9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d96dc0 0x558f68d9a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab621d9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9b850 0x558f68db88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab621d9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68d9c240 0x558f68db8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab621d9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fab581a7b40 con 0x558f68d96dc0 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab621d9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fab581a97a0 con 0x558f68d9c240 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab621d9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fab581a8210 con 0x558f68d9b850 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab5d38f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d96dc0 0x558f68d9a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab5d38f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d96dc0 0x558f68d9a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60950/0 (socket says 172.21.15.77:60950) 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab5cb8e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9b850 0x558f68db88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab5d38f640 1 -- 172.21.15.77:0/497055908 learned_addr learned my addr 172.21.15.77:0/497055908 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab5db90640 1 --2- 172.21.15.77:0/497055908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68d9c240 0x558f68db8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab5d38f640 1 -- 172.21.15.77:0/497055908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9b850 msgr2=0x558f68db88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab5d38f640 1 --2- 172.21.15.77:0/497055908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9b850 0x558f68db88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab5d38f640 1 -- 172.21.15.77:0/497055908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68d9c240 msgr2=0x558f68db8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab5d38f640 1 --2- 172.21.15.77:0/497055908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68d9c240 0x558f68db8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab5d38f640 1 -- 172.21.15.77:0/497055908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f68da58c0 con 0x558f68d96dc0 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.469+0000 7fab5cb8e640 1 --2- 172.21.15.77:0/497055908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9b850 0x558f68db88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.470+0000 7fab5d38f640 1 --2- 172.21.15.77:0/497055908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d96dc0 0x558f68d9a730 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fab54009230 tx=0x7fab5402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3e7792a141354e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.470+0000 7fab42ffd640 1 -- 172.21.15.77:0/497055908 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab54057020 con 0x558f68d96dc0 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.470+0000 7fab42ffd640 1 -- 172.21.15.77:0/497055908 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab54048bb0 con 0x558f68d96dc0 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.470+0000 7fab42ffd640 1 -- 172.21.15.77:0/497055908 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5404e040 con 0x558f68d96dc0 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.470+0000 7fab621d9740 1 -- 172.21.15.77:0/497055908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d96dc0 msgr2=0x558f68d9a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.470+0000 7fab621d9740 1 --2- 172.21.15.77:0/497055908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d96dc0 0x558f68d9a730 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fab54009230 tx=0x7fab5402bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.470+0000 7fab621d9740 1 -- 172.21.15.77:0/497055908 shutdown_connections 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.470+0000 7fab621d9740 1 --2- 172.21.15.77:0/497055908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68d9c240 0x558f68db8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.470+0000 7fab621d9740 1 --2- 172.21.15.77:0/497055908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d9b850 0x558f68db88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.470+0000 7fab621d9740 1 --2- 172.21.15.77:0/497055908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d96dc0 0x558f68d9a730 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.470+0000 7fab621d9740 1 -- 172.21.15.77:0/497055908 >> 172.21.15.77:0/497055908 conn(0x558f68d9d020 msgr2=0x558f68d9a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab621d9740 1 -- 172.21.15.77:0/497055908 shutdown_connections 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab621d9740 1 -- 172.21.15.77:0/497055908 wait complete. 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab621d9740 1 -- start start 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab621d9740 1 Processor -- start 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab621d9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d96dc0 0x558f68e13ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab621d9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68e16410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab621d9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68d9c240 0x558f68e18960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab621d9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fab581a9010 con 0x558f68d9b850 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab621d9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fab581a8940 con 0x558f68d9c240 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab621d9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fab581a5b40 con 0x558f68d96dc0 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab621d9740 10 client.0 _ll_register_callbacks cb 0x7fab54009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab5cb8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68e16410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab5d38f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d96dc0 0x558f68e13ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab5cb8e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68e16410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60958/0 (socket says 172.21.15.77:60958) 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab5cb8e640 1 -- 172.21.15.77:0/1627104285 learned_addr learned my addr 172.21.15.77:0/1627104285 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.471+0000 7fab5db90640 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68d9c240 0x558f68e18960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab5cb8e640 1 -- 172.21.15.77:0/1627104285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d96dc0 msgr2=0x558f68e13ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab5cb8e640 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d96dc0 0x558f68e13ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab5cb8e640 1 -- 172.21.15.77:0/1627104285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68d9c240 msgr2=0x558f68e18960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab5cb8e640 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68d9c240 0x558f68e18960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab5cb8e640 1 -- 172.21.15.77:0/1627104285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f68ade7d0 con 0x558f68d9b850 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab5db90640 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68d9c240 0x558f68e18960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab5d38f640 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d96dc0 0x558f68e13ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab5cb8e640 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68e16410 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x558f68d9caa0 tx=0x7fab48009400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00f76b36e393e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab40ff9640 1 -- 172.21.15.77:0/1627104285 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab48031060 con 0x558f68d9b850 2024-03-16T10:19:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab40ff9640 1 -- 172.21.15.77:0/1627104285 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fab4802cb90 con 0x558f68d9b850 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab621d9740 1 -- 172.21.15.77:0/1627104285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab54048bb0 con 0x558f68d9b850 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab40ff9640 1 -- 172.21.15.77:0/1627104285 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fab4802cd90 con 0x558f68d9b850 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab621d9740 10 client.16245 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab621d9740 1 -- 172.21.15.77:0/1627104285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558f68e1a060 con 0x558f68d9b850 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.472+0000 7fab40ff9640 1 -- 172.21.15.77:0/1627104285 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab54048bb0 con 0x558f68d9b850 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab40ff9640 1 -- 172.21.15.77:0/1627104285 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fab4801e070 con 0x558f68d9b850 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab621d9740 10 client.16245 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab621d9740 10 client.16245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab621d9740 20 client.16245 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab621d9740 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f68da13b0 0x558f68e1c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab621d9740 4 client.16245 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab621d9740 1 -- 172.21.15.77:0/1627104285 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558f68d07d20 con 0x558f68da13b0 2024-03-16T10:19:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab5d38f640 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f68da13b0 0x558f68e1c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab40ff9640 10 client.16245 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab5d38f640 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f68da13b0 0x558f68e1c970 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8609eb21d26b5f3 server_cookie=e9540c5d74a32d00 in_seq=0 out_seq=0 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab40ff9640 1 -- 172.21.15.77:0/1627104285 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558f68d07d20 con 0x558f68da13b0 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab40ff9640 10 client.16245 handle_command_reply: tid=0 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.473+0000 7fab621d9740 1 client.16245 shutdown 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab621d9740 1 -- 172.21.15.77:0/1627104285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 msgr2=0x558f68e16410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab621d9740 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68e16410 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x558f68d9caa0 tx=0x7fab48009400 comp rx=0 tx=0).stop 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab621d9740 1 -- 172.21.15.77:0/1627104285 shutdown_connections 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab621d9740 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f68d9c240 0x558f68e18960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab621d9740 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f68da13b0 0x558f68e1c970 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab621d9740 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f68d9b850 0x558f68e16410 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab40ff9640 0 client.16245 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab621d9740 1 --2- 172.21.15.77:0/1627104285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f68d96dc0 0x558f68e13ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab621d9740 1 -- 172.21.15.77:0/1627104285 >> 172.21.15.77:0/1627104285 conn(0x558f68d9d020 msgr2=0x558f68d9a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab621d9740 1 -- 172.21.15.77:0/1627104285 shutdown_connections 2024-03-16T10:19:20.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab621d9740 1 -- 172.21.15.77:0/1627104285 wait complete. 2024-03-16T10:19:20.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab621d9740 20 client.16245 trim_cache size 0 max 16384 2024-03-16T10:19:20.482 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:19:20.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3912087971 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fab440a2880 msgr2=0x7fab440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab5f5f3640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fab440a2880 0x7fab440a4d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fab54000c00 tx=0x7fab54005fd0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3912087971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab58102290 msgr2=0x7fab5819dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:20.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab5f5f3640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab58102290 0x7fab5819dfd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fab5000eb10 tx=0x7fab5000c2d0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3912087971 shutdown_connections 2024-03-16T10:19:20.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab5f5f3640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab58102290 0x7fab5819dfd0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab5f5f3640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab581018d0 0x7fab58196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab5f5f3640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fab440a2880 0x7fab440a4d40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab5f5f3640 1 --2- 172.21.15.77:0/3912087971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab581006e0 0x7fab58196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:20.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.474+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3912087971 >> 172.21.15.77:0/3912087971 conn(0x7fab580fbc70 msgr2=0x7fab58104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:20.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.475+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3912087971 shutdown_connections 2024-03-16T10:19:20.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:20.475+0000 7fab5f5f3640 1 -- 172.21.15.77:0/3912087971 wait complete. 2024-03-16T10:19:21.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:20 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:19:21.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:20 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:19:21.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:20 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:19:21.487 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T10:19:21.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:19:21.487 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T10:19:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.595+0000 7f3ed777e640 1 -- 172.21.15.77:0/1566720741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ed01018f0 msgr2=0x7f3ed0101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.595+0000 7f3ed777e640 1 --2- 172.21.15.77:0/1566720741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ed01018f0 0x7f3ed0101d70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0009900 tx=0x7f3ec002bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.595+0000 7f3ed777e640 1 -- 172.21.15.77:0/1566720741 shutdown_connections 2024-03-16T10:19:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.595+0000 7f3ed777e640 1 --2- 172.21.15.77:0/1566720741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ed01022b0 0x7f3ed0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.595+0000 7f3ed777e640 1 --2- 172.21.15.77:0/1566720741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ed01018f0 0x7f3ed0101d70 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.595+0000 7f3ed777e640 1 --2- 172.21.15.77:0/1566720741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ed0100700 0x7f3ed0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.595+0000 7f3ed777e640 1 -- 172.21.15.77:0/1566720741 >> 172.21.15.77:0/1566720741 conn(0x7f3ed00fbc70 msgr2=0x7f3ed00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.595+0000 7f3ed777e640 1 -- 172.21.15.77:0/1566720741 shutdown_connections 2024-03-16T10:19:21.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.595+0000 7f3ed777e640 1 -- 172.21.15.77:0/1566720741 wait complete. 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.595+0000 7f3ed777e640 1 Processor -- start 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.595+0000 7f3ed777e640 1 -- start start 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed777e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ed0100700 0x7f3ed01047e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed777e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ed01018f0 0x7f3ed0102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed777e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ed01022b0 0x7f3ed01033b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed777e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed010c380 con 0x7f3ed0100700 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed777e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed010c200 con 0x7f3ed01018f0 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed777e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed010c500 con 0x7f3ed01022b0 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed551a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ed0100700 0x7f3ed01047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed551a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ed0100700 0x7f3ed01047e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44616/0 (socket says 172.21.15.77:44616) 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed551a640 1 -- 172.21.15.77:0/3253265064 learned_addr learned my addr 172.21.15.77:0/3253265064 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed4d19640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ed01018f0 0x7f3ed0102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed5d1b640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ed01022b0 0x7f3ed01033b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed551a640 1 -- 172.21.15.77:0/3253265064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ed01022b0 msgr2=0x7f3ed01033b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed551a640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ed01022b0 0x7f3ed01033b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed551a640 1 -- 172.21.15.77:0/3253265064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ed01018f0 msgr2=0x7f3ed0102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed551a640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ed01018f0 0x7f3ed0102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed551a640 1 -- 172.21.15.77:0/3253265064 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed0103c70 con 0x7f3ed0100700 2024-03-16T10:19:21.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed4d19640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ed01018f0 0x7f3ed0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:21.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed551a640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ed0100700 0x7f3ed01047e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc00e9c0 tx=0x7f3ecc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ec67fc640 1 -- 172.21.15.77:0/3253265064 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc00cd80 con 0x7f3ed0100700 2024-03-16T10:19:21.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ec67fc640 1 -- 172.21.15.77:0/3253265064 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ecc0035f0 con 0x7f3ed0100700 2024-03-16T10:19:21.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed777e640 1 -- 172.21.15.77:0/3253265064 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed01a1d10 con 0x7f3ed0100700 2024-03-16T10:19:21.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ec67fc640 1 -- 172.21.15.77:0/3253265064 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc010690 con 0x7f3ed0100700 2024-03-16T10:19:21.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.596+0000 7f3ed777e640 1 -- 172.21.15.77:0/3253265064 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed01a2180 con 0x7f3ed0100700 2024-03-16T10:19:21.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3ec67fc640 1 -- 172.21.15.77:0/3253265064 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f3ecc0037a0 con 0x7f3ed0100700 2024-03-16T10:19:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3eda36b740 1 Processor -- start 2024-03-16T10:19:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3eda36b740 1 -- start start 2024-03-16T10:19:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3eda36b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f40982990 0x558f40982db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3eda36b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f40987e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3eda36b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f409884f0 0x558f4098a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3eda36b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558f408e70d0 con 0x558f40983380 2024-03-16T10:19:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3eda36b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x558f4088fc70 con 0x558f40982990 2024-03-16T10:19:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3eda36b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558f408e3170 con 0x558f409884f0 2024-03-16T10:19:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3ed551a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f40987e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3ed551a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f40987e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44622/0 (socket says 172.21.15.77:44622) 2024-03-16T10:19:21.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3ed551a640 1 -- 172.21.15.77:0/548015574 learned_addr learned my addr 172.21.15.77:0/548015574 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3ed4d19640 1 --2- 172.21.15.77:0/548015574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f409884f0 0x558f4098a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3ed5d1b640 1 --2- 172.21.15.77:0/548015574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f40982990 0x558f40982db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.599+0000 7f3ec67fc640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3ebc0a2800 0x7f3ebc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ed551a640 1 -- 172.21.15.77:0/548015574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f409884f0 msgr2=0x558f4098a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ed551a640 1 --2- 172.21.15.77:0/548015574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f409884f0 0x558f4098a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ed551a640 1 -- 172.21.15.77:0/548015574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f40982990 msgr2=0x558f40982db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ed551a640 1 --2- 172.21.15.77:0/548015574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f40982990 0x558f40982db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ed551a640 1 -- 172.21.15.77:0/548015574 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f406c37e0 con 0x558f40983380 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ed4d19640 1 --2- 172.21.15.77:0/548015574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f409884f0 0x558f4098a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ed5d1b640 1 --2- 172.21.15.77:0/548015574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f40982990 0x558f40982db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ed5d1b640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3ebc0a2800 0x7f3ebc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ec67fc640 1 -- 172.21.15.77:0/3253265064 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f3ecc0ced30 con 0x7f3ed0100700 2024-03-16T10:19:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ed551a640 1 --2- 172.21.15.77:0/548015574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f40987e80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc04d200 tx=0x7f3ecc021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f266c424a8172ad4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ebb7fe640 1 -- 172.21.15.77:0/548015574 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc09b530 con 0x558f40983380 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ebb7fe640 1 -- 172.21.15.77:0/548015574 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ecc04aeb0 con 0x558f40983380 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.600+0000 7f3ebb7fe640 1 -- 172.21.15.77:0/548015574 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc09b0c0 con 0x558f40983380 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.601+0000 7f3eda36b740 1 -- 172.21.15.77:0/548015574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 msgr2=0x558f40987e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.601+0000 7f3eda36b740 1 --2- 172.21.15.77:0/548015574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f40987e80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc04d200 tx=0x7f3ecc021000 comp rx=0 tx=0).stop 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.601+0000 7f3ed5d1b640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3ebc0a2800 0x7f3ebc0a4cc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x558f40983be0 tx=0x7f3ec8004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.601+0000 7f3eda36b740 1 -- 172.21.15.77:0/548015574 shutdown_connections 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.601+0000 7f3eda36b740 1 --2- 172.21.15.77:0/548015574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f409884f0 0x558f4098a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.601+0000 7f3eda36b740 1 --2- 172.21.15.77:0/548015574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f40987e80 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.601+0000 7f3eda36b740 1 --2- 172.21.15.77:0/548015574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f40982990 0x558f40982db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.601+0000 7f3eda36b740 1 -- 172.21.15.77:0/548015574 >> 172.21.15.77:0/548015574 conn(0x558f4094ac50 msgr2=0x558f4097e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.601+0000 7f3eda36b740 1 -- 172.21.15.77:0/548015574 shutdown_connections 2024-03-16T10:19:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.601+0000 7f3eda36b740 1 -- 172.21.15.77:0/548015574 wait complete. 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.601+0000 7f3eda36b740 1 -- start start 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.601+0000 7f3eda36b740 1 Processor -- start 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3eda36b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40982990 0x558f4098ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3eda36b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f4098d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3eda36b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f409884f0 0x558f409943f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3eda36b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558f4098b440 con 0x558f40983380 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3eda36b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x558f408a30d0 con 0x558f409884f0 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3eda36b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558f4098b5c0 con 0x558f40982990 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3eda36b740 10 client.0 _ll_register_callbacks cb 0x558f4086c760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed4d19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f4098d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed4d19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f4098d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44638/0 (socket says 172.21.15.77:44638) 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed4d19640 1 -- 172.21.15.77:0/1480590179 learned_addr learned my addr 172.21.15.77:0/1480590179 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed551a640 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40982990 0x558f4098ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed5d1b640 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f409884f0 0x558f409943f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed4d19640 1 -- 172.21.15.77:0/1480590179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40982990 msgr2=0x558f4098ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed4d19640 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40982990 0x558f4098ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed4d19640 1 -- 172.21.15.77:0/1480590179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f409884f0 msgr2=0x558f409943f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed4d19640 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f409884f0 0x558f409943f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed4d19640 1 -- 172.21.15.77:0/1480590179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f40723f70 con 0x558f40983380 2024-03-16T10:19:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed551a640 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40982990 0x558f4098ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed5d1b640 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f409884f0 0x558f409943f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:19:21.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3ed4d19640 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f4098d3b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0006010 tx=0x7f3ec00095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe338c7fdbcff3e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3eb8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:21.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3eb8ff9640 1 -- 172.21.15.77:0/1480590179 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00048c0 con 0x558f40983380 2024-03-16T10:19:21.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3eb8ff9640 1 -- 172.21.15.77:0/1480590179 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f3ec0004a60 con 0x558f40983380 2024-03-16T10:19:21.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.602+0000 7f3eda36b740 1 -- 172.21.15.77:0/1480590179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558f408f4610 con 0x558f40983380 2024-03-16T10:19:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.603+0000 7f3eb8ff9640 1 -- 172.21.15.77:0/1480590179 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3ec0004c60 con 0x558f40983380 2024-03-16T10:19:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.603+0000 7f3eda36b740 10 client.16257 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.603+0000 7f3eda36b740 1 -- 172.21.15.77:0/1480590179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558f405bb170 con 0x558f40983380 2024-03-16T10:19:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.603+0000 7f3eb8ff9640 1 -- 172.21.15.77:0/1480590179 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558f408f4610 con 0x558f40983380 2024-03-16T10:19:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.603+0000 7f3eb8ff9640 1 -- 172.21.15.77:0/1480590179 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f3ec0050050 con 0x558f40983380 2024-03-16T10:19:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.603+0000 7f3eda36b740 10 client.16257 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.603+0000 7f3eda36b740 10 client.16257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:21.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.603+0000 7f3eda36b740 20 client.16257 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:21.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.603+0000 7f3eda36b740 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f40998980 0x558f4099adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.603+0000 7f3eda36b740 4 client.16257 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:21.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.603+0000 7f3eda36b740 1 -- 172.21.15.77:0/1480590179 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f408ee960 con 0x558f40998980 2024-03-16T10:19:21.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.603+0000 7f3ed551a640 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f40998980 0x558f4099adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.604+0000 7f3eb8ff9640 10 client.16257 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:21.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.604+0000 7f3ed551a640 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f40998980 0x558f4099adc0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1773ced28a4fda0e server_cookie=aba3de6f783417fb in_seq=0 out_seq=0 2024-03-16T10:19:21.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eb8ff9640 1 -- 172.21.15.77:0/1480590179 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558f408ee960 con 0x558f40998980 2024-03-16T10:19:21.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eb8ff9640 10 client.16257 handle_command_reply: tid=0 2024-03-16T10:19:21.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eda36b740 1 client.16257 shutdown 2024-03-16T10:19:21.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eda36b740 1 -- 172.21.15.77:0/1480590179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 msgr2=0x558f4098d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eda36b740 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f4098d3b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0006010 tx=0x7f3ec00095f0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eda36b740 1 -- 172.21.15.77:0/1480590179 shutdown_connections 2024-03-16T10:19:21.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eda36b740 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f409884f0 0x558f409943f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eda36b740 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f40998980 0x558f4099adc0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eda36b740 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f4098d3b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eb8ff9640 0 client.16257 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:21.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eda36b740 1 --2- 172.21.15.77:0/1480590179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40982990 0x558f4098ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eda36b740 1 -- 172.21.15.77:0/1480590179 >> 172.21.15.77:0/1480590179 conn(0x558f4094ac50 msgr2=0x558f4097d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eda36b740 1 -- 172.21.15.77:0/1480590179 shutdown_connections 2024-03-16T10:19:21.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eda36b740 1 -- 172.21.15.77:0/1480590179 wait complete. 2024-03-16T10:19:21.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.605+0000 7f3eda36b740 20 client.16257 trim_cache size 0 max 16384 2024-03-16T10:19:21.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.623+0000 7f3eda36b740 1 Processor -- start 2024-03-16T10:19:21.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.623+0000 7f3eda36b740 1 -- start start 2024-03-16T10:19:21.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.623+0000 7f3eda36b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40982990 0x558f40981540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.623+0000 7f3eda36b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f409943b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.623+0000 7f3eda36b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f409884f0 0x558f40981a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.623+0000 7f3eda36b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01aad80 con 0x558f40983380 2024-03-16T10:19:21.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.623+0000 7f3eda36b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01ac9e0 con 0x558f409884f0 2024-03-16T10:19:21.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.623+0000 7f3eda36b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01ab450 con 0x558f40982990 2024-03-16T10:19:21.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ed4d19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f409943b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ed4d19640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f409943b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44648/0 (socket says 172.21.15.77:44648) 2024-03-16T10:19:21.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ed4d19640 1 -- 172.21.15.77:0/1569566324 learned_addr learned my addr 172.21.15.77:0/1569566324 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ed551a640 1 --2- 172.21.15.77:0/1569566324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40982990 0x558f40981540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ed5d1b640 1 --2- 172.21.15.77:0/1569566324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f409884f0 0x558f40981a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ed4d19640 1 -- 172.21.15.77:0/1569566324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40982990 msgr2=0x558f40981540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ed4d19640 1 --2- 172.21.15.77:0/1569566324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40982990 0x558f40981540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ed4d19640 1 -- 172.21.15.77:0/1569566324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f409884f0 msgr2=0x558f40981a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ed4d19640 1 --2- 172.21.15.77:0/1569566324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f409884f0 0x558f40981a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ed4d19640 1 -- 172.21.15.77:0/1569566324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f40983760 con 0x558f40983380 2024-03-16T10:19:21.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ed4d19640 1 --2- 172.21.15.77:0/1569566324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f409943b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0009850 tx=0x7f3ec002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab95aee931db819 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ebaffd640 1 -- 172.21.15.77:0/1569566324 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0056030 con 0x558f40983380 2024-03-16T10:19:21.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ebaffd640 1 -- 172.21.15.77:0/1569566324 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec0003ca0 con 0x558f40983380 2024-03-16T10:19:21.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.624+0000 7f3ebaffd640 1 -- 172.21.15.77:0/1569566324 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0005110 con 0x558f40983380 2024-03-16T10:19:21.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 -- 172.21.15.77:0/1569566324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 msgr2=0x558f409943b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 --2- 172.21.15.77:0/1569566324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f409943b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0009850 tx=0x7f3ec002bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:21.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 -- 172.21.15.77:0/1569566324 shutdown_connections 2024-03-16T10:19:21.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 --2- 172.21.15.77:0/1569566324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f409884f0 0x558f40981a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 --2- 172.21.15.77:0/1569566324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f40983380 0x558f409943b0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 --2- 172.21.15.77:0/1569566324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40982990 0x558f40981540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 -- 172.21.15.77:0/1569566324 >> 172.21.15.77:0/1569566324 conn(0x558f4094ac50 msgr2=0x558f40984820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 -- 172.21.15.77:0/1569566324 shutdown_connections 2024-03-16T10:19:21.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 -- 172.21.15.77:0/1569566324 wait complete. 2024-03-16T10:19:21.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 -- start start 2024-03-16T10:19:21.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 Processor -- start 2024-03-16T10:19:21.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f40982990 0x558f40993d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40983380 0x558f40994260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.625+0000 7f3eda36b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f409884f0 0x558f409923d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3eda36b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01ac250 con 0x558f409884f0 2024-03-16T10:19:21.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3eda36b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01abb80 con 0x558f40982990 2024-03-16T10:19:21.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3eda36b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01a8d60 con 0x558f40983380 2024-03-16T10:19:21.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3eda36b740 10 client.0 _ll_register_callbacks cb 0x7f3ec0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:21.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3ed5d1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f409884f0 0x558f409923d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3ed551a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f40982990 0x558f40993d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3ed5d1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f409884f0 0x558f409923d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44664/0 (socket says 172.21.15.77:44664) 2024-03-16T10:19:21.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3ed5d1b640 1 -- 172.21.15.77:0/947648606 learned_addr learned my addr 172.21.15.77:0/947648606 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3ed4d19640 1 --2- 172.21.15.77:0/947648606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40983380 0x558f40994260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3ed5d1b640 1 -- 172.21.15.77:0/947648606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40983380 msgr2=0x558f40994260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3ed5d1b640 1 --2- 172.21.15.77:0/947648606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40983380 0x558f40994260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3ed5d1b640 1 -- 172.21.15.77:0/947648606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f40982990 msgr2=0x558f40993d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3ed5d1b640 1 --2- 172.21.15.77:0/947648606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f40982990 0x558f40993d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3ed5d1b640 1 -- 172.21.15.77:0/947648606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f40992b60 con 0x558f409884f0 2024-03-16T10:19:21.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3ed5d1b640 1 --2- 172.21.15.77:0/947648606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f409884f0 0x558f409923d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80209a0 tx=0x7f3ec8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7524a39d9c8901f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3eb8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:21.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3eb8ff9640 1 -- 172.21.15.77:0/947648606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec802f060 con 0x558f409884f0 2024-03-16T10:19:21.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3eda36b740 1 -- 172.21.15.77:0/947648606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ec0003ca0 con 0x558f409884f0 2024-03-16T10:19:21.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3eb8ff9640 1 -- 172.21.15.77:0/947648606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f3ec801edc0 con 0x558f409884f0 2024-03-16T10:19:21.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3eb8ff9640 1 -- 172.21.15.77:0/947648606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3ec8010430 con 0x558f409884f0 2024-03-16T10:19:21.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3eda36b740 10 client.16263 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:21.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.626+0000 7f3eda36b740 1 -- 172.21.15.77:0/947648606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558f409ff890 con 0x558f409884f0 2024-03-16T10:19:21.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.627+0000 7f3eb8ff9640 1 -- 172.21.15.77:0/947648606 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec0003ca0 con 0x558f409884f0 2024-03-16T10:19:21.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.627+0000 7f3eb8ff9640 1 -- 172.21.15.77:0/947648606 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f3ec8029070 con 0x558f409884f0 2024-03-16T10:19:21.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.627+0000 7f3eda36b740 10 client.16263 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.627+0000 7f3eda36b740 10 client.16263 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.627+0000 7f3eda36b740 20 client.16263 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.627+0000 7f3eda36b740 1 --2- 172.21.15.77:0/947648606 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f40a00360 0x558f40a02780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.627+0000 7f3eda36b740 4 client.16263 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T10:19:21.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.627+0000 7f3eda36b740 1 -- 172.21.15.77:0/947648606 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558f4098cc60 con 0x558f40a00360 2024-03-16T10:19:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.627+0000 7f3ed551a640 1 --2- 172.21.15.77:0/947648606 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f40a00360 0x558f40a02780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.627+0000 7f3eb8ff9640 10 client.16263 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.627+0000 7f3ed551a640 1 --2- 172.21.15.77:0/947648606 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f40a00360 0x558f40a02780 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a7f3bb0e453830e server_cookie=6ac8c01f98595d62 in_seq=0 out_seq=0 2024-03-16T10:19:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eb8ff9640 1 -- 172.21.15.77:0/947648606 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558f4098cc60 con 0x558f40a00360 2024-03-16T10:19:21.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eb8ff9640 10 client.16263 handle_command_reply: tid=0 2024-03-16T10:19:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eda36b740 1 client.16263 shutdown 2024-03-16T10:19:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eda36b740 1 -- 172.21.15.77:0/947648606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f409884f0 msgr2=0x558f409923d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eda36b740 1 --2- 172.21.15.77:0/947648606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f409884f0 0x558f409923d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80209a0 tx=0x7f3ec8004ab0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eda36b740 1 -- 172.21.15.77:0/947648606 shutdown_connections 2024-03-16T10:19:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eda36b740 1 --2- 172.21.15.77:0/947648606 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558f40a00360 0x558f40a02780 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eda36b740 1 --2- 172.21.15.77:0/947648606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f409884f0 0x558f409923d0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eb8ff9640 0 client.16263 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:21.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eda36b740 1 --2- 172.21.15.77:0/947648606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f40983380 0x558f40994260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eda36b740 1 --2- 172.21.15.77:0/947648606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558f40982990 0x558f40993d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eda36b740 1 -- 172.21.15.77:0/947648606 >> 172.21.15.77:0/947648606 conn(0x558f4094ac50 msgr2=0x558f40981540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eda36b740 1 -- 172.21.15.77:0/947648606 shutdown_connections 2024-03-16T10:19:21.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.629+0000 7f3eda36b740 1 -- 172.21.15.77:0/947648606 wait complete. 2024-03-16T10:19:21.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.630+0000 7f3eda36b740 20 client.16263 trim_cache size 0 max 16384 2024-03-16T10:19:21.651 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:19:21.651 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T10:19:21.651 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "beb30b81-8f3e-4064-9c56-547f603e44cd", 2024-03-16T10:19:21.651 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T10:19:21.651 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:19:21.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.630+0000 7f3ed777e640 1 -- 172.21.15.77:0/3253265064 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3ebc0a2800 msgr2=0x7f3ebc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.630+0000 7f3ed777e640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3ebc0a2800 0x7f3ebc0a4cc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x558f40983be0 tx=0x7f3ec8004a40 comp rx=0 tx=0).stop 2024-03-16T10:19:21.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.630+0000 7f3ed777e640 1 -- 172.21.15.77:0/3253265064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ed0100700 msgr2=0x7f3ed01047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.630+0000 7f3ed777e640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ed0100700 0x7f3ed01047e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc00e9c0 tx=0x7f3ecc00ee80 comp rx=0 tx=0).stop 2024-03-16T10:19:21.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.630+0000 7f3ed777e640 1 -- 172.21.15.77:0/3253265064 shutdown_connections 2024-03-16T10:19:21.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.630+0000 7f3ed777e640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3ebc0a2800 0x7f3ebc0a4cc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.630+0000 7f3ed777e640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ed01022b0 0x7f3ed01033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.630+0000 7f3ed777e640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ed01018f0 0x7f3ed0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.630+0000 7f3ed777e640 1 --2- 172.21.15.77:0/3253265064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ed0100700 0x7f3ed01047e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.630+0000 7f3ed777e640 1 -- 172.21.15.77:0/3253265064 >> 172.21.15.77:0/3253265064 conn(0x7f3ed00fbc70 msgr2=0x7f3ed0069a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.631+0000 7f3ed777e640 1 -- 172.21.15.77:0/3253265064 shutdown_connections 2024-03-16T10:19:21.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.631+0000 7f3ed777e640 1 -- 172.21.15.77:0/3253265064 wait complete. 2024-03-16T10:19:21.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:19:21.656 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:19:21.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.763+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2161611857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41018d0 msgr2=0x7f40c4101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.763+0000 7f40cb9e6640 1 --2- 172.21.15.77:0/2161611857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41018d0 0x7f40c4101d50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009900 tx=0x7f40b402bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:21.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.763+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2161611857 shutdown_connections 2024-03-16T10:19:21.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.763+0000 7f40cb9e6640 1 --2- 172.21.15.77:0/2161611857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c4102290 0x7f40c4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.763+0000 7f40cb9e6640 1 --2- 172.21.15.77:0/2161611857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41018d0 0x7f40c4101d50 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.763+0000 7f40cb9e6640 1 --2- 172.21.15.77:0/2161611857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40c41006e0 0x7f40c4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2161611857 >> 172.21.15.77:0/2161611857 conn(0x7f40c40fbc70 msgr2=0x7f40c40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2161611857 shutdown_connections 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2161611857 wait complete. 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40cb9e6640 1 Processor -- start 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40cb9e6640 1 -- start start 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40cb9e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41006e0 0x7f40c4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40cb9e6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c41018d0 0x7f40c4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40cb9e6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40c4102290 0x7f40c419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40cb9e6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f40c410c360 con 0x7f40c41006e0 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40cb9e6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f40c410c1e0 con 0x7f40c4102290 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40cb9e6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40c410c4e0 con 0x7f40c41018d0 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40c9782640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41006e0 0x7f40c4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40c9782640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41006e0 0x7f40c4197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44678/0 (socket says 172.21.15.77:44678) 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40c9782640 1 -- 172.21.15.77:0/2411964381 learned_addr learned my addr 172.21.15.77:0/2411964381 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40c9f83640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40c4102290 0x7f40c419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.764+0000 7f40c8f81640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c41018d0 0x7f40c4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.765+0000 7f40c9782640 1 -- 172.21.15.77:0/2411964381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c41018d0 msgr2=0x7f40c4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.765+0000 7f40c9782640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c41018d0 0x7f40c4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.765+0000 7f40c9782640 1 -- 172.21.15.77:0/2411964381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40c4102290 msgr2=0x7f40c419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.765+0000 7f40c9782640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40c4102290 0x7f40c419f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.765+0000 7f40c9782640 1 -- 172.21.15.77:0/2411964381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c419f720 con 0x7f40c41006e0 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.765+0000 7f40c9782640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41006e0 0x7f40c4197b50 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00e9c0 tx=0x7f40bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.765+0000 7f40ba7fc640 1 -- 172.21.15.77:0/2411964381 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc00cd80 con 0x7f40c41006e0 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.765+0000 7f40ba7fc640 1 -- 172.21.15.77:0/2411964381 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40bc0035f0 con 0x7f40c41006e0 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.765+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2411964381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c419fa10 con 0x7f40c41006e0 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.765+0000 7f40ba7fc640 1 -- 172.21.15.77:0/2411964381 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc010690 con 0x7f40c41006e0 2024-03-16T10:19:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.765+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2411964381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c419ff30 con 0x7f40c41006e0 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.767+0000 7f40ba7fc640 1 -- 172.21.15.77:0/2411964381 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f40bc0037a0 con 0x7f40c41006e0 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40ce5d0740 1 Processor -- start 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40ce5d0740 1 -- start start 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40ce5d0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad33e170 0x55c7ad33e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40ce5d0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 0x55c7ad343660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40ce5d0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 0x55c7ad346160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40ce5d0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c7ad2a3500 con 0x55c7ad343cd0 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40ce5d0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c7ad24bbe0 con 0x55c7ad33eb60 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40ce5d0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c7ad29f5a0 con 0x55c7ad33e170 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40c8f81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 0x55c7ad346160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40c9782640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 0x55c7ad343660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40c8f81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 0x55c7ad346160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44682/0 (socket says 172.21.15.77:44682) 2024-03-16T10:19:21.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40c8f81640 1 -- 172.21.15.77:0/2373217116 learned_addr learned my addr 172.21.15.77:0/2373217116 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40c9f83640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad33e170 0x55c7ad33e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40ba7fc640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f40b00a2800 0x7f40b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40c8f81640 1 -- 172.21.15.77:0/2373217116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad33e170 msgr2=0x55c7ad33e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40c8f81640 1 --2- 172.21.15.77:0/2373217116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad33e170 0x55c7ad33e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40c8f81640 1 -- 172.21.15.77:0/2373217116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 msgr2=0x55c7ad343660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40c8f81640 1 --2- 172.21.15.77:0/2373217116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 0x55c7ad343660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40c8f81640 1 -- 172.21.15.77:0/2373217116 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7ad25f4b0 con 0x55c7ad343cd0 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40c9782640 1 --2- 172.21.15.77:0/2373217116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 0x55c7ad343660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40c9f83640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f40b00a2800 0x7f40b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.768+0000 7f40ba7fc640 1 -- 172.21.15.77:0/2411964381 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f40bc0ced80 con 0x7f40c41006e0 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.769+0000 7f40c8f81640 1 --2- 172.21.15.77:0/2373217116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 0x55c7ad346160 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009900 tx=0x7f40b4009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2d44f7ef628ee5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.769+0000 7f40af7fe640 1 -- 172.21.15.77:0/2373217116 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b402cb80 con 0x55c7ad343cd0 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.769+0000 7f40af7fe640 1 -- 172.21.15.77:0/2373217116 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40b4005040 con 0x55c7ad343cd0 2024-03-16T10:19:21.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.769+0000 7f40af7fe640 1 -- 172.21.15.77:0/2373217116 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4004900 con 0x55c7ad343cd0 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.769+0000 7f40ce5d0740 1 -- 172.21.15.77:0/2373217116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 msgr2=0x55c7ad346160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.769+0000 7f40c9f83640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f40b00a2800 0x7f40b00a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009fd0 tx=0x7f40c0004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.769+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/2373217116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 0x55c7ad346160 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009900 tx=0x7f40b4009560 comp rx=0 tx=0).stop 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.769+0000 7f40ce5d0740 1 -- 172.21.15.77:0/2373217116 shutdown_connections 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.769+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/2373217116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 0x55c7ad346160 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/2373217116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 0x55c7ad343660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/2373217116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad33e170 0x55c7ad33e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 -- 172.21.15.77:0/2373217116 >> 172.21.15.77:0/2373217116 conn(0x55c7ad3396e0 msgr2=0x55c7ad339ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 -- 172.21.15.77:0/2373217116 shutdown_connections 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 -- 172.21.15.77:0/2373217116 wait complete. 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 -- start start 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 Processor -- start 2024-03-16T10:19:21.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33e170 0x55c7ad348650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad33eb60 0x55c7ad348b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 0x55c7ad34fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c7ad346c70 con 0x55c7ad343cd0 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c7ad25f160 con 0x55c7ad33e170 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c7ad346df0 con 0x55c7ad33eb60 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40ce5d0740 10 client.0 _ll_register_callbacks cb 0x55c7ad243150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40c9f83640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 0x55c7ad34fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40c9782640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33e170 0x55c7ad348650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40c9f83640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 0x55c7ad34fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44698/0 (socket says 172.21.15.77:44698) 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40c9f83640 1 -- 172.21.15.77:0/155249824 learned_addr learned my addr 172.21.15.77:0/155249824 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.770+0000 7f40c8f81640 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad33eb60 0x55c7ad348b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40c9f83640 1 -- 172.21.15.77:0/155249824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad33eb60 msgr2=0x55c7ad348b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40c9f83640 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad33eb60 0x55c7ad348b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40c9f83640 1 -- 172.21.15.77:0/155249824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33e170 msgr2=0x55c7ad348650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40c9f83640 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33e170 0x55c7ad348650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40c9f83640 1 -- 172.21.15.77:0/155249824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7ad0817d0 con 0x55c7ad343cd0 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40c8f81640 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad33eb60 0x55c7ad348b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40c9f83640 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 0x55c7ad34fbd0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f40c000fb90 tx=0x7f40c0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b74f912ab57fff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40acff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40acff9640 1 -- 172.21.15.77:0/155249824 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c001eac0 con 0x55c7ad343cd0 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40acff9640 1 -- 172.21.15.77:0/155249824 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f40c001ec60 con 0x55c7ad343cd0 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40ce5d0740 1 -- 172.21.15.77:0/155249824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40b4005040 con 0x55c7ad343cd0 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40acff9640 1 -- 172.21.15.77:0/155249824 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f40c001ee60 con 0x55c7ad343cd0 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40ce5d0740 10 client.16275 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40ce5d0740 1 -- 172.21.15.77:0/155249824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c7acfb73f0 con 0x55c7ad343cd0 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.771+0000 7f40acff9640 1 -- 172.21.15.77:0/155249824 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40b4005040 con 0x55c7ad343cd0 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.772+0000 7f40acff9640 1 -- 172.21.15.77:0/155249824 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f40c0031050 con 0x55c7ad343cd0 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.772+0000 7f40ce5d0740 10 client.16275 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:21.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.772+0000 7f40ce5d0740 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.772+0000 7f40ce5d0740 20 client.16275 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.772+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c7ad3541a0 0x55c7ad356600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.772+0000 7f40ce5d0740 4 client.16275 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.772+0000 7f40ce5d0740 1 -- 172.21.15.77:0/155249824 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7ad2b0a40 con 0x55c7ad3541a0 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.772+0000 7f40c9782640 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c7ad3541a0 0x55c7ad356600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.772+0000 7f40c9782640 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c7ad3541a0 0x55c7ad356600 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=224938a86128ff07 server_cookie=4e3f801f1cec3cd2 in_seq=0 out_seq=0 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.772+0000 7f40acff9640 10 client.16275 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.773+0000 7f40acff9640 1 -- 172.21.15.77:0/155249824 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c7ad2b0a40 con 0x55c7ad3541a0 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.773+0000 7f40acff9640 10 client.16275 handle_command_reply: tid=0 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.773+0000 7f40ce5d0740 1 client.16275 shutdown 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.773+0000 7f40ce5d0740 1 -- 172.21.15.77:0/155249824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 msgr2=0x55c7ad34fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.773+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 0x55c7ad34fbd0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f40c000fb90 tx=0x7f40c0004ab0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.773+0000 7f40ce5d0740 1 -- 172.21.15.77:0/155249824 shutdown_connections 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.773+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c7ad3541a0 0x55c7ad356600 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.773+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad343cd0 0x55c7ad34fbd0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.773+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad33eb60 0x55c7ad348b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.773+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/155249824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33e170 0x55c7ad348650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.773+0000 7f40ce5d0740 1 -- 172.21.15.77:0/155249824 >> 172.21.15.77:0/155249824 conn(0x55c7ad3396e0 msgr2=0x55c7ad338900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.773+0000 7f40acff9640 0 client.16275 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:21.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.774+0000 7f40ce5d0740 1 -- 172.21.15.77:0/155249824 shutdown_connections 2024-03-16T10:19:21.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.774+0000 7f40ce5d0740 1 -- 172.21.15.77:0/155249824 wait complete. 2024-03-16T10:19:21.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.774+0000 7f40ce5d0740 20 client.16275 trim_cache size 0 max 16384 2024-03-16T10:19:21.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.792+0000 7f40ce5d0740 1 Processor -- start 2024-03-16T10:19:21.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.792+0000 7f40ce5d0740 1 -- start start 2024-03-16T10:19:21.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.792+0000 7f40ce5d0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33e170 0x55c7ad340060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.792+0000 7f40ce5d0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad33eb60 0x55c7ad3405a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.792+0000 7f40ce5d0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 0x55c7ad34fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.792+0000 7f40ce5d0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f40c41a8b40 con 0x55c7ad33eb60 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.792+0000 7f40ce5d0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f40c41aa7a0 con 0x55c7ad33e170 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.792+0000 7f40ce5d0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40c41a9210 con 0x55c7ad343cd0 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40c9782640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33e170 0x55c7ad340060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40c9f83640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 0x55c7ad34fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40c9782640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33e170 0x55c7ad340060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41134/0 (socket says 172.21.15.77:41134) 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40c9f83640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 0x55c7ad34fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:37312/0 (socket says 172.21.15.77:37312) 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40c9782640 1 -- 172.21.15.77:0/3186424008 learned_addr learned my addr 172.21.15.77:0/3186424008 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40c8f81640 1 --2- 172.21.15.77:0/3186424008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad33eb60 0x55c7ad3405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40c9f83640 1 -- 172.21.15.77:0/3186424008 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33e170 msgr2=0x55c7ad340060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40c9f83640 1 --2- 172.21.15.77:0/3186424008 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33e170 0x55c7ad340060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40c9f83640 1 -- 172.21.15.77:0/3186424008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad33eb60 msgr2=0x55c7ad3405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40c9f83640 1 --2- 172.21.15.77:0/3186424008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad33eb60 0x55c7ad3405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40c9f83640 1 -- 172.21.15.77:0/3186424008 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7acf770f0 con 0x55c7ad343cd0 2024-03-16T10:19:21.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40c9f83640 1 --2- 172.21.15.77:0/3186424008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 0x55c7ad34fb90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f40c001e410 tx=0x7f40c00104b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ded23d9bfe501824 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40aeffd640 1 -- 172.21.15.77:0/3186424008 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c000f950 con 0x55c7ad343cd0 2024-03-16T10:19:21.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40aeffd640 1 -- 172.21.15.77:0/3186424008 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c0037070 con 0x55c7ad343cd0 2024-03-16T10:19:21.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.793+0000 7f40aeffd640 1 -- 172.21.15.77:0/3186424008 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0033bb0 con 0x55c7ad343cd0 2024-03-16T10:19:21.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.794+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3186424008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 msgr2=0x55c7ad34fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.794+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/3186424008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 0x55c7ad34fb90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f40c001e410 tx=0x7f40c00104b0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.794+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3186424008 shutdown_connections 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.794+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/3186424008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 0x55c7ad34fb90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.794+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/3186424008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad33eb60 0x55c7ad3405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.794+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/3186424008 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33e170 0x55c7ad340060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.794+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3186424008 >> 172.21.15.77:0/3186424008 conn(0x55c7ad3396e0 msgr2=0x55c7ad348650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.794+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3186424008 shutdown_connections 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.794+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3186424008 wait complete. 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40ce5d0740 1 -- start start 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40ce5d0740 1 Processor -- start 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40ce5d0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad33e170 0x55c7ad34f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40ce5d0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 0x55c7ad34c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40ce5d0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 0x55c7ad34c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40ce5d0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f40c41aa010 con 0x55c7ad33e170 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40ce5d0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f40c41a9940 con 0x55c7ad33eb60 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40ce5d0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40c41a6b20 con 0x55c7ad343cd0 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40ce5d0740 10 client.0 _ll_register_callbacks cb 0x7f40c0009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40c8f81640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 0x55c7ad34c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40c8f81640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 0x55c7ad34c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41136/0 (socket says 172.21.15.77:41136) 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40c8f81640 1 -- 172.21.15.77:0/3658955445 learned_addr learned my addr 172.21.15.77:0/3658955445 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.795+0000 7f40c9782640 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad33e170 0x55c7ad34f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40c9f83640 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 0x55c7ad34c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40c8f81640 1 -- 172.21.15.77:0/3658955445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 msgr2=0x55c7ad34c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40c8f81640 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 0x55c7ad34c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40c8f81640 1 -- 172.21.15.77:0/3658955445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad33e170 msgr2=0x55c7ad34f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40c8f81640 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad33e170 0x55c7ad34f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40c8f81640 1 -- 172.21.15.77:0/3658955445 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7ad26a9d0 con 0x55c7ad33eb60 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40c9f83640 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 0x55c7ad34c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40c8f81640 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 0x55c7ad34c340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009900 tx=0x7f40b40044f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14f05f3f0e391c57 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40acff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40acff9640 1 -- 172.21.15.77:0/3658955445 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4005440 con 0x55c7ad33eb60 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40acff9640 1 -- 172.21.15.77:0/3658955445 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f40b40055e0 con 0x55c7ad33eb60 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3658955445 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40c0037070 con 0x55c7ad33eb60 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40acff9640 1 -- 172.21.15.77:0/3658955445 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f40b40046c0 con 0x55c7ad33eb60 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40ce5d0740 10 client.26066 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.796+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3658955445 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c7ad3bd740 con 0x55c7ad33eb60 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.797+0000 7f40acff9640 1 -- 172.21.15.77:0/3658955445 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c0037070 con 0x55c7ad33eb60 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.797+0000 7f40acff9640 1 -- 172.21.15.77:0/3658955445 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f40b4003bb0 con 0x55c7ad33eb60 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.797+0000 7f40ce5d0740 10 client.26066 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:21.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.797+0000 7f40ce5d0740 10 client.26066 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.797+0000 7f40ce5d0740 20 client.26066 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.797+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c7ad3bde40 0x55c7ad3c0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.797+0000 7f40ce5d0740 4 client.26066 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.797+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3658955445 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c7ad2aad90 con 0x55c7ad3bde40 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.797+0000 7f40c9782640 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c7ad3bde40 0x55c7ad3c0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40acff9640 10 client.26066 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40c9782640 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c7ad3bde40 0x55c7ad3c0260 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=904630f29f589d21 server_cookie=30a0b083c8c74756 in_seq=0 out_seq=0 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40acff9640 1 -- 172.21.15.77:0/3658955445 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c7ad2aad90 con 0x55c7ad3bde40 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40acff9640 10 client.26066 handle_command_reply: tid=0 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40ce5d0740 1 client.26066 shutdown 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3658955445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 msgr2=0x55c7ad34c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 0x55c7ad34c340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009900 tx=0x7f40b40044f0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3658955445 shutdown_connections 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7ad343cd0 0x55c7ad34c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.804 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c7ad33eb60 0x55c7ad34c340 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c7ad3bde40 0x55c7ad3c0260 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40ce5d0740 1 --2- 172.21.15.77:0/3658955445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c7ad33e170 0x55c7ad34f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40acff9640 0 client.26066 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:21.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3658955445 >> 172.21.15.77:0/3658955445 conn(0x55c7ad3396e0 msgr2=0x55c7ad348650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3658955445 shutdown_connections 2024-03-16T10:19:21.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40ce5d0740 1 -- 172.21.15.77:0/3658955445 wait complete. 2024-03-16T10:19:21.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.798+0000 7f40ce5d0740 20 client.26066 trim_cache size 0 max 16384 2024-03-16T10:19:21.805 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:19:21.805 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:19:21.805 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:19:21.806 INFO:teuthology.orchestra.run.smithi077.stdout: "beb30b81-8f3e-4064-9c56-547f603e44cd": { 2024-03-16T10:19:21.806 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:19:21.806 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "beb30b81-8f3e-4064-9c56-547f603e44cd", 2024-03-16T10:19:21.806 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:19:21.806 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:19:21.806 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:19:21.806 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:19:21.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.799+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2411964381 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f40b00a2800 msgr2=0x7f40b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.799+0000 7f40cb9e6640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f40b00a2800 0x7f40b00a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009fd0 tx=0x7f40c0004a40 comp rx=0 tx=0).stop 2024-03-16T10:19:21.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.799+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2411964381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41006e0 msgr2=0x7f40c4197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.799+0000 7f40cb9e6640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41006e0 0x7f40c4197b50 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00e9c0 tx=0x7f40bc00ee80 comp rx=0 tx=0).stop 2024-03-16T10:19:21.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.799+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2411964381 shutdown_connections 2024-03-16T10:19:21.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.799+0000 7f40cb9e6640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40c4102290 0x7f40c419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.799+0000 7f40cb9e6640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f40b00a2800 0x7f40b00a4cc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.799+0000 7f40cb9e6640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c41018d0 0x7f40c4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.799+0000 7f40cb9e6640 1 --2- 172.21.15.77:0/2411964381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40c41006e0 0x7f40c4197b50 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.799+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2411964381 >> 172.21.15.77:0/2411964381 conn(0x7f40c40fbc70 msgr2=0x7f40c4068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.799+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2411964381 shutdown_connections 2024-03-16T10:19:21.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:21.799+0000 7f40cb9e6640 1 -- 172.21.15.77:0/2411964381 wait complete. 2024-03-16T10:19:21.811 INFO:tasks.cephfs.filesystem:scrub status for tag:beb30b81-8f3e-4064-9c56-547f603e44cd - {'path': '/', 'tag': 'beb30b81-8f3e-4064-9c56-547f603e44cd', 'options': 'recursive,force'} 2024-03-16T10:19:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.926+0000 7fc819986640 1 -- 172.21.15.77:0/2161254979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8140733f0 msgr2=0x7fc814108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.926+0000 7fc819986640 1 --2- 172.21.15.77:0/2161254979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8140733f0 0x7fc814108e10 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009ca0 tx=0x7fc80c02c140 comp rx=0 tx=0).stop 2024-03-16T10:19:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.927+0000 7fc819986640 1 -- 172.21.15.77:0/2161254979 shutdown_connections 2024-03-16T10:19:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.927+0000 7fc819986640 1 --2- 172.21.15.77:0/2161254979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8141094d0 0x7fc81410e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.927+0000 7fc819986640 1 --2- 172.21.15.77:0/2161254979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8140733f0 0x7fc814108e10 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.927+0000 7fc819986640 1 --2- 172.21.15.77:0/2161254979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc814072a90 0x7fc814072eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.927+0000 7fc819986640 1 -- 172.21.15.77:0/2161254979 >> 172.21.15.77:0/2161254979 conn(0x7fc81406e650 msgr2=0x7fc814070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.927+0000 7fc819986640 1 -- 172.21.15.77:0/2161254979 shutdown_connections 2024-03-16T10:19:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.927+0000 7fc819986640 1 -- 172.21.15.77:0/2161254979 wait complete. 2024-03-16T10:19:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.928+0000 7fc819986640 1 Processor -- start 2024-03-16T10:19:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.928+0000 7fc819986640 1 -- start start 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.928+0000 7fc819986640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc814072a90 0x7fc814114ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.928+0000 7fc819986640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8140733f0 0x7fc814115430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.928+0000 7fc819986640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8141094d0 0x7fc8141b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.928+0000 7fc819986640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc81411c210 con 0x7fc814072a90 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.928+0000 7fc819986640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc81411c090 con 0x7fc8140733f0 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.928+0000 7fc819986640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc81411c390 con 0x7fc8141094d0 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.929+0000 7fc819185640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8141094d0 0x7fc8141b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.929+0000 7fc818984640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc814072a90 0x7fc814114ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.929+0000 7fc818984640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc814072a90 0x7fc814114ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44720/0 (socket says 172.21.15.77:44720) 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.929+0000 7fc818984640 1 -- 172.21.15.77:0/3813405138 learned_addr learned my addr 172.21.15.77:0/3813405138 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.929+0000 7fc813fff640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8140733f0 0x7fc814115430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.929+0000 7fc813fff640 1 -- 172.21.15.77:0/3813405138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8141094d0 msgr2=0x7fc8141b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.929+0000 7fc813fff640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8141094d0 0x7fc8141b6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.929+0000 7fc813fff640 1 -- 172.21.15.77:0/3813405138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc814072a90 msgr2=0x7fc814114ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.929+0000 7fc813fff640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc814072a90 0x7fc814114ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.929+0000 7fc813fff640 1 -- 172.21.15.77:0/3813405138 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8141b6e60 con 0x7fc8140733f0 2024-03-16T10:19:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.929+0000 7fc818984640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc814072a90 0x7fc814114ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.929+0000 7fc813fff640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8140733f0 0x7fc814115430 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc80c037040 tx=0x7fc80c035dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.930+0000 7fc811ffb640 1 -- 172.21.15.77:0/3813405138 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c039040 con 0x7fc8140733f0 2024-03-16T10:19:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.930+0000 7fc819986640 1 -- 172.21.15.77:0/3813405138 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8141b7030 con 0x7fc8140733f0 2024-03-16T10:19:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.930+0000 7fc811ffb640 1 -- 172.21.15.77:0/3813405138 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc80c003e50 con 0x7fc8140733f0 2024-03-16T10:19:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.930+0000 7fc811ffb640 1 -- 172.21.15.77:0/3813405138 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c03f6a0 con 0x7fc8140733f0 2024-03-16T10:19:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.930+0000 7fc819986640 1 -- 172.21.15.77:0/3813405138 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8141b7570 con 0x7fc8140733f0 2024-03-16T10:19:21.934 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.933+0000 7fc811ffb640 1 -- 172.21.15.77:0/3813405138 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fc80c046030 con 0x7fc8140733f0 2024-03-16T10:19:21.935 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.934+0000 7fc81d7cb740 1 Processor -- start 2024-03-16T10:19:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.934+0000 7fc811ffb640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7fc0a2930 0x7fc7fc0a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.934+0000 7fc811ffb640 1 -- 172.21.15.77:0/3813405138 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc80c0f0790 con 0x7fc8140733f0 2024-03-16T10:19:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.935+0000 7fc819185640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7fc0a2930 0x7fc7fc0a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.935+0000 7fc81d7cb740 1 -- start start 2024-03-16T10:19:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.935+0000 7fc81d7cb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 0x5589449278f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.935+0000 7fc81d7cb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558944927ec0 0x55894492c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.935+0000 7fc81d7cb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55894492d030 0x55894492f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.935+0000 7fc81d7cb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55894488ca40 con 0x558944927ec0 2024-03-16T10:19:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.935+0000 7fc81d7cb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x558944835920 con 0x5589449274d0 2024-03-16T10:19:21.937 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.935+0000 7fc81d7cb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558944888ae0 con 0x55894492d030 2024-03-16T10:19:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.935+0000 7fc818984640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 0x5589449278f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.935+0000 7fc818984640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 0x5589449278f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41150/0 (socket says 172.21.15.77:41150) 2024-03-16T10:19:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.935+0000 7fc818984640 1 -- 172.21.15.77:0/2126486339 learned_addr learned my addr 172.21.15.77:0/2126486339 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.936+0000 7fc813fff640 1 --2- 172.21.15.77:0/2126486339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558944927ec0 0x55894492c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.936+0000 7fc818984640 1 -- 172.21.15.77:0/2126486339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55894492d030 msgr2=0x55894492f4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T10:19:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.936+0000 7fc818984640 1 --2- 172.21.15.77:0/2126486339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55894492d030 0x55894492f4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.936+0000 7fc818984640 1 -- 172.21.15.77:0/2126486339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558944927ec0 msgr2=0x55894492c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.936+0000 7fc818984640 1 --2- 172.21.15.77:0/2126486339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558944927ec0 0x55894492c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.936+0000 7fc818984640 1 -- 172.21.15.77:0/2126486339 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55894484d460 con 0x5589449274d0 2024-03-16T10:19:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.936+0000 7fc813fff640 1 --2- 172.21.15.77:0/2126486339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558944927ec0 0x55894492c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:19:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.936+0000 7fc818984640 1 --2- 172.21.15.77:0/2126486339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 0x5589449278f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc808009ed0 tx=0x7fc80800e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=608922cc14741e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.936+0000 7fc819185640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7fc0a2930 0x7fc7fc0a4df0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc804006fd0 tx=0x7fc804009060 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.963 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.962+0000 7fc802ffd640 1 -- 172.21.15.77:0/2126486339 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808017070 con 0x5589449274d0 2024-03-16T10:19:21.963 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.962+0000 7fc802ffd640 1 -- 172.21.15.77:0/2126486339 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc808013950 con 0x5589449274d0 2024-03-16T10:19:21.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.963+0000 7fc81d7cb740 1 -- 172.21.15.77:0/2126486339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 msgr2=0x5589449278f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.963+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/2126486339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 0x5589449278f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc808009ed0 tx=0x7fc80800e5a0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.963+0000 7fc81d7cb740 1 -- 172.21.15.77:0/2126486339 shutdown_connections 2024-03-16T10:19:21.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.963+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/2126486339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55894492d030 0x55894492f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.963+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/2126486339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558944927ec0 0x55894492c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.963+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/2126486339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 0x5589449278f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.964 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.963+0000 7fc81d7cb740 1 -- 172.21.15.77:0/2126486339 >> 172.21.15.77:0/2126486339 conn(0x558944922a40 msgr2=0x558944922e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.968+0000 7fc81d7cb740 1 -- 172.21.15.77:0/2126486339 shutdown_connections 2024-03-16T10:19:21.969 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.968+0000 7fc81d7cb740 1 -- 172.21.15.77:0/2126486339 wait complete. 2024-03-16T10:19:21.970 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.969+0000 7fc81d7cb740 1 -- start start 2024-03-16T10:19:21.970 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.969+0000 7fc81d7cb740 1 Processor -- start 2024-03-16T10:19:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.970+0000 7fc81d7cb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 0x558944931a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.970+0000 7fc81d7cb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558944927ec0 0x558944931f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.970+0000 7fc81d7cb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55894492d030 0x558944938fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.970+0000 7fc81d7cb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558944930030 con 0x558944927ec0 2024-03-16T10:19:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.970+0000 7fc81d7cb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x558944848d70 con 0x55894492d030 2024-03-16T10:19:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.970+0000 7fc81d7cb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5589449301b0 con 0x5589449274d0 2024-03-16T10:19:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.970+0000 7fc81d7cb740 10 client.0 _ll_register_callbacks cb 0x5589445a0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.970+0000 7fc818984640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 0x558944931a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.970+0000 7fc818984640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 0x558944931a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:37372/0 (socket says 172.21.15.77:37372) 2024-03-16T10:19:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.970+0000 7fc818984640 1 -- 172.21.15.77:0/1599783291 learned_addr learned my addr 172.21.15.77:0/1599783291 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.970+0000 7fc813fff640 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558944927ec0 0x558944931f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.970+0000 7fc819185640 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55894492d030 0x558944938fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.971+0000 7fc818984640 1 -- 172.21.15.77:0/1599783291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55894492d030 msgr2=0x558944938fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.971+0000 7fc818984640 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55894492d030 0x558944938fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.971+0000 7fc818984640 1 -- 172.21.15.77:0/1599783291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558944927ec0 msgr2=0x558944931f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.971+0000 7fc818984640 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558944927ec0 0x558944931f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.971+0000 7fc818984640 1 -- 172.21.15.77:0/1599783291 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589445a1210 con 0x5589449274d0 2024-03-16T10:19:21.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.971+0000 7fc813fff640 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558944927ec0 0x558944931f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:21.973 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.972+0000 7fc818984640 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 0x558944931a50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc808006ae0 tx=0x7fc808012740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ec1355de2293a6e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:21.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.972+0000 7fc7f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:19:21.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.972+0000 7fc7f7fff640 1 -- 172.21.15.77:0/1599783291 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808017040 con 0x5589449274d0 2024-03-16T10:19:21.974 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.972+0000 7fc7f7fff640 1 -- 172.21.15.77:0/1599783291 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc808004450 con 0x5589449274d0 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.973+0000 7fc81d7cb740 1 -- 172.21.15.77:0/1599783291 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558944899f80 con 0x5589449274d0 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.973+0000 7fc7f7fff640 1 -- 172.21.15.77:0/1599783291 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc808005990 con 0x5589449274d0 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.973+0000 7fc81d7cb740 10 client.26059 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.973+0000 7fc81d7cb740 1 -- 172.21.15.77:0/1599783291 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55894466d620 con 0x5589449274d0 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.973+0000 7fc7f7fff640 1 -- 172.21.15.77:0/1599783291 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558944899f80 con 0x5589449274d0 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.973+0000 7fc7f7fff640 1 -- 172.21.15.77:0/1599783291 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fc808023070 con 0x5589449274d0 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.974+0000 7fc81d7cb740 10 client.26059 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.974+0000 7fc81d7cb740 10 client.26059 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.974+0000 7fc81d7cb740 20 client.26059 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.974+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55894493db70 0x55894493ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.974+0000 7fc81d7cb740 4 client.26059 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.974+0000 7fc81d7cb740 1 -- 172.21.15.77:0/1599783291 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589448942d0 con 0x55894493db70 2024-03-16T10:19:21.975 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.974+0000 7fc813fff640 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55894493db70 0x55894493ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:21.976 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.975+0000 7fc813fff640 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55894493db70 0x55894493ffb0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9be916c358090a7b server_cookie=62a0488487727136 in_seq=0 out_seq=0 2024-03-16T10:19:21.976 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.975+0000 7fc7f7fff640 10 client.26059 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc7f7fff640 1 -- 172.21.15.77:0/1599783291 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5589448942d0 con 0x55894493db70 2024-03-16T10:19:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc7f7fff640 10 client.26059 handle_command_reply: tid=0 2024-03-16T10:19:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc81d7cb740 1 client.26059 shutdown 2024-03-16T10:19:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc81d7cb740 1 -- 172.21.15.77:0/1599783291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 msgr2=0x558944931a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 0x558944931a50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc808006ae0 tx=0x7fc808012740 comp rx=0 tx=0).stop 2024-03-16T10:19:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc81d7cb740 1 -- 172.21.15.77:0/1599783291 shutdown_connections 2024-03-16T10:19:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55894492d030 0x558944938fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55894493db70 0x55894493ffb0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558944927ec0 0x558944931f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/1599783291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 0x558944931a50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc81d7cb740 1 -- 172.21.15.77:0/1599783291 >> 172.21.15.77:0/1599783291 conn(0x558944922a40 msgr2=0x558944921c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc7f7fff640 0 client.26059 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.985+0000 7fc81d7cb740 1 -- 172.21.15.77:0/1599783291 shutdown_connections 2024-03-16T10:19:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.986+0000 7fc81d7cb740 1 -- 172.21.15.77:0/1599783291 wait complete. 2024-03-16T10:19:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:21.986+0000 7fc81d7cb740 20 client.26059 trim_cache size 0 max 16384 2024-03-16T10:19:22.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:21 smithi192 ceph-mon[36354]: pgmap v418: 97 pgs: 97 active+clean; 131 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.0 MiB/s wr, 487 op/s 2024-03-16T10:19:22.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:21 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:19:22.178 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.178+0000 7fc81d7cb740 1 Processor -- start 2024-03-16T10:19:22.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.181+0000 7fc81d7cb740 1 -- start start 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.181+0000 7fc81d7cb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 0x558944931070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.181+0000 7fc81d7cb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558944927ec0 0x558944938ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.181+0000 7fc81d7cb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55894492d030 0x5589449315b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.181+0000 7fc81d7cb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc8141c0100 con 0x55894492d030 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc81d7cb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc8141c1d60 con 0x558944927ec0 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc81d7cb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc8141c07d0 con 0x5589449274d0 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc818984640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 0x558944931070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc818984640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 0x558944931070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:37378/0 (socket says 172.21.15.77:37378) 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc818984640 1 -- 172.21.15.77:0/987436796 learned_addr learned my addr 172.21.15.77:0/987436796 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc819185640 1 --2- 172.21.15.77:0/987436796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55894492d030 0x5589449315b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc818984640 1 -- 172.21.15.77:0/987436796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558944927ec0 msgr2=0x558944938ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc818984640 1 --2- 172.21.15.77:0/987436796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558944927ec0 0x558944938ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc818984640 1 -- 172.21.15.77:0/987436796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55894492d030 msgr2=0x5589449315b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc818984640 1 --2- 172.21.15.77:0/987436796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55894492d030 0x5589449315b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc818984640 1 -- 172.21.15.77:0/987436796 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589449282a0 con 0x5589449274d0 2024-03-16T10:19:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc819185640 1 --2- 172.21.15.77:0/987436796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55894492d030 0x5589449315b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:19:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.182+0000 7fc818984640 1 --2- 172.21.15.77:0/987436796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 0x558944931070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc808010010 tx=0x7fc80800e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcdb5cd5d7ee2d5d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.216+0000 7fc8027fc640 1 -- 172.21.15.77:0/987436796 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808034070 con 0x5589449274d0 2024-03-16T10:19:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.216+0000 7fc8027fc640 1 -- 172.21.15.77:0/987436796 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc808006390 con 0x5589449274d0 2024-03-16T10:19:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.216+0000 7fc8027fc640 1 -- 172.21.15.77:0/987436796 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808017040 con 0x5589449274d0 2024-03-16T10:19:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.217+0000 7fc81d7cb740 1 -- 172.21.15.77:0/987436796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 msgr2=0x558944931070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.217+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/987436796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 0x558944931070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc808010010 tx=0x7fc80800e5a0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.218+0000 7fc81d7cb740 1 -- 172.21.15.77:0/987436796 shutdown_connections 2024-03-16T10:19:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.218+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/987436796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55894492d030 0x5589449315b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.218+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/987436796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558944927ec0 0x558944938ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.218+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/987436796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589449274d0 0x558944931070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.218+0000 7fc81d7cb740 1 -- 172.21.15.77:0/987436796 >> 172.21.15.77:0/987436796 conn(0x558944922a40 msgr2=0x558944926530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.231+0000 7fc81d7cb740 1 -- 172.21.15.77:0/987436796 shutdown_connections 2024-03-16T10:19:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.231+0000 7fc81d7cb740 1 -- 172.21.15.77:0/987436796 wait complete. 2024-03-16T10:19:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.232+0000 7fc81d7cb740 1 -- start start 2024-03-16T10:19:22.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.232+0000 7fc81d7cb740 1 Processor -- start 2024-03-16T10:19:22.233 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc81d7cb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 0x558944935df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc81d7cb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558944927ec0 0x558944936330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc81d7cb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55894492d030 0x5589449a1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc81d7cb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc8141c15d0 con 0x55894492d030 2024-03-16T10:19:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc81d7cb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc8141c0f00 con 0x5589449274d0 2024-03-16T10:19:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc81d7cb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc8141be100 con 0x558944927ec0 2024-03-16T10:19:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc81d7cb740 10 client.0 _ll_register_callbacks cb 0x7fc808009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc813fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558944927ec0 0x558944936330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc813fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558944927ec0 0x558944936330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:37392/0 (socket says 172.21.15.77:37392) 2024-03-16T10:19:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc813fff640 1 -- 172.21.15.77:0/3507283130 learned_addr learned my addr 172.21.15.77:0/3507283130 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc818984640 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 0x558944935df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:22.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc819185640 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55894492d030 0x5589449a1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc818984640 1 -- 172.21.15.77:0/3507283130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558944927ec0 msgr2=0x558944936330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc818984640 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558944927ec0 0x558944936330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.233+0000 7fc818984640 1 -- 172.21.15.77:0/3507283130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55894492d030 msgr2=0x5589449a1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.234+0000 7fc818984640 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55894492d030 0x5589449a1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.234+0000 7fc818984640 1 -- 172.21.15.77:0/3507283130 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558944936970 con 0x5589449274d0 2024-03-16T10:19:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.234+0000 7fc819185640 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55894492d030 0x5589449a1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.234+0000 7fc818984640 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 0x558944935df0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc808013040 tx=0x7fc808004480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10eb2fde3d7b04d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.235+0000 7fc7f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:19:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.235+0000 7fc7f7fff640 1 -- 172.21.15.77:0/3507283130 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808034070 con 0x5589449274d0 2024-03-16T10:19:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.235+0000 7fc81d7cb740 1 -- 172.21.15.77:0/3507283130 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc808006390 con 0x5589449274d0 2024-03-16T10:19:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.235+0000 7fc7f7fff640 1 -- 172.21.15.77:0/3507283130 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc8080039c0 con 0x5589449274d0 2024-03-16T10:19:22.236 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.235+0000 7fc7f7fff640 1 -- 172.21.15.77:0/3507283130 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc808017040 con 0x5589449274d0 2024-03-16T10:19:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.235+0000 7fc81d7cb740 10 client.26078 fetch_fsmap learned FSMap version 31 2024-03-16T10:19:22.237 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.236+0000 7fc81d7cb740 1 -- 172.21.15.77:0/3507283130 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5589449a34c0 con 0x5589449274d0 2024-03-16T10:19:22.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.239+0000 7fc7f7fff640 1 -- 172.21.15.77:0/3507283130 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc808006390 con 0x5589449274d0 2024-03-16T10:19:22.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.239+0000 7fc7f7fff640 1 -- 172.21.15.77:0/3507283130 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fc808023070 con 0x5589449274d0 2024-03-16T10:19:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.239+0000 7fc81d7cb740 10 client.26078 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T10:19:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.239+0000 7fc81d7cb740 10 client.26078 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.239+0000 7fc81d7cb740 20 client.26078 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.239+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5589449a3e00 0x5589449a6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.239+0000 7fc81d7cb740 4 client.26078 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:19:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.239+0000 7fc81d7cb740 1 -- 172.21.15.77:0/3507283130 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558944812770 con 0x5589449a3e00 2024-03-16T10:19:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.240+0000 7fc813fff640 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5589449a3e00 0x5589449a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.241+0000 7fc813fff640 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5589449a3e00 0x5589449a6220 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a4c97da0bda38b5 server_cookie=d9eeb2ac2470968 in_seq=0 out_seq=0 2024-03-16T10:19:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.241+0000 7fc7f7fff640 10 client.26078 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:22.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:21 smithi077 ceph-mon[32400]: pgmap v418: 97 pgs: 97 active+clean; 131 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.0 MiB/s wr, 487 op/s 2024-03-16T10:19:22.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:21 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:19:22.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:21 smithi193 ceph-mon[36311]: pgmap v418: 97 pgs: 97 active+clean; 131 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.0 MiB/s wr, 487 op/s 2024-03-16T10:19:22.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:21 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:19:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc7f7fff640 1 -- 172.21.15.77:0/3507283130 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558944812770 con 0x5589449a3e00 2024-03-16T10:19:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc7f7fff640 10 client.26078 handle_command_reply: tid=0 2024-03-16T10:19:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc81d7cb740 1 client.26078 shutdown 2024-03-16T10:19:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc81d7cb740 1 -- 172.21.15.77:0/3507283130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 msgr2=0x558944935df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 0x558944935df0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc808013040 tx=0x7fc808004480 comp rx=0 tx=0).stop 2024-03-16T10:19:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc81d7cb740 1 -- 172.21.15.77:0/3507283130 shutdown_connections 2024-03-16T10:19:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5589449a3e00 0x5589449a6220 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55894492d030 0x5589449a1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558944927ec0 0x558944936330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc81d7cb740 1 --2- 172.21.15.77:0/3507283130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5589449274d0 0x558944935df0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc81d7cb740 1 -- 172.21.15.77:0/3507283130 >> 172.21.15.77:0/3507283130 conn(0x558944922a40 msgr2=0x5589449213d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc7f7fff640 0 client.26078 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc81d7cb740 1 -- 172.21.15.77:0/3507283130 shutdown_connections 2024-03-16T10:19:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.800+0000 7fc81d7cb740 1 -- 172.21.15.77:0/3507283130 wait complete. 2024-03-16T10:19:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc81d7cb740 20 client.26078 trim_cache size 0 max 16384 2024-03-16T10:19:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:19:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:19:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:19:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:19:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc819986640 1 -- 172.21.15.77:0/3813405138 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7fc0a2930 msgr2=0x7fc7fc0a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc819986640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7fc0a2930 0x7fc7fc0a4df0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc804006fd0 tx=0x7fc804009060 comp rx=0 tx=0).stop 2024-03-16T10:19:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc819986640 1 -- 172.21.15.77:0/3813405138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8140733f0 msgr2=0x7fc814115430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc819986640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8140733f0 0x7fc814115430 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc80c037040 tx=0x7fc80c035dc0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc819986640 1 -- 172.21.15.77:0/3813405138 shutdown_connections 2024-03-16T10:19:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc819986640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7fc0a2930 0x7fc7fc0a4df0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc819986640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8141094d0 0x7fc8141b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc819986640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8140733f0 0x7fc814115430 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc819986640 1 --2- 172.21.15.77:0/3813405138 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc814072a90 0x7fc814114ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc819986640 1 -- 172.21.15.77:0/3813405138 >> 172.21.15.77:0/3813405138 conn(0x7fc81406e650 msgr2=0x7fc814070300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc819986640 1 -- 172.21.15.77:0/3813405138 shutdown_connections 2024-03-16T10:19:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:22.801+0000 7fc819986640 1 -- 172.21.15.77:0/3813405138 wait complete. 2024-03-16T10:19:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:22 smithi192 ceph-mon[36354]: scrub queued for path: / 2024-03-16T10:19:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:22 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:22 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:19:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:22 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:23.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:22 smithi077 ceph-mon[32400]: scrub queued for path: / 2024-03-16T10:19:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:22 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:22 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:19:23.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:22 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:23.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:22 smithi193 ceph-mon[36311]: scrub queued for path: / 2024-03-16T10:19:23.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:22 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:23.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:22 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:19:23.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:22 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:23 smithi192 ceph-mon[36354]: pgmap v419: 97 pgs: 97 active+clean; 136 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.6 MiB/s wr, 531 op/s 2024-03-16T10:19:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:23 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:19:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:23 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:24.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:23 smithi077 ceph-mon[32400]: pgmap v419: 97 pgs: 97 active+clean; 136 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.6 MiB/s wr, 531 op/s 2024-03-16T10:19:24.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:23 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:19:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:23 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:23 smithi193 ceph-mon[36311]: pgmap v419: 97 pgs: 97 active+clean; 136 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.6 MiB/s wr, 531 op/s 2024-03-16T10:19:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:23 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:19:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:23 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:25.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:24 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:19:25.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:24 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:25.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:19:25.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:19:25.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:24 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:19:25.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:24 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:19:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:24 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:19:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:19:25.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:24 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:19:25.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:24 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:19:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:24 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:19:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:19:25.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:24 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:19:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:25 smithi192 ceph-mon[36354]: pgmap v420: 97 pgs: 97 active+clean; 137 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.1 MiB/s wr, 533 op/s 2024-03-16T10:19:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:25 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:19:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:25 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:19:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:25 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:19:26.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:25 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:19:26.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:25 smithi077 ceph-mon[32400]: pgmap v420: 97 pgs: 97 active+clean; 137 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.1 MiB/s wr, 533 op/s 2024-03-16T10:19:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:19:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:19:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:19:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:25 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:19:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:25 smithi193 ceph-mon[36311]: pgmap v420: 97 pgs: 97 active+clean; 137 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.1 MiB/s wr, 533 op/s 2024-03-16T10:19:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:25 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:19:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:25 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:19:26.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:25 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:19:26.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:25 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:19:27.102 INFO:tasks.workunit.client.0.smithi077.stdout: 35 727858 376 462044 421 368827 1259 2024-03-16T10:19:28.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:27 smithi192 ceph-mon[36354]: pgmap v421: 97 pgs: 97 active+clean; 136 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.1 MiB/s wr, 475 op/s 2024-03-16T10:19:28.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:27 smithi077 ceph-mon[32400]: pgmap v421: 97 pgs: 97 active+clean; 136 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.1 MiB/s wr, 475 op/s 2024-03-16T10:19:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:27 smithi193 ceph-mon[36311]: pgmap v421: 97 pgs: 97 active+clean; 136 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.1 MiB/s wr, 475 op/s 2024-03-16T10:19:30.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:29 smithi192 ceph-mon[36354]: pgmap v422: 97 pgs: 97 active+clean; 139 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 417 op/s 2024-03-16T10:19:30.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:29 smithi077 ceph-mon[32400]: pgmap v422: 97 pgs: 97 active+clean; 139 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 417 op/s 2024-03-16T10:19:30.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:29 smithi193 ceph-mon[36311]: pgmap v422: 97 pgs: 97 active+clean; 139 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 417 op/s 2024-03-16T10:19:32.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:31 smithi192 ceph-mon[36354]: pgmap v423: 97 pgs: 97 active+clean; 143 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 193 op/s 2024-03-16T10:19:32.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:31 smithi077 ceph-mon[32400]: pgmap v423: 97 pgs: 97 active+clean; 143 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 193 op/s 2024-03-16T10:19:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:31 smithi193 ceph-mon[36311]: pgmap v423: 97 pgs: 97 active+clean; 143 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 193 op/s 2024-03-16T10:19:34.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:33 smithi192 ceph-mon[36354]: pgmap v424: 97 pgs: 97 active+clean; 146 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 186 op/s 2024-03-16T10:19:34.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:33 smithi077 ceph-mon[32400]: pgmap v424: 97 pgs: 97 active+clean; 146 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 186 op/s 2024-03-16T10:19:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:33 smithi193 ceph-mon[36311]: pgmap v424: 97 pgs: 97 active+clean; 146 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 186 op/s 2024-03-16T10:19:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.899+0000 7fbe1a636640 1 -- 172.21.15.77:0/1170914097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe141018f0 msgr2=0x7fbe14101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.899+0000 7fbe1a636640 1 --2- 172.21.15.77:0/1170914097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe141018f0 0x7fbe14101d70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009900 tx=0x7fbe0802bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.899+0000 7fbe1a636640 1 -- 172.21.15.77:0/1170914097 shutdown_connections 2024-03-16T10:19:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.899+0000 7fbe1a636640 1 --2- 172.21.15.77:0/1170914097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe141022b0 0x7fbe14108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.899+0000 7fbe1a636640 1 --2- 172.21.15.77:0/1170914097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe141018f0 0x7fbe14101d70 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.899+0000 7fbe1a636640 1 --2- 172.21.15.77:0/1170914097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe14100700 0x7fbe14100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.899+0000 7fbe1a636640 1 -- 172.21.15.77:0/1170914097 >> 172.21.15.77:0/1170914097 conn(0x7fbe140fbc70 msgr2=0x7fbe140fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.899+0000 7fbe1a636640 1 -- 172.21.15.77:0/1170914097 shutdown_connections 2024-03-16T10:19:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.899+0000 7fbe1a636640 1 -- 172.21.15.77:0/1170914097 wait complete. 2024-03-16T10:19:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.900+0000 7fbe1a636640 1 Processor -- start 2024-03-16T10:19:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.900+0000 7fbe1a636640 1 -- start start 2024-03-16T10:19:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe1a636640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe14100700 0x7fbe1419e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe1a636640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe141018f0 0x7fbe1419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe1a636640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe141022b0 0x7fbe141a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe1a636640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1410d4a0 con 0x7fbe141018f0 2024-03-16T10:19:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe1a636640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1410d320 con 0x7fbe14100700 2024-03-16T10:19:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe1a636640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1410d620 con 0x7fbe141022b0 2024-03-16T10:19:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe141018f0 0x7fbe1419e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe18bd3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe141022b0 0x7fbe141a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe141018f0 0x7fbe1419e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59254/0 (socket says 172.21.15.77:59254) 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe137fe640 1 -- 172.21.15.77:0/1558951779 learned_addr learned my addr 172.21.15.77:0/1558951779 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe13fff640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe14100700 0x7fbe1419e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe18bd3640 1 -- 172.21.15.77:0/1558951779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe14100700 msgr2=0x7fbe1419e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe18bd3640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe14100700 0x7fbe1419e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe18bd3640 1 -- 172.21.15.77:0/1558951779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe141018f0 msgr2=0x7fbe1419e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe18bd3640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe141018f0 0x7fbe1419e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe18bd3640 1 -- 172.21.15.77:0/1558951779 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe141a5c90 con 0x7fbe141022b0 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.901+0000 7fbe137fe640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe141018f0 0x7fbe1419e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.902+0000 7fbe13fff640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe14100700 0x7fbe1419e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.902+0000 7fbe18bd3640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe141022b0 0x7fbe141a5620 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009fd0 tx=0x7fbe0c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.902+0000 7fbe117fa640 1 -- 172.21.15.77:0/1558951779 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c00ce20 con 0x7fbe141022b0 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.902+0000 7fbe117fa640 1 -- 172.21.15.77:0/1558951779 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe0c005020 con 0x7fbe141022b0 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.902+0000 7fbe1a636640 1 -- 172.21.15.77:0/1558951779 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe141a5f80 con 0x7fbe141022b0 2024-03-16T10:19:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.903+0000 7fbe117fa640 1 -- 172.21.15.77:0/1558951779 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c010720 con 0x7fbe141022b0 2024-03-16T10:19:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.903+0000 7fbe1a636640 1 -- 172.21.15.77:0/1558951779 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe141a6490 con 0x7fbe141022b0 2024-03-16T10:19:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.904+0000 7fbe117fa640 1 -- 172.21.15.77:0/1558951779 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fbe0c005540 con 0x7fbe141022b0 2024-03-16T10:19:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.905+0000 7fbe117fa640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbdfc0a2920 0x7fbdfc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.905+0000 7fbe117fa640 1 -- 172.21.15.77:0/1558951779 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fbe0c0ce640 con 0x7fbe141022b0 2024-03-16T10:19:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.905+0000 7fbe13fff640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbdfc0a2920 0x7fbdfc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.906+0000 7fbe13fff640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbdfc0a2920 0x7fbdfc0a4de0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbe14101750 tx=0x7fbe04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe1d21e740 1 Processor -- start 2024-03-16T10:19:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe1d21e740 1 -- start start 2024-03-16T10:19:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe1d21e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38340 0x55a7b4c38760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe1d21e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38d30 0x55a7b4c3d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe1d21e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c3dea0 0x55a7b4c40330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe1d21e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a7b4b9da40 con 0x55a7b4c3dea0 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe1d21e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a7b4b46920 con 0x55a7b4c38340 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe1d21e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a7b4b99ae0 con 0x55a7b4c38d30 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe13fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38340 0x55a7b4c38760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe137fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38d30 0x55a7b4c3d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe13fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38340 0x55a7b4c38760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:43526/0 (socket says 172.21.15.77:43526) 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe18bd3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c3dea0 0x55a7b4c40330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.907+0000 7fbe13fff640 1 -- 172.21.15.77:0/3884262416 learned_addr learned my addr 172.21.15.77:0/3884262416 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.908+0000 7fbe137fe640 1 -- 172.21.15.77:0/3884262416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38340 msgr2=0x55a7b4c38760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.908+0000 7fbe137fe640 1 --2- 172.21.15.77:0/3884262416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38340 0x55a7b4c38760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.908+0000 7fbe137fe640 1 -- 172.21.15.77:0/3884262416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c3dea0 msgr2=0x55a7b4c40330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.908+0000 7fbe137fe640 1 --2- 172.21.15.77:0/3884262416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c3dea0 0x55a7b4c40330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.908+0000 7fbe137fe640 1 -- 172.21.15.77:0/3884262416 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7b4b5e460 con 0x55a7b4c38d30 2024-03-16T10:19:35.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.908+0000 7fbe137fe640 1 --2- 172.21.15.77:0/3884262416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38d30 0x55a7b4c3d830 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbe0802c7b0 tx=0x7fbe0802cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d3ec212517452f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:35.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.908+0000 7fbdfa7fc640 1 -- 172.21.15.77:0/3884262416 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe08042070 con 0x55a7b4c38d30 2024-03-16T10:19:35.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.908+0000 7fbdfa7fc640 1 -- 172.21.15.77:0/3884262416 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe08004ba0 con 0x55a7b4c38d30 2024-03-16T10:19:35.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.908+0000 7fbdfa7fc640 1 -- 172.21.15.77:0/3884262416 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080052e0 con 0x55a7b4c38d30 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.909+0000 7fbe1d21e740 1 -- 172.21.15.77:0/3884262416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38d30 msgr2=0x55a7b4c3d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.909+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/3884262416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38d30 0x55a7b4c3d830 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbe0802c7b0 tx=0x7fbe0802cd30 comp rx=0 tx=0).stop 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.909+0000 7fbe1d21e740 1 -- 172.21.15.77:0/3884262416 shutdown_connections 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.909+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/3884262416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c3dea0 0x55a7b4c40330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.909+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/3884262416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38d30 0x55a7b4c3d830 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.909+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/3884262416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38340 0x55a7b4c38760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.909+0000 7fbe1d21e740 1 -- 172.21.15.77:0/3884262416 >> 172.21.15.77:0/3884262416 conn(0x55a7b4c338b0 msgr2=0x55a7b4c33cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.909+0000 7fbe1d21e740 1 -- 172.21.15.77:0/3884262416 shutdown_connections 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.910+0000 7fbe1d21e740 1 -- 172.21.15.77:0/3884262416 wait complete. 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.910+0000 7fbe1d21e740 1 -- start start 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.910+0000 7fbe1d21e740 1 Processor -- start 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.910+0000 7fbe1d21e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 0x55a7b4c42610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.910+0000 7fbe1d21e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 0x55a7b4c42b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.910+0000 7fbe1d21e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c3dea0 0x55a7b4c49b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.910+0000 7fbe1d21e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a7b4c40df0 con 0x55a7b4c38d30 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.910+0000 7fbe1d21e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a7b4b59d70 con 0x55a7b4c3dea0 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.910+0000 7fbe1d21e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a7b4c40f70 con 0x55a7b4c38340 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.910+0000 7fbe13fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 0x55a7b4c42610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.910+0000 7fbe137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 0x55a7b4c42b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbe18bd3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c3dea0 0x55a7b4c49b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.910+0000 7fbe1d21e740 10 client.0 _ll_register_callbacks cb 0x55a7b48b1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbe137fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 0x55a7b4c42b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59272/0 (socket says 172.21.15.77:59272) 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbe13fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 0x55a7b4c42610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:54404/0 (socket says 172.21.15.77:54404) 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbe137fe640 1 -- 172.21.15.77:0/91859151 learned_addr learned my addr 172.21.15.77:0/91859151 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbe18bd3640 1 -- 172.21.15.77:0/91859151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 msgr2=0x55a7b4c42610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbe18bd3640 1 --2- 172.21.15.77:0/91859151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 0x55a7b4c42610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbe18bd3640 1 -- 172.21.15.77:0/91859151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 msgr2=0x55a7b4c42b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbe18bd3640 1 --2- 172.21.15.77:0/91859151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 0x55a7b4c42b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbe18bd3640 1 -- 172.21.15.77:0/91859151 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7b48b2210 con 0x55a7b4c3dea0 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbe13fff640 1 --2- 172.21.15.77:0/91859151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 0x55a7b4c42610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbe18bd3640 1 --2- 172.21.15.77:0/91859151 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c3dea0 0x55a7b4c49b90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c0cc130 tx=0x7fbe0c0054b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=763de6965ae56c89 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbdf37fe640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbdf37fe640 1 -- 172.21.15.77:0/91859151 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c0a3770 con 0x55a7b4c3dea0 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbdf37fe640 1 -- 172.21.15.77:0/91859151 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fbe0c095b40 con 0x55a7b4c3dea0 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.911+0000 7fbe1d21e740 1 -- 172.21.15.77:0/91859151 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe08004ba0 con 0x55a7b4c3dea0 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.912+0000 7fbdf37fe640 1 -- 172.21.15.77:0/91859151 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbe0c095d40 con 0x55a7b4c3dea0 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.912+0000 7fbe1d21e740 10 client.26090 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.912+0000 7fbe1d21e740 1 -- 172.21.15.77:0/91859151 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a7b497e620 con 0x55a7b4c3dea0 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.912+0000 7fbdf37fe640 1 -- 172.21.15.77:0/91859151 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe08004ba0 con 0x55a7b4c3dea0 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.912+0000 7fbdf37fe640 1 -- 172.21.15.77:0/91859151 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fbe0c0b9cd0 con 0x55a7b4c3dea0 2024-03-16T10:19:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.912+0000 7fbe1d21e740 10 client.26090 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.912+0000 7fbe1d21e740 10 client.26090 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.912+0000 7fbe1d21e740 20 client.26090 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.912+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/91859151 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b4c4e540 0x55a7b4c50990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.912+0000 7fbe1d21e740 4 client.26090 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.912+0000 7fbe1d21e740 1 -- 172.21.15.77:0/91859151 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7b4baaf80 con 0x55a7b4c4e540 2024-03-16T10:19:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.912+0000 7fbe13fff640 1 --2- 172.21.15.77:0/91859151 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b4c4e540 0x55a7b4c50990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.913+0000 7fbdf37fe640 10 client.26090 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.913+0000 7fbe13fff640 1 --2- 172.21.15.77:0/91859151 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b4c4e540 0x55a7b4c50990 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a5ddff708c07bdf server_cookie=97bdf7679285e19c in_seq=0 out_seq=0 2024-03-16T10:19:35.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbdf37fe640 1 -- 172.21.15.77:0/91859151 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a7b4baaf80 con 0x55a7b4c4e540 2024-03-16T10:19:35.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbdf37fe640 10 client.26090 handle_command_reply: tid=0 2024-03-16T10:19:35.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbe1d21e740 1 client.26090 shutdown 2024-03-16T10:19:35.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbe1d21e740 1 -- 172.21.15.77:0/91859151 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c3dea0 msgr2=0x55a7b4c49b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:35.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/91859151 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c3dea0 0x55a7b4c49b90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c0cc130 tx=0x7fbe0c0054b0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.918 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbe1d21e740 1 -- 172.21.15.77:0/91859151 shutdown_connections 2024-03-16T10:19:35.919 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/91859151 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c3dea0 0x55a7b4c49b90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.919 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/91859151 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b4c4e540 0x55a7b4c50990 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.919 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/91859151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 0x55a7b4c42b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.919 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbdf37fe640 0 client.26090 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:35.919 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/91859151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 0x55a7b4c42610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.919 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbe1d21e740 1 -- 172.21.15.77:0/91859151 >> 172.21.15.77:0/91859151 conn(0x55a7b4c338b0 msgr2=0x55a7b4c32bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:35.919 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbe1d21e740 1 -- 172.21.15.77:0/91859151 shutdown_connections 2024-03-16T10:19:35.919 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbe1d21e740 1 -- 172.21.15.77:0/91859151 wait complete. 2024-03-16T10:19:35.919 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.915+0000 7fbe1d21e740 20 client.26090 trim_cache size 0 max 16384 2024-03-16T10:19:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe1d21e740 1 Processor -- start 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe1d21e740 1 -- start start 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe1d21e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c338b0 0x55a7b4c2ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe1d21e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38340 0x55a7b4c49b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe1d21e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 0x55a7b4c32320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe1d21e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbe141af040 con 0x55a7b4c38d30 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe1d21e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbe141b0d00 con 0x55a7b4c38340 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe1d21e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbe141af710 con 0x55a7b4c338b0 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe18bd3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 0x55a7b4c32320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe18bd3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 0x55a7b4c32320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59286/0 (socket says 172.21.15.77:59286) 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe18bd3640 1 -- 172.21.15.77:0/3360915042 learned_addr learned my addr 172.21.15.77:0/3360915042 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe137fe640 1 --2- 172.21.15.77:0/3360915042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38340 0x55a7b4c49b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe13fff640 1 --2- 172.21.15.77:0/3360915042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c338b0 0x55a7b4c2ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe18bd3640 1 -- 172.21.15.77:0/3360915042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c338b0 msgr2=0x55a7b4c2ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe18bd3640 1 --2- 172.21.15.77:0/3360915042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c338b0 0x55a7b4c2ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe18bd3640 1 -- 172.21.15.77:0/3360915042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38340 msgr2=0x55a7b4c49b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe18bd3640 1 --2- 172.21.15.77:0/3360915042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38340 0x55a7b4c49b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe18bd3640 1 -- 172.21.15.77:0/3360915042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7b4c39110 con 0x55a7b4c38d30 2024-03-16T10:19:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.940+0000 7fbe18bd3640 1 --2- 172.21.15.77:0/3360915042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 0x55a7b4c32320 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c0979a0 tx=0x7fbe0c0cc070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc1cfaa3b4d6abc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.941+0000 7fbdf9ffb640 1 -- 172.21.15.77:0/3360915042 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c0051d0 con 0x55a7b4c38d30 2024-03-16T10:19:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.941+0000 7fbdf9ffb640 1 -- 172.21.15.77:0/3360915042 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe0c0ce1f0 con 0x55a7b4c38d30 2024-03-16T10:19:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.941+0000 7fbdf9ffb640 1 -- 172.21.15.77:0/3360915042 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c09a420 con 0x55a7b4c38d30 2024-03-16T10:19:35.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.941+0000 7fbe1d21e740 1 -- 172.21.15.77:0/3360915042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 msgr2=0x55a7b4c32320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:35.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.941+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/3360915042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 0x55a7b4c32320 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c0979a0 tx=0x7fbe0c0cc070 comp rx=0 tx=0).stop 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.941+0000 7fbe1d21e740 1 -- 172.21.15.77:0/3360915042 shutdown_connections 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.941+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/3360915042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c38d30 0x55a7b4c32320 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.941+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/3360915042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38340 0x55a7b4c49b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.941+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/3360915042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c338b0 0x55a7b4c2ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.941+0000 7fbe1d21e740 1 -- 172.21.15.77:0/3360915042 >> 172.21.15.77:0/3360915042 conn(0x55a7b4c3d9e0 msgr2=0x55a7b4c3a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe1d21e740 1 -- 172.21.15.77:0/3360915042 shutdown_connections 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe1d21e740 1 -- 172.21.15.77:0/3360915042 wait complete. 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe1d21e740 1 -- start start 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe1d21e740 1 Processor -- start 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe1d21e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c338b0 0x55a7b4c492f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe1d21e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 0x55a7b4c49830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe1d21e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38d30 0x55a7b4c47900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe1d21e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbe141b0570 con 0x55a7b4c338b0 2024-03-16T10:19:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe1d21e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbe141afea0 con 0x55a7b4c38d30 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe1d21e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbe141ad020 con 0x55a7b4c38340 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe1d21e740 10 client.0 _ll_register_callbacks cb 0x7fbe0c005b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c338b0 0x55a7b4c492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe13fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c338b0 0x55a7b4c492f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59300/0 (socket says 172.21.15.77:59300) 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe13fff640 1 -- 172.21.15.77:0/2390750099 learned_addr learned my addr 172.21.15.77:0/2390750099 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe137fe640 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 0x55a7b4c49830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.942+0000 7fbe18bd3640 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38d30 0x55a7b4c47900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbe137fe640 1 -- 172.21.15.77:0/2390750099 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38d30 msgr2=0x55a7b4c47900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbe137fe640 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38d30 0x55a7b4c47900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbe137fe640 1 -- 172.21.15.77:0/2390750099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c338b0 msgr2=0x55a7b4c492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbe137fe640 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c338b0 0x55a7b4c492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbe137fe640 1 -- 172.21.15.77:0/2390750099 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7b4c47f40 con 0x55a7b4c38340 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbe13fff640 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c338b0 0x55a7b4c492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbe137fe640 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 0x55a7b4c49830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbe08000c00 tx=0x7fbe08005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b394d0308f8fc4ea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbdf37fe640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbdf37fe640 1 -- 172.21.15.77:0/2390750099 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe08042040 con 0x55a7b4c38340 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbe1d21e740 1 -- 172.21.15.77:0/2390750099 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe0c0ce1f0 con 0x55a7b4c38340 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbdf37fe640 1 -- 172.21.15.77:0/2390750099 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fbe08004020 con 0x55a7b4c38340 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbdf37fe640 1 -- 172.21.15.77:0/2390750099 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbe08005d50 con 0x55a7b4c38340 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbe1d21e740 10 client.26083 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.943+0000 7fbe1d21e740 1 -- 172.21.15.77:0/2390750099 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a7b4cb5230 con 0x55a7b4c38340 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.944+0000 7fbdf37fe640 1 -- 172.21.15.77:0/2390750099 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe0c0ce1f0 con 0x55a7b4c38340 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.944+0000 7fbdf37fe640 1 -- 172.21.15.77:0/2390750099 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fbe08003860 con 0x55a7b4c38340 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.944+0000 7fbe1d21e740 10 client.26083 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.944+0000 7fbe1d21e740 10 client.26083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.944+0000 7fbe1d21e740 20 client.26083 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.944+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b4c3dea0 0x55a7b4cb7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.944+0000 7fbe1d21e740 4 client.26083 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.944+0000 7fbe1d21e740 1 -- 172.21.15.77:0/2390750099 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a7b4ba52d0 con 0x55a7b4c3dea0 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.944+0000 7fbe13fff640 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b4c3dea0 0x55a7b4cb7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.944+0000 7fbe13fff640 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b4c3dea0 0x55a7b4cb7870 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f10f3bb079f7629 server_cookie=c5c09a9d38ce63b7 in_seq=0 out_seq=0 2024-03-16T10:19:35.949 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:35.944+0000 7fbdf37fe640 10 client.26083 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.119+0000 7fbdf37fe640 1 -- 172.21.15.77:0/2390750099 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a7b4ba52d0 con 0x55a7b4c3dea0 2024-03-16T10:19:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.119+0000 7fbdf37fe640 10 client.26083 handle_command_reply: tid=0 2024-03-16T10:19:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.119+0000 7fbe1d21e740 1 client.26083 shutdown 2024-03-16T10:19:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.119+0000 7fbe1d21e740 1 -- 172.21.15.77:0/2390750099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 msgr2=0x55a7b4c49830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.119+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 0x55a7b4c49830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbe08000c00 tx=0x7fbe08005850 comp rx=0 tx=0).stop 2024-03-16T10:19:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1d21e740 1 -- 172.21.15.77:0/2390750099 shutdown_connections 2024-03-16T10:19:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b4c38d30 0x55a7b4c47900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b4c38340 0x55a7b4c49830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b4c3dea0 0x55a7b4cb7870 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1d21e740 1 --2- 172.21.15.77:0/2390750099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b4c338b0 0x55a7b4c492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbdf37fe640 0 client.26083 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1d21e740 1 -- 172.21.15.77:0/2390750099 >> 172.21.15.77:0/2390750099 conn(0x55a7b4c3d9e0 msgr2=0x55a7b4c36b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1d21e740 1 -- 172.21.15.77:0/2390750099 shutdown_connections 2024-03-16T10:19:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1d21e740 1 -- 172.21.15.77:0/2390750099 wait complete. 2024-03-16T10:19:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1d21e740 20 client.26083 trim_cache size 0 max 16384 2024-03-16T10:19:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:19:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:19:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:19:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:19:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1a636640 1 -- 172.21.15.77:0/1558951779 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbdfc0a2920 msgr2=0x7fbdfc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1a636640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbdfc0a2920 0x7fbdfc0a4de0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbe14101750 tx=0x7fbe04005f90 comp rx=0 tx=0).stop 2024-03-16T10:19:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1a636640 1 -- 172.21.15.77:0/1558951779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe141022b0 msgr2=0x7fbe141a5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1a636640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe141022b0 0x7fbe141a5620 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009fd0 tx=0x7fbe0c00ef10 comp rx=0 tx=0).stop 2024-03-16T10:19:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1a636640 1 -- 172.21.15.77:0/1558951779 shutdown_connections 2024-03-16T10:19:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1a636640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fbdfc0a2920 0x7fbdfc0a4de0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1a636640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe141022b0 0x7fbe141a5620 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1a636640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe141018f0 0x7fbe1419e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1a636640 1 --2- 172.21.15.77:0/1558951779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe14100700 0x7fbe1419e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1a636640 1 -- 172.21.15.77:0/1558951779 >> 172.21.15.77:0/1558951779 conn(0x7fbe140fbc70 msgr2=0x7fbe14102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1a636640 1 -- 172.21.15.77:0/1558951779 shutdown_connections 2024-03-16T10:19:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:36.120+0000 7fbe1a636640 1 -- 172.21.15.77:0/1558951779 wait complete. 2024-03-16T10:19:36.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:35 smithi192 ceph-mon[36354]: pgmap v425: 97 pgs: 97 active+clean; 146 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-03-16T10:19:36.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:35 smithi077 ceph-mon[32400]: pgmap v425: 97 pgs: 97 active+clean; 146 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-03-16T10:19:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:35 smithi193 ceph-mon[36311]: pgmap v425: 97 pgs: 97 active+clean; 146 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-03-16T10:19:37.102 INFO:tasks.workunit.client.0.smithi077.stdout: 43 882102 439 567063 362 437625 1227 2024-03-16T10:19:38.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:37 smithi192 ceph-mon[36354]: pgmap v426: 97 pgs: 97 active+clean; 149 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-03-16T10:19:38.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:37 smithi077 ceph-mon[32400]: pgmap v426: 97 pgs: 97 active+clean; 149 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-03-16T10:19:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:37 smithi193 ceph-mon[36311]: pgmap v426: 97 pgs: 97 active+clean; 149 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-03-16T10:19:40.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:39 smithi192 ceph-mon[36354]: pgmap v427: 97 pgs: 97 active+clean; 153 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-03-16T10:19:40.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:39 smithi077 ceph-mon[32400]: pgmap v427: 97 pgs: 97 active+clean; 153 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-03-16T10:19:40.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:39 smithi193 ceph-mon[36311]: pgmap v427: 97 pgs: 97 active+clean; 153 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-03-16T10:19:41.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:40 smithi192 ceph-mon[36354]: pgmap v428: 97 pgs: 97 active+clean; 177 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 299 op/s 2024-03-16T10:19:41.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:40 smithi077 ceph-mon[32400]: pgmap v428: 97 pgs: 97 active+clean; 177 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 299 op/s 2024-03-16T10:19:41.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:40 smithi193 ceph-mon[36311]: pgmap v428: 97 pgs: 97 active+clean; 177 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 299 op/s 2024-03-16T10:19:42.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:42 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:43.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:42 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:43.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:42 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:43.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:43 smithi077 ceph-mon[32400]: pgmap v429: 97 pgs: 97 active+clean; 211 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 539 op/s 2024-03-16T10:19:44.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:43 smithi193 ceph-mon[36311]: pgmap v429: 97 pgs: 97 active+clean; 211 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 539 op/s 2024-03-16T10:19:44.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:43 smithi192 ceph-mon[36354]: pgmap v429: 97 pgs: 97 active+clean; 211 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 539 op/s 2024-03-16T10:19:46.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:45 smithi192 ceph-mon[36354]: pgmap v430: 97 pgs: 97 active+clean; 216 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.6 MiB/s wr, 576 op/s 2024-03-16T10:19:46.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:45 smithi077 ceph-mon[32400]: pgmap v430: 97 pgs: 97 active+clean; 216 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.6 MiB/s wr, 576 op/s 2024-03-16T10:19:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:45 smithi193 ceph-mon[36311]: pgmap v430: 97 pgs: 97 active+clean; 216 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.6 MiB/s wr, 576 op/s 2024-03-16T10:19:47.103 INFO:tasks.workunit.client.0.smithi077.stdout: 47 111600 205 69612 276 57416 652 2024-03-16T10:19:48.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:47 smithi192 ceph-mon[36354]: pgmap v431: 97 pgs: 97 active+clean; 215 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.6 MiB/s wr, 593 op/s 2024-03-16T10:19:48.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:47 smithi077 ceph-mon[32400]: pgmap v431: 97 pgs: 97 active+clean; 215 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.6 MiB/s wr, 593 op/s 2024-03-16T10:19:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:47 smithi193 ceph-mon[36311]: pgmap v431: 97 pgs: 97 active+clean; 215 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.6 MiB/s wr, 593 op/s 2024-03-16T10:19:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.283+0000 7f8cf6953640 1 -- 172.21.15.77:0/489398394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cf0074240 msgr2=0x7f8cf00746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.283+0000 7f8cf6953640 1 --2- 172.21.15.77:0/489398394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cf0074240 0x7f8cf00746c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009860 tx=0x7f8ce402bb70 comp rx=0 tx=0).stop 2024-03-16T10:19:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.283+0000 7f8cf6953640 1 -- 172.21.15.77:0/489398394 shutdown_connections 2024-03-16T10:19:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.283+0000 7f8cf6953640 1 --2- 172.21.15.77:0/489398394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cf01096f0 0x7f8cf010bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.283+0000 7f8cf6953640 1 --2- 172.21.15.77:0/489398394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cf0074240 0x7f8cf00746c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.283+0000 7f8cf6953640 1 --2- 172.21.15.77:0/489398394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cf0075820 0x7f8cf0073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.283+0000 7f8cf6953640 1 -- 172.21.15.77:0/489398394 >> 172.21.15.77:0/489398394 conn(0x7f8cf00fcda0 msgr2=0x7f8cf00ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.283+0000 7f8cf6953640 1 -- 172.21.15.77:0/489398394 shutdown_connections 2024-03-16T10:19:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.283+0000 7f8cf6953640 1 -- 172.21.15.77:0/489398394 wait complete. 2024-03-16T10:19:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.284+0000 7f8cf6953640 1 Processor -- start 2024-03-16T10:19:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.284+0000 7f8cf6953640 1 -- start start 2024-03-16T10:19:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cf6953640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cf0074240 0x7f8cf019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cf6953640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cf0075820 0x7f8cf019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.287 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cf6953640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cf01096f0 0x7f8cf01a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cf6953640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf010f690 con 0x7f8cf0074240 2024-03-16T10:19:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cf6953640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf010f510 con 0x7f8cf01096f0 2024-03-16T10:19:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cf6953640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf010f810 con 0x7f8cf0075820 2024-03-16T10:19:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cef7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cf0075820 0x7f8cf019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cf4ef0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cf01096f0 0x7f8cf01a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cf0074240 0x7f8cf019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cef7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cf0075820 0x7f8cf019e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:43412/0 (socket says 172.21.15.77:43412) 2024-03-16T10:19:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cf4ef0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cf01096f0 0x7f8cf01a54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:40924/0 (socket says 172.21.15.77:40924) 2024-03-16T10:19:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cef7fe640 1 -- 172.21.15.77:0/1340134165 learned_addr learned my addr 172.21.15.77:0/1340134165 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cef7fe640 1 -- 172.21.15.77:0/1340134165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cf01096f0 msgr2=0x7f8cf01a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cf01096f0 0x7f8cf01a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cef7fe640 1 -- 172.21.15.77:0/1340134165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cf0074240 msgr2=0x7f8cf019def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cf0074240 0x7f8cf019def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8cef7fe640 1 -- 172.21.15.77:0/1340134165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf01a5b10 con 0x7f8cf0075820 2024-03-16T10:19:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.285+0000 7f8ceffff640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cf0074240 0x7f8cf019def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.286+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cf0075820 0x7f8cf019e430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8ce402c180 tx=0x7f8ce4035f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.286+0000 7f8ced7fa640 1 -- 172.21.15.77:0/1340134165 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4039040 con 0x7f8cf0075820 2024-03-16T10:19:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.286+0000 7f8ced7fa640 1 -- 172.21.15.77:0/1340134165 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce4053070 con 0x7f8cf0075820 2024-03-16T10:19:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.286+0000 7f8cf6953640 1 -- 172.21.15.77:0/1340134165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf006b200 con 0x7f8cf0075820 2024-03-16T10:19:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.286+0000 7f8ced7fa640 1 -- 172.21.15.77:0/1340134165 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4004a80 con 0x7f8cf0075820 2024-03-16T10:19:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.286+0000 7f8cf6953640 1 -- 172.21.15.77:0/1340134165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf006b6f0 con 0x7f8cf0075820 2024-03-16T10:19:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.289+0000 7f8ced7fa640 1 -- 172.21.15.77:0/1340134165 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f8ce4004c20 con 0x7f8cf0075820 2024-03-16T10:19:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.290+0000 7f8cf9618740 1 Processor -- start 2024-03-16T10:19:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.292+0000 7f8cf9618740 1 -- start start 2024-03-16T10:19:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.292+0000 7f8ced7fa640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8cd80a2920 0x7f8cd80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.292+0000 7f8ced7fa640 1 -- 172.21.15.77:0/1340134165 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8ce41014b0 con 0x7f8cf0075820 2024-03-16T10:19:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.292+0000 7f8cf4ef0640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8cd80a2920 0x7f8cd80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.292+0000 7f8cf9618740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 0x55afd1acb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cf9618740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acbd30 0x55afd1ad0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cf9618740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ad0ea0 0x55afd1ad3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cf9618740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55afd1a30a40 con 0x55afd1acb340 2024-03-16T10:19:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cf9618740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55afd19d9920 con 0x55afd1ad0ea0 2024-03-16T10:19:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cf9618740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55afd1a2cae0 con 0x55afd1acbd30 2024-03-16T10:19:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cef7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acbd30 0x55afd1ad0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 0x55afd1acb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cef7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acbd30 0x55afd1ad0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:43418/0 (socket says 172.21.15.77:43418) 2024-03-16T10:19:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cef7fe640 1 -- 172.21.15.77:0/386591975 learned_addr learned my addr 172.21.15.77:0/386591975 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cef7fe640 1 -- 172.21.15.77:0/386591975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ad0ea0 msgr2=0x55afd1ad3330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:19:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/386591975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ad0ea0 0x55afd1ad3330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cef7fe640 1 -- 172.21.15.77:0/386591975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 msgr2=0x55afd1acb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/386591975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 0x55afd1acb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cf4ef0640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8cd80a2920 0x7f8cd80a4de0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8005fd0 tx=0x7f8ce8010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8cef7fe640 1 -- 172.21.15.77:0/386591975 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd19f1460 con 0x55afd1acbd30 2024-03-16T10:19:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.293+0000 7f8ceffff640 1 --2- 172.21.15.77:0/386591975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 0x55afd1acb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.294+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/386591975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acbd30 0x55afd1ad0830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40ca2a0 tx=0x7f8ce40cd980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9413c784ecd31ee1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.294+0000 7f8cd67fc640 1 -- 172.21.15.77:0/386591975 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce40d0070 con 0x55afd1acbd30 2024-03-16T10:19:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.294+0000 7f8cd67fc640 1 -- 172.21.15.77:0/386591975 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce40cde40 con 0x55afd1acbd30 2024-03-16T10:19:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.294+0000 7f8cd67fc640 1 -- 172.21.15.77:0/386591975 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce40ce300 con 0x55afd1acbd30 2024-03-16T10:19:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.295+0000 7f8cf9618740 1 -- 172.21.15.77:0/386591975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acbd30 msgr2=0x55afd1ad0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.295+0000 7f8cf9618740 1 --2- 172.21.15.77:0/386591975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acbd30 0x55afd1ad0830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40ca2a0 tx=0x7f8ce40cd980 comp rx=0 tx=0).stop 2024-03-16T10:19:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.295+0000 7f8cf9618740 1 -- 172.21.15.77:0/386591975 shutdown_connections 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.295+0000 7f8cf9618740 1 --2- 172.21.15.77:0/386591975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ad0ea0 0x55afd1ad3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.295+0000 7f8cf9618740 1 --2- 172.21.15.77:0/386591975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acbd30 0x55afd1ad0830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.295+0000 7f8cf9618740 1 --2- 172.21.15.77:0/386591975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 0x55afd1acb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.295+0000 7f8cf9618740 1 -- 172.21.15.77:0/386591975 >> 172.21.15.77:0/386591975 conn(0x55afd1ac68b0 msgr2=0x55afd1ac6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.295+0000 7f8cf9618740 1 -- 172.21.15.77:0/386591975 shutdown_connections 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.295+0000 7f8cf9618740 1 -- 172.21.15.77:0/386591975 wait complete. 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.296+0000 7f8cf9618740 1 -- start start 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.296+0000 7f8cf9618740 1 Processor -- start 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cf9618740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 0x55afd1ad5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cf9618740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acbd30 0x55afd1ad59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cf9618740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1ad0ea0 0x55afd1adc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cf9618740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55afd1ad3c10 con 0x55afd1acb340 2024-03-16T10:19:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cf9618740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55afd19ecd70 con 0x55afd1acbd30 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cf9618740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55afd1ad3d90 con 0x55afd1ad0ea0 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cf9618740 10 client.0 _ll_register_callbacks cb 0x55afd1744a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 0x55afd1ad5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 0x55afd1ad5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54662/0 (socket says 172.21.15.77:54662) 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8ceffff640 1 -- 172.21.15.77:0/2254562818 learned_addr learned my addr 172.21.15.77:0/2254562818 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acbd30 0x55afd1ad59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cf4ef0640 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1ad0ea0 0x55afd1adc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cef7fe640 1 -- 172.21.15.77:0/2254562818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1ad0ea0 msgr2=0x55afd1adc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1ad0ea0 0x55afd1adc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cef7fe640 1 -- 172.21.15.77:0/2254562818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 msgr2=0x55afd1ad5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 0x55afd1ad5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.297+0000 7f8cef7fe640 1 -- 172.21.15.77:0/2254562818 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd1745210 con 0x55afd1acbd30 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.298+0000 7f8ceffff640 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 0x55afd1ad5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.298+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acbd30 0x55afd1ad59c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55afd1acc130 tx=0x7f8ce40ca700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42fd614e060a55dc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.298+0000 7f8ccf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.298+0000 7f8ccf7fe640 1 -- 172.21.15.77:0/2254562818 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce40d0050 con 0x55afd1acbd30 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.298+0000 7f8ccf7fe640 1 -- 172.21.15.77:0/2254562818 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8ce40d6360 con 0x55afd1acbd30 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.298+0000 7f8cf9618740 1 -- 172.21.15.77:0/2254562818 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ce40cde40 con 0x55afd1acbd30 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.298+0000 7f8ccf7fe640 1 -- 172.21.15.77:0/2254562818 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8ce40d65e0 con 0x55afd1acbd30 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.298+0000 7f8cf9618740 10 client.26105 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.298+0000 7f8cf9618740 1 -- 172.21.15.77:0/2254562818 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55afd1811620 con 0x55afd1acbd30 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.299+0000 7f8ccf7fe640 1 -- 172.21.15.77:0/2254562818 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce40cde40 con 0x55afd1acbd30 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.299+0000 7f8ccf7fe640 1 -- 172.21.15.77:0/2254562818 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f8ce40b5030 con 0x55afd1acbd30 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.299+0000 7f8cf9618740 10 client.26105 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.299+0000 7f8cf9618740 10 client.26105 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.299+0000 7f8cf9618740 20 client.26105 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.299+0000 7f8cf9618740 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55afd1ae1100 0x55afd1ae34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.305 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.299+0000 7f8cf9618740 4 client.26105 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:49.305 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.299+0000 7f8cf9618740 1 -- 172.21.15.77:0/2254562818 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afd1a3df80 con 0x55afd1ae1100 2024-03-16T10:19:49.305 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.299+0000 7f8ceffff640 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55afd1ae1100 0x55afd1ae34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.305 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.300+0000 7f8ccf7fe640 10 client.26105 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:49.305 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.300+0000 7f8ceffff640 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55afd1ae1100 0x55afd1ae34f0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d740a45bccce03e1 server_cookie=276223b29aa354cb in_seq=0 out_seq=0 2024-03-16T10:19:49.305 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8ccf7fe640 1 -- 172.21.15.77:0/2254562818 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55afd1a3df80 con 0x55afd1ae1100 2024-03-16T10:19:49.305 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8ccf7fe640 10 client.26105 handle_command_reply: tid=0 2024-03-16T10:19:49.305 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8cf9618740 1 client.26105 shutdown 2024-03-16T10:19:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8cf9618740 1 -- 172.21.15.77:0/2254562818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acbd30 msgr2=0x55afd1ad59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8cf9618740 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acbd30 0x55afd1ad59c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55afd1acc130 tx=0x7f8ce40ca700 comp rx=0 tx=0).stop 2024-03-16T10:19:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8cf9618740 1 -- 172.21.15.77:0/2254562818 shutdown_connections 2024-03-16T10:19:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8cf9618740 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1ad0ea0 0x55afd1adc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8cf9618740 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acbd30 0x55afd1ad59c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8cf9618740 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55afd1ae1100 0x55afd1ae34f0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8cf9618740 1 --2- 172.21.15.77:0/2254562818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acb340 0x55afd1ad5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8cf9618740 1 -- 172.21.15.77:0/2254562818 >> 172.21.15.77:0/2254562818 conn(0x55afd1ac68b0 msgr2=0x55afd1ac5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8ccf7fe640 0 client.26105 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8cf9618740 1 -- 172.21.15.77:0/2254562818 shutdown_connections 2024-03-16T10:19:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8cf9618740 1 -- 172.21.15.77:0/2254562818 wait complete. 2024-03-16T10:19:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.301+0000 7f8cf9618740 20 client.26105 trim_cache size 0 max 16384 2024-03-16T10:19:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.332+0000 7f8cf9618740 1 Processor -- start 2024-03-16T10:19:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.332+0000 7f8cf9618740 1 -- start start 2024-03-16T10:19:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.332+0000 7f8cf9618740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acb340 0x55afd1ad5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.332+0000 7f8cf9618740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acbd30 0x55afd1adc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.332+0000 7f8cf9618740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1ad0ea0 0x55afd1ad59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.332+0000 7f8cf9618740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc00e320 con 0x55afd1ad0ea0 2024-03-16T10:19:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acb340 0x55afd1ad5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acb340 0x55afd1ad5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:43436/0 (socket says 172.21.15.77:43436) 2024-03-16T10:19:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8ceffff640 1 -- 172.21.15.77:0/412689485 learned_addr learned my addr 172.21.15.77:0/412689485 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/412689485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acbd30 0x55afd1adc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8cf4ef0640 1 --2- 172.21.15.77:0/412689485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1ad0ea0 0x55afd1ad59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8cf9618740 1 -- 172.21.15.77:0/412689485 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc00ffe0 con 0x55afd1acbd30 2024-03-16T10:19:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8cf9618740 1 -- 172.21.15.77:0/412689485 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc00e9f0 con 0x55afd1acb340 2024-03-16T10:19:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8ceffff640 1 -- 172.21.15.77:0/412689485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acbd30 msgr2=0x55afd1adc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8ceffff640 1 --2- 172.21.15.77:0/412689485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acbd30 0x55afd1adc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8ceffff640 1 -- 172.21.15.77:0/412689485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1ad0ea0 msgr2=0x55afd1ad59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8ceffff640 1 --2- 172.21.15.77:0/412689485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1ad0ea0 0x55afd1ad59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8ceffff640 1 -- 172.21.15.77:0/412689485 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd1ad03b0 con 0x55afd1acb340 2024-03-16T10:19:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/412689485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acbd30 0x55afd1adc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.333+0000 7f8ceffff640 1 --2- 172.21.15.77:0/412689485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acb340 0x55afd1ad5480 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0004700 tx=0x7f8ce0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbe2b0a90ff0fde3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.334+0000 7f8cd5ffb640 1 -- 172.21.15.77:0/412689485 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0004050 con 0x55afd1acb340 2024-03-16T10:19:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.334+0000 7f8cd5ffb640 1 -- 172.21.15.77:0/412689485 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce000fe00 con 0x55afd1acb340 2024-03-16T10:19:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.334+0000 7f8cd5ffb640 1 -- 172.21.15.77:0/412689485 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce00047a0 con 0x55afd1acb340 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.335+0000 7f8cf9618740 1 -- 172.21.15.77:0/412689485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acb340 msgr2=0x55afd1ad5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.335+0000 7f8cf9618740 1 --2- 172.21.15.77:0/412689485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acb340 0x55afd1ad5480 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0004700 tx=0x7f8ce0005f90 comp rx=0 tx=0).stop 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.335+0000 7f8cf9618740 1 -- 172.21.15.77:0/412689485 shutdown_connections 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.335+0000 7f8cf9618740 1 --2- 172.21.15.77:0/412689485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1ad0ea0 0x55afd1ad59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.335+0000 7f8cf9618740 1 --2- 172.21.15.77:0/412689485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acbd30 0x55afd1adc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.335+0000 7f8cf9618740 1 --2- 172.21.15.77:0/412689485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1acb340 0x55afd1ad5480 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.335+0000 7f8cf9618740 1 -- 172.21.15.77:0/412689485 >> 172.21.15.77:0/412689485 conn(0x55afd1ac68b0 msgr2=0x55afd1ac6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.335+0000 7f8cf9618740 1 -- 172.21.15.77:0/412689485 shutdown_connections 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.335+0000 7f8cf9618740 1 -- 172.21.15.77:0/412689485 wait complete. 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.335+0000 7f8cf9618740 1 -- start start 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.336+0000 7f8cf9618740 1 Processor -- start 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.336+0000 7f8cf9618740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acb340 0x55afd1ada0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.336+0000 7f8cf9618740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acbd30 0x55afd1ada610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.336+0000 7f8cf9618740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1ad0ea0 0x55afd1adb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.336+0000 7f8cf9618740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc00f850 con 0x55afd1acbd30 2024-03-16T10:19:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.336+0000 7f8cf9618740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc00f180 con 0x55afd1acb340 2024-03-16T10:19:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.336+0000 7f8cf9618740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc00c300 con 0x55afd1ad0ea0 2024-03-16T10:19:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.336+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acb340 0x55afd1ada0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.336+0000 7f8cef7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acbd30 0x55afd1ada610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.336+0000 7f8cf9618740 10 client.0 _ll_register_callbacks cb 0x7f8ce0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.336+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acb340 0x55afd1ada0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:40946/0 (socket says 172.21.15.77:40946) 2024-03-16T10:19:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.336+0000 7f8ceffff640 1 -- 172.21.15.77:0/3513062981 learned_addr learned my addr 172.21.15.77:0/3513062981 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.337+0000 7f8cef7fe640 1 -- 172.21.15.77:0/3513062981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1ad0ea0 msgr2=0x55afd1adb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.337+0000 7f8cf4ef0640 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1ad0ea0 0x55afd1adb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.338+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1ad0ea0 0x55afd1adb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.338+0000 7f8cef7fe640 1 -- 172.21.15.77:0/3513062981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acb340 msgr2=0x55afd1ada0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.338+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acb340 0x55afd1ada0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.338+0000 7f8cef7fe640 1 -- 172.21.15.77:0/3513062981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd1adbc10 con 0x55afd1acbd30 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.338+0000 7f8ceffff640 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acb340 0x55afd1ada0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.338+0000 7f8cf4ef0640 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1ad0ea0 0x55afd1adb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.338+0000 7f8cef7fe640 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acbd30 0x55afd1ada610 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009780 tx=0x7f8ce402c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beca1a2411cf0cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.338+0000 7f8ccf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.338+0000 7f8ccf7fe640 1 -- 172.21.15.77:0/3513062981 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce40d0040 con 0x55afd1acbd30 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.338+0000 7f8ccf7fe640 1 -- 172.21.15.77:0/3513062981 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8ce40d2f00 con 0x55afd1acbd30 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.339+0000 7f8cf9618740 1 -- 172.21.15.77:0/3513062981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ce000fe00 con 0x55afd1acbd30 2024-03-16T10:19:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.340+0000 7f8ccf7fe640 1 -- 172.21.15.77:0/3513062981 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8ce40d3100 con 0x55afd1acbd30 2024-03-16T10:19:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.340+0000 7f8cf9618740 10 client.16332 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.340+0000 7f8cf9618740 1 -- 172.21.15.77:0/3513062981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55afd1adbc10 con 0x55afd1acbd30 2024-03-16T10:19:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.340+0000 7f8ccf7fe640 1 -- 172.21.15.77:0/3513062981 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce000fe00 con 0x55afd1acbd30 2024-03-16T10:19:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.342+0000 7f8ccf7fe640 1 -- 172.21.15.77:0/3513062981 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f8ce40c9b90 con 0x55afd1acbd30 2024-03-16T10:19:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.342+0000 7f8cf9618740 10 client.16332 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.342+0000 7f8cf9618740 10 client.16332 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.342+0000 7f8cf9618740 20 client.16332 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.342+0000 7f8cf9618740 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55afd1b47b80 0x55afd1b49fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.342+0000 7f8cf9618740 4 client.16332 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:19:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.343+0000 7f8cf9618740 1 -- 172.21.15.77:0/3513062981 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55afd1a382d0 con 0x55afd1b47b80 2024-03-16T10:19:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.343+0000 7f8ceffff640 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55afd1b47b80 0x55afd1b49fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.344+0000 7f8ceffff640 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55afd1b47b80 0x55afd1b49fa0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=780d2e10f263c74d server_cookie=853f7b678081f6a4 in_seq=0 out_seq=0 2024-03-16T10:19:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:49.344+0000 7f8ccf7fe640 10 client.16332 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:49 smithi192 ceph-mon[36354]: pgmap v432: 97 pgs: 97 active+clean; 220 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 626 op/s 2024-03-16T10:19:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:49 smithi077 ceph-mon[32400]: pgmap v432: 97 pgs: 97 active+clean; 220 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 626 op/s 2024-03-16T10:19:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:49 smithi193 ceph-mon[36311]: pgmap v432: 97 pgs: 97 active+clean; 220 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 626 op/s 2024-03-16T10:19:50.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.742+0000 7f8ccf7fe640 1 -- 172.21.15.77:0/3513062981 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55afd1a382d0 con 0x55afd1b47b80 2024-03-16T10:19:50.743 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.742+0000 7f8ccf7fe640 10 client.16332 handle_command_reply: tid=0 2024-03-16T10:19:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.742+0000 7f8cf9618740 1 client.16332 shutdown 2024-03-16T10:19:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.742+0000 7f8cf9618740 1 -- 172.21.15.77:0/3513062981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acbd30 msgr2=0x55afd1ada610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.742+0000 7f8cf9618740 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acbd30 0x55afd1ada610 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009780 tx=0x7f8ce402c390 comp rx=0 tx=0).stop 2024-03-16T10:19:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.742+0000 7f8cf9618740 1 -- 172.21.15.77:0/3513062981 shutdown_connections 2024-03-16T10:19:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.742+0000 7f8cf9618740 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afd1ad0ea0 0x55afd1adb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.742+0000 7f8cf9618740 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55afd1b47b80 0x55afd1b49fa0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.742+0000 7f8cf9618740 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55afd1acbd30 0x55afd1ada610 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.742+0000 7f8cf9618740 1 --2- 172.21.15.77:0/3513062981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1acb340 0x55afd1ada0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.742+0000 7f8cf9618740 1 -- 172.21.15.77:0/3513062981 >> 172.21.15.77:0/3513062981 conn(0x55afd1ac68b0 msgr2=0x55afd1ad5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.742+0000 7f8ccf7fe640 0 client.16332 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.743+0000 7f8cf9618740 1 -- 172.21.15.77:0/3513062981 shutdown_connections 2024-03-16T10:19:50.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.743+0000 7f8cf9618740 1 -- 172.21.15.77:0/3513062981 wait complete. 2024-03-16T10:19:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.743+0000 7f8cf9618740 20 client.16332 trim_cache size 0 max 16384 2024-03-16T10:19:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:19:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:19:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:19:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:19:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.749+0000 7f8cf6953640 1 -- 172.21.15.77:0/1340134165 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8cd80a2920 msgr2=0x7f8cd80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.749+0000 7f8cf6953640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8cd80a2920 0x7f8cd80a4de0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8005fd0 tx=0x7f8ce8010040 comp rx=0 tx=0).stop 2024-03-16T10:19:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.749+0000 7f8cf6953640 1 -- 172.21.15.77:0/1340134165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cf0075820 msgr2=0x7f8cf019e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.749+0000 7f8cf6953640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cf0075820 0x7f8cf019e430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8ce402c180 tx=0x7f8ce4035f70 comp rx=0 tx=0).stop 2024-03-16T10:19:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.749+0000 7f8cf6953640 1 -- 172.21.15.77:0/1340134165 shutdown_connections 2024-03-16T10:19:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.749+0000 7f8cf6953640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cf01096f0 0x7f8cf01a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.750+0000 7f8cf6953640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8cd80a2920 0x7f8cd80a4de0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.750+0000 7f8cf6953640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cf0075820 0x7f8cf019e430 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.750+0000 7f8cf6953640 1 --2- 172.21.15.77:0/1340134165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cf0074240 0x7f8cf019def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.750+0000 7f8cf6953640 1 -- 172.21.15.77:0/1340134165 >> 172.21.15.77:0/1340134165 conn(0x7f8cf00fcda0 msgr2=0x7f8cf01011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.750+0000 7f8cf6953640 1 -- 172.21.15.77:0/1340134165 shutdown_connections 2024-03-16T10:19:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:19:50.750+0000 7f8cf6953640 1 -- 172.21.15.77:0/1340134165 wait complete. 2024-03-16T10:19:51.811 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:19:51.812 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:19:52.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:51 smithi192 ceph-mon[36354]: pgmap v433: 97 pgs: 97 active+clean; 223 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 600 op/s 2024-03-16T10:19:52.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:51 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:19:52.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:51 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:19:52.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:51 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:19:52.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:51 smithi077 ceph-mon[32400]: pgmap v433: 97 pgs: 97 active+clean; 223 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 600 op/s 2024-03-16T10:19:52.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:51 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:19:52.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:51 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:19:52.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:51 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:19:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:51 smithi193 ceph-mon[36311]: pgmap v433: 97 pgs: 97 active+clean; 223 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 600 op/s 2024-03-16T10:19:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:51 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:19:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:51 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:19:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:51 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:19:52.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.980+0000 7fdf83ef8640 1 -- 172.21.15.77:0/2449435869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c1087e0 msgr2=0x7fdf7c10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:52.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.980+0000 7fdf83ef8640 1 --2- 172.21.15.77:0/2449435869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c1087e0 0x7fdf7c10ac10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009900 tx=0x7fdf6c02bc80 comp rx=0 tx=0).stop 2024-03-16T10:19:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.981+0000 7fdf83ef8640 1 -- 172.21.15.77:0/2449435869 shutdown_connections 2024-03-16T10:19:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.981+0000 7fdf83ef8640 1 --2- 172.21.15.77:0/2449435869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf7c10db00 0x7fdf7c10df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.981+0000 7fdf83ef8640 1 --2- 172.21.15.77:0/2449435869 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf7c10b150 0x7fdf7c10d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.981+0000 7fdf83ef8640 1 --2- 172.21.15.77:0/2449435869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c1087e0 0x7fdf7c10ac10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:52.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.981+0000 7fdf83ef8640 1 -- 172.21.15.77:0/2449435869 >> 172.21.15.77:0/2449435869 conn(0x7fdf7c06e640 msgr2=0x7fdf7c070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.982+0000 7fdf83ef8640 1 -- 172.21.15.77:0/2449435869 shutdown_connections 2024-03-16T10:19:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.982+0000 7fdf83ef8640 1 -- 172.21.15.77:0/2449435869 wait complete. 2024-03-16T10:19:52.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.983+0000 7fdf83ef8640 1 Processor -- start 2024-03-16T10:19:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.983+0000 7fdf83ef8640 1 -- start start 2024-03-16T10:19:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.983+0000 7fdf83ef8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf7c1087e0 0x7fdf7c1a6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.983+0000 7fdf83ef8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf7c10b150 0x7fdf7c1a6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.983+0000 7fdf83ef8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c10db00 0x7fdf7c1adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.983+0000 7fdf83ef8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c11c3a0 con 0x7fdf7c10b150 2024-03-16T10:19:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.983+0000 7fdf83ef8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c11c220 con 0x7fdf7c1087e0 2024-03-16T10:19:52.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.983+0000 7fdf83ef8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c11c520 con 0x7fdf7c10db00 2024-03-16T10:19:52.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.983+0000 7fdf81493640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf7c10b150 0x7fdf7c1a6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:52.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.983+0000 7fdf81493640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf7c10b150 0x7fdf7c1a6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44014/0 (socket says 172.21.15.77:44014) 2024-03-16T10:19:52.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.983+0000 7fdf81493640 1 -- 172.21.15.77:0/1460627451 learned_addr learned my addr 172.21.15.77:0/1460627451 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:52.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.984+0000 7fdf81c94640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf7c1087e0 0x7fdf7c1a6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:52.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.984+0000 7fdf81493640 1 -- 172.21.15.77:0/1460627451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c10db00 msgr2=0x7fdf7c1adf80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T10:19:52.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.984+0000 7fdf81493640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c10db00 0x7fdf7c1adf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:52.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.984+0000 7fdf81493640 1 -- 172.21.15.77:0/1460627451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf7c1087e0 msgr2=0x7fdf7c1a6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:52.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.984+0000 7fdf81493640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf7c1087e0 0x7fdf7c1a6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:52.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.984+0000 7fdf81493640 1 -- 172.21.15.77:0/1460627451 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c1ae5f0 con 0x7fdf7c10b150 2024-03-16T10:19:52.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.984+0000 7fdf81c94640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf7c1087e0 0x7fdf7c1a6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:19:52.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.984+0000 7fdf81493640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf7c10b150 0x7fdf7c1a6f60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800b450 tx=0x7fdf7800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:52.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.985+0000 7fdf72ffd640 1 -- 172.21.15.77:0/1460627451 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78015020 con 0x7fdf7c10b150 2024-03-16T10:19:52.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.985+0000 7fdf72ffd640 1 -- 172.21.15.77:0/1460627451 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf78011d70 con 0x7fdf7c10b150 2024-03-16T10:19:52.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.985+0000 7fdf72ffd640 1 -- 172.21.15.77:0/1460627451 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7800f340 con 0x7fdf7c10b150 2024-03-16T10:19:52.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.985+0000 7fdf83ef8640 1 -- 172.21.15.77:0/1460627451 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7c1ae8e0 con 0x7fdf7c10b150 2024-03-16T10:19:52.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.985+0000 7fdf83ef8640 1 -- 172.21.15.77:0/1460627451 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf7c1aede0 con 0x7fdf7c10b150 2024-03-16T10:19:52.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.992+0000 7fdf86bc2740 1 Processor -- start 2024-03-16T10:19:52.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.992+0000 7fdf86bc2740 1 -- start start 2024-03-16T10:19:52.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.992+0000 7fdf86bc2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 0x563f81494590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:52.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.992+0000 7fdf86bc2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 0x563f81499660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:52.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.992+0000 7fdf86bc2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 0x563f8149c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:52.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.992+0000 7fdf86bc2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563f813f9500 con 0x563f81494170 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.992+0000 7fdf86bc2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563f813a1be0 con 0x563f81499cd0 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.992+0000 7fdf86bc2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563f813f55a0 con 0x563f81494b60 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.992+0000 7fdf72ffd640 1 -- 172.21.15.77:0/1460627451 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fdf78018020 con 0x7fdf7c10b150 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf82495640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 0x563f81494590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf81c94640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 0x563f81499660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf81c94640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 0x563f81499660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44050/0 (socket says 172.21.15.77:44050) 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf82495640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 0x563f81494590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44018/0 (socket says 172.21.15.77:44018) 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf81c94640 1 -- 172.21.15.77:0/1420119134 learned_addr learned my addr 172.21.15.77:0/1420119134 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf81493640 1 --2- 172.21.15.77:0/1420119134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 0x563f8149c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf81c94640 1 -- 172.21.15.77:0/1420119134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 msgr2=0x563f8149c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf81c94640 1 --2- 172.21.15.77:0/1420119134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 0x563f8149c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf81c94640 1 -- 172.21.15.77:0/1420119134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 msgr2=0x563f81494590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf81c94640 1 --2- 172.21.15.77:0/1420119134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 0x563f81494590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf81c94640 1 -- 172.21.15.77:0/1420119134 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f813b54b0 con 0x563f81494b60 2024-03-16T10:19:52.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf82495640 1 --2- 172.21.15.77:0/1420119134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 0x563f81494590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:52.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.993+0000 7fdf81c94640 1 --2- 172.21.15.77:0/1420119134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 0x563f81499660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c02c210 tx=0x7fdf6c02c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c434ca77bb03a56 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:52.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.994+0000 7fdf72ffd640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdf680a2990 0x7fdf680a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:52.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.994+0000 7fdf67fff640 1 -- 172.21.15.77:0/1420119134 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c02cd00 con 0x563f81494b60 2024-03-16T10:19:52.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.994+0000 7fdf67fff640 1 -- 172.21.15.77:0/1420119134 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf6c005000 con 0x563f81494b60 2024-03-16T10:19:52.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.994+0000 7fdf67fff640 1 -- 172.21.15.77:0/1420119134 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c004b50 con 0x563f81494b60 2024-03-16T10:19:52.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.994+0000 7fdf82495640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdf680a2990 0x7fdf680a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:52.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.994+0000 7fdf72ffd640 1 -- 172.21.15.77:0/1460627451 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fdf780ce4a0 con 0x7fdf7c10b150 2024-03-16T10:19:52.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.994+0000 7fdf86bc2740 1 -- 172.21.15.77:0/1420119134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 msgr2=0x563f81499660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:52.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.994+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/1420119134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 0x563f81499660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c02c210 tx=0x7fdf6c02c6d0 comp rx=0 tx=0).stop 2024-03-16T10:19:52.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.994+0000 7fdf82495640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdf680a2990 0x7fdf680a4e50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x563f814953c0 tx=0x7fdf7400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.995+0000 7fdf86bc2740 1 -- 172.21.15.77:0/1420119134 shutdown_connections 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.995+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/1420119134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 0x563f8149c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.995+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/1420119134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 0x563f81499660 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.995+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/1420119134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 0x563f81494590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.995+0000 7fdf86bc2740 1 -- 172.21.15.77:0/1420119134 >> 172.21.15.77:0/1420119134 conn(0x563f8148f6e0 msgr2=0x563f8148fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.995+0000 7fdf86bc2740 1 -- 172.21.15.77:0/1420119134 shutdown_connections 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.995+0000 7fdf86bc2740 1 -- 172.21.15.77:0/1420119134 wait complete. 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.995+0000 7fdf86bc2740 1 -- start start 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.995+0000 7fdf86bc2740 1 Processor -- start 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.995+0000 7fdf86bc2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 0x563f8149e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf86bc2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 0x563f8149ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf86bc2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 0x563f814a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf86bc2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563f8149cc20 con 0x563f81494170 2024-03-16T10:19:53.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf86bc2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563f813b5160 con 0x563f81499cd0 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf86bc2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563f8149cda0 con 0x563f81494b60 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf86bc2740 10 client.0 _ll_register_callbacks cb 0x563f81399150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf81c94640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 0x563f8149e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf81493640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 0x563f8149ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf81493640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 0x563f8149ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44066/0 (socket says 172.21.15.77:44066) 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf81493640 1 -- 172.21.15.77:0/3920947790 learned_addr learned my addr 172.21.15.77:0/3920947790 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf81c94640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 0x563f8149e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44032/0 (socket says 172.21.15.77:44032) 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf82495640 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 0x563f814a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf81493640 1 -- 172.21.15.77:0/3920947790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 msgr2=0x563f814a5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf81493640 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 0x563f814a5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf81493640 1 -- 172.21.15.77:0/3920947790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 msgr2=0x563f8149e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf81493640 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 0x563f8149e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf81493640 1 -- 172.21.15.77:0/3920947790 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f811d77d0 con 0x563f81494b60 2024-03-16T10:19:53.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf81c94640 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 0x563f8149e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.996+0000 7fdf81493640 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 0x563f8149ebd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdf78010a80 tx=0x7fdf7809f170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1da52c55976f32ec server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.997+0000 7fdf657fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.997+0000 7fdf657fa640 1 -- 172.21.15.77:0/3920947790 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780ce0d0 con 0x563f81494b60 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.997+0000 7fdf657fa640 1 -- 172.21.15.77:0/3920947790 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fdf780ce270 con 0x563f81494b60 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.997+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3920947790 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf6c005000 con 0x563f81494b60 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.997+0000 7fdf657fa640 1 -- 172.21.15.77:0/3920947790 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdf78096a90 con 0x563f81494b60 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.997+0000 7fdf86bc2740 10 client.26107 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.997+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3920947790 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563f8110d3f0 con 0x563f81494b60 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.997+0000 7fdf657fa640 1 -- 172.21.15.77:0/3920947790 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf6c005000 con 0x563f81494b60 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.997+0000 7fdf657fa640 1 -- 172.21.15.77:0/3920947790 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fdf780d0c40 con 0x563f81494b60 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.997+0000 7fdf86bc2740 10 client.26107 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.997+0000 7fdf86bc2740 10 client.26107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:53.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.997+0000 7fdf86bc2740 20 client.26107 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:53.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.998+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563f814aa2c0 0x563f814ac780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.998+0000 7fdf86bc2740 4 client.26107 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:53.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.998+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3920947790 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f81406a40 con 0x563f814aa2c0 2024-03-16T10:19:53.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.998+0000 7fdf81c94640 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563f814aa2c0 0x563f814ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.998+0000 7fdf657fa640 10 client.26107 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:53.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:52.999+0000 7fdf81c94640 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563f814aa2c0 0x563f814ac780 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db1707a09d9308c3 server_cookie=5deea439c5068db7 in_seq=0 out_seq=0 2024-03-16T10:19:53.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf657fa640 1 -- 172.21.15.77:0/3920947790 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563f81406a40 con 0x563f814aa2c0 2024-03-16T10:19:53.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf657fa640 10 client.26107 handle_command_reply: tid=0 2024-03-16T10:19:53.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf86bc2740 1 client.26107 shutdown 2024-03-16T10:19:53.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3920947790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 msgr2=0x563f8149ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:53.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 0x563f8149ebd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdf78010a80 tx=0x7fdf7809f170 comp rx=0 tx=0).stop 2024-03-16T10:19:53.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3920947790 shutdown_connections 2024-03-16T10:19:53.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 0x563f814a5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494b60 0x563f8149ebd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563f814aa2c0 0x563f814ac780 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/3920947790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494170 0x563f8149e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf657fa640 0 client.26107 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:53.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3920947790 >> 172.21.15.77:0/3920947790 conn(0x563f8148f6e0 msgr2=0x563f8148e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:53.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3920947790 shutdown_connections 2024-03-16T10:19:53.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3920947790 wait complete. 2024-03-16T10:19:53.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.000+0000 7fdf86bc2740 20 client.26107 trim_cache size 0 max 16384 2024-03-16T10:19:53.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.018+0000 7fdf86bc2740 1 Processor -- start 2024-03-16T10:19:53.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.018+0000 7fdf86bc2740 1 -- start start 2024-03-16T10:19:53.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf86bc2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 0x563f814a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf86bc2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494b60 0x563f814a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf86bc2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 0x563f814a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf86bc2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c1b7a20 con 0x563f81494b60 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf86bc2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c1b96e0 con 0x563f81499cd0 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf86bc2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c1b80f0 con 0x563f81494170 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf81493640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494b60 0x563f814a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf81493640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494b60 0x563f814a5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44046/0 (socket says 172.21.15.77:44046) 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf81c94640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 0x563f814a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf81493640 1 -- 172.21.15.77:0/3520201478 learned_addr learned my addr 172.21.15.77:0/3520201478 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf82495640 1 --2- 172.21.15.77:0/3520201478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 0x563f814a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf81493640 1 -- 172.21.15.77:0/3520201478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 msgr2=0x563f814a62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf81493640 1 --2- 172.21.15.77:0/3520201478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 0x563f814a62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf81493640 1 -- 172.21.15.77:0/3520201478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 msgr2=0x563f814a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf81493640 1 --2- 172.21.15.77:0/3520201478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 0x563f814a67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf81493640 1 -- 172.21.15.77:0/3520201478 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f810cd0f0 con 0x563f81494b60 2024-03-16T10:19:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.019+0000 7fdf81c94640 1 --2- 172.21.15.77:0/3520201478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 0x563f814a62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:53.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.020+0000 7fdf81493640 1 --2- 172.21.15.77:0/3520201478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494b60 0x563f814a5bd0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fdf78010830 tx=0x7fdf78098780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d75104a9b2d1c47e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:53.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.020+0000 7fdf677fe640 1 -- 172.21.15.77:0/3520201478 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780b7070 con 0x563f81494b60 2024-03-16T10:19:53.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.020+0000 7fdf677fe640 1 -- 172.21.15.77:0/3520201478 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf780b9a20 con 0x563f81494b60 2024-03-16T10:19:53.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.020+0000 7fdf677fe640 1 -- 172.21.15.77:0/3520201478 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780b9300 con 0x563f81494b60 2024-03-16T10:19:53.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.021+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3520201478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494b60 msgr2=0x563f814a5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:53.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.021+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/3520201478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494b60 0x563f814a5bd0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fdf78010830 tx=0x7fdf78098780 comp rx=0 tx=0).stop 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.021+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3520201478 shutdown_connections 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.021+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/3520201478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81499cd0 0x563f814a67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.021+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/3520201478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81494b60 0x563f814a5bd0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.021+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/3520201478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 0x563f814a62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.021+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3520201478 >> 172.21.15.77:0/3520201478 conn(0x563f8148f6e0 msgr2=0x563f814960f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.021+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3520201478 shutdown_connections 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.021+0000 7fdf86bc2740 1 -- 172.21.15.77:0/3520201478 wait complete. 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.021+0000 7fdf86bc2740 1 -- start start 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.021+0000 7fdf86bc2740 1 Processor -- start 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf86bc2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 0x563f814a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf86bc2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81494b60 0x563f814a2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf86bc2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81499cd0 0x563f814a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf86bc2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c1b8f50 con 0x563f81499cd0 2024-03-16T10:19:53.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf86bc2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c1b8880 con 0x563f81494b60 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf86bc2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c1b5a00 con 0x563f81494170 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf86bc2740 10 client.0 _ll_register_callbacks cb 0x7fdf780d1af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf81c94640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 0x563f814a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf81493640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81494b60 0x563f814a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf81c94640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 0x563f814a5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44082/0 (socket says 172.21.15.77:44082) 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf81493640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81494b60 0x563f814a2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41536/0 (socket says 172.21.15.77:41536) 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf81c94640 1 -- 172.21.15.77:0/901017457 learned_addr learned my addr 172.21.15.77:0/901017457 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf82495640 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81499cd0 0x563f814a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf81c94640 1 -- 172.21.15.77:0/901017457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81494b60 msgr2=0x563f814a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf81c94640 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81494b60 0x563f814a2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf81c94640 1 -- 172.21.15.77:0/901017457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81499cd0 msgr2=0x563f814a2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf81c94640 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81499cd0 0x563f814a2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf81c94640 1 -- 172.21.15.77:0/901017457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f813c09d0 con 0x563f81494170 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf81493640 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81494b60 0x563f814a2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:53.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf81c94640 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 0x563f814a5c20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c02c210 tx=0x7fdf6c02c240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=687e3b964cfcaec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.022+0000 7fdf657fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf657fa640 1 -- 172.21.15.77:0/901017457 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c052020 con 0x563f81494170 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf657fa640 1 -- 172.21.15.77:0/901017457 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fdf6c04e7b0 con 0x563f81494170 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf86bc2740 1 -- 172.21.15.77:0/901017457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf780b9a20 con 0x563f81494170 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf657fa640 1 -- 172.21.15.77:0/901017457 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdf6c0053b0 con 0x563f81494170 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf86bc2740 10 client.26113 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf86bc2740 1 -- 172.21.15.77:0/901017457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563f8150f860 con 0x563f81494170 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf657fa640 1 -- 172.21.15.77:0/901017457 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf780b9a20 con 0x563f81494170 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf657fa640 1 -- 172.21.15.77:0/901017457 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fdf6c057030 con 0x563f81494170 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf86bc2740 10 client.26113 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf86bc2740 10 client.26113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf86bc2740 20 client.26113 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563f814aa2c0 0x563f81511ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf86bc2740 4 client.26113 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:19:53.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.023+0000 7fdf86bc2740 1 -- 172.21.15.77:0/901017457 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563f81400d90 con 0x563f814aa2c0 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.024+0000 7fdf81493640 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563f814aa2c0 0x563f81511ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.024+0000 7fdf81493640 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563f814aa2c0 0x563f81511ef0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6487ef8e1ee29869 server_cookie=637f4c0d66b8510f in_seq=0 out_seq=0 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.024+0000 7fdf657fa640 10 client.26113 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.024+0000 7fdf657fa640 1 -- 172.21.15.77:0/901017457 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563f81400d90 con 0x563f814aa2c0 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.024+0000 7fdf657fa640 10 client.26113 handle_command_reply: tid=0 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.024+0000 7fdf86bc2740 1 client.26113 shutdown 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.024+0000 7fdf86bc2740 1 -- 172.21.15.77:0/901017457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 msgr2=0x563f814a5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.024+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 0x563f814a5c20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c02c210 tx=0x7fdf6c02c240 comp rx=0 tx=0).stop 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf86bc2740 1 -- 172.21.15.77:0/901017457 shutdown_connections 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563f814aa2c0 0x563f81511ef0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81499cd0 0x563f814a2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81494b60 0x563f814a2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf86bc2740 1 --2- 172.21.15.77:0/901017457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81494170 0x563f814a5c20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf657fa640 0 client.26113 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:53.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf86bc2740 1 -- 172.21.15.77:0/901017457 >> 172.21.15.77:0/901017457 conn(0x563f8148f6e0 msgr2=0x563f814a62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:53.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf86bc2740 1 -- 172.21.15.77:0/901017457 shutdown_connections 2024-03-16T10:19:53.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf86bc2740 1 -- 172.21.15.77:0/901017457 wait complete. 2024-03-16T10:19:53.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf86bc2740 20 client.26113 trim_cache size 0 max 16384 2024-03-16T10:19:53.031 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:19:53.031 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T10:19:53.031 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T10:19:53.031 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:19:53.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf83ef8640 1 -- 172.21.15.77:0/1460627451 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdf680a2990 msgr2=0x7fdf680a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:53.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf83ef8640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdf680a2990 0x7fdf680a4e50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x563f814953c0 tx=0x7fdf7400d040 comp rx=0 tx=0).stop 2024-03-16T10:19:53.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf83ef8640 1 -- 172.21.15.77:0/1460627451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf7c10b150 msgr2=0x7fdf7c1a6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:53.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf83ef8640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf7c10b150 0x7fdf7c1a6f60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800b450 tx=0x7fdf7800b910 comp rx=0 tx=0).stop 2024-03-16T10:19:53.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf83ef8640 1 -- 172.21.15.77:0/1460627451 shutdown_connections 2024-03-16T10:19:53.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf83ef8640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fdf680a2990 0x7fdf680a4e50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf83ef8640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c10db00 0x7fdf7c1adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf83ef8640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf7c10b150 0x7fdf7c1a6f60 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf83ef8640 1 --2- 172.21.15.77:0/1460627451 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf7c1087e0 0x7fdf7c1a6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.025+0000 7fdf83ef8640 1 -- 172.21.15.77:0/1460627451 >> 172.21.15.77:0/1460627451 conn(0x7fdf7c06e640 msgr2=0x7fdf7c10d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:53.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.026+0000 7fdf83ef8640 1 -- 172.21.15.77:0/1460627451 shutdown_connections 2024-03-16T10:19:53.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.026+0000 7fdf83ef8640 1 -- 172.21.15.77:0/1460627451 wait complete. 2024-03-16T10:19:53.041 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T10:19:53.041 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:19:53.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.173+0000 7fa35a663640 1 -- 172.21.15.77:0/831686915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa354100a10 msgr2=0x7fa354100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:53.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.173+0000 7fa35a663640 1 --2- 172.21.15.77:0/831686915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa354100a10 0x7fa354100e70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa348009900 tx=0x7fa34802bc80 comp rx=0 tx=0).stop 2024-03-16T10:19:53.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.173+0000 7fa35a663640 1 -- 172.21.15.77:0/831686915 shutdown_connections 2024-03-16T10:19:53.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.173+0000 7fa35a663640 1 --2- 172.21.15.77:0/831686915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3541013b0 0x7fa354101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.173+0000 7fa35a663640 1 --2- 172.21.15.77:0/831686915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa354100a10 0x7fa354100e70 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.173+0000 7fa35a663640 1 --2- 172.21.15.77:0/831686915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa3541000b0 0x7fa3541004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.173+0000 7fa35a663640 1 -- 172.21.15.77:0/831686915 >> 172.21.15.77:0/831686915 conn(0x7fa3540fbc70 msgr2=0x7fa3540fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:53.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.173+0000 7fa35a663640 1 -- 172.21.15.77:0/831686915 shutdown_connections 2024-03-16T10:19:53.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.173+0000 7fa35a663640 1 -- 172.21.15.77:0/831686915 wait complete. 2024-03-16T10:19:53.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.174+0000 7fa35a663640 1 Processor -- start 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.174+0000 7fa35a663640 1 -- start start 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.174+0000 7fa35a663640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3541000b0 0x7fa354199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.174+0000 7fa35a663640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa354100a10 0x7fa35419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.174+0000 7fa35a663640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3541013b0 0x7fa3541a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.174+0000 7fa35a663640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa35410f650 con 0x7fa3541013b0 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.174+0000 7fa35a663640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa35410f4d0 con 0x7fa354100a10 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.174+0000 7fa35a663640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa35410f7d0 con 0x7fa3541000b0 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.174+0000 7fa353fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3541000b0 0x7fa354199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa353fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3541000b0 0x7fa354199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44098/0 (socket says 172.21.15.77:44098) 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa353fff640 1 -- 172.21.15.77:0/2377591355 learned_addr learned my addr 172.21.15.77:0/2377591355 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa358c00640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3541013b0 0x7fa3541a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa3537fe640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa354100a10 0x7fa35419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa353fff640 1 -- 172.21.15.77:0/2377591355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa354100a10 msgr2=0x7fa35419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:53.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa353fff640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa354100a10 0x7fa35419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa353fff640 1 -- 172.21.15.77:0/2377591355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3541013b0 msgr2=0x7fa3541a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:53.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa353fff640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3541013b0 0x7fa3541a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa353fff640 1 -- 172.21.15.77:0/2377591355 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3541a1970 con 0x7fa3541000b0 2024-03-16T10:19:53.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa3537fe640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa354100a10 0x7fa35419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:19:53.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa353fff640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3541000b0 0x7fa354199d50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa34400e9f0 tx=0x7fa34400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:53.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa3517fa640 1 -- 172.21.15.77:0/2377591355 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34400ccc0 con 0x7fa3541000b0 2024-03-16T10:19:53.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa3517fa640 1 -- 172.21.15.77:0/2377591355 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa344010910 con 0x7fa3541000b0 2024-03-16T10:19:53.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa35a663640 1 -- 172.21.15.77:0/2377591355 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3541a1c60 con 0x7fa3541000b0 2024-03-16T10:19:53.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa3517fa640 1 -- 172.21.15.77:0/2377591355 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3440188a0 con 0x7fa3541000b0 2024-03-16T10:19:53.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.175+0000 7fa35a663640 1 -- 172.21.15.77:0/2377591355 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3541a21d0 con 0x7fa3541000b0 2024-03-16T10:19:53.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.177+0000 7fa33affd640 1 -- 172.21.15.77:0/2377591355 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa334005290 con 0x7fa3541000b0 2024-03-16T10:19:53.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.178+0000 7fa3517fa640 1 -- 172.21.15.77:0/2377591355 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fa344010430 con 0x7fa3541000b0 2024-03-16T10:19:53.180 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.179+0000 7fa3517fa640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa33c0a2900 0x7fa33c0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:53.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.180+0000 7fa3517fa640 1 -- 172.21.15.77:0/2377591355 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa3440ceed0 con 0x7fa3541000b0 2024-03-16T10:19:53.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.180+0000 7fa3537fe640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa33c0a2900 0x7fa33c0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:53.181 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.181+0000 7fa3537fe640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa33c0a2900 0x7fa33c0a4dc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa348006010 tx=0x7fa34802c890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:53.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.188+0000 7fa3517fa640 1 -- 172.21.15.77:0/2377591355 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa34409bf60 con 0x7fa3541000b0 2024-03-16T10:19:53.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.486+0000 7fa33affd640 1 -- 172.21.15.77:0/2377591355 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa334005580 con 0x7fa3541000b0 2024-03-16T10:19:53.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.488+0000 7fa3517fa640 1 -- 172.21.15.77:0/2377591355 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11952 (secure 0 0 0) 0x7fa34409b860 con 0x7fa3541000b0 2024-03-16T10:19:53.490 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:19:53.490 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:19:24.666985+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24455,14658,24482,24475,24454]},"id":1}]} 2024-03-16T10:19:53.490 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 32 2024-03-16T10:19:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.493+0000 7fa35a663640 1 -- 172.21.15.77:0/2377591355 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa33c0a2900 msgr2=0x7fa33c0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.493+0000 7fa35a663640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa33c0a2900 0x7fa33c0a4dc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa348006010 tx=0x7fa34802c890 comp rx=0 tx=0).stop 2024-03-16T10:19:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.493+0000 7fa35a663640 1 -- 172.21.15.77:0/2377591355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3541000b0 msgr2=0x7fa354199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:53.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.493+0000 7fa35a663640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3541000b0 0x7fa354199d50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa34400e9f0 tx=0x7fa34400edf0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.498+0000 7fa35a663640 1 -- 172.21.15.77:0/2377591355 shutdown_connections 2024-03-16T10:19:53.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.498+0000 7fa35a663640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3541013b0 0x7fa3541a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.498+0000 7fa35a663640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa354100a10 0x7fa35419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.498+0000 7fa35a663640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa33c0a2900 0x7fa33c0a4dc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.498+0000 7fa35a663640 1 --2- 172.21.15.77:0/2377591355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3541000b0 0x7fa354199d50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:53.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.498+0000 7fa35a663640 1 -- 172.21.15.77:0/2377591355 >> 172.21.15.77:0/2377591355 conn(0x7fa3540fbc70 msgr2=0x7fa3540fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.510+0000 7fa35a663640 1 -- 172.21.15.77:0/2377591355 shutdown_connections 2024-03-16T10:19:53.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:53.516+0000 7fa35a663640 1 -- 172.21.15.77:0/2377591355 wait complete. 2024-03-16T10:19:53.542 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-16T10:19:54.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.108+0000 7f8507d7e640 1 -- 172.21.15.77:0/3665263538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001006f0 msgr2=0x7f8500100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.108+0000 7f8507d7e640 1 --2- 172.21.15.77:0/3665263538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001006f0 0x7f8500100b10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009860 tx=0x7f84fc02bb70 comp rx=0 tx=0).stop 2024-03-16T10:19:54.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.108+0000 7f8507d7e640 1 -- 172.21.15.77:0/3665263538 shutdown_connections 2024-03-16T10:19:54.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.108+0000 7f8507d7e640 1 --2- 172.21.15.77:0/3665263538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85001022a0 0x7f8500108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.108+0000 7f8507d7e640 1 --2- 172.21.15.77:0/3665263538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85001018e0 0x7f8500101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.108+0000 7f8507d7e640 1 --2- 172.21.15.77:0/3665263538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001006f0 0x7f8500100b10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.108+0000 7f8507d7e640 1 -- 172.21.15.77:0/3665263538 >> 172.21.15.77:0/3665263538 conn(0x7f85000fbc80 msgr2=0x7f85000fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.108+0000 7f8507d7e640 1 -- 172.21.15.77:0/3665263538 shutdown_connections 2024-03-16T10:19:54.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.108+0000 7f8507d7e640 1 -- 172.21.15.77:0/3665263538 wait complete. 2024-03-16T10:19:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.109+0000 7f8507d7e640 1 Processor -- start 2024-03-16T10:19:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.109+0000 7f8507d7e640 1 -- start start 2024-03-16T10:19:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.109+0000 7f8507d7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85001006f0 0x7f8500197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.109+0000 7f8507d7e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85001018e0 0x7f8500198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8507d7e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001022a0 0x7f850019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8505b1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85001006f0 0x7f8500197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8505b1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85001006f0 0x7f8500197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44078/0 (socket says 172.21.15.77:44078) 2024-03-16T10:19:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8505319640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85001018e0 0x7f8500198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8507d7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f850010d490 con 0x7f85001006f0 2024-03-16T10:19:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8507d7e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f850010d310 con 0x7f85001018e0 2024-03-16T10:19:54.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8505319640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85001018e0 0x7f8500198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41564/0 (socket says 172.21.15.77:41564) 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8505319640 1 -- 172.21.15.77:0/1154865433 learned_addr learned my addr 172.21.15.77:0/1154865433 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8507d7e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f850010d610 con 0x7f85001022a0 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f850631b640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001022a0 0x7f850019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8505b1a640 1 -- 172.21.15.77:0/1154865433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001022a0 msgr2=0x7f850019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8505b1a640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001022a0 0x7f850019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8505b1a640 1 -- 172.21.15.77:0/1154865433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85001018e0 msgr2=0x7f8500198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8505b1a640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85001018e0 0x7f8500198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8505b1a640 1 -- 172.21.15.77:0/1154865433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850019f760 con 0x7f85001006f0 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f850631b640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001022a0 0x7f850019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8505319640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85001018e0 0x7f8500198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.110+0000 7f8505b1a640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85001006f0 0x7f8500197b40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f84fc02c0b0 tx=0x7f84fc032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.111+0000 7f84f6ffd640 1 -- 172.21.15.77:0/1154865433 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc041020 con 0x7f85001006f0 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.111+0000 7f84f6ffd640 1 -- 172.21.15.77:0/1154865433 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84fc02c9f0 con 0x7f85001006f0 2024-03-16T10:19:54.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.111+0000 7f8507d7e640 1 -- 172.21.15.77:0/1154865433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850019f9f0 con 0x7f85001006f0 2024-03-16T10:19:54.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.111+0000 7f8507d7e640 1 -- 172.21.15.77:0/1154865433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f850019ff30 con 0x7f85001006f0 2024-03-16T10:19:54.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.111+0000 7f84f6ffd640 1 -- 172.21.15.77:0/1154865433 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc039040 con 0x7f85001006f0 2024-03-16T10:19:54.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.114+0000 7f84f6ffd640 1 -- 172.21.15.77:0/1154865433 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f84fc044020 con 0x7f85001006f0 2024-03-16T10:19:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f850a96d740 1 Processor -- start 2024-03-16T10:19:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f850a96d740 1 -- start start 2024-03-16T10:19:54.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f84f6ffd640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84ec0a2880 0x7f84ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f850a96d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 0x55bf7f447c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f850a96d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f448240 0x55bf7f44cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f850a96d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f44d3b0 0x55bf7f44f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f850a96d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bf7f3ac490 con 0x55bf7f447850 2024-03-16T10:19:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f850a96d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55bf7f354c90 con 0x55bf7f448240 2024-03-16T10:19:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f850a96d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bf7f3a8530 con 0x55bf7f44d3b0 2024-03-16T10:19:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f8505b1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 0x55bf7f447c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f850631b640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84ec0a2880 0x7f84ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f8505b1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 0x55bf7f447c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44094/0 (socket says 172.21.15.77:44094) 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.115+0000 7f8505b1a640 1 -- 172.21.15.77:0/2344412066 learned_addr learned my addr 172.21.15.77:0/2344412066 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f850631b640 1 --2- 172.21.15.77:0/2344412066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f44d3b0 0x55bf7f44f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f8505319640 1 --2- 172.21.15.77:0/2344412066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f448240 0x55bf7f44cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f8505b1a640 1 -- 172.21.15.77:0/2344412066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f44d3b0 msgr2=0x55bf7f44f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f8505b1a640 1 --2- 172.21.15.77:0/2344412066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f44d3b0 0x55bf7f44f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f8505b1a640 1 -- 172.21.15.77:0/2344412066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f448240 msgr2=0x55bf7f44cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f8505b1a640 1 --2- 172.21.15.77:0/2344412066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f448240 0x55bf7f44cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f8505b1a640 1 -- 172.21.15.77:0/2344412066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf7f40f500 con 0x55bf7f447850 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f84f6ffd640 1 -- 172.21.15.77:0/1154865433 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f84fc0f0260 con 0x7f85001006f0 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f850631b640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84ec0a2880 0x7f84ec0a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f84f8005c40 tx=0x7f84f8009e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f8505b1a640 1 --2- 172.21.15.77:0/2344412066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 0x55bf7f447c70 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f84fc02cbc0 tx=0x7f84fc02bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea4abf77f9bb9e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f84ebfff640 1 -- 172.21.15.77:0/2344412066 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc09a300 con 0x55bf7f447850 2024-03-16T10:19:54.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f84ebfff640 1 -- 172.21.15.77:0/2344412066 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84fc0bd390 con 0x55bf7f447850 2024-03-16T10:19:54.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.116+0000 7f84ebfff640 1 -- 172.21.15.77:0/2344412066 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc02f160 con 0x55bf7f447850 2024-03-16T10:19:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.117+0000 7f850a96d740 1 -- 172.21.15.77:0/2344412066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 msgr2=0x55bf7f447c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.117+0000 7f850a96d740 1 --2- 172.21.15.77:0/2344412066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 0x55bf7f447c70 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f84fc02cbc0 tx=0x7f84fc02bb70 comp rx=0 tx=0).stop 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.117+0000 7f850a96d740 1 -- 172.21.15.77:0/2344412066 shutdown_connections 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.117+0000 7f850a96d740 1 --2- 172.21.15.77:0/2344412066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f44d3b0 0x55bf7f44f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.117+0000 7f850a96d740 1 --2- 172.21.15.77:0/2344412066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f448240 0x55bf7f44cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.117+0000 7f850a96d740 1 --2- 172.21.15.77:0/2344412066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 0x55bf7f447c70 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.117+0000 7f850a96d740 1 -- 172.21.15.77:0/2344412066 >> 172.21.15.77:0/2344412066 conn(0x55bf7f442dc0 msgr2=0x55bf7f4431c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.117+0000 7f850a96d740 1 -- 172.21.15.77:0/2344412066 shutdown_connections 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.117+0000 7f850a96d740 1 -- 172.21.15.77:0/2344412066 wait complete. 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.118+0000 7f850a96d740 1 -- start start 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.118+0000 7f850a96d740 1 Processor -- start 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.118+0000 7f850a96d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f447850 0x55bf7f455f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.118+0000 7f850a96d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f448240 0x55bf7f456460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.118+0000 7f850a96d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f44d3b0 0x55bf7f45d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.118+0000 7f850a96d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bf7f454530 con 0x55bf7f44d3b0 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.118+0000 7f850a96d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55bf7f368230 con 0x55bf7f448240 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.118+0000 7f850a96d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bf7f4546b0 con 0x55bf7f447850 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.118+0000 7f850a96d740 10 client.0 _ll_register_callbacks cb 0x55bf7f3a4010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f8505b1a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f447850 0x55bf7f455f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f8505b1a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f447850 0x55bf7f455f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44136/0 (socket says 172.21.15.77:44136) 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f8505b1a640 1 -- 172.21.15.77:0/2574008797 learned_addr learned my addr 172.21.15.77:0/2574008797 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f8505319640 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f448240 0x55bf7f456460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f850631b640 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f44d3b0 0x55bf7f45d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f8505b1a640 1 -- 172.21.15.77:0/2574008797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f448240 msgr2=0x55bf7f456460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f8505b1a640 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f448240 0x55bf7f456460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f8505b1a640 1 -- 172.21.15.77:0/2574008797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f44d3b0 msgr2=0x55bf7f45d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f8505b1a640 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f44d3b0 0x55bf7f45d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f8505b1a640 1 -- 172.21.15.77:0/2574008797 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf7f377100 con 0x55bf7f447850 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f850631b640 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f44d3b0 0x55bf7f45d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f8505b1a640 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f447850 0x55bf7f455f20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f84fc004650 tx=0x7f84fc0095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=637c33e550771f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f84e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f84e97fa640 1 -- 172.21.15.77:0/2574008797 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc09b8a0 con 0x55bf7f447850 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f850a96d740 1 -- 172.21.15.77:0/2574008797 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84fc0bd390 con 0x55bf7f447850 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f84e97fa640 1 -- 172.21.15.77:0/2574008797 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f84fc032640 con 0x55bf7f447850 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f84e97fa640 1 -- 172.21.15.77:0/2574008797 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f84fc09beb0 con 0x55bf7f447850 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f850a96d740 10 client.26131 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.119+0000 7f850a96d740 1 -- 172.21.15.77:0/2574008797 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bf7f0c03f0 con 0x55bf7f447850 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.120+0000 7f84e97fa640 1 -- 172.21.15.77:0/2574008797 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84fc0bd390 con 0x55bf7f447850 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.120+0000 7f84e97fa640 1 -- 172.21.15.77:0/2574008797 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f84fc03c030 con 0x55bf7f447850 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.120+0000 7f850a96d740 10 client.26131 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.120+0000 7f850a96d740 10 client.26131 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.120+0000 7f850a96d740 20 client.26131 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.120+0000 7f850a96d740 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55bf7f461bb0 0x55bf7f464030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.120+0000 7f850a96d740 4 client.26131 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.120+0000 7f850a96d740 1 -- 172.21.15.77:0/2574008797 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf7f3b99d0 con 0x55bf7f461bb0 2024-03-16T10:19:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.120+0000 7f8505319640 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55bf7f461bb0 0x55bf7f464030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.121+0000 7f84e97fa640 10 client.26131 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:19:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.121+0000 7f8505319640 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55bf7f461bb0 0x55bf7f464030 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e18ef5e4db2abb0d server_cookie=ecb129a9938b8c6b in_seq=0 out_seq=0 2024-03-16T10:19:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.121+0000 7f84e97fa640 1 -- 172.21.15.77:0/2574008797 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bf7f3b99d0 con 0x55bf7f461bb0 2024-03-16T10:19:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.121+0000 7f84e97fa640 10 client.26131 handle_command_reply: tid=0 2024-03-16T10:19:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.121+0000 7f850a96d740 1 client.26131 shutdown 2024-03-16T10:19:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.122+0000 7f850a96d740 1 -- 172.21.15.77:0/2574008797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f447850 msgr2=0x55bf7f455f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.122+0000 7f850a96d740 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f447850 0x55bf7f455f20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f84fc004650 tx=0x7f84fc0095b0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.122+0000 7f850a96d740 1 -- 172.21.15.77:0/2574008797 shutdown_connections 2024-03-16T10:19:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.122+0000 7f850a96d740 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f44d3b0 0x55bf7f45d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.122+0000 7f850a96d740 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f448240 0x55bf7f456460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.122+0000 7f850a96d740 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55bf7f461bb0 0x55bf7f464030 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.122+0000 7f850a96d740 1 --2- 172.21.15.77:0/2574008797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f447850 0x55bf7f455f20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.122+0000 7f84e97fa640 0 client.26131 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:19:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.122+0000 7f850a96d740 1 -- 172.21.15.77:0/2574008797 >> 172.21.15.77:0/2574008797 conn(0x55bf7f442dc0 msgr2=0x55bf7f441fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.122+0000 7f850a96d740 1 -- 172.21.15.77:0/2574008797 shutdown_connections 2024-03-16T10:19:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.122+0000 7f850a96d740 1 -- 172.21.15.77:0/2574008797 wait complete. 2024-03-16T10:19:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.122+0000 7f850a96d740 20 client.26131 trim_cache size 0 max 16384 2024-03-16T10:19:54.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.136+0000 7f850a96d740 1 Processor -- start 2024-03-16T10:19:54.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.136+0000 7f850a96d740 1 -- start start 2024-03-16T10:19:54.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.136+0000 7f850a96d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 0x55bf7f441750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f850a96d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f448240 0x55bf7f45d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f850a96d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f44d3b0 0x55bf7f441c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f850a96d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f85001a69c0 con 0x55bf7f447850 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f850a96d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f85001a8680 con 0x55bf7f44d3b0 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f850a96d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f85001a7090 con 0x55bf7f448240 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f8505b1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 0x55bf7f441750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f8505b1a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 0x55bf7f441750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44114/0 (socket says 172.21.15.77:44114) 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f8505b1a640 1 -- 172.21.15.77:0/1839111318 learned_addr learned my addr 172.21.15.77:0/1839111318 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f8505319640 1 --2- 172.21.15.77:0/1839111318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f448240 0x55bf7f45d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f850631b640 1 --2- 172.21.15.77:0/1839111318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f44d3b0 0x55bf7f441c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f8505b1a640 1 -- 172.21.15.77:0/1839111318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f448240 msgr2=0x55bf7f45d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f8505b1a640 1 --2- 172.21.15.77:0/1839111318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f448240 0x55bf7f45d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f8505b1a640 1 -- 172.21.15.77:0/1839111318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f44d3b0 msgr2=0x55bf7f441c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f8505b1a640 1 --2- 172.21.15.77:0/1839111318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f44d3b0 0x55bf7f441c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f8505b1a640 1 -- 172.21.15.77:0/1839111318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf7f18a7d0 con 0x55bf7f447850 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f8505b1a640 1 --2- 172.21.15.77:0/1839111318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 0x55bf7f441750 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f84fc0b9660 tx=0x7f84fc0bc6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a749c75f1c6ce5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f84eb7fe640 1 -- 172.21.15.77:0/1839111318 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc09a120 con 0x55bf7f447850 2024-03-16T10:19:54.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f84eb7fe640 1 -- 172.21.15.77:0/1839111318 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84fc0bd390 con 0x55bf7f447850 2024-03-16T10:19:54.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.137+0000 7f84eb7fe640 1 -- 172.21.15.77:0/1839111318 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc0c5430 con 0x55bf7f447850 2024-03-16T10:19:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.138+0000 7f850a96d740 1 -- 172.21.15.77:0/1839111318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 msgr2=0x55bf7f441750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.138+0000 7f850a96d740 1 --2- 172.21.15.77:0/1839111318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 0x55bf7f441750 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f84fc0b9660 tx=0x7f84fc0bc6b0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.138+0000 7f850a96d740 1 -- 172.21.15.77:0/1839111318 shutdown_connections 2024-03-16T10:19:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.138+0000 7f850a96d740 1 --2- 172.21.15.77:0/1839111318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f44d3b0 0x55bf7f441c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.138+0000 7f850a96d740 1 --2- 172.21.15.77:0/1839111318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f448240 0x55bf7f45d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.138+0000 7f850a96d740 1 --2- 172.21.15.77:0/1839111318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f447850 0x55bf7f441750 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.138+0000 7f850a96d740 1 -- 172.21.15.77:0/1839111318 >> 172.21.15.77:0/1839111318 conn(0x55bf7f442dc0 msgr2=0x55bf7f455b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.138+0000 7f850a96d740 1 -- 172.21.15.77:0/1839111318 shutdown_connections 2024-03-16T10:19:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.138+0000 7f850a96d740 1 -- 172.21.15.77:0/1839111318 wait complete. 2024-03-16T10:19:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.138+0000 7f850a96d740 1 -- start start 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.139+0000 7f850a96d740 1 Processor -- start 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.139+0000 7f850a96d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f447850 0x55bf7f45a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.139+0000 7f850a96d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f448240 0x55bf7f45abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.139+0000 7f850a96d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f44d3b0 0x55bf7f45bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.139+0000 7f850a96d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f85001a7ef0 con 0x55bf7f448240 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.139+0000 7f850a96d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f85001a7820 con 0x55bf7f447850 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.139+0000 7f850a96d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f85001a49a0 con 0x55bf7f44d3b0 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.139+0000 7f850a96d740 10 client.0 _ll_register_callbacks cb 0x7f84f80127f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.139+0000 7f8505319640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f448240 0x55bf7f45abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.139+0000 7f8505319640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f448240 0x55bf7f45abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44122/0 (socket says 172.21.15.77:44122) 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.139+0000 7f8505319640 1 -- 172.21.15.77:0/1028025273 learned_addr learned my addr 172.21.15.77:0/1028025273 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.140+0000 7f8505319640 1 -- 172.21.15.77:0/1028025273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f44d3b0 msgr2=0x55bf7f45bc00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.140+0000 7f8505319640 1 --2- 172.21.15.77:0/1028025273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f44d3b0 0x55bf7f45bc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.140+0000 7f8505319640 1 -- 172.21.15.77:0/1028025273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f447850 msgr2=0x55bf7f45a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.140+0000 7f8505319640 1 --2- 172.21.15.77:0/1028025273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f447850 0x55bf7f45a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.140+0000 7f8505319640 1 -- 172.21.15.77:0/1028025273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf7f45c1f0 con 0x55bf7f448240 2024-03-16T10:19:54.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.140+0000 7f8505319640 1 --2- 172.21.15.77:0/1028025273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f448240 0x55bf7f45abf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x55bf7f448cb0 tx=0x7f84f0004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e6e826c58b7ce8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.140+0000 7f84e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:54.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.140+0000 7f84e97fa640 1 -- 172.21.15.77:0/1028025273 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0024030 con 0x55bf7f448240 2024-03-16T10:19:54.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.140+0000 7f84e97fa640 1 -- 172.21.15.77:0/1028025273 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f84f0003a40 con 0x55bf7f448240 2024-03-16T10:19:54.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.140+0000 7f850a96d740 1 -- 172.21.15.77:0/1028025273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84fc0bd390 con 0x55bf7f448240 2024-03-16T10:19:54.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.141+0000 7f84e97fa640 1 -- 172.21.15.77:0/1028025273 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f84f0003c40 con 0x55bf7f448240 2024-03-16T10:19:54.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.141+0000 7f850a96d740 10 client.16371 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:54.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.141+0000 7f850a96d740 1 -- 172.21.15.77:0/1028025273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bf7f45d5f0 con 0x55bf7f448240 2024-03-16T10:19:54.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.141+0000 7f84e97fa640 1 -- 172.21.15.77:0/1028025273 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84fc0bd390 con 0x55bf7f448240 2024-03-16T10:19:54.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.141+0000 7f84e97fa640 1 -- 172.21.15.77:0/1028025273 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f84f0004b80 con 0x55bf7f448240 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.141+0000 7f850a96d740 10 client.16371 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.141+0000 7f850a96d740 10 client.16371 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.141+0000 7f850a96d740 20 client.16371 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.141+0000 7f850a96d740 1 --2- 172.21.15.77:0/1028025273 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55bf7f4c85d0 0x55bf7f4ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.141+0000 7f850a96d740 4 client.16371 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.141+0000 7f850a96d740 1 -- 172.21.15.77:0/1028025273 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bf7f3b3d20 con 0x55bf7f4c85d0 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.142+0000 7f8505b1a640 1 --2- 172.21.15.77:0/1028025273 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55bf7f4c85d0 0x55bf7f4ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f84e97fa640 10 client.16371 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f8505b1a640 1 --2- 172.21.15.77:0/1028025273 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55bf7f4c85d0 0x55bf7f4ca9f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=34fd5a3076639d2c server_cookie=891f6d8038207f32 in_seq=0 out_seq=0 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f84e97fa640 1 -- 172.21.15.77:0/1028025273 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bf7f3b3d20 con 0x55bf7f4c85d0 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f84e97fa640 10 client.16371 handle_command_reply: tid=0 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f850a96d740 1 client.16371 shutdown 2024-03-16T10:19:54.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f850a96d740 1 -- 172.21.15.77:0/1028025273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f448240 msgr2=0x55bf7f45abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f850a96d740 1 --2- 172.21.15.77:0/1028025273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f448240 0x55bf7f45abf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x55bf7f448cb0 tx=0x7f84f0004410 comp rx=0 tx=0).stop 2024-03-16T10:19:54.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f850a96d740 1 -- 172.21.15.77:0/1028025273 shutdown_connections 2024-03-16T10:19:54.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f850a96d740 1 --2- 172.21.15.77:0/1028025273 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55bf7f4c85d0 0x55bf7f4ca9f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f850a96d740 1 --2- 172.21.15.77:0/1028025273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf7f44d3b0 0x55bf7f45bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f850a96d740 1 --2- 172.21.15.77:0/1028025273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf7f448240 0x55bf7f45abf0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f850a96d740 1 --2- 172.21.15.77:0/1028025273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf7f447850 0x55bf7f45a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f850a96d740 1 -- 172.21.15.77:0/1028025273 >> 172.21.15.77:0/1028025273 conn(0x55bf7f442dc0 msgr2=0x55bf7f446020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f84e97fa640 0 client.16371 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:19:54.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f850a96d740 1 -- 172.21.15.77:0/1028025273 shutdown_connections 2024-03-16T10:19:54.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.143+0000 7f850a96d740 1 -- 172.21.15.77:0/1028025273 wait complete. 2024-03-16T10:19:54.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f850a96d740 20 client.16371 trim_cache size 0 max 16384 2024-03-16T10:19:54.147 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:19:54.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f8507d7e640 1 -- 172.21.15.77:0/1154865433 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84ec0a2880 msgr2=0x7f84ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f8507d7e640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84ec0a2880 0x7f84ec0a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f84f8005c40 tx=0x7f84f8009e50 comp rx=0 tx=0).stop 2024-03-16T10:19:54.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f8507d7e640 1 -- 172.21.15.77:0/1154865433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85001006f0 msgr2=0x7f8500197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f8507d7e640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85001006f0 0x7f8500197b40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f84fc02c0b0 tx=0x7f84fc032430 comp rx=0 tx=0).stop 2024-03-16T10:19:54.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f8507d7e640 1 -- 172.21.15.77:0/1154865433 shutdown_connections 2024-03-16T10:19:54.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f8507d7e640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84ec0a2880 0x7f84ec0a4d40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f8507d7e640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001022a0 0x7f850019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f8507d7e640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85001018e0 0x7f8500198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f8507d7e640 1 --2- 172.21.15.77:0/1154865433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85001006f0 0x7f8500197b40 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f8507d7e640 1 -- 172.21.15.77:0/1154865433 >> 172.21.15.77:0/1154865433 conn(0x7f85000fbc80 msgr2=0x7f8500104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f8507d7e640 1 -- 172.21.15.77:0/1154865433 shutdown_connections 2024-03-16T10:19:54.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.144+0000 7f8507d7e640 1 -- 172.21.15.77:0/1154865433 wait complete. 2024-03-16T10:19:54.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:53 smithi077 ceph-mon[32400]: pgmap v434: 97 pgs: 97 active+clean; 222 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 425 op/s 2024-03-16T10:19:54.162 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:53 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2377591355' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:19:54.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-16T10:19:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:53 smithi192 ceph-mon[36354]: pgmap v434: 97 pgs: 97 active+clean; 222 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 425 op/s 2024-03-16T10:19:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:53 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2377591355' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:19:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:53 smithi193 ceph-mon[36311]: pgmap v434: 97 pgs: 97 active+clean; 222 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 425 op/s 2024-03-16T10:19:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:53 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2377591355' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:19:54.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.274+0000 7fb1dfe85640 1 -- 172.21.15.77:0/349281759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d81018f0 msgr2=0x7fb1d8101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.274+0000 7fb1dfe85640 1 --2- 172.21.15.77:0/349281759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d81018f0 0x7fb1d8101d70 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009900 tx=0x7fb1d402bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:54.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.274+0000 7fb1dfe85640 1 -- 172.21.15.77:0/349281759 shutdown_connections 2024-03-16T10:19:54.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.274+0000 7fb1dfe85640 1 --2- 172.21.15.77:0/349281759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1d81022b0 0x7fb1d8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.274+0000 7fb1dfe85640 1 --2- 172.21.15.77:0/349281759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d81018f0 0x7fb1d8101d70 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.274+0000 7fb1dfe85640 1 --2- 172.21.15.77:0/349281759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1d8100700 0x7fb1d8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.274+0000 7fb1dfe85640 1 -- 172.21.15.77:0/349281759 >> 172.21.15.77:0/349281759 conn(0x7fb1d80fbc90 msgr2=0x7fb1d80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.274+0000 7fb1dfe85640 1 -- 172.21.15.77:0/349281759 shutdown_connections 2024-03-16T10:19:54.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.274+0000 7fb1dfe85640 1 -- 172.21.15.77:0/349281759 wait complete. 2024-03-16T10:19:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1dfe85640 1 Processor -- start 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1dfe85640 1 -- start start 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1dfe85640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1d8100700 0x7fb1d8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1dfe85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d81018f0 0x7fb1d81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1dfe85640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1d81022b0 0x7fb1d819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1dfe85640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d810c360 con 0x7fb1d81018f0 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1dfe85640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d810c1e0 con 0x7fb1d8100700 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1dfe85640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d810c4e0 con 0x7fb1d81022b0 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1dd420640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d81018f0 0x7fb1d81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1ddc21640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1d8100700 0x7fb1d8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1dd420640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d81018f0 0x7fb1d81980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44148/0 (socket says 172.21.15.77:44148) 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1dd420640 1 -- 172.21.15.77:0/2674895008 learned_addr learned my addr 172.21.15.77:0/2674895008 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.275+0000 7fb1de422640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1d81022b0 0x7fb1d819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.276+0000 7fb1dd420640 1 -- 172.21.15.77:0/2674895008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1d81022b0 msgr2=0x7fb1d819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.276+0000 7fb1dd420640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1d81022b0 0x7fb1d819f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.276+0000 7fb1dd420640 1 -- 172.21.15.77:0/2674895008 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1d8100700 msgr2=0x7fb1d8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.276+0000 7fb1dd420640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1d8100700 0x7fb1d8197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.276+0000 7fb1dd420640 1 -- 172.21.15.77:0/2674895008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d819f780 con 0x7fb1d81018f0 2024-03-16T10:19:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.276+0000 7fb1dd420640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d81018f0 0x7fb1d81980a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009230 tx=0x7fb1d4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.276+0000 7fb1ceffd640 1 -- 172.21.15.77:0/2674895008 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4041070 con 0x7fb1d81018f0 2024-03-16T10:19:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.276+0000 7fb1dfe85640 1 -- 172.21.15.77:0/2674895008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d819fa10 con 0x7fb1d81018f0 2024-03-16T10:19:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.276+0000 7fb1ceffd640 1 -- 172.21.15.77:0/2674895008 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d40044d0 con 0x7fb1d81018f0 2024-03-16T10:19:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.276+0000 7fb1ceffd640 1 -- 172.21.15.77:0/2674895008 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4034440 con 0x7fb1d81018f0 2024-03-16T10:19:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.276+0000 7fb1dfe85640 1 -- 172.21.15.77:0/2674895008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1d819ff50 con 0x7fb1d81018f0 2024-03-16T10:19:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.279+0000 7fb1ceffd640 1 -- 172.21.15.77:0/2674895008 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fb1d4044070 con 0x7fb1d81018f0 2024-03-16T10:19:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1e2a73740 1 Processor -- start 2024-03-16T10:19:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1e2a73740 1 -- start start 2024-03-16T10:19:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1e2a73740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 0x562600a46bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1e2a73740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a471c0 0x562600a49cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1ceffd640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb1c40a2800 0x7fb1c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1e2a73740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 0x562600a57190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1e2a73740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5626009ab490 con 0x562600a471c0 2024-03-16T10:19:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1e2a73740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x562600953c90 con 0x562600a4a320 2024-03-16T10:19:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1e2a73740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5626009a7530 con 0x562600a467d0 2024-03-16T10:19:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1de422640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 0x562600a46bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1dd420640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a471c0 0x562600a49cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1de422640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 0x562600a46bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44188/0 (socket says 172.21.15.77:44188) 2024-03-16T10:19:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1de422640 1 -- 172.21.15.77:0/528792431 learned_addr learned my addr 172.21.15.77:0/528792431 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1ddc21640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb1c40a2800 0x7fb1c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1de422640 1 --2- 172.21.15.77:0/528792431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 0x562600a57190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.280+0000 7fb1ceffd640 1 -- 172.21.15.77:0/2674895008 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb1d40f05b0 con 0x7fb1d81018f0 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.281+0000 7fb1de422640 1 -- 172.21.15.77:0/528792431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 msgr2=0x562600a57190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.281+0000 7fb1de422640 1 --2- 172.21.15.77:0/528792431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 0x562600a57190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.281+0000 7fb1de422640 1 -- 172.21.15.77:0/528792431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a471c0 msgr2=0x562600a49cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.281+0000 7fb1de422640 1 --2- 172.21.15.77:0/528792431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a471c0 0x562600a49cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.281+0000 7fb1de422640 1 -- 172.21.15.77:0/528792431 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562600a0e500 con 0x562600a467d0 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.281+0000 7fb1dd420640 1 --2- 172.21.15.77:0/528792431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a471c0 0x562600a49cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.281+0000 7fb1ddc21640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb1c40a2800 0x7fb1c40a4cc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8101750 tx=0x7fb1c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.281+0000 7fb1de422640 1 --2- 172.21.15.77:0/528792431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 0x562600a46bf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009990 tx=0x7fb1d0009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93d814d24f35ab server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.281+0000 7fb1c3fff640 1 -- 172.21.15.77:0/528792431 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0004b20 con 0x562600a467d0 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.281+0000 7fb1c3fff640 1 -- 172.21.15.77:0/528792431 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d0004cc0 con 0x562600a467d0 2024-03-16T10:19:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.281+0000 7fb1c3fff640 1 -- 172.21.15.77:0/528792431 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d00055d0 con 0x562600a467d0 2024-03-16T10:19:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.282+0000 7fb1e2a73740 1 -- 172.21.15.77:0/528792431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 msgr2=0x562600a46bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.282+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/528792431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 0x562600a46bf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009990 tx=0x7fb1d0009920 comp rx=0 tx=0).stop 2024-03-16T10:19:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.282+0000 7fb1e2a73740 1 -- 172.21.15.77:0/528792431 shutdown_connections 2024-03-16T10:19:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.282+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/528792431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 0x562600a57190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.282+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/528792431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a471c0 0x562600a49cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.282+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/528792431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 0x562600a46bf0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.282+0000 7fb1e2a73740 1 -- 172.21.15.77:0/528792431 >> 172.21.15.77:0/528792431 conn(0x562600a41dc0 msgr2=0x562600a421c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.282+0000 7fb1e2a73740 1 -- 172.21.15.77:0/528792431 shutdown_connections 2024-03-16T10:19:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.282+0000 7fb1e2a73740 1 -- 172.21.15.77:0/528792431 wait complete. 2024-03-16T10:19:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.282+0000 7fb1e2a73740 1 -- start start 2024-03-16T10:19:54.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1e2a73740 1 Processor -- start 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1e2a73740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a467d0 0x562600a59590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1e2a73740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a471c0 0x562600a59ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1e2a73740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 0x562600a60b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1e2a73740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562600a57b90 con 0x562600a467d0 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1e2a73740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x562600967230 con 0x562600a4a320 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1e2a73740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562600a57d10 con 0x562600a471c0 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1e2a73740 10 client.0 _ll_register_callbacks cb 0x5626009a3010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1dd420640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a471c0 0x562600a59ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1ddc21640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a467d0 0x562600a59590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1dd420640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a471c0 0x562600a59ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44194/0 (socket says 172.21.15.77:44194) 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1dd420640 1 -- 172.21.15.77:0/809989366 learned_addr learned my addr 172.21.15.77:0/809989366 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1de422640 1 --2- 172.21.15.77:0/809989366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 0x562600a60b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1dd420640 1 -- 172.21.15.77:0/809989366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 msgr2=0x562600a60b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1dd420640 1 --2- 172.21.15.77:0/809989366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 0x562600a60b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1dd420640 1 -- 172.21.15.77:0/809989366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a467d0 msgr2=0x562600a59590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1dd420640 1 --2- 172.21.15.77:0/809989366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a467d0 0x562600a59590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.283+0000 7fb1dd420640 1 -- 172.21.15.77:0/809989366 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562600976100 con 0x562600a471c0 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.284+0000 7fb1dd420640 1 --2- 172.21.15.77:0/809989366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a471c0 0x562600a59ad0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb1d402c7b0 tx=0x7fb1d4087e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=920ce8a452c9ce68 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.284+0000 7fb1c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.284+0000 7fb1c17fa640 1 -- 172.21.15.77:0/809989366 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d404b040 con 0x562600a471c0 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.284+0000 7fb1c17fa640 1 -- 172.21.15.77:0/809989366 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb1d40035f0 con 0x562600a471c0 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.284+0000 7fb1e2a73740 1 -- 172.21.15.77:0/809989366 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1d0004cc0 con 0x562600a471c0 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.284+0000 7fb1c17fa640 1 -- 172.21.15.77:0/809989366 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1d40bc9f0 con 0x562600a471c0 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.284+0000 7fb1e2a73740 10 client.26146 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.284+0000 7fb1e2a73740 1 -- 172.21.15.77:0/809989366 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5626006bf3f0 con 0x562600a471c0 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.284+0000 7fb1c17fa640 1 -- 172.21.15.77:0/809989366 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d0004cc0 con 0x562600a471c0 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.284+0000 7fb1c17fa640 1 -- 172.21.15.77:0/809989366 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fb1d40c52f0 con 0x562600a471c0 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.284+0000 7fb1e2a73740 10 client.26146 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:54.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.284+0000 7fb1e2a73740 10 client.26146 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.285+0000 7fb1e2a73740 20 client.26146 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.285+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/809989366 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x562600a65340 0x562600a677e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.285+0000 7fb1e2a73740 4 client.26146 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.285+0000 7fb1e2a73740 1 -- 172.21.15.77:0/809989366 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626009b89d0 con 0x562600a65340 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.285+0000 7fb1ddc21640 1 --2- 172.21.15.77:0/809989366 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x562600a65340 0x562600a677e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.285+0000 7fb1ddc21640 1 --2- 172.21.15.77:0/809989366 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x562600a65340 0x562600a677e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=55a1ff5da68207f6 server_cookie=b02bd86edc701efc in_seq=0 out_seq=0 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.285+0000 7fb1c17fa640 10 client.26146 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1c17fa640 1 -- 172.21.15.77:0/809989366 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5626009b89d0 con 0x562600a65340 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1c17fa640 10 client.26146 handle_command_reply: tid=0 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1e2a73740 1 client.26146 shutdown 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1e2a73740 1 -- 172.21.15.77:0/809989366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a471c0 msgr2=0x562600a59ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/809989366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a471c0 0x562600a59ad0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb1d402c7b0 tx=0x7fb1d4087e50 comp rx=0 tx=0).stop 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1e2a73740 1 -- 172.21.15.77:0/809989366 shutdown_connections 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/809989366 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x562600a65340 0x562600a677e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/809989366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 0x562600a60b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/809989366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a471c0 0x562600a59ad0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1c17fa640 0 client.26146 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:19:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/809989366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a467d0 0x562600a59590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1e2a73740 1 -- 172.21.15.77:0/809989366 >> 172.21.15.77:0/809989366 conn(0x562600a41dc0 msgr2=0x562600a40fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1e2a73740 1 -- 172.21.15.77:0/809989366 shutdown_connections 2024-03-16T10:19:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1e2a73740 1 -- 172.21.15.77:0/809989366 wait complete. 2024-03-16T10:19:54.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.286+0000 7fb1e2a73740 20 client.26146 trim_cache size 0 max 16384 2024-03-16T10:19:54.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1e2a73740 1 Processor -- start 2024-03-16T10:19:54.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1e2a73740 1 -- start start 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1e2a73740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a467d0 0x562600a53b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1e2a73740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a471c0 0x562600a60ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1e2a73740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a4a320 0x562600a54070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1e2a73740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d81a6970 con 0x562600a467d0 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1e2a73740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d81a8630 con 0x562600a471c0 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1e2a73740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d81a7040 con 0x562600a4a320 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1ddc21640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a467d0 0x562600a53b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1de422640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a4a320 0x562600a54070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1dd420640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a471c0 0x562600a60ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1de422640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a4a320 0x562600a54070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44202/0 (socket says 172.21.15.77:44202) 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1ddc21640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a467d0 0x562600a53b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44180/0 (socket says 172.21.15.77:44180) 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1de422640 1 -- 172.21.15.77:0/1249284523 learned_addr learned my addr 172.21.15.77:0/1249284523 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1de422640 1 -- 172.21.15.77:0/1249284523 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a471c0 msgr2=0x562600a60ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1de422640 1 --2- 172.21.15.77:0/1249284523 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a471c0 0x562600a60ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1de422640 1 -- 172.21.15.77:0/1249284523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a467d0 msgr2=0x562600a53b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1de422640 1 --2- 172.21.15.77:0/1249284523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a467d0 0x562600a53b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.302+0000 7fb1de422640 1 -- 172.21.15.77:0/1249284523 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626007897d0 con 0x562600a4a320 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.303+0000 7fb1de422640 1 --2- 172.21.15.77:0/1249284523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a4a320 0x562600a54070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0004b90 tx=0x7fb1d001b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=427fb03a9019f6d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.303+0000 7fb1c37fe640 1 -- 172.21.15.77:0/1249284523 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0012050 con 0x562600a4a320 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.303+0000 7fb1c37fe640 1 -- 172.21.15.77:0/1249284523 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d0016040 con 0x562600a4a320 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.303+0000 7fb1c37fe640 1 -- 172.21.15.77:0/1249284523 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d00052a0 con 0x562600a4a320 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.303+0000 7fb1e2a73740 1 -- 172.21.15.77:0/1249284523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a4a320 msgr2=0x562600a54070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.303+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/1249284523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a4a320 0x562600a54070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0004b90 tx=0x7fb1d001b040 comp rx=0 tx=0).stop 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.304+0000 7fb1e2a73740 1 -- 172.21.15.77:0/1249284523 shutdown_connections 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.304+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/1249284523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a4a320 0x562600a54070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.304+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/1249284523 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a471c0 0x562600a60ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.304+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/1249284523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a467d0 0x562600a53b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.304+0000 7fb1e2a73740 1 -- 172.21.15.77:0/1249284523 >> 172.21.15.77:0/1249284523 conn(0x562600a41dc0 msgr2=0x562600a59590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.304+0000 7fb1e2a73740 1 -- 172.21.15.77:0/1249284523 shutdown_connections 2024-03-16T10:19:54.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.304+0000 7fb1e2a73740 1 -- 172.21.15.77:0/1249284523 wait complete. 2024-03-16T10:19:54.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.304+0000 7fb1e2a73740 1 -- start start 2024-03-16T10:19:54.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.304+0000 7fb1e2a73740 1 Processor -- start 2024-03-16T10:19:54.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1e2a73740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 0x562600a5d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1e2a73740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a471c0 0x562600a5df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1e2a73740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 0x562600ac9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1e2a73740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d81a7ea0 con 0x562600a471c0 2024-03-16T10:19:54.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1e2a73740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d81a77d0 con 0x562600a4a320 2024-03-16T10:19:54.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1e2a73740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d81a4950 con 0x562600a467d0 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1e2a73740 10 client.0 _ll_register_callbacks cb 0x7fb1d4005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1dd420640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a471c0 0x562600a5df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1ddc21640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 0x562600a5d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1ddc21640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 0x562600a5d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44204/0 (socket says 172.21.15.77:44204) 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1ddc21640 1 -- 172.21.15.77:0/4254275641 learned_addr learned my addr 172.21.15.77:0/4254275641 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1de422640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 0x562600ac9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1dd420640 1 -- 172.21.15.77:0/4254275641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 msgr2=0x562600a5d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1dd420640 1 --2- 172.21.15.77:0/4254275641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 0x562600a5d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1dd420640 1 -- 172.21.15.77:0/4254275641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 msgr2=0x562600ac9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1dd420640 1 --2- 172.21.15.77:0/4254275641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 0x562600ac9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1dd420640 1 -- 172.21.15.77:0/4254275641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562600ac9c50 con 0x562600a471c0 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1ddc21640 1 --2- 172.21.15.77:0/4254275641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 0x562600a5d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1dd420640 1 --2- 172.21.15.77:0/4254275641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a471c0 0x562600a5df00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb1d40b94f0 tx=0x7fb1d4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40409d2323179d9f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.305+0000 7fb1c17fa640 1 -- 172.21.15.77:0/4254275641 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d404b040 con 0x562600a471c0 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1c17fa640 1 -- 172.21.15.77:0/4254275641 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb1d40bc4e0 con 0x562600a471c0 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1e2a73740 1 -- 172.21.15.77:0/4254275641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1d0016040 con 0x562600a471c0 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1c17fa640 1 -- 172.21.15.77:0/4254275641 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1d40c5780 con 0x562600a471c0 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1e2a73740 10 client.16389 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1e2a73740 1 -- 172.21.15.77:0/4254275641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562600ac9c50 con 0x562600a471c0 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1c17fa640 1 -- 172.21.15.77:0/4254275641 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d0016040 con 0x562600a471c0 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1c17fa640 1 -- 172.21.15.77:0/4254275641 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fb1d40b8de0 con 0x562600a471c0 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1e2a73740 10 client.16389 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1e2a73740 10 client.16389 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1e2a73740 20 client.16389 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/4254275641 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x562600a65340 0x562600acd500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1e2a73740 4 client.16389 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:19:54.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1e2a73740 1 -- 172.21.15.77:0/4254275641 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5626009b2d20 con 0x562600a65340 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.306+0000 7fb1ddc21640 1 --2- 172.21.15.77:0/4254275641 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x562600a65340 0x562600acd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1ddc21640 1 --2- 172.21.15.77:0/4254275641 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x562600a65340 0x562600acd500 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=625c7661bf4fc9d2 server_cookie=a5dcd7b8f43cbdcd in_seq=0 out_seq=0 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1c17fa640 10 client.16389 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1c17fa640 1 -- 172.21.15.77:0/4254275641 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5626009b2d20 con 0x562600a65340 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1c17fa640 10 client.16389 handle_command_reply: tid=0 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1e2a73740 1 client.16389 shutdown 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1e2a73740 1 -- 172.21.15.77:0/4254275641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a471c0 msgr2=0x562600a5df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/4254275641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a471c0 0x562600a5df00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb1d40b94f0 tx=0x7fb1d4009260 comp rx=0 tx=0).stop 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1e2a73740 1 -- 172.21.15.77:0/4254275641 shutdown_connections 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/4254275641 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x562600a65340 0x562600acd500 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/4254275641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562600a4a320 0x562600ac9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/4254275641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562600a471c0 0x562600a5df00 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1c17fa640 0 client.16389 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1e2a73740 1 --2- 172.21.15.77:0/4254275641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562600a467d0 0x562600a5d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1e2a73740 1 -- 172.21.15.77:0/4254275641 >> 172.21.15.77:0/4254275641 conn(0x562600a41dc0 msgr2=0x562600a59550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1e2a73740 1 -- 172.21.15.77:0/4254275641 shutdown_connections 2024-03-16T10:19:54.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1e2a73740 1 -- 172.21.15.77:0/4254275641 wait complete. 2024-03-16T10:19:54.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.307+0000 7fb1e2a73740 20 client.16389 trim_cache size 0 max 16384 2024-03-16T10:19:54.314 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:19:54.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.308+0000 7fb1dfe85640 1 -- 172.21.15.77:0/2674895008 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb1c40a2800 msgr2=0x7fb1c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.308+0000 7fb1dfe85640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb1c40a2800 0x7fb1c40a4cc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8101750 tx=0x7fb1c8005f90 comp rx=0 tx=0).stop 2024-03-16T10:19:54.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.308+0000 7fb1dfe85640 1 -- 172.21.15.77:0/2674895008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d81018f0 msgr2=0x7fb1d81980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.308+0000 7fb1dfe85640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d81018f0 0x7fb1d81980a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009230 tx=0x7fb1d4005d50 comp rx=0 tx=0).stop 2024-03-16T10:19:54.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.308+0000 7fb1dfe85640 1 -- 172.21.15.77:0/2674895008 shutdown_connections 2024-03-16T10:19:54.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.308+0000 7fb1dfe85640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb1c40a2800 0x7fb1c40a4cc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.308+0000 7fb1dfe85640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1d81022b0 0x7fb1d819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.308+0000 7fb1dfe85640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1d81018f0 0x7fb1d81980a0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.308+0000 7fb1dfe85640 1 --2- 172.21.15.77:0/2674895008 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1d8100700 0x7fb1d8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.308+0000 7fb1dfe85640 1 -- 172.21.15.77:0/2674895008 >> 172.21.15.77:0/2674895008 conn(0x7fb1d80fbc90 msgr2=0x7fb1d80fc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.308+0000 7fb1dfe85640 1 -- 172.21.15.77:0/2674895008 shutdown_connections 2024-03-16T10:19:54.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.308+0000 7fb1dfe85640 1 -- 172.21.15.77:0/2674895008 wait complete. 2024-03-16T10:19:54.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T10:19:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.431+0000 7f7cfb801640 1 -- 172.21.15.77:0/1259206152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf40ffaa0 msgr2=0x7f7cf40ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.431+0000 7f7cfb801640 1 --2- 172.21.15.77:0/1259206152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf40ffaa0 0x7f7cf40ffec0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009860 tx=0x7f7cf002bb70 comp rx=0 tx=0).stop 2024-03-16T10:19:54.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.431+0000 7f7cfb801640 1 -- 172.21.15.77:0/1259206152 shutdown_connections 2024-03-16T10:19:54.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.431+0000 7f7cfb801640 1 --2- 172.21.15.77:0/1259206152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cf41034d0 0x7f7cf4109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.431+0000 7f7cfb801640 1 --2- 172.21.15.77:0/1259206152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4102b10 0x7f7cf4102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.431+0000 7f7cfb801640 1 --2- 172.21.15.77:0/1259206152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf40ffaa0 0x7f7cf40ffec0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.431+0000 7f7cfb801640 1 -- 172.21.15.77:0/1259206152 >> 172.21.15.77:0/1259206152 conn(0x7f7cf40fb3d0 msgr2=0x7f7cf40fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.432+0000 7f7cfb801640 1 -- 172.21.15.77:0/1259206152 shutdown_connections 2024-03-16T10:19:54.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.432+0000 7f7cfb801640 1 -- 172.21.15.77:0/1259206152 wait complete. 2024-03-16T10:19:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.432+0000 7f7cfb801640 1 Processor -- start 2024-03-16T10:19:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.432+0000 7f7cfb801640 1 -- start start 2024-03-16T10:19:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.432+0000 7f7cfb801640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cf40ffaa0 0x7f7cf4102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.432+0000 7f7cfb801640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4102b10 0x7f7cf4100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.432+0000 7f7cfb801640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf41034d0 0x7f7cf4101000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.432+0000 7f7cfb801640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf410d440 con 0x7f7cf4102b10 2024-03-16T10:19:54.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.432+0000 7f7cfb801640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf410d2c0 con 0x7f7cf40ffaa0 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.432+0000 7f7cfb801640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf410d5c0 con 0x7f7cf41034d0 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cf8d9c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4102b10 0x7f7cf4100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cf959d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cf40ffaa0 0x7f7cf4102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cf8d9c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4102b10 0x7f7cf4100ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44210/0 (socket says 172.21.15.77:44210) 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cf8d9c640 1 -- 172.21.15.77:0/2940031384 learned_addr learned my addr 172.21.15.77:0/2940031384 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cf9d9e640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf41034d0 0x7f7cf4101000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cf9d9e640 1 -- 172.21.15.77:0/2940031384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cf40ffaa0 msgr2=0x7f7cf4102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cf9d9e640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cf40ffaa0 0x7f7cf4102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cf9d9e640 1 -- 172.21.15.77:0/2940031384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4102b10 msgr2=0x7f7cf4100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cf9d9e640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4102b10 0x7f7cf4100ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cf9d9e640 1 -- 172.21.15.77:0/2940031384 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf4101890 con 0x7f7cf41034d0 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cf8d9c640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4102b10 0x7f7cf4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cf9d9e640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf41034d0 0x7f7cf4101000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00e940 tx=0x7f7cec00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cea7fc640 1 -- 172.21.15.77:0/2940031384 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec004d20 con 0x7f7cf41034d0 2024-03-16T10:19:54.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cea7fc640 1 -- 172.21.15.77:0/2940031384 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cec005020 con 0x7f7cf41034d0 2024-03-16T10:19:54.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cea7fc640 1 -- 172.21.15.77:0/2940031384 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec0116b0 con 0x7f7cf41034d0 2024-03-16T10:19:54.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.433+0000 7f7cfb801640 1 -- 172.21.15.77:0/2940031384 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf41a1c60 con 0x7f7cf41034d0 2024-03-16T10:19:54.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.434+0000 7f7cfb801640 1 -- 172.21.15.77:0/2940031384 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf41a2160 con 0x7f7cf41034d0 2024-03-16T10:19:54.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.436+0000 7f7cea7fc640 1 -- 172.21.15.77:0/2940031384 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f7cec0054d0 con 0x7f7cf41034d0 2024-03-16T10:19:54.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.436+0000 7f7cfe3ea740 1 Processor -- start 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cea7fc640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7ce00a2800 0x7f7ce00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cfe3ea740 1 -- start start 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cfe3ea740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d98ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cfe3ea740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d98fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cfe3ea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d9903b0 0x55d31d992840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cfe3ea740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d31d8ef490 con 0x55d31d98b240 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cfe3ea740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d31d897c90 con 0x55d31d98a850 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cfe3ea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d31d8eb530 con 0x55d31d9903b0 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cea7fc640 1 -- 172.21.15.77:0/2940031384 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7cec0ceb60 con 0x7f7cf41034d0 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cf8d9c640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7ce00a2800 0x7f7ce00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cf959d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d98ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cf959d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d98ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41666/0 (socket says 172.21.15.77:41666) 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cf959d640 1 -- 172.21.15.77:0/2889709712 learned_addr learned my addr 172.21.15.77:0/2889709712 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cf8d9c640 1 --2- 172.21.15.77:0/2889709712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d98fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cf9d9e640 1 --2- 172.21.15.77:0/2889709712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d9903b0 0x55d31d992840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cf959d640 1 -- 172.21.15.77:0/2889709712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d9903b0 msgr2=0x55d31d992840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cf959d640 1 --2- 172.21.15.77:0/2889709712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d9903b0 0x55d31d992840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cf959d640 1 -- 172.21.15.77:0/2889709712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 msgr2=0x55d31d98fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cf959d640 1 --2- 172.21.15.77:0/2889709712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d98fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cf959d640 1 -- 172.21.15.77:0/2889709712 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d31d952500 con 0x55d31d98a850 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.437+0000 7f7cf8d9c640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7ce00a2800 0x7f7ce00a4cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009fd0 tx=0x7f7ce4013d70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.438+0000 7f7cf959d640 1 --2- 172.21.15.77:0/2889709712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d98ac70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7cf002c180 tx=0x7f7cf0005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e03c619cb76b04e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.438+0000 7f7cdf7fe640 1 -- 172.21.15.77:0/2889709712 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0043020 con 0x55d31d98a850 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.438+0000 7f7cdf7fe640 1 -- 172.21.15.77:0/2889709712 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cf0034db0 con 0x55d31d98a850 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.438+0000 7f7cdf7fe640 1 -- 172.21.15.77:0/2889709712 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0039040 con 0x55d31d98a850 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/2889709712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 msgr2=0x55d31d98ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/2889709712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d98ac70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7cf002c180 tx=0x7f7cf0005b20 comp rx=0 tx=0).stop 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/2889709712 shutdown_connections 2024-03-16T10:19:54.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/2889709712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d9903b0 0x55d31d992840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/2889709712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d98fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/2889709712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d98ac70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/2889709712 >> 172.21.15.77:0/2889709712 conn(0x55d31d985dc0 msgr2=0x55d31d9861c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/2889709712 shutdown_connections 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/2889709712 wait complete. 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 -- start start 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 Processor -- start 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d994cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d98b240 0x55d31d9951f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d9903b0 0x55d31d99c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d31d993300 con 0x55d31d9903b0 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d31d8ab230 con 0x55d31d98a850 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d31d993480 con 0x55d31d98b240 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.439+0000 7f7cfe3ea740 10 client.0 _ll_register_callbacks cb 0x55d31d8e7010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:54.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf9d9e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d9903b0 0x55d31d99c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf959d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d994cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf8d9c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d98b240 0x55d31d9951f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf9d9e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d9903b0 0x55d31d99c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44230/0 (socket says 172.21.15.77:44230) 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf959d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d994cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41672/0 (socket says 172.21.15.77:41672) 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf9d9e640 1 -- 172.21.15.77:0/257312933 learned_addr learned my addr 172.21.15.77:0/257312933 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf8d9c640 1 -- 172.21.15.77:0/257312933 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 msgr2=0x55d31d994cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf8d9c640 1 --2- 172.21.15.77:0/257312933 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d994cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf8d9c640 1 -- 172.21.15.77:0/257312933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d9903b0 msgr2=0x55d31d99c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf8d9c640 1 --2- 172.21.15.77:0/257312933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d9903b0 0x55d31d99c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf8d9c640 1 -- 172.21.15.77:0/257312933 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d31d8ba100 con 0x55d31d98b240 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf9d9e640 1 --2- 172.21.15.77:0/257312933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d9903b0 0x55d31d99c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cf8d9c640 1 --2- 172.21.15.77:0/257312933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d98b240 0x55d31d9951f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55d31d98bcb0 tx=0x7f7ce4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=992da445a243b6e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cdcff9640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cdcff9640 1 -- 172.21.15.77:0/257312933 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4005220 con 0x55d31d98b240 2024-03-16T10:19:54.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cdcff9640 1 -- 172.21.15.77:0/257312933 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7ce40053c0 con 0x55d31d98b240 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/257312933 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cf0034db0 con 0x55d31d98b240 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cdcff9640 1 -- 172.21.15.77:0/257312933 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7ce40055c0 con 0x55d31d98b240 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cfe3ea740 10 client.26164 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.440+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/257312933 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d31d6033f0 con 0x55d31d98b240 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.441+0000 7f7cdcff9640 1 -- 172.21.15.77:0/257312933 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cf0034db0 con 0x55d31d98b240 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.441+0000 7f7cdcff9640 1 -- 172.21.15.77:0/257312933 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f7ce401f070 con 0x55d31d98b240 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.441+0000 7f7cfe3ea740 10 client.26164 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.441+0000 7f7cfe3ea740 10 client.26164 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.441+0000 7f7cfe3ea740 20 client.26164 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.441+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/257312933 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55d31d9a0c00 0x55d31d9a3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.441+0000 7f7cfe3ea740 4 client.26164 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.441+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/257312933 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d31d8fc9d0 con 0x55d31d9a0c00 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.441+0000 7f7cf959d640 1 --2- 172.21.15.77:0/257312933 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55d31d9a0c00 0x55d31d9a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.441+0000 7f7cdcff9640 10 client.26164 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:19:54.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.442+0000 7f7cf959d640 1 --2- 172.21.15.77:0/257312933 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55d31d9a0c00 0x55d31d9a3060 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f57b5fd697b84571 server_cookie=11a4ff8e0877dccc in_seq=0 out_seq=0 2024-03-16T10:19:54.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cdcff9640 1 -- 172.21.15.77:0/257312933 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d31d8fc9d0 con 0x55d31d9a0c00 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cdcff9640 10 client.26164 handle_command_reply: tid=0 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cfe3ea740 1 client.26164 shutdown 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/257312933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d98b240 msgr2=0x55d31d9951f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/257312933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d98b240 0x55d31d9951f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55d31d98bcb0 tx=0x7f7ce4004730 comp rx=0 tx=0).stop 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/257312933 shutdown_connections 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/257312933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d9903b0 0x55d31d99c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/257312933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d98b240 0x55d31d9951f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/257312933 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55d31d9a0c00 0x55d31d9a3060 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/257312933 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d994cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cdcff9640 0 client.26164 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/257312933 >> 172.21.15.77:0/257312933 conn(0x55d31d985dc0 msgr2=0x55d31d984fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/257312933 shutdown_connections 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.445+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/257312933 wait complete. 2024-03-16T10:19:54.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.446+0000 7f7cfe3ea740 20 client.26164 trim_cache size 0 max 16384 2024-03-16T10:19:54.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.468+0000 7f7cfe3ea740 1 Processor -- start 2024-03-16T10:19:54.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cfe3ea740 1 -- start start 2024-03-16T10:19:54.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cfe3ea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d98a850 0x55d31d984750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cfe3ea740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d99c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cfe3ea740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d9903b0 0x55d31d984c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cfe3ea740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf41aad30 con 0x55d31d98b240 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cfe3ea740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf41ac9f0 con 0x55d31d9903b0 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cfe3ea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf41ab400 con 0x55d31d98a850 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cf8d9c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d99c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cf9d9e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d9903b0 0x55d31d984c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cf959d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d98a850 0x55d31d984750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cf8d9c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d99c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44244/0 (socket says 172.21.15.77:44244) 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cf8d9c640 1 -- 172.21.15.77:0/2723501410 learned_addr learned my addr 172.21.15.77:0/2723501410 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cf959d640 1 -- 172.21.15.77:0/2723501410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d9903b0 msgr2=0x55d31d984c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cf959d640 1 --2- 172.21.15.77:0/2723501410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d9903b0 0x55d31d984c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cf959d640 1 -- 172.21.15.77:0/2723501410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 msgr2=0x55d31d99c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cf959d640 1 --2- 172.21.15.77:0/2723501410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d99c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cf959d640 1 -- 172.21.15.77:0/2723501410 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d31d6cd7d0 con 0x55d31d98a850 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cf8d9c640 1 --2- 172.21.15.77:0/2723501410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d99c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.469+0000 7f7cf959d640 1 --2- 172.21.15.77:0/2723501410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d98a850 0x55d31d984750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00349e0 tx=0x7f7cf002bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cec5e31d0893cf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.470+0000 7f7cdeffd640 1 -- 172.21.15.77:0/2723501410 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0043040 con 0x55d31d98a850 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.470+0000 7f7cdeffd640 1 -- 172.21.15.77:0/2723501410 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cf0056070 con 0x55d31d98a850 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.470+0000 7f7cdeffd640 1 -- 172.21.15.77:0/2723501410 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0039040 con 0x55d31d98a850 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.470+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/2723501410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d98a850 msgr2=0x55d31d984750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.470+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/2723501410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d98a850 0x55d31d984750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00349e0 tx=0x7f7cf002bb70 comp rx=0 tx=0).stop 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.470+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/2723501410 shutdown_connections 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.470+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/2723501410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d9903b0 0x55d31d984c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.470+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/2723501410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d99c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.470+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/2723501410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d98a850 0x55d31d984750 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.470+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/2723501410 >> 172.21.15.77:0/2723501410 conn(0x55d31d985dc0 msgr2=0x55d31d994cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.470+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/2723501410 shutdown_connections 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.470+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/2723501410 wait complete. 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.471+0000 7f7cfe3ea740 1 -- start start 2024-03-16T10:19:54.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.471+0000 7f7cfe3ea740 1 Processor -- start 2024-03-16T10:19:54.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.471+0000 7f7cfe3ea740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d999b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.471+0000 7f7cfe3ea740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d99a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.471+0000 7f7cfe3ea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d9903b0 0x55d31d99aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.471+0000 7f7cfe3ea740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf41ac260 con 0x55d31d98b240 2024-03-16T10:19:54.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.471+0000 7f7cfe3ea740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf41abb90 con 0x55d31d98a850 2024-03-16T10:19:54.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.471+0000 7f7cfe3ea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf41a8d10 con 0x55d31d9903b0 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cfe3ea740 10 client.0 _ll_register_callbacks cb 0x7f7ce4012050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf959d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d999b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf8d9c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d99a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf959d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d999b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41690/0 (socket says 172.21.15.77:41690) 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf8d9c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d99a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44260/0 (socket says 172.21.15.77:44260) 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf959d640 1 -- 172.21.15.77:0/718147370 learned_addr learned my addr 172.21.15.77:0/718147370 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf9d9e640 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d9903b0 0x55d31d99aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf959d640 1 -- 172.21.15.77:0/718147370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d9903b0 msgr2=0x55d31d99aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf959d640 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d9903b0 0x55d31d99aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf959d640 1 -- 172.21.15.77:0/718147370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 msgr2=0x55d31d99a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf959d640 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d99a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf959d640 1 -- 172.21.15.77:0/718147370 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d31d99b5e0 con 0x55d31d98a850 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf9d9e640 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d9903b0 0x55d31d99aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cf959d640 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d999b10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7cf002c180 tx=0x7f7cf0009450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab3874fce23204f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cdcff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:19:54.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.472+0000 7f7cdcff9640 1 -- 172.21.15.77:0/718147370 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0039050 con 0x55d31d98a850 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cdcff9640 1 -- 172.21.15.77:0/718147370 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7cf00516a0 con 0x55d31d98a850 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/718147370 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cf0056070 con 0x55d31d98a850 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cdcff9640 1 -- 172.21.15.77:0/718147370 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7cf0043040 con 0x55d31d98a850 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cfe3ea740 10 client.26180 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/718147370 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d31da07070 con 0x55d31d98a850 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cdcff9640 1 -- 172.21.15.77:0/718147370 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cf0056070 con 0x55d31d98a850 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cdcff9640 1 -- 172.21.15.77:0/718147370 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f7cf0004200 con 0x55d31d98a850 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cfe3ea740 10 client.26180 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cfe3ea740 10 client.26180 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cfe3ea740 20 client.26180 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55d31da07710 0x55d31da09b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cfe3ea740 4 client.26180 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.473+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/718147370 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d31d8f6d20 con 0x55d31da07710 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.474+0000 7f7cf8d9c640 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55d31da07710 0x55d31da09b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.474+0000 7f7cdcff9640 10 client.26180 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:19:54.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.474+0000 7f7cf8d9c640 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55d31da07710 0x55d31da09b30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3734b2fb6cd32b20 server_cookie=3cc2ef2cd0bdeba2 in_seq=0 out_seq=0 2024-03-16T10:19:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.478+0000 7f7cdcff9640 1 -- 172.21.15.77:0/718147370 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d31d8f6d20 con 0x55d31da07710 2024-03-16T10:19:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.478+0000 7f7cdcff9640 10 client.26180 handle_command_reply: tid=0 2024-03-16T10:19:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.478+0000 7f7cfe3ea740 1 client.26180 shutdown 2024-03-16T10:19:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.478+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/718147370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 msgr2=0x55d31d999b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.478+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d999b10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7cf002c180 tx=0x7f7cf0009450 comp rx=0 tx=0).stop 2024-03-16T10:19:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.478+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/718147370 shutdown_connections 2024-03-16T10:19:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.478+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d31d9903b0 0x55d31d99aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.478+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d31d98b240 0x55d31d99a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.478+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55d31da07710 0x55d31da09b30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.478+0000 7f7cfe3ea740 1 --2- 172.21.15.77:0/718147370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d31d98a850 0x55d31d999b10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.478+0000 7f7cdcff9640 0 client.26180 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:19:54.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.478+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/718147370 >> 172.21.15.77:0/718147370 conn(0x55d31d985dc0 msgr2=0x55d31d989020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/718147370 shutdown_connections 2024-03-16T10:19:54.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfe3ea740 1 -- 172.21.15.77:0/718147370 wait complete. 2024-03-16T10:19:54.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfe3ea740 20 client.26180 trim_cache size 0 max 16384 2024-03-16T10:19:54.481 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:19:54.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfb801640 1 -- 172.21.15.77:0/2940031384 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7ce00a2800 msgr2=0x7f7ce00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfb801640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7ce00a2800 0x7f7ce00a4cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009fd0 tx=0x7f7ce4013d70 comp rx=0 tx=0).stop 2024-03-16T10:19:54.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfb801640 1 -- 172.21.15.77:0/2940031384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf41034d0 msgr2=0x7f7cf4101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfb801640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf41034d0 0x7f7cf4101000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00e940 tx=0x7f7cec00ee00 comp rx=0 tx=0).stop 2024-03-16T10:19:54.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfb801640 1 -- 172.21.15.77:0/2940031384 shutdown_connections 2024-03-16T10:19:54.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfb801640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7ce00a2800 0x7f7ce00a4cc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfb801640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf41034d0 0x7f7cf4101000 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfb801640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cf4102b10 0x7f7cf4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfb801640 1 --2- 172.21.15.77:0/2940031384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cf40ffaa0 0x7f7cf4102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.479+0000 7f7cfb801640 1 -- 172.21.15.77:0/2940031384 >> 172.21.15.77:0/2940031384 conn(0x7f7cf40fb3d0 msgr2=0x7f7cf4105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.480+0000 7f7cfb801640 1 -- 172.21.15.77:0/2940031384 shutdown_connections 2024-03-16T10:19:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.480+0000 7f7cfb801640 1 -- 172.21.15.77:0/2940031384 wait complete. 2024-03-16T10:19:54.498 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T10:19:54.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.619+0000 7f234d036640 1 -- 172.21.15.77:0/3718531917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2348101d90 msgr2=0x7f23481021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.619+0000 7f234d036640 1 --2- 172.21.15.77:0/3718531917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2348101d90 0x7f23481021b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2338009900 tx=0x7f233802bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:54.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.619+0000 7f234d036640 1 -- 172.21.15.77:0/3718531917 shutdown_connections 2024-03-16T10:19:54.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.619+0000 7f234d036640 1 --2- 172.21.15.77:0/3718531917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2348100240 0x7f23481006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.619+0000 7f234d036640 1 --2- 172.21.15.77:0/3718531917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23480ff880 0x7f23480ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.619+0000 7f234d036640 1 --2- 172.21.15.77:0/3718531917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2348101d90 0x7f23481021b0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.619+0000 7f234d036640 1 -- 172.21.15.77:0/3718531917 >> 172.21.15.77:0/3718531917 conn(0x7f23480fb400 msgr2=0x7f23480fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.619+0000 7f234d036640 1 -- 172.21.15.77:0/3718531917 shutdown_connections 2024-03-16T10:19:54.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.619+0000 7f234d036640 1 -- 172.21.15.77:0/3718531917 wait complete. 2024-03-16T10:19:54.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f234d036640 1 Processor -- start 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f234d036640 1 -- start start 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f234d036640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23480ff880 0x7f2348199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f234d036640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2348100240 0x7f234819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f234d036640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2348101d90 0x7f23481a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f234d036640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f234810f690 con 0x7f23480ff880 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f234d036640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f234810f510 con 0x7f2348100240 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f234d036640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f234810f810 con 0x7f2348101d90 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f2346d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23480ff880 0x7f2348199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f234659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2348100240 0x7f234819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f2346d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23480ff880 0x7f2348199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44292/0 (socket says 172.21.15.77:44292) 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f2346d9d640 1 -- 172.21.15.77:0/2464483170 learned_addr learned my addr 172.21.15.77:0/2464483170 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f234659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2348100240 0x7f234819a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41714/0 (socket says 172.21.15.77:41714) 2024-03-16T10:19:54.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.620+0000 7f234759e640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2348101d90 0x7f23481a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f234759e640 1 -- 172.21.15.77:0/2464483170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2348100240 msgr2=0x7f234819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f234759e640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2348100240 0x7f234819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f234759e640 1 -- 172.21.15.77:0/2464483170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23480ff880 msgr2=0x7f2348199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f234759e640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23480ff880 0x7f2348199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f234759e640 1 -- 172.21.15.77:0/2464483170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23481a1960 con 0x7f2348101d90 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f2346d9d640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23480ff880 0x7f2348199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f234659c640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2348100240 0x7f234819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f234759e640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2348101d90 0x7f23481a12f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2340009fd0 tx=0x7f234000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f232ffff640 1 -- 172.21.15.77:0/2464483170 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234000ce20 con 0x7f2348101d90 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f234d036640 1 -- 172.21.15.77:0/2464483170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23481a1c50 con 0x7f2348101d90 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f232ffff640 1 -- 172.21.15.77:0/2464483170 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2340005020 con 0x7f2348101d90 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f232ffff640 1 -- 172.21.15.77:0/2464483170 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2340010720 con 0x7f2348101d90 2024-03-16T10:19:54.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.621+0000 7f234d036640 1 -- 172.21.15.77:0/2464483170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23481a21a0 con 0x7f2348101d90 2024-03-16T10:19:54.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.623+0000 7f232ffff640 1 -- 172.21.15.77:0/2464483170 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f2340004b20 con 0x7f2348101d90 2024-03-16T10:19:54.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f232ffff640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f23300a2800 0x7f23300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f234fc23740 1 Processor -- start 2024-03-16T10:19:54.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f2346d9d640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f23300a2800 0x7f23300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f234fc23740 1 -- start start 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f232ffff640 1 -- 172.21.15.77:0/2464483170 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f23400cf0c0 con 0x7f2348101d90 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f234fc23740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c85480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f234fc23740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c8a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f234fc23740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999c8d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f234fc23740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e999be1490 con 0x55e999c85a50 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f234fc23740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55e999b89c90 con 0x55e999c8abc0 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f234fc23740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e999bdd530 con 0x55e999c85060 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f234659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c8a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f2346d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c85480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f234659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c8a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44298/0 (socket says 172.21.15.77:44298) 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f234659c640 1 -- 172.21.15.77:0/988232025 learned_addr learned my addr 172.21.15.77:0/988232025 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.624+0000 7f234759e640 1 --2- 172.21.15.77:0/988232025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999c8d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.625+0000 7f2346d9d640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f23300a2800 0x7f23300a4cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2338000c00 tx=0x7f233802c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.625+0000 7f234659c640 1 -- 172.21.15.77:0/988232025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 msgr2=0x55e999c85480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.625+0000 7f234659c640 1 --2- 172.21.15.77:0/988232025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c85480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.625+0000 7f234659c640 1 -- 172.21.15.77:0/988232025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 msgr2=0x55e999c8d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.625+0000 7f234659c640 1 --2- 172.21.15.77:0/988232025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999c8d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.625+0000 7f234659c640 1 -- 172.21.15.77:0/988232025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e999c44500 con 0x55e999c85a50 2024-03-16T10:19:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.625+0000 7f2346d9d640 1 --2- 172.21.15.77:0/988232025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c85480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.625+0000 7f234659c640 1 --2- 172.21.15.77:0/988232025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c8a550 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f233c0114f0 tx=0x7f233c0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1437f0b8f85a96de server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.625+0000 7f232d7fa640 1 -- 172.21.15.77:0/988232025 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c01e070 con 0x55e999c85a50 2024-03-16T10:19:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.625+0000 7f232d7fa640 1 -- 172.21.15.77:0/988232025 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f233c0035f0 con 0x55e999c85a50 2024-03-16T10:19:54.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.625+0000 7f232d7fa640 1 -- 172.21.15.77:0/988232025 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c00f880 con 0x55e999c85a50 2024-03-16T10:19:54.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.626+0000 7f234fc23740 1 -- 172.21.15.77:0/988232025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 msgr2=0x55e999c8a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.626+0000 7f234fc23740 1 --2- 172.21.15.77:0/988232025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c8a550 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f233c0114f0 tx=0x7f233c0119b0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.626+0000 7f234fc23740 1 -- 172.21.15.77:0/988232025 shutdown_connections 2024-03-16T10:19:54.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.626+0000 7f234fc23740 1 --2- 172.21.15.77:0/988232025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999c8d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.626+0000 7f234fc23740 1 --2- 172.21.15.77:0/988232025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c8a550 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.626+0000 7f234fc23740 1 --2- 172.21.15.77:0/988232025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c85480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.626+0000 7f234fc23740 1 -- 172.21.15.77:0/988232025 >> 172.21.15.77:0/988232025 conn(0x55e999c77dc0 msgr2=0x55e999c781c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.627+0000 7f234fc23740 1 -- 172.21.15.77:0/988232025 shutdown_connections 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.627+0000 7f234fc23740 1 -- 172.21.15.77:0/988232025 wait complete. 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.627+0000 7f234fc23740 1 -- start start 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.627+0000 7f234fc23740 1 Processor -- start 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234fc23740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c85060 0x55e999c8f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234fc23740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c8fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234fc23740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c8abc0 0x55e999c96b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234fc23740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e999c8db70 con 0x55e999c85a50 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234fc23740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55e999b9d230 con 0x55e999c85060 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234fc23740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e999c8dcf0 con 0x55e999c8abc0 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234fc23740 10 client.0 _ll_register_callbacks cb 0x55e999bd9010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c8fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c8fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44302/0 (socket says 172.21.15.77:44302) 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234659c640 1 -- 172.21.15.77:0/161277367 learned_addr learned my addr 172.21.15.77:0/161277367 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234759e640 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c8abc0 0x55e999c96b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f2346d9d640 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c85060 0x55e999c8f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234659c640 1 -- 172.21.15.77:0/161277367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c8abc0 msgr2=0x55e999c96b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234659c640 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c8abc0 0x55e999c96b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234659c640 1 -- 172.21.15.77:0/161277367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c85060 msgr2=0x55e999c8f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234659c640 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c85060 0x55e999c8f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f234659c640 1 -- 172.21.15.77:0/161277367 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e999bac100 con 0x55e999c85a50 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.628+0000 7f2346d9d640 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c85060 0x55e999c8f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.629+0000 7f234659c640 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c8fae0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f233c0114c0 tx=0x7f233c0039d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a17859bdc2b89c18 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.629+0000 7f232aff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.629+0000 7f232aff5640 1 -- 172.21.15.77:0/161277367 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c027020 con 0x55e999c85a50 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.629+0000 7f234fc23740 1 -- 172.21.15.77:0/161277367 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f233c0035f0 con 0x55e999c85a50 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.629+0000 7f232aff5640 1 -- 172.21.15.77:0/161277367 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f233c005040 con 0x55e999c85a50 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.629+0000 7f232aff5640 1 -- 172.21.15.77:0/161277367 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f233c032050 con 0x55e999c85a50 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.629+0000 7f234fc23740 10 client.16419 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.629+0000 7f234fc23740 1 -- 172.21.15.77:0/161277367 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e9998f53f0 con 0x55e999c85a50 2024-03-16T10:19:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.630+0000 7f232aff5640 1 -- 172.21.15.77:0/161277367 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f233c0035f0 con 0x55e999c85a50 2024-03-16T10:19:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.630+0000 7f232aff5640 1 -- 172.21.15.77:0/161277367 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f233c00f620 con 0x55e999c85a50 2024-03-16T10:19:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.630+0000 7f234fc23740 10 client.16419 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.630+0000 7f234fc23740 10 client.16419 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:19:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.630+0000 7f234fc23740 20 client.16419 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.630+0000 7f234fc23740 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55e999c9b060 0x55e999c9d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.630+0000 7f234fc23740 4 client.16419 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.630+0000 7f234fc23740 1 -- 172.21.15.77:0/161277367 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e999bee9d0 con 0x55e999c9b060 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.630+0000 7f2346d9d640 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55e999c9b060 0x55e999c9d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.631+0000 7f2346d9d640 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55e999c9b060 0x55e999c9d4d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=86aa8b7ab2ae5696 server_cookie=230d0f15507ec252 in_seq=0 out_seq=0 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.631+0000 7f232aff5640 10 client.16419 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.632+0000 7f232aff5640 1 -- 172.21.15.77:0/161277367 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e999bee9d0 con 0x55e999c9b060 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.632+0000 7f232aff5640 10 client.16419 handle_command_reply: tid=0 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f234fc23740 1 client.16419 shutdown 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f234fc23740 1 -- 172.21.15.77:0/161277367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 msgr2=0x55e999c8fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f234fc23740 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c8fae0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f233c0114c0 tx=0x7f233c0039d0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f234fc23740 1 -- 172.21.15.77:0/161277367 shutdown_connections 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f234fc23740 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55e999c9b060 0x55e999c9d4d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f234fc23740 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c8abc0 0x55e999c96b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f234fc23740 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c8fae0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f234fc23740 1 --2- 172.21.15.77:0/161277367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c85060 0x55e999c8f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f234fc23740 1 -- 172.21.15.77:0/161277367 >> 172.21.15.77:0/161277367 conn(0x55e999c77dc0 msgr2=0x55e999c76fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f232aff5640 0 client.16419 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:19:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f234fc23740 1 -- 172.21.15.77:0/161277367 shutdown_connections 2024-03-16T10:19:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f234fc23740 1 -- 172.21.15.77:0/161277367 wait complete. 2024-03-16T10:19:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.633+0000 7f234fc23740 20 client.16419 trim_cache size 0 max 16384 2024-03-16T10:19:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.646+0000 7f234fc23740 1 Processor -- start 2024-03-16T10:19:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.646+0000 7f234fc23740 1 -- start start 2024-03-16T10:19:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234fc23740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c9d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234fc23740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c96ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234fc23740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999c86f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234fc23740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f23481a69a0 con 0x55e999c85a50 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234fc23740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f23481a8660 con 0x55e999c8abc0 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234fc23740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f23481a7070 con 0x55e999c85060 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c96ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c96ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44316/0 (socket says 172.21.15.77:44316) 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234659c640 1 -- 172.21.15.77:0/68574375 learned_addr learned my addr 172.21.15.77:0/68574375 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f2346d9d640 1 --2- 172.21.15.77:0/68574375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c9d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234759e640 1 --2- 172.21.15.77:0/68574375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999c86f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234659c640 1 -- 172.21.15.77:0/68574375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 msgr2=0x55e999c9d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234659c640 1 --2- 172.21.15.77:0/68574375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c9d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234659c640 1 -- 172.21.15.77:0/68574375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 msgr2=0x55e999c86f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234659c640 1 --2- 172.21.15.77:0/68574375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999c86f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234659c640 1 -- 172.21.15.77:0/68574375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e999c9d840 con 0x55e999c85a50 2024-03-16T10:19:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f234659c640 1 --2- 172.21.15.77:0/68574375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c96ae0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f233c003f80 tx=0x7f233c003a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4297ee37fed2cf18 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f232cff9640 1 -- 172.21.15.77:0/68574375 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c027070 con 0x55e999c85a50 2024-03-16T10:19:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f232cff9640 1 -- 172.21.15.77:0/68574375 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f233c01f040 con 0x55e999c85a50 2024-03-16T10:19:54.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.647+0000 7f232cff9640 1 -- 172.21.15.77:0/68574375 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c01e070 con 0x55e999c85a50 2024-03-16T10:19:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.648+0000 7f234fc23740 1 -- 172.21.15.77:0/68574375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 msgr2=0x55e999c96ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.648+0000 7f234fc23740 1 --2- 172.21.15.77:0/68574375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c96ae0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f233c003f80 tx=0x7f233c003a40 comp rx=0 tx=0).stop 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.648+0000 7f234fc23740 1 -- 172.21.15.77:0/68574375 shutdown_connections 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.648+0000 7f234fc23740 1 --2- 172.21.15.77:0/68574375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999c86f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.648+0000 7f234fc23740 1 --2- 172.21.15.77:0/68574375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c96ae0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.648+0000 7f234fc23740 1 --2- 172.21.15.77:0/68574375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c9d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.648+0000 7f234fc23740 1 -- 172.21.15.77:0/68574375 >> 172.21.15.77:0/68574375 conn(0x55e999c77dc0 msgr2=0x55e999c8e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.648+0000 7f234fc23740 1 -- 172.21.15.77:0/68574375 shutdown_connections 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.648+0000 7f234fc23740 1 -- 172.21.15.77:0/68574375 wait complete. 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.648+0000 7f234fc23740 1 -- start start 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234fc23740 1 Processor -- start 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234fc23740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c93af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234fc23740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c94030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234fc23740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999cff400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234fc23740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f23481a7ed0 con 0x55e999c85a50 2024-03-16T10:19:54.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234fc23740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f23481a7800 con 0x55e999c8abc0 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234fc23740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f23481a4980 con 0x55e999c85060 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c94030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f2346d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c93af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c94030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44318/0 (socket says 172.21.15.77:44318) 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234659c640 1 -- 172.21.15.77:0/2587690724 learned_addr learned my addr 172.21.15.77:0/2587690724 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234759e640 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999cff400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234fc23740 10 client.0 _ll_register_callbacks cb 0x7f233c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234659c640 1 -- 172.21.15.77:0/2587690724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 msgr2=0x55e999c93af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234659c640 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c93af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234659c640 1 -- 172.21.15.77:0/2587690724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 msgr2=0x55e999cff400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234659c640 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999cff400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234659c640 1 -- 172.21.15.77:0/2587690724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9999bf7d0 con 0x55e999c85a50 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234759e640 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999cff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f234659c640 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c94030 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f233c00fa40 tx=0x7f233c009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63804e5beb312741 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.649+0000 7f232aff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f232aff5640 1 -- 172.21.15.77:0/2587690724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c01e040 con 0x55e999c85a50 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f232aff5640 1 -- 172.21.15.77:0/2587690724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f233c00f560 con 0x55e999c85a50 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f234fc23740 1 -- 172.21.15.77:0/2587690724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f233c01f040 con 0x55e999c85a50 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f232aff5640 1 -- 172.21.15.77:0/2587690724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f233c045050 con 0x55e999c85a50 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f234fc23740 10 client.16425 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f234fc23740 1 -- 172.21.15.77:0/2587690724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e999d00ae0 con 0x55e999c85a50 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f232aff5640 1 -- 172.21.15.77:0/2587690724 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f233c01f040 con 0x55e999c85a50 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f232aff5640 1 -- 172.21.15.77:0/2587690724 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f233c019ba0 con 0x55e999c85a50 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f234fc23740 10 client.16425 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f234fc23740 10 client.16425 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f234fc23740 20 client.16425 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f234fc23740 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55e999d01510 0x55e999d03930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f234fc23740 4 client.16425 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f234fc23740 1 -- 172.21.15.77:0/2587690724 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e999be8d20 con 0x55e999d01510 2024-03-16T10:19:54.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.650+0000 7f2346d9d640 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55e999d01510 0x55e999d03930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f232aff5640 10 client.16425 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f2346d9d640 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55e999d01510 0x55e999d03930 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=468eb20e6a5e32aa server_cookie=704cf06c5546c04d in_seq=0 out_seq=0 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f232aff5640 1 -- 172.21.15.77:0/2587690724 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e999be8d20 con 0x55e999d01510 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f232aff5640 10 client.16425 handle_command_reply: tid=0 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f234fc23740 1 client.16425 shutdown 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f234fc23740 1 -- 172.21.15.77:0/2587690724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 msgr2=0x55e999c94030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f234fc23740 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c94030 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f233c00fa40 tx=0x7f233c009370 comp rx=0 tx=0).stop 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f234fc23740 1 -- 172.21.15.77:0/2587690724 shutdown_connections 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f234fc23740 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55e999d01510 0x55e999d03930 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f232aff5640 0 client.16425 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f234fc23740 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e999c8abc0 0x55e999cff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f234fc23740 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e999c85a50 0x55e999c94030 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f234fc23740 1 --2- 172.21.15.77:0/2587690724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e999c85060 0x55e999c93af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f234fc23740 1 -- 172.21.15.77:0/2587690724 >> 172.21.15.77:0/2587690724 conn(0x55e999c77dc0 msgr2=0x55e999c86f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.651+0000 7f234fc23740 1 -- 172.21.15.77:0/2587690724 shutdown_connections 2024-03-16T10:19:54.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234fc23740 1 -- 172.21.15.77:0/2587690724 wait complete. 2024-03-16T10:19:54.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234fc23740 20 client.16425 trim_cache size 0 max 16384 2024-03-16T10:19:54.659 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:19:54.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234d036640 1 -- 172.21.15.77:0/2464483170 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f23300a2800 msgr2=0x7f23300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234d036640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f23300a2800 0x7f23300a4cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2338000c00 tx=0x7f233802c880 comp rx=0 tx=0).stop 2024-03-16T10:19:54.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234d036640 1 -- 172.21.15.77:0/2464483170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2348101d90 msgr2=0x7f23481a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234d036640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2348101d90 0x7f23481a12f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2340009fd0 tx=0x7f234000ef10 comp rx=0 tx=0).stop 2024-03-16T10:19:54.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234d036640 1 -- 172.21.15.77:0/2464483170 shutdown_connections 2024-03-16T10:19:54.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234d036640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f23300a2800 0x7f23300a4cc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234d036640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2348101d90 0x7f23481a12f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234d036640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2348100240 0x7f234819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234d036640 1 --2- 172.21.15.77:0/2464483170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23480ff880 0x7f2348199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234d036640 1 -- 172.21.15.77:0/2464483170 >> 172.21.15.77:0/2464483170 conn(0x7f23480fb400 msgr2=0x7f23480fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234d036640 1 -- 172.21.15.77:0/2464483170 shutdown_connections 2024-03-16T10:19:54.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.652+0000 7f234d036640 1 -- 172.21.15.77:0/2464483170 wait complete. 2024-03-16T10:19:54.666 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T10:19:54.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.812+0000 7fb7c1a63640 1 -- 172.21.15.77:0/2196591248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7bc1029e0 msgr2=0x7fb7bc102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.812+0000 7fb7c1a63640 1 --2- 172.21.15.77:0/2196591248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7bc1029e0 0x7fb7bc102e60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009900 tx=0x7fb7b002bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:54.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.812+0000 7fb7c1a63640 1 -- 172.21.15.77:0/2196591248 shutdown_connections 2024-03-16T10:19:54.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.813+0000 7fb7c1a63640 1 --2- 172.21.15.77:0/2196591248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc1033a0 0x7fb7bc109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.813+0000 7fb7c1a63640 1 --2- 172.21.15.77:0/2196591248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7bc1029e0 0x7fb7bc102e60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.813+0000 7fb7c1a63640 1 --2- 172.21.15.77:0/2196591248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7bc1017f0 0x7fb7bc101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.813+0000 7fb7c1a63640 1 -- 172.21.15.77:0/2196591248 >> 172.21.15.77:0/2196591248 conn(0x7fb7bc0fcd80 msgr2=0x7fb7bc0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.813+0000 7fb7c1a63640 1 -- 172.21.15.77:0/2196591248 shutdown_connections 2024-03-16T10:19:54.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.813+0000 7fb7c1a63640 1 -- 172.21.15.77:0/2196591248 wait complete. 2024-03-16T10:19:54.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.813+0000 7fb7c1a63640 1 Processor -- start 2024-03-16T10:19:54.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.813+0000 7fb7c1a63640 1 -- start start 2024-03-16T10:19:54.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7c1a63640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7bc1017f0 0x7fb7bc199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7c1a63640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7bc1029e0 0x7fb7bc19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7c1a63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc1033a0 0x7fb7bc1a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7c1a63640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc10d450 con 0x7fb7bc1033a0 2024-03-16T10:19:54.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7c1a63640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc10d2d0 con 0x7fb7bc1017f0 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7c1a63640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc10d5d0 con 0x7fb7bc1029e0 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7baffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7bc1029e0 0x7fb7bc19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7bb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7bc1017f0 0x7fb7bc199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7baffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7bc1029e0 0x7fb7bc19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44314/0 (socket says 172.21.15.77:44314) 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7bb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7bc1017f0 0x7fb7bc199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41770/0 (socket says 172.21.15.77:41770) 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7baffd640 1 -- 172.21.15.77:0/3139551852 learned_addr learned my addr 172.21.15.77:0/3139551852 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7bbfff640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc1033a0 0x7fb7bc1a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7baffd640 1 -- 172.21.15.77:0/3139551852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7bc1017f0 msgr2=0x7fb7bc199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7baffd640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7bc1017f0 0x7fb7bc199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7baffd640 1 -- 172.21.15.77:0/3139551852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc1033a0 msgr2=0x7fb7bc1a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7baffd640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc1033a0 0x7fb7bc1a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7baffd640 1 -- 172.21.15.77:0/3139551852 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7bc1a1960 con 0x7fb7bc1029e0 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.814+0000 7fb7bbfff640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc1033a0 0x7fb7bc1a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.815+0000 7fb7baffd640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7bc1029e0 0x7fb7bc19a280 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009230 tx=0x7fb7b002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.815+0000 7fb7b8ff9640 1 -- 172.21.15.77:0/3139551852 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0041070 con 0x7fb7bc1029e0 2024-03-16T10:19:54.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.815+0000 7fb7b8ff9640 1 -- 172.21.15.77:0/3139551852 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b0004450 con 0x7fb7bc1029e0 2024-03-16T10:19:54.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.815+0000 7fb7c1a63640 1 -- 172.21.15.77:0/3139551852 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7bc1a1bf0 con 0x7fb7bc1029e0 2024-03-16T10:19:54.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.815+0000 7fb7b8ff9640 1 -- 172.21.15.77:0/3139551852 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0003e20 con 0x7fb7bc1029e0 2024-03-16T10:19:54.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.815+0000 7fb7c1a63640 1 -- 172.21.15.77:0/3139551852 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7bc1a2070 con 0x7fb7bc1029e0 2024-03-16T10:19:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.818+0000 7fb7c4648740 1 Processor -- start 2024-03-16T10:19:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.818+0000 7fb7b8ff9640 1 -- 172.21.15.77:0/3139551852 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fb7b0044070 con 0x7fb7bc1029e0 2024-03-16T10:19:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.818+0000 7fb7c4648740 1 -- start start 2024-03-16T10:19:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.818+0000 7fb7c4648740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb848850 0x5567bb848c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.818+0000 7fb7c4648740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb849240 0x5567bb84dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.818+0000 7fb7c4648740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb850840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.818+0000 7fb7c4648740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5567bb7ad490 con 0x5567bb849240 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.818+0000 7fb7c4648740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5567bb755c90 con 0x5567bb84e3b0 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.818+0000 7fb7c4648740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5567bb7a9530 con 0x5567bb848850 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7bbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb848850 0x5567bb848c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7bbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb848850 0x5567bb848c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44322/0 (socket says 172.21.15.77:44322) 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7bbfff640 1 -- 172.21.15.77:0/2313511281 learned_addr learned my addr 172.21.15.77:0/2313511281 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/2313511281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb849240 0x5567bb84dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7baffd640 1 --2- 172.21.15.77:0/2313511281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb850840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7bb7fe640 1 -- 172.21.15.77:0/2313511281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb848850 msgr2=0x5567bb848c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/2313511281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb848850 0x5567bb848c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7bb7fe640 1 -- 172.21.15.77:0/2313511281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 msgr2=0x5567bb850840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/2313511281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb850840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7bb7fe640 1 -- 172.21.15.77:0/2313511281 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567bb810500 con 0x5567bb849240 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7b8ff9640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb7a40a2880 0x7fb7a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7bbfff640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb7a40a2880 0x7fb7a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7b8ff9640 1 -- 172.21.15.77:0/3139551852 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb7b00f0580 con 0x7fb7bc1029e0 2024-03-16T10:19:54.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.819+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/2313511281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb849240 0x5567bb84dd40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009bf0 tx=0x7fb7ac011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6e66655651414f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.820+0000 7fb7a1ffb640 1 -- 172.21.15.77:0/2313511281 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac009da0 con 0x5567bb849240 2024-03-16T10:19:54.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.820+0000 7fb7a1ffb640 1 -- 172.21.15.77:0/2313511281 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7ac005020 con 0x5567bb849240 2024-03-16T10:19:54.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.820+0000 7fb7a1ffb640 1 -- 172.21.15.77:0/2313511281 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac004bc0 con 0x5567bb849240 2024-03-16T10:19:54.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.821+0000 7fb7bbfff640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb7a40a2880 0x7fb7a40a4d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x5567bb849aa0 tx=0x7fb7b4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.821+0000 7fb7c4648740 1 -- 172.21.15.77:0/2313511281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb849240 msgr2=0x5567bb84dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.821+0000 7fb7c4648740 1 --2- 172.21.15.77:0/2313511281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb849240 0x5567bb84dd40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009bf0 tx=0x7fb7ac011810 comp rx=0 tx=0).stop 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.821+0000 7fb7c4648740 1 -- 172.21.15.77:0/2313511281 shutdown_connections 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.821+0000 7fb7c4648740 1 --2- 172.21.15.77:0/2313511281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb850840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.821+0000 7fb7c4648740 1 --2- 172.21.15.77:0/2313511281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb849240 0x5567bb84dd40 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.821+0000 7fb7c4648740 1 --2- 172.21.15.77:0/2313511281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb848850 0x5567bb848c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.821+0000 7fb7c4648740 1 -- 172.21.15.77:0/2313511281 >> 172.21.15.77:0/2313511281 conn(0x5567bb843dc0 msgr2=0x5567bb8441c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.821+0000 7fb7c4648740 1 -- 172.21.15.77:0/2313511281 shutdown_connections 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.821+0000 7fb7c4648740 1 -- 172.21.15.77:0/2313511281 wait complete. 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.821+0000 7fb7c4648740 1 -- start start 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7c4648740 1 Processor -- start 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7c4648740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb852de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7c4648740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 0x5567bb853320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7c4648740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb85a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7c4648740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5567bb851360 con 0x5567bb848850 2024-03-16T10:19:54.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7c4648740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5567bb769230 con 0x5567bb84e3b0 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7c4648740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5567bb8514e0 con 0x5567bb849240 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7c4648740 10 client.0 _ll_register_callbacks cb 0x5567bb7a5010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7bbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb85a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7baffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 0x5567bb853320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7bbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb85a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41792/0 (socket says 172.21.15.77:41792) 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7bbfff640 1 -- 172.21.15.77:0/1320773511 learned_addr learned my addr 172.21.15.77:0/1320773511 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7baffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 0x5567bb853320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44332/0 (socket says 172.21.15.77:44332) 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb852de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7baffd640 1 -- 172.21.15.77:0/1320773511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 msgr2=0x5567bb85a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7baffd640 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb85a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.822+0000 7fb7baffd640 1 -- 172.21.15.77:0/1320773511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 msgr2=0x5567bb852de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.823+0000 7fb7baffd640 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb852de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.823+0000 7fb7baffd640 1 -- 172.21.15.77:0/1320773511 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567bb778100 con 0x5567bb849240 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.823+0000 7fb7bbfff640 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb85a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.823+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb852de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.823+0000 7fb7baffd640 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 0x5567bb853320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0006010 tx=0x7fb7b00b9320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=854085a4890c2347 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.823+0000 7fb79affd640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.823+0000 7fb79affd640 1 -- 172.21.15.77:0/1320773511 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b004b070 con 0x5567bb849240 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.823+0000 7fb7c4648740 1 -- 172.21.15.77:0/1320773511 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7ac005020 con 0x5567bb849240 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.823+0000 7fb79affd640 1 -- 172.21.15.77:0/1320773511 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb7b00c5220 con 0x5567bb849240 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.823+0000 7fb79affd640 1 -- 172.21.15.77:0/1320773511 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb7b0004c60 con 0x5567bb849240 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.823+0000 7fb7c4648740 10 client.26200 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.823+0000 7fb7c4648740 1 -- 172.21.15.77:0/1320773511 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5567bb4c13f0 con 0x5567bb849240 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.824+0000 7fb79affd640 1 -- 172.21.15.77:0/1320773511 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7ac005020 con 0x5567bb849240 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.824+0000 7fb79affd640 1 -- 172.21.15.77:0/1320773511 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fb7b00d2050 con 0x5567bb849240 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.824+0000 7fb7c4648740 10 client.26200 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.824+0000 7fb7c4648740 10 client.26200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.824+0000 7fb7c4648740 20 client.26200 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.824+0000 7fb7c4648740 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5567bb85ec10 0x5567bb8610d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.824+0000 7fb7c4648740 4 client.26200 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.824+0000 7fb7c4648740 1 -- 172.21.15.77:0/1320773511 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567bb7ba9d0 con 0x5567bb85ec10 2024-03-16T10:19:54.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.824+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5567bb85ec10 0x5567bb8610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.825+0000 7fb79affd640 10 client.26200 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.825+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5567bb85ec10 0x5567bb8610d0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7a286227fa1db26 server_cookie=9844a2852303b2d3 in_seq=0 out_seq=0 2024-03-16T10:19:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb79affd640 1 -- 172.21.15.77:0/1320773511 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5567bb7ba9d0 con 0x5567bb85ec10 2024-03-16T10:19:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb79affd640 10 client.26200 handle_command_reply: tid=0 2024-03-16T10:19:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb7c4648740 1 client.26200 shutdown 2024-03-16T10:19:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb7c4648740 1 -- 172.21.15.77:0/1320773511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 msgr2=0x5567bb853320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb7c4648740 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 0x5567bb853320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0006010 tx=0x7fb7b00b9320 comp rx=0 tx=0).stop 2024-03-16T10:19:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb7c4648740 1 -- 172.21.15.77:0/1320773511 shutdown_connections 2024-03-16T10:19:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb7c4648740 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb85a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb7c4648740 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 0x5567bb853320 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb7c4648740 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5567bb85ec10 0x5567bb8610d0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb7c4648740 1 --2- 172.21.15.77:0/1320773511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb852de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb79affd640 0 client.26200 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb7c4648740 1 -- 172.21.15.77:0/1320773511 >> 172.21.15.77:0/1320773511 conn(0x5567bb843dc0 msgr2=0x5567bb842fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb7c4648740 1 -- 172.21.15.77:0/1320773511 shutdown_connections 2024-03-16T10:19:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb7c4648740 1 -- 172.21.15.77:0/1320773511 wait complete. 2024-03-16T10:19:54.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.826+0000 7fb7c4648740 20 client.26200 trim_cache size 0 max 16384 2024-03-16T10:19:54.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.843+0000 7fb7c4648740 1 Processor -- start 2024-03-16T10:19:54.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.843+0000 7fb7c4648740 1 -- start start 2024-03-16T10:19:54.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.843+0000 7fb7c4648740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb852de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.843+0000 7fb7c4648740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 0x5567bb853320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.843+0000 7fb7c4648740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb85a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.843+0000 7fb7c4648740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1aad70 con 0x5567bb848850 2024-03-16T10:19:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.843+0000 7fb7c4648740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1aca30 con 0x5567bb84e3b0 2024-03-16T10:19:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.843+0000 7fb7c4648740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1ab440 con 0x5567bb849240 2024-03-16T10:19:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7baffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 0x5567bb853320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7bbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb85a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7baffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 0x5567bb853320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44340/0 (socket says 172.21.15.77:44340) 2024-03-16T10:19:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7baffd640 1 -- 172.21.15.77:0/4277230012 learned_addr learned my addr 172.21.15.77:0/4277230012 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/4277230012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb852de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7baffd640 1 -- 172.21.15.77:0/4277230012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 msgr2=0x5567bb85a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7baffd640 1 --2- 172.21.15.77:0/4277230012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb85a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7baffd640 1 -- 172.21.15.77:0/4277230012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 msgr2=0x5567bb852de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7baffd640 1 --2- 172.21.15.77:0/4277230012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb852de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7baffd640 1 -- 172.21.15.77:0/4277230012 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567bb852980 con 0x5567bb849240 2024-03-16T10:19:54.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7bbfff640 1 --2- 172.21.15.77:0/4277230012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb85a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/4277230012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb852de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.844+0000 7fb7baffd640 1 --2- 172.21.15.77:0/4277230012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 0x5567bb853320 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00bccc0 tx=0x7fb7b002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b56bb43e8e69d69 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.845+0000 7fb7a17fa640 1 -- 172.21.15.77:0/4277230012 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b004b070 con 0x5567bb849240 2024-03-16T10:19:54.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.845+0000 7fb7a17fa640 1 -- 172.21.15.77:0/4277230012 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b00c56b0 con 0x5567bb849240 2024-03-16T10:19:54.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.845+0000 7fb7a17fa640 1 -- 172.21.15.77:0/4277230012 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b00b7b30 con 0x5567bb849240 2024-03-16T10:19:54.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.846+0000 7fb7c4648740 1 -- 172.21.15.77:0/4277230012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 msgr2=0x5567bb853320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.846+0000 7fb7c4648740 1 --2- 172.21.15.77:0/4277230012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 0x5567bb853320 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00bccc0 tx=0x7fb7b002bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:54.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.846+0000 7fb7c4648740 1 -- 172.21.15.77:0/4277230012 shutdown_connections 2024-03-16T10:19:54.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.846+0000 7fb7c4648740 1 --2- 172.21.15.77:0/4277230012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb84e3b0 0x5567bb85a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.846+0000 7fb7c4648740 1 --2- 172.21.15.77:0/4277230012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb849240 0x5567bb853320 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.846+0000 7fb7c4648740 1 --2- 172.21.15.77:0/4277230012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb852de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.846+0000 7fb7c4648740 1 -- 172.21.15.77:0/4277230012 >> 172.21.15.77:0/4277230012 conn(0x5567bb843dc0 msgr2=0x5567bb8441a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.846+0000 7fb7c4648740 1 -- 172.21.15.77:0/4277230012 shutdown_connections 2024-03-16T10:19:54.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.846+0000 7fb7c4648740 1 -- 172.21.15.77:0/4277230012 wait complete. 2024-03-16T10:19:54.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.847+0000 7fb7c4648740 1 -- start start 2024-03-16T10:19:54.848 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.847+0000 7fb7c4648740 1 Processor -- start 2024-03-16T10:19:54.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.847+0000 7fb7c4648740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb857380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.848+0000 7fb7c4648740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb849240 0x5567bb8578c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.848+0000 7fb7bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb857380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.848+0000 7fb7bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb857380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44396/0 (socket says 172.21.15.77:44396) 2024-03-16T10:19:54.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.848+0000 7fb7bb7fe640 1 -- 172.21.15.77:0/878792815 learned_addr learned my addr 172.21.15.77:0/878792815 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:54.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.848+0000 7fb7c4648740 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb84e3b0 0x5567bb8588d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.848+0000 7fb7baffd640 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb849240 0x5567bb8578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.848+0000 7fb7c4648740 1 -- 172.21.15.77:0/878792815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1ac2a0 con 0x5567bb848850 2024-03-16T10:19:54.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.848+0000 7fb7bbfff640 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb84e3b0 0x5567bb8588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.850 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.849+0000 7fb7c4648740 1 -- 172.21.15.77:0/878792815 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1abbd0 con 0x5567bb849240 2024-03-16T10:19:54.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.849+0000 7fb7c4648740 1 -- 172.21.15.77:0/878792815 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1a8d50 con 0x5567bb84e3b0 2024-03-16T10:19:54.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.849+0000 7fb7c4648740 10 client.0 _ll_register_callbacks cb 0x7fb7b0009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:54.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.849+0000 7fb7bb7fe640 1 -- 172.21.15.77:0/878792815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb84e3b0 msgr2=0x5567bb8588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.849+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb84e3b0 0x5567bb8588d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.849+0000 7fb7bb7fe640 1 -- 172.21.15.77:0/878792815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb849240 msgr2=0x5567bb8578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:54.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.849+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb849240 0x5567bb8578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.849+0000 7fb7bb7fe640 1 -- 172.21.15.77:0/878792815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567bb58b7d0 con 0x5567bb848850 2024-03-16T10:19:54.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.849+0000 7fb7bbfff640 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb84e3b0 0x5567bb8588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.849+0000 7fb7baffd640 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb849240 0x5567bb8578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:54.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.849+0000 7fb7bb7fe640 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb857380 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac011d10 tx=0x7fb7ac011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3523bd22261dcba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:54.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.850+0000 7fb7c4648740 1 -- 172.21.15.77:0/878792815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7b00c56b0 con 0x5567bb848850 2024-03-16T10:19:54.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb79affd640 10 client.16443 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:54.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb79affd640 1 -- 172.21.15.77:0/878792815 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac005020 con 0x5567bb848850 2024-03-16T10:19:54.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb79affd640 1 -- 172.21.15.77:0/878792815 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb7ac02ba30 con 0x5567bb848850 2024-03-16T10:19:54.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb79affd640 1 -- 172.21.15.77:0/878792815 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb7ac02bc70 con 0x5567bb848850 2024-03-16T10:19:54.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb7c4648740 10 client.16443 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:54.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb7c4648740 1 -- 172.21.15.77:0/878792815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=32}) -- 0x5567bb859ec0 con 0x5567bb848850 2024-03-16T10:19:54.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb79affd640 1 -- 172.21.15.77:0/878792815 <== mon.0 v2:172.21.15.77:3300/0 4 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fb7ac02b280 con 0x5567bb848850 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb7c4648740 10 client.16443 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb7c4648740 10 client.16443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb7c4648740 20 client.16443 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb7c4648740 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5567bb8c5010 0x5567bb8c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb7c4648740 4 client.16443 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.862+0000 7fb7c4648740 1 -- 172.21.15.77:0/878792815 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb7b00c56b0 con 0x5567bb8c5010 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.863+0000 7fb7baffd640 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5567bb8c5010 0x5567bb8c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.863+0000 7fb7baffd640 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5567bb8c5010 0x5567bb8c7400 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30565b0229890975 server_cookie=f25ecbc011adf03f in_seq=0 out_seq=0 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.863+0000 7fb79affd640 10 client.16443 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.863+0000 7fb79affd640 1 -- 172.21.15.77:0/878792815 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb7b00c56b0 con 0x5567bb8c5010 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.863+0000 7fb79affd640 10 client.16443 handle_command_reply: tid=0 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.863+0000 7fb7c4648740 1 client.16443 shutdown 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.864+0000 7fb7c4648740 1 -- 172.21.15.77:0/878792815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 msgr2=0x5567bb857380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.864+0000 7fb7c4648740 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb857380 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac011d10 tx=0x7fb7ac011dc0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.864+0000 7fb7c4648740 1 -- 172.21.15.77:0/878792815 shutdown_connections 2024-03-16T10:19:54.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.864+0000 7fb7c4648740 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5567bb84e3b0 0x5567bb8588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.864+0000 7fb7c4648740 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5567bb849240 0x5567bb8578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.864+0000 7fb7c4648740 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5567bb8c5010 0x5567bb8c7400 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.864+0000 7fb7c4648740 1 --2- 172.21.15.77:0/878792815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5567bb848850 0x5567bb857380 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.864+0000 7fb79affd640 0 client.16443 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:54.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.864+0000 7fb7c4648740 1 -- 172.21.15.77:0/878792815 >> 172.21.15.77:0/878792815 conn(0x5567bb843dc0 msgr2=0x5567bb852d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.864+0000 7fb7c4648740 1 -- 172.21.15.77:0/878792815 shutdown_connections 2024-03-16T10:19:54.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.864+0000 7fb7c4648740 1 -- 172.21.15.77:0/878792815 wait complete. 2024-03-16T10:19:54.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.865+0000 7fb7c4648740 20 client.16443 trim_cache size 0 max 16384 2024-03-16T10:19:54.868 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:19:54.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.868+0000 7fb7c1a63640 1 -- 172.21.15.77:0/3139551852 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb7a40a2880 msgr2=0x7fb7a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.868+0000 7fb7c1a63640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb7a40a2880 0x7fb7a40a4d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x5567bb849aa0 tx=0x7fb7b4009500 comp rx=0 tx=0).stop 2024-03-16T10:19:54.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.868+0000 7fb7c1a63640 1 -- 172.21.15.77:0/3139551852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7bc1029e0 msgr2=0x7fb7bc19a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:54.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.868+0000 7fb7c1a63640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7bc1029e0 0x7fb7bc19a280 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009230 tx=0x7fb7b002c650 comp rx=0 tx=0).stop 2024-03-16T10:19:54.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.868+0000 7fb7c1a63640 1 -- 172.21.15.77:0/3139551852 shutdown_connections 2024-03-16T10:19:54.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.868+0000 7fb7c1a63640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc1033a0 0x7fb7bc1a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.868+0000 7fb7c1a63640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb7a40a2880 0x7fb7a40a4d40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.868+0000 7fb7c1a63640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7bc1029e0 0x7fb7bc19a280 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.868+0000 7fb7c1a63640 1 --2- 172.21.15.77:0/3139551852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7bc1017f0 0x7fb7bc199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:54.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.868+0000 7fb7c1a63640 1 -- 172.21.15.77:0/3139551852 >> 172.21.15.77:0/3139551852 conn(0x7fb7bc0fcd80 msgr2=0x7fb7bc073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:54.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.869+0000 7fb7c1a63640 1 -- 172.21.15.77:0/3139551852 shutdown_connections 2024-03-16T10:19:54.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:54.869+0000 7fb7c1a63640 1 -- 172.21.15.77:0/3139551852 wait complete. 2024-03-16T10:19:55.888 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T10:19:55.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:19:55.889 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T10:19:56.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:55 smithi192 ceph-mon[36354]: pgmap v435: 97 pgs: 97 active+clean; 226 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 205 op/s 2024-03-16T10:19:56.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.185+0000 7fa04719e640 1 -- 172.21.15.77:0/2053687171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0401022e0 msgr2=0x7fa0401046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.185+0000 7fa04719e640 1 --2- 172.21.15.77:0/2053687171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0401022e0 0x7fa0401046d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fa034009900 tx=0x7fa03402bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:56.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.185+0000 7fa04719e640 1 -- 172.21.15.77:0/2053687171 shutdown_connections 2024-03-16T10:19:56.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.185+0000 7fa04719e640 1 --2- 172.21.15.77:0/2053687171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa040107540 0x7fa040109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.185+0000 7fa04719e640 1 --2- 172.21.15.77:0/2053687171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa040104c10 0x7fa040107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.185+0000 7fa04719e640 1 --2- 172.21.15.77:0/2053687171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0401022e0 0x7fa0401046d0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.185+0000 7fa04719e640 1 -- 172.21.15.77:0/2053687171 >> 172.21.15.77:0/2053687171 conn(0x7fa0400fbc70 msgr2=0x7fa0400fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:56.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.185+0000 7fa04719e640 1 -- 172.21.15.77:0/2053687171 shutdown_connections 2024-03-16T10:19:56.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.185+0000 7fa04719e640 1 -- 172.21.15.77:0/2053687171 wait complete. 2024-03-16T10:19:56.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa04719e640 1 Processor -- start 2024-03-16T10:19:56.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa04719e640 1 -- start start 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa04719e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0401022e0 0x7fa040197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa04719e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa040104c10 0x7fa0401980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa04719e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa040107540 0x7fa04019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa04719e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa04010d4a0 con 0x7fa040104c10 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa04719e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa04010d320 con 0x7fa040107540 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa04719e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa04010d620 con 0x7fa0401022e0 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa03ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa040104c10 0x7fa0401980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa03ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa040104c10 0x7fa0401980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44412/0 (socket says 172.21.15.77:44412) 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa03ffff640 1 -- 172.21.15.77:0/2094243403 learned_addr learned my addr 172.21.15.77:0/2094243403 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa044f3a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0401022e0 0x7fa040197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa04573b640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa040107540 0x7fa04019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa03ffff640 1 -- 172.21.15.77:0/2094243403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0401022e0 msgr2=0x7fa040197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa03ffff640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0401022e0 0x7fa040197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa03ffff640 1 -- 172.21.15.77:0/2094243403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa040107540 msgr2=0x7fa04019f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa03ffff640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa040107540 0x7fa04019f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.186+0000 7fa03ffff640 1 -- 172.21.15.77:0/2094243403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04019f7c0 con 0x7fa040104c10 2024-03-16T10:19:56.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.187+0000 7fa03ffff640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa040104c10 0x7fa0401980e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00e9c0 tx=0x7fa02c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:56.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.187+0000 7fa03dffb640 1 -- 172.21.15.77:0/2094243403 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c00cd80 con 0x7fa040104c10 2024-03-16T10:19:56.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.187+0000 7fa04719e640 1 -- 172.21.15.77:0/2094243403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa04019fab0 con 0x7fa040104c10 2024-03-16T10:19:56.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.187+0000 7fa03dffb640 1 -- 172.21.15.77:0/2094243403 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c005500 con 0x7fa040104c10 2024-03-16T10:19:56.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.187+0000 7fa03dffb640 1 -- 172.21.15.77:0/2094243403 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c010690 con 0x7fa040104c10 2024-03-16T10:19:56.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.187+0000 7fa04719e640 1 -- 172.21.15.77:0/2094243403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0401a0050 con 0x7fa040104c10 2024-03-16T10:19:56.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.189+0000 7fa03dffb640 1 -- 172.21.15.77:0/2094243403 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fa02c0046c0 con 0x7fa040104c10 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.189+0000 7fa049d89740 1 Processor -- start 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.189+0000 7fa049d89740 1 -- start start 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa049d89740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6141990 0x5640a6141db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa049d89740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6142380 0x5640a6146e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa049d89740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a61474f0 0x5640a6149980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa049d89740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5640a60a60d0 con 0x5640a6142380 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa049d89740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5640a604ec70 con 0x5640a61474f0 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa049d89740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5640a60a2170 con 0x5640a6141990 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa03dffb640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0300a2800 0x7fa0300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa044f3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6142380 0x5640a6146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa03ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a61474f0 0x5640a6149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa044f3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6142380 0x5640a6146e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44428/0 (socket says 172.21.15.77:44428) 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa044f3a640 1 -- 172.21.15.77:0/2728510640 learned_addr learned my addr 172.21.15.77:0/2728510640 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa04573b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6141990 0x5640a6141db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa04573b640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0300a2800 0x7fa0300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa04573b640 1 -- 172.21.15.77:0/2728510640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a61474f0 msgr2=0x5640a6149980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa03dffb640 1 -- 172.21.15.77:0/2094243403 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa02c0ced00 con 0x7fa040104c10 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa04573b640 1 --2- 172.21.15.77:0/2728510640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a61474f0 0x5640a6149980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa04573b640 1 -- 172.21.15.77:0/2728510640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6142380 msgr2=0x5640a6146e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa04573b640 1 --2- 172.21.15.77:0/2728510640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6142380 0x5640a6146e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa04573b640 1 -- 172.21.15.77:0/2728510640 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640a5e827e0 con 0x5640a6141990 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa044f3a640 1 --2- 172.21.15.77:0/2728510640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6142380 0x5640a6146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.190+0000 7fa04573b640 1 --2- 172.21.15.77:0/2728510640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6141990 0x5640a6141db0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa038005fd0 tx=0x7fa038011d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d096297ea2a72083 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa04573b640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0300a2800 0x7fa0300a4cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa03801d000 tx=0x7fa03801f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa02affd640 1 -- 172.21.15.77:0/2728510640 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038003970 con 0x5640a6141990 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa02affd640 1 -- 172.21.15.77:0/2728510640 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa038022030 con 0x5640a6141990 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa02affd640 1 -- 172.21.15.77:0/2728510640 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038004320 con 0x5640a6141990 2024-03-16T10:19:56.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa049d89740 1 -- 172.21.15.77:0/2728510640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6141990 msgr2=0x5640a6141db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa049d89740 1 --2- 172.21.15.77:0/2728510640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6141990 0x5640a6141db0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa038005fd0 tx=0x7fa038011d40 comp rx=0 tx=0).stop 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa049d89740 1 -- 172.21.15.77:0/2728510640 shutdown_connections 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa049d89740 1 --2- 172.21.15.77:0/2728510640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a61474f0 0x5640a6149980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa049d89740 1 --2- 172.21.15.77:0/2728510640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6142380 0x5640a6146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa049d89740 1 --2- 172.21.15.77:0/2728510640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6141990 0x5640a6141db0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa049d89740 1 -- 172.21.15.77:0/2728510640 >> 172.21.15.77:0/2728510640 conn(0x5640a6109c50 msgr2=0x5640a613d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa049d89740 1 -- 172.21.15.77:0/2728510640 shutdown_connections 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.191+0000 7fa049d89740 1 -- 172.21.15.77:0/2728510640 wait complete. 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa049d89740 1 -- start start 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa049d89740 1 Processor -- start 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa049d89740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a6141990 0x5640a614bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa049d89740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a614c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa049d89740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a61474f0 0x5640a6153460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa049d89740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5640a614a440 con 0x5640a61474f0 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa049d89740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5640a60620d0 con 0x5640a6141990 2024-03-16T10:19:56.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa049d89740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5640a614a5c0 con 0x5640a6142380 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa049d89740 10 client.0 _ll_register_callbacks cb 0x5640a602b760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa04573b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a61474f0 0x5640a6153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa03ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a614c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa044f3a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a6141990 0x5640a614bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa04573b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a61474f0 0x5640a6153460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44430/0 (socket says 172.21.15.77:44430) 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa03ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a614c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44404/0 (socket says 172.21.15.77:44404) 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa04573b640 1 -- 172.21.15.77:0/1586057703 learned_addr learned my addr 172.21.15.77:0/1586057703 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa04573b640 1 -- 172.21.15.77:0/1586057703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 msgr2=0x5640a614c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa04573b640 1 --2- 172.21.15.77:0/1586057703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a614c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa04573b640 1 -- 172.21.15.77:0/1586057703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a6141990 msgr2=0x5640a614bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa04573b640 1 --2- 172.21.15.77:0/1586057703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a6141990 0x5640a614bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa04573b640 1 -- 172.21.15.77:0/1586057703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640a5ee2f70 con 0x5640a61474f0 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.192+0000 7fa044f3a640 1 --2- 172.21.15.77:0/1586057703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a6141990 0x5640a614bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa04573b640 1 --2- 172.21.15.77:0/1586057703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a61474f0 0x5640a6153460 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa038011fd0 tx=0x7fa03801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d6b6e5dfa896798 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:56.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa01ffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa01ffff640 1 -- 172.21.15.77:0/1586057703 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038003b80 con 0x5640a61474f0 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa01ffff640 1 -- 172.21.15.77:0/1586057703 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa038003d20 con 0x5640a61474f0 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa049d89740 1 -- 172.21.15.77:0/1586057703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa038022030 con 0x5640a61474f0 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa01ffff640 1 -- 172.21.15.77:0/1586057703 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa038005140 con 0x5640a61474f0 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa049d89740 10 client.16455 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa049d89740 1 -- 172.21.15.77:0/1586057703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5640a5d7a170 con 0x5640a61474f0 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa01ffff640 1 -- 172.21.15.77:0/1586057703 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa038022030 con 0x5640a61474f0 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa01ffff640 1 -- 172.21.15.77:0/1586057703 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fa0380267a0 con 0x5640a61474f0 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa049d89740 10 client.16455 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa049d89740 10 client.16455 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.193+0000 7fa049d89740 20 client.16455 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.194+0000 7fa049d89740 1 --2- 172.21.15.77:0/1586057703 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5640a6157830 0x5640a6159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.194+0000 7fa049d89740 4 client.16455 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.194+0000 7fa049d89740 1 -- 172.21.15.77:0/1586057703 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640a60b3610 con 0x5640a6157830 2024-03-16T10:19:56.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.194+0000 7fa044f3a640 1 --2- 172.21.15.77:0/1586057703 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5640a6157830 0x5640a6159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.194+0000 7fa01ffff640 10 client.16455 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.194+0000 7fa044f3a640 1 --2- 172.21.15.77:0/1586057703 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5640a6157830 0x5640a6159cb0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63240d701b50b28 server_cookie=fc45f357b7c23458 in_seq=0 out_seq=0 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa01ffff640 1 -- 172.21.15.77:0/1586057703 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5640a60b3610 con 0x5640a6157830 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa01ffff640 10 client.16455 handle_command_reply: tid=0 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa049d89740 1 client.16455 shutdown 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa049d89740 1 -- 172.21.15.77:0/1586057703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a61474f0 msgr2=0x5640a6153460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa049d89740 1 --2- 172.21.15.77:0/1586057703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a61474f0 0x5640a6153460 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa038011fd0 tx=0x7fa03801ed30 comp rx=0 tx=0).stop 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa049d89740 1 -- 172.21.15.77:0/1586057703 shutdown_connections 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa049d89740 1 --2- 172.21.15.77:0/1586057703 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5640a6157830 0x5640a6159cb0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa049d89740 1 --2- 172.21.15.77:0/1586057703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a61474f0 0x5640a6153460 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa049d89740 1 --2- 172.21.15.77:0/1586057703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a614c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa01ffff640 0 client.16455 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:56.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa049d89740 1 --2- 172.21.15.77:0/1586057703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a6141990 0x5640a614bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa049d89740 1 -- 172.21.15.77:0/1586057703 >> 172.21.15.77:0/1586057703 conn(0x5640a6109c50 msgr2=0x5640a613c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:56.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa049d89740 1 -- 172.21.15.77:0/1586057703 shutdown_connections 2024-03-16T10:19:56.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa049d89740 1 -- 172.21.15.77:0/1586057703 wait complete. 2024-03-16T10:19:56.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.195+0000 7fa049d89740 20 client.16455 trim_cache size 0 max 16384 2024-03-16T10:19:56.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa049d89740 1 Processor -- start 2024-03-16T10:19:56.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa049d89740 1 -- start start 2024-03-16T10:19:56.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa049d89740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a6141990 0x5640a6140540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa049d89740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a6153380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa049d89740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a61474f0 0x5640a6140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa049d89740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401a8c80 con 0x5640a61474f0 2024-03-16T10:19:56.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa049d89740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401aa940 con 0x5640a6141990 2024-03-16T10:19:56.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa049d89740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401a9350 con 0x5640a6142380 2024-03-16T10:19:56.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa04573b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a61474f0 0x5640a6140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa03ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a6153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa03ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a6153380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44412/0 (socket says 172.21.15.77:44412) 2024-03-16T10:19:56.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa03ffff640 1 -- 172.21.15.77:0/2542660754 learned_addr learned my addr 172.21.15.77:0/2542660754 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:56.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.213+0000 7fa044f3a640 1 --2- 172.21.15.77:0/2542660754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a6141990 0x5640a6140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.214+0000 7fa03ffff640 1 -- 172.21.15.77:0/2542660754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a6141990 msgr2=0x5640a6140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.214+0000 7fa03ffff640 1 --2- 172.21.15.77:0/2542660754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a6141990 0x5640a6140540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.214+0000 7fa03ffff640 1 -- 172.21.15.77:0/2542660754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a61474f0 msgr2=0x5640a6140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.214+0000 7fa03ffff640 1 --2- 172.21.15.77:0/2542660754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a61474f0 0x5640a6140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.214+0000 7fa03ffff640 1 -- 172.21.15.77:0/2542660754 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640a6154550 con 0x5640a6142380 2024-03-16T10:19:56.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.214+0000 7fa03ffff640 1 --2- 172.21.15.77:0/2542660754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a6153380 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa02c09b6c0 tx=0x7fa02c0973d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b27b6d29408c8ec4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:56.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.214+0000 7fa02a7fc640 1 -- 172.21.15.77:0/2542660754 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c09b270 con 0x5640a6142380 2024-03-16T10:19:56.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.214+0000 7fa02a7fc640 1 -- 172.21.15.77:0/2542660754 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c09be30 con 0x5640a6142380 2024-03-16T10:19:56.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.214+0000 7fa02a7fc640 1 -- 172.21.15.77:0/2542660754 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c09b090 con 0x5640a6142380 2024-03-16T10:19:56.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.215+0000 7fa049d89740 1 -- 172.21.15.77:0/2542660754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 msgr2=0x5640a6153380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.215+0000 7fa049d89740 1 --2- 172.21.15.77:0/2542660754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a6153380 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa02c09b6c0 tx=0x7fa02c0973d0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.215+0000 7fa049d89740 1 -- 172.21.15.77:0/2542660754 shutdown_connections 2024-03-16T10:19:56.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.215+0000 7fa049d89740 1 --2- 172.21.15.77:0/2542660754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a61474f0 0x5640a6140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.215+0000 7fa049d89740 1 --2- 172.21.15.77:0/2542660754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a6153380 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.215+0000 7fa049d89740 1 --2- 172.21.15.77:0/2542660754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a6141990 0x5640a6140540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.215+0000 7fa049d89740 1 -- 172.21.15.77:0/2542660754 >> 172.21.15.77:0/2542660754 conn(0x5640a6109c50 msgr2=0x5640a614a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:56.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.215+0000 7fa049d89740 1 -- 172.21.15.77:0/2542660754 shutdown_connections 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.215+0000 7fa049d89740 1 -- 172.21.15.77:0/2542660754 wait complete. 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.215+0000 7fa049d89740 1 -- start start 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.215+0000 7fa049d89740 1 Processor -- start 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa049d89740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6141990 0x5640a6151100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa049d89740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a6151640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa049d89740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a61474f0 0x5640a6152650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa049d89740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401aa1b0 con 0x5640a6141990 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa049d89740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401a9ae0 con 0x5640a61474f0 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa049d89740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401a6c80 con 0x5640a6142380 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa049d89740 10 client.0 _ll_register_callbacks cb 0x7fa0380093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa044f3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6141990 0x5640a6151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa044f3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6141990 0x5640a6151100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44458/0 (socket says 172.21.15.77:44458) 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa044f3a640 1 -- 172.21.15.77:0/3608549090 learned_addr learned my addr 172.21.15.77:0/3608549090 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:56.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa04573b640 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a61474f0 0x5640a6152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa03ffff640 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a6151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa044f3a640 1 -- 172.21.15.77:0/3608549090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 msgr2=0x5640a6151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa044f3a640 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a6151640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa044f3a640 1 -- 172.21.15.77:0/3608549090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a61474f0 msgr2=0x5640a6152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa044f3a640 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a61474f0 0x5640a6152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa044f3a640 1 -- 172.21.15.77:0/3608549090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640a6152c40 con 0x5640a6141990 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa044f3a640 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6141990 0x5640a6151100 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa034036040 tx=0x7fa03402c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98bbc8c7be08ca06 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa01ffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa01ffff640 1 -- 172.21.15.77:0/3608549090 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa034004140 con 0x5640a6141990 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa01ffff640 1 -- 172.21.15.77:0/3608549090 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa0340042e0 con 0x5640a6141990 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa049d89740 1 -- 172.21.15.77:0/3608549090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa02c09be30 con 0x5640a6141990 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa01ffff640 1 -- 172.21.15.77:0/3608549090 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa0340044e0 con 0x5640a6141990 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.216+0000 7fa049d89740 10 client.16461 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:56.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.217+0000 7fa049d89740 1 -- 172.21.15.77:0/3608549090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5640a61be190 con 0x5640a6141990 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.217+0000 7fa01ffff640 1 -- 172.21.15.77:0/3608549090 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c09be30 con 0x5640a6141990 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.217+0000 7fa01ffff640 1 -- 172.21.15.77:0/3608549090 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fa034004ef0 con 0x5640a6141990 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.217+0000 7fa049d89740 10 client.16461 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.217+0000 7fa049d89740 10 client.16461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.217+0000 7fa049d89740 20 client.16461 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.217+0000 7fa049d89740 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5640a61bec80 0x5640a61c10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.217+0000 7fa049d89740 4 client.16461 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.217+0000 7fa049d89740 1 -- 172.21.15.77:0/3608549090 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5640a60ad960 con 0x5640a61bec80 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.217+0000 7fa03ffff640 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5640a61bec80 0x5640a61c10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.217+0000 7fa01ffff640 10 client.16461 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.217+0000 7fa03ffff640 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5640a61bec80 0x5640a61c10a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=192ab8bfa2e402f8 server_cookie=1d431373c8585b33 in_seq=0 out_seq=0 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.218+0000 7fa01ffff640 1 -- 172.21.15.77:0/3608549090 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5640a60ad960 con 0x5640a61bec80 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.218+0000 7fa01ffff640 10 client.16461 handle_command_reply: tid=0 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.218+0000 7fa049d89740 1 client.16461 shutdown 2024-03-16T10:19:56.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa049d89740 1 -- 172.21.15.77:0/3608549090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6141990 msgr2=0x5640a6151100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa049d89740 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6141990 0x5640a6151100 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa034036040 tx=0x7fa03402c3c0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa049d89740 1 -- 172.21.15.77:0/3608549090 shutdown_connections 2024-03-16T10:19:56.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa049d89740 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5640a61474f0 0x5640a6152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa049d89740 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640a6142380 0x5640a6151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa049d89740 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5640a61bec80 0x5640a61c10a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa049d89740 1 --2- 172.21.15.77:0/3608549090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640a6141990 0x5640a6151100 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa01ffff640 0 client.16461 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:56.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa049d89740 1 -- 172.21.15.77:0/3608549090 >> 172.21.15.77:0/3608549090 conn(0x5640a6109c50 msgr2=0x5640a6140540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:56.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa049d89740 1 -- 172.21.15.77:0/3608549090 shutdown_connections 2024-03-16T10:19:56.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa049d89740 1 -- 172.21.15.77:0/3608549090 wait complete. 2024-03-16T10:19:56.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa049d89740 20 client.16461 trim_cache size 0 max 16384 2024-03-16T10:19:56.226 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:19:56.226 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T10:19:56.226 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "b9f7336e-f2e2-4b8a-9fb4-bff0ee5d1be5", 2024-03-16T10:19:56.226 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T10:19:56.226 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:19:56.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa04719e640 1 -- 172.21.15.77:0/2094243403 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0300a2800 msgr2=0x7fa0300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa04719e640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0300a2800 0x7fa0300a4cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa03801d000 tx=0x7fa03801f000 comp rx=0 tx=0).stop 2024-03-16T10:19:56.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa04719e640 1 -- 172.21.15.77:0/2094243403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa040104c10 msgr2=0x7fa0401980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.219+0000 7fa04719e640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa040104c10 0x7fa0401980e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00e9c0 tx=0x7fa02c00ee80 comp rx=0 tx=0).stop 2024-03-16T10:19:56.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.220+0000 7fa04719e640 1 -- 172.21.15.77:0/2094243403 shutdown_connections 2024-03-16T10:19:56.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.220+0000 7fa04719e640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa040107540 0x7fa04019f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.220+0000 7fa04719e640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa040104c10 0x7fa0401980e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.220+0000 7fa04719e640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0300a2800 0x7fa0300a4cc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.220+0000 7fa04719e640 1 --2- 172.21.15.77:0/2094243403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0401022e0 0x7fa040197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.220+0000 7fa04719e640 1 -- 172.21.15.77:0/2094243403 >> 172.21.15.77:0/2094243403 conn(0x7fa0400fbc70 msgr2=0x7fa040102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:56.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.220+0000 7fa04719e640 1 -- 172.21.15.77:0/2094243403 shutdown_connections 2024-03-16T10:19:56.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.220+0000 7fa04719e640 1 -- 172.21.15.77:0/2094243403 wait complete. 2024-03-16T10:19:56.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:55 smithi077 ceph-mon[32400]: pgmap v435: 97 pgs: 97 active+clean; 226 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 205 op/s 2024-03-16T10:19:56.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:19:56.231 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:19:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:55 smithi193 ceph-mon[36311]: pgmap v435: 97 pgs: 97 active+clean; 226 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 205 op/s 2024-03-16T10:19:56.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.608+0000 7f59f0c63640 1 -- 172.21.15.77:0/3666357908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59ec10e990 msgr2=0x7f59ec110e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.608+0000 7f59f0c63640 1 --2- 172.21.15.77:0/3666357908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59ec10e990 0x7f59ec110e50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009900 tx=0x7f59dc02bc30 comp rx=0 tx=0).stop 2024-03-16T10:19:56.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.613+0000 7f59f0c63640 1 -- 172.21.15.77:0/3666357908 shutdown_connections 2024-03-16T10:19:56.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.613+0000 7f59f0c63640 1 --2- 172.21.15.77:0/3666357908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59ec10e990 0x7f59ec110e50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.613+0000 7f59f0c63640 1 --2- 172.21.15.77:0/3666357908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ec10bed0 0x7f59ec10e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.613+0000 7f59f0c63640 1 --2- 172.21.15.77:0/3666357908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59ec0735f0 0x7f59ec10b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.613+0000 7f59f0c63640 1 -- 172.21.15.77:0/3666357908 >> 172.21.15.77:0/3666357908 conn(0x7f59ec06e600 msgr2=0x7f59ec070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:56.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.614+0000 7f59f0c63640 1 -- 172.21.15.77:0/3666357908 shutdown_connections 2024-03-16T10:19:56.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.614+0000 7f59f0c63640 1 -- 172.21.15.77:0/3666357908 wait complete. 2024-03-16T10:19:56.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.615+0000 7f59f0c63640 1 Processor -- start 2024-03-16T10:19:56.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.615+0000 7f59f0c63640 1 -- start start 2024-03-16T10:19:56.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.616+0000 7f59f0c63640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59ec0735f0 0x7f59ec1a0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.616+0000 7f59f0c63640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59ec10bed0 0x7f59ec1a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.616+0000 7f59f0c63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ec10e990 0x7f59ec1a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.616+0000 7f59f0c63640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec115b20 con 0x7f59ec10e990 2024-03-16T10:19:56.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.616+0000 7f59f0c63640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec1159a0 con 0x7f59ec10bed0 2024-03-16T10:19:56.617 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.616+0000 7f59f0c63640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec115ca0 con 0x7f59ec0735f0 2024-03-16T10:19:56.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.616+0000 7f59e9d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59ec10bed0 0x7f59ec1a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.616+0000 7f59ead9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ec10e990 0x7f59ec1a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.616+0000 7f59e9d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59ec10bed0 0x7f59ec1a0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41866/0 (socket says 172.21.15.77:41866) 2024-03-16T10:19:56.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.616+0000 7f59e9d9b640 1 -- 172.21.15.77:0/669597031 learned_addr learned my addr 172.21.15.77:0/669597031 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:56.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.616+0000 7f59ea59c640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59ec0735f0 0x7f59ec1a0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.617+0000 7f59ea59c640 1 -- 172.21.15.77:0/669597031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59ec10bed0 msgr2=0x7f59ec1a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.617+0000 7f59ea59c640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59ec10bed0 0x7f59ec1a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.618 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.617+0000 7f59ea59c640 1 -- 172.21.15.77:0/669597031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ec10e990 msgr2=0x7f59ec1a75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.617+0000 7f59ea59c640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ec10e990 0x7f59ec1a75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.617+0000 7f59ea59c640 1 -- 172.21.15.77:0/669597031 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ec1a7c60 con 0x7f59ec0735f0 2024-03-16T10:19:56.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.617+0000 7f59e9d9b640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59ec10bed0 0x7f59ec1a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:19:56.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.617+0000 7f59ead9d640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ec10e990 0x7f59ec1a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:56.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.617+0000 7f59ea59c640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59ec0735f0 0x7f59ec1a0040 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f59e000d6d0 tx=0x7f59e000db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.635+0000 7f59db7fe640 1 -- 172.21.15.77:0/669597031 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0017070 con 0x7f59ec0735f0 2024-03-16T10:19:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.635+0000 7f59db7fe640 1 -- 172.21.15.77:0/669597031 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59e0009820 con 0x7f59ec0735f0 2024-03-16T10:19:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.635+0000 7f59f0c63640 1 -- 172.21.15.77:0/669597031 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59ec1a7f50 con 0x7f59ec0735f0 2024-03-16T10:19:56.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.635+0000 7f59f0c63640 1 -- 172.21.15.77:0/669597031 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59ec1a83b0 con 0x7f59ec0735f0 2024-03-16T10:19:56.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.635+0000 7f59db7fe640 1 -- 172.21.15.77:0/669597031 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0004d00 con 0x7f59ec0735f0 2024-03-16T10:19:56.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.641+0000 7f59f384a740 1 Processor -- start 2024-03-16T10:19:56.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.641+0000 7f59f384a740 1 -- start start 2024-03-16T10:19:56.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.641+0000 7f59db7fe640 1 -- 172.21.15.77:0/669597031 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f59e0010460 con 0x7f59ec0735f0 2024-03-16T10:19:56.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.642+0000 7f59f384a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4360 0x563ba34d4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.642+0000 7f59f384a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 0x563ba34d9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.642+0000 7f59f384a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d9ec0 0x563ba34dc350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.642+0000 7f59f384a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563ba3439500 con 0x563ba34d4360 2024-03-16T10:19:56.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.642+0000 7f59f384a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563ba33e1be0 con 0x563ba34d4d50 2024-03-16T10:19:56.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.642+0000 7f59f384a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563ba34355a0 con 0x563ba34d9ec0 2024-03-16T10:19:56.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.643+0000 7f59ea59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 0x563ba34d9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.643+0000 7f59ead9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4360 0x563ba34d4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.643+0000 7f59ea59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 0x563ba34d9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41870/0 (socket says 172.21.15.77:41870) 2024-03-16T10:19:56.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.643+0000 7f59ea59c640 1 -- 172.21.15.77:0/2158770502 learned_addr learned my addr 172.21.15.77:0/2158770502 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:56.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.643+0000 7f59e9d9b640 1 --2- 172.21.15.77:0/2158770502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d9ec0 0x563ba34dc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.654+0000 7f59db7fe640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f59d40a2a10 0x7f59d40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.654+0000 7f59db7fe640 1 -- 172.21.15.77:0/669597031 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f59e00cf2f0 con 0x7f59ec0735f0 2024-03-16T10:19:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.654+0000 7f59ead9d640 1 -- 172.21.15.77:0/2158770502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d9ec0 msgr2=0x563ba34dc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.654+0000 7f59ead9d640 1 --2- 172.21.15.77:0/2158770502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d9ec0 0x563ba34dc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.654+0000 7f59ead9d640 1 -- 172.21.15.77:0/2158770502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 msgr2=0x563ba34d9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.654+0000 7f59ead9d640 1 --2- 172.21.15.77:0/2158770502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 0x563ba34d9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.654+0000 7f59ead9d640 1 -- 172.21.15.77:0/2158770502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ba33f54b0 con 0x563ba34d4360 2024-03-16T10:19:56.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.655+0000 7f59e9d9b640 1 --2- 172.21.15.77:0/2158770502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d9ec0 0x563ba34dc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:19:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.655+0000 7f59ead9d640 1 --2- 172.21.15.77:0/2158770502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4360 0x563ba34d4780 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009850 tx=0x7f59dc02c170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed8a802489c46b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:56.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.656+0000 7f59ead9d640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f59d40a2a10 0x7f59d40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.656+0000 7f59ead9d640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f59d40a2a10 0x7f59d40a4ed0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f59dc004650 tx=0x7f59dc034a80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:56.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.661+0000 7f59d8ff9640 1 -- 172.21.15.77:0/2158770502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc02cc20 con 0x563ba34d4360 2024-03-16T10:19:56.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.661+0000 7f59d8ff9640 1 -- 172.21.15.77:0/2158770502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59dc0035f0 con 0x563ba34d4360 2024-03-16T10:19:56.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.662+0000 7f59f384a740 1 -- 172.21.15.77:0/2158770502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4360 msgr2=0x563ba34d4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.662+0000 7f59f384a740 1 --2- 172.21.15.77:0/2158770502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4360 0x563ba34d4780 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009850 tx=0x7f59dc02c170 comp rx=0 tx=0).stop 2024-03-16T10:19:56.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.668+0000 7f59f384a740 1 -- 172.21.15.77:0/2158770502 shutdown_connections 2024-03-16T10:19:56.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.668+0000 7f59f384a740 1 --2- 172.21.15.77:0/2158770502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d9ec0 0x563ba34dc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.668+0000 7f59f384a740 1 --2- 172.21.15.77:0/2158770502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 0x563ba34d9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.668+0000 7f59f384a740 1 --2- 172.21.15.77:0/2158770502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4360 0x563ba34d4780 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.668 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.668+0000 7f59f384a740 1 -- 172.21.15.77:0/2158770502 >> 172.21.15.77:0/2158770502 conn(0x563ba34cf8d0 msgr2=0x563ba34cfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:56.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.672+0000 7f59f384a740 1 -- 172.21.15.77:0/2158770502 shutdown_connections 2024-03-16T10:19:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.676+0000 7f59f384a740 1 -- 172.21.15.77:0/2158770502 wait complete. 2024-03-16T10:19:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.676+0000 7f59f384a740 1 -- start start 2024-03-16T10:19:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.676+0000 7f59f384a740 1 Processor -- start 2024-03-16T10:19:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.677+0000 7f59f384a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4360 0x563ba34e2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.677 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.677+0000 7f59f384a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4d50 0x563ba34e30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.677+0000 7f59f384a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d9ec0 0x563ba34ea130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.677+0000 7f59f384a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563ba34e1270 con 0x563ba34d4d50 2024-03-16T10:19:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.677+0000 7f59f384a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563ba33f5160 con 0x563ba34d4360 2024-03-16T10:19:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.677+0000 7f59f384a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563ba34e13f0 con 0x563ba34d9ec0 2024-03-16T10:19:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.677+0000 7f59f384a740 10 client.0 _ll_register_callbacks cb 0x563ba33d9150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.677+0000 7f59e9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4d50 0x563ba34e30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.677+0000 7f59e9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4d50 0x563ba34e30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44494/0 (socket says 172.21.15.77:44494) 2024-03-16T10:19:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.677+0000 7f59e9d9b640 1 -- 172.21.15.77:0/2571897238 learned_addr learned my addr 172.21.15.77:0/2571897238 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.678+0000 7f59ead9d640 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d9ec0 0x563ba34ea130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.678+0000 7f59ea59c640 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4360 0x563ba34e2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.678+0000 7f59ead9d640 1 -- 172.21.15.77:0/2571897238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4360 msgr2=0x563ba34e2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.678+0000 7f59ead9d640 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4360 0x563ba34e2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.678+0000 7f59ead9d640 1 -- 172.21.15.77:0/2571897238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4d50 msgr2=0x563ba34e30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.678+0000 7f59ead9d640 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4d50 0x563ba34e30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.678+0000 7f59ead9d640 1 -- 172.21.15.77:0/2571897238 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ba32177d0 con 0x563ba34d9ec0 2024-03-16T10:19:56.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.678+0000 7f59e9d9b640 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4d50 0x563ba34e30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:56.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.678+0000 7f59ead9d640 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d9ec0 0x563ba34ea130 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x563ba34d55b0 tx=0x7f59dc046000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93444716021f30f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:56.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.685+0000 7f59d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:19:56.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.685+0000 7f59d1ffb640 1 -- 172.21.15.77:0/2571897238 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc059070 con 0x563ba34d9ec0 2024-03-16T10:19:56.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.686+0000 7f59f384a740 1 -- 172.21.15.77:0/2571897238 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59dc0035f0 con 0x563ba34d9ec0 2024-03-16T10:19:56.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.686+0000 7f59d1ffb640 1 -- 172.21.15.77:0/2571897238 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f59dc052460 con 0x563ba34d9ec0 2024-03-16T10:19:56.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.690+0000 7f59d1ffb640 1 -- 172.21.15.77:0/2571897238 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f59dc065050 con 0x563ba34d9ec0 2024-03-16T10:19:56.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.690+0000 7f59f384a740 10 client.26236 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:56.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.690+0000 7f59f384a740 1 -- 172.21.15.77:0/2571897238 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563ba314d3f0 con 0x563ba34d9ec0 2024-03-16T10:19:56.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.691+0000 7f59d1ffb640 1 -- 172.21.15.77:0/2571897238 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59dc0035f0 con 0x563ba34d9ec0 2024-03-16T10:19:56.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.691+0000 7f59d1ffb640 1 -- 172.21.15.77:0/2571897238 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f59dc050430 con 0x563ba34d9ec0 2024-03-16T10:19:56.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.691+0000 7f59f384a740 10 client.26236 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:56.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.691+0000 7f59f384a740 10 client.26236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:56.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.691+0000 7f59f384a740 20 client.26236 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:56.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.691+0000 7f59f384a740 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563ba34ee850 0x563ba34f0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.691+0000 7f59f384a740 4 client.26236 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:19:56.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.691+0000 7f59f384a740 1 -- 172.21.15.77:0/2571897238 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ba3446a40 con 0x563ba34ee850 2024-03-16T10:19:56.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.691+0000 7f59ea59c640 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563ba34ee850 0x563ba34f0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.692+0000 7f59ea59c640 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563ba34ee850 0x563ba34f0c70 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bfa22b67612c0c9 server_cookie=4d518fbf1623a01e in_seq=0 out_seq=0 2024-03-16T10:19:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.693+0000 7f59d1ffb640 10 client.26236 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.693+0000 7f59d1ffb640 1 -- 172.21.15.77:0/2571897238 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ba3446a40 con 0x563ba34ee850 2024-03-16T10:19:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.693+0000 7f59d1ffb640 10 client.26236 handle_command_reply: tid=0 2024-03-16T10:19:56.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.693+0000 7f59f384a740 1 client.26236 shutdown 2024-03-16T10:19:56.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.700+0000 7f59f384a740 1 -- 172.21.15.77:0/2571897238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d9ec0 msgr2=0x563ba34ea130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.700+0000 7f59f384a740 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d9ec0 0x563ba34ea130 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x563ba34d55b0 tx=0x7f59dc046000 comp rx=0 tx=0).stop 2024-03-16T10:19:56.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.701+0000 7f59f384a740 1 -- 172.21.15.77:0/2571897238 shutdown_connections 2024-03-16T10:19:56.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.701+0000 7f59f384a740 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d9ec0 0x563ba34ea130 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.701+0000 7f59f384a740 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563ba34ee850 0x563ba34f0c70 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.701+0000 7f59f384a740 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d4d50 0x563ba34e30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.701+0000 7f59f384a740 1 --2- 172.21.15.77:0/2571897238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4360 0x563ba34e2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.701+0000 7f59d1ffb640 0 client.26236 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:56.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.701+0000 7f59f384a740 1 -- 172.21.15.77:0/2571897238 >> 172.21.15.77:0/2571897238 conn(0x563ba34cf8d0 msgr2=0x563ba34ceb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:56.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.701+0000 7f59f384a740 1 -- 172.21.15.77:0/2571897238 shutdown_connections 2024-03-16T10:19:56.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.701+0000 7f59f384a740 1 -- 172.21.15.77:0/2571897238 wait complete. 2024-03-16T10:19:56.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.701+0000 7f59f384a740 20 client.26236 trim_cache size 0 max 16384 2024-03-16T10:19:56.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.906+0000 7f59f384a740 1 Processor -- start 2024-03-16T10:19:56.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.907+0000 7f59f384a740 1 -- start start 2024-03-16T10:19:56.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.911+0000 7f59f384a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d4360 0x563ba34e1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.911+0000 7f59f384a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 0x563ba34ea0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.911+0000 7f59f384a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 0x563ba34e23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.911+0000 7f59ea59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d4360 0x563ba34e1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.911+0000 7f59ea59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d4360 0x563ba34e1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44470/0 (socket says 172.21.15.77:44470) 2024-03-16T10:19:56.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.911+0000 7f59ea59c640 1 -- 172.21.15.77:0/320157110 learned_addr learned my addr 172.21.15.77:0/320157110 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:56.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.911+0000 7f59ead9d640 1 --2- 172.21.15.77:0/320157110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 0x563ba34e23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.912+0000 7f59f384a740 1 -- 172.21.15.77:0/320157110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc00e470 con 0x563ba34d9ec0 2024-03-16T10:19:56.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.912+0000 7f59f384a740 1 -- 172.21.15.77:0/320157110 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc010130 con 0x563ba34d4d50 2024-03-16T10:19:56.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.912+0000 7f59f384a740 1 -- 172.21.15.77:0/320157110 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc00eb40 con 0x563ba34d4360 2024-03-16T10:19:56.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.912+0000 7f59ea59c640 1 -- 172.21.15.77:0/320157110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 msgr2=0x563ba34ea0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T10:19:56.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.912+0000 7f59ea59c640 1 --2- 172.21.15.77:0/320157110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 0x563ba34ea0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.912+0000 7f59ea59c640 1 -- 172.21.15.77:0/320157110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 msgr2=0x563ba34e23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:19:56.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.912+0000 7f59ea59c640 1 --2- 172.21.15.77:0/320157110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 0x563ba34e23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.912+0000 7f59ea59c640 1 -- 172.21.15.77:0/320157110 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ba310d0f0 con 0x563ba34d4360 2024-03-16T10:19:56.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.912+0000 7f59ead9d640 1 --2- 172.21.15.77:0/320157110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 0x563ba34e23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:19:56.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.913+0000 7f59ea59c640 1 --2- 172.21.15.77:0/320157110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d4360 0x563ba34e1ea0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f59e009b980 tx=0x7f59e00cccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab05724eb913f2ce server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.921+0000 7f59d3fff640 1 -- 172.21.15.77:0/320157110 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0020040 con 0x563ba34d4360 2024-03-16T10:19:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.921+0000 7f59d3fff640 1 -- 172.21.15.77:0/320157110 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59e009c380 con 0x563ba34d4360 2024-03-16T10:19:56.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.921+0000 7f59d3fff640 1 -- 172.21.15.77:0/320157110 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e00099d0 con 0x563ba34d4360 2024-03-16T10:19:56.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.926+0000 7f59f384a740 1 -- 172.21.15.77:0/320157110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d4360 msgr2=0x563ba34e1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.926+0000 7f59f384a740 1 --2- 172.21.15.77:0/320157110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d4360 0x563ba34e1ea0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f59e009b980 tx=0x7f59e00cccb0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.926+0000 7f59f384a740 1 -- 172.21.15.77:0/320157110 shutdown_connections 2024-03-16T10:19:56.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.926+0000 7f59f384a740 1 --2- 172.21.15.77:0/320157110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 0x563ba34e23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.926+0000 7f59f384a740 1 --2- 172.21.15.77:0/320157110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 0x563ba34ea0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.926+0000 7f59f384a740 1 --2- 172.21.15.77:0/320157110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d4360 0x563ba34e1ea0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.926+0000 7f59f384a740 1 -- 172.21.15.77:0/320157110 >> 172.21.15.77:0/320157110 conn(0x563ba34cf8d0 msgr2=0x563ba34caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:56.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.931+0000 7f59f384a740 1 -- 172.21.15.77:0/320157110 shutdown_connections 2024-03-16T10:19:56.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.931+0000 7f59f384a740 1 -- 172.21.15.77:0/320157110 wait complete. 2024-03-16T10:19:56.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.931+0000 7f59f384a740 1 -- start start 2024-03-16T10:19:56.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.931+0000 7f59f384a740 1 Processor -- start 2024-03-16T10:19:56.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.932+0000 7f59f384a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d4360 0x563ba34e6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.932 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.932+0000 7f59f384a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 0x563ba34e72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.932+0000 7f59f384a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 0x563ba3552b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.932+0000 7f59ead9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 0x563ba3552b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.932+0000 7f59ead9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 0x563ba3552b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44504/0 (socket says 172.21.15.77:44504) 2024-03-16T10:19:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.932+0000 7f59ead9d640 1 -- 172.21.15.77:0/3599410628 learned_addr learned my addr 172.21.15.77:0/3599410628 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:19:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.932+0000 7f59f384a740 1 -- 172.21.15.77:0/3599410628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc00f9a0 con 0x563ba34d9ec0 2024-03-16T10:19:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.933+0000 7f59f384a740 1 -- 172.21.15.77:0/3599410628 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc00f2d0 con 0x563ba34d4d50 2024-03-16T10:19:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.933+0000 7f59f384a740 1 -- 172.21.15.77:0/3599410628 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc00c450 con 0x563ba34d4360 2024-03-16T10:19:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.933+0000 7f59ead9d640 1 -- 172.21.15.77:0/3599410628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d4360 msgr2=0x563ba34e6db0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T10:19:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.933+0000 7f59ead9d640 1 --2- 172.21.15.77:0/3599410628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d4360 0x563ba34e6db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.933+0000 7f59ead9d640 1 -- 172.21.15.77:0/3599410628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 msgr2=0x563ba34e72f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T10:19:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.933+0000 7f59ead9d640 1 --2- 172.21.15.77:0/3599410628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 0x563ba34e72f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.933+0000 7f59ead9d640 1 -- 172.21.15.77:0/3599410628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ba34009d0 con 0x563ba34d9ec0 2024-03-16T10:19:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.933+0000 7f59ead9d640 1 --2- 172.21.15.77:0/3599410628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 0x563ba3552b60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0509a0 tx=0x7f59dc050660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ec7da528a97cc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:19:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.934+0000 7f59f384a740 10 client.0 _ll_register_callbacks cb 0x7f59e0009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:19:56.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.934+0000 7f59f384a740 1 -- 172.21.15.77:0/3599410628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59e009c380 con 0x563ba34d9ec0 2024-03-16T10:19:56.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.942+0000 7f59d1ffb640 10 client.16479 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:19:56.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.942+0000 7f59d1ffb640 1 -- 172.21.15.77:0/3599410628 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc059070 con 0x563ba34d9ec0 2024-03-16T10:19:56.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.942+0000 7f59d1ffb640 1 -- 172.21.15.77:0/3599410628 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f59dc050c20 con 0x563ba34d9ec0 2024-03-16T10:19:56.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.942+0000 7f59d1ffb640 1 -- 172.21.15.77:0/3599410628 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f59dc009780 con 0x563ba34d9ec0 2024-03-16T10:19:56.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.942+0000 7f59f384a740 10 client.16479 fetch_fsmap learned FSMap version 32 2024-03-16T10:19:56.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.942+0000 7f59f384a740 1 -- 172.21.15.77:0/3599410628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=32}) -- 0x563ba3553d70 con 0x563ba34d9ec0 2024-03-16T10:19:56.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.943+0000 7f59d1ffb640 1 -- 172.21.15.77:0/3599410628 <== mon.0 v2:172.21.15.77:3300/0 4 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f59dc073050 con 0x563ba34d9ec0 2024-03-16T10:19:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.943+0000 7f59f384a740 10 client.16479 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:19:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.943+0000 7f59f384a740 10 client.16479 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:19:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.943+0000 7f59f384a740 20 client.16479 populate_metadata read hostname 'smithi077' 2024-03-16T10:19:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.943+0000 7f59f384a740 1 --2- 172.21.15.77:0/3599410628 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563ba3554640 0x563ba3556b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:19:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.943+0000 7f59f384a740 4 client.16479 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:19:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.943+0000 7f59f384a740 1 -- 172.21.15.77:0/3599410628 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563ba3440d90 con 0x563ba3554640 2024-03-16T10:19:56.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.943+0000 7f59ea59c640 1 --2- 172.21.15.77:0/3599410628 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563ba3554640 0x563ba3556b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:19:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.944+0000 7f59ea59c640 1 --2- 172.21.15.77:0/3599410628 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563ba3554640 0x563ba3556b00 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa054679d89a8f8 server_cookie=341ba56cb5a55142 in_seq=0 out_seq=0 2024-03-16T10:19:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.944+0000 7f59d1ffb640 10 client.16479 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.944+0000 7f59d1ffb640 1 -- 172.21.15.77:0/3599410628 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563ba3440d90 con 0x563ba3554640 2024-03-16T10:19:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.944+0000 7f59d1ffb640 10 client.16479 handle_command_reply: tid=0 2024-03-16T10:19:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.944+0000 7f59f384a740 1 client.16479 shutdown 2024-03-16T10:19:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.945+0000 7f59f384a740 1 -- 172.21.15.77:0/3599410628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 msgr2=0x563ba3552b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.945+0000 7f59f384a740 1 --2- 172.21.15.77:0/3599410628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 0x563ba3552b60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0509a0 tx=0x7f59dc050660 comp rx=0 tx=0).stop 2024-03-16T10:19:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.945+0000 7f59f384a740 1 -- 172.21.15.77:0/3599410628 shutdown_connections 2024-03-16T10:19:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.945+0000 7f59f384a740 1 --2- 172.21.15.77:0/3599410628 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563ba3554640 0x563ba3556b00 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.945+0000 7f59f384a740 1 --2- 172.21.15.77:0/3599410628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563ba34d9ec0 0x563ba3552b60 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.945+0000 7f59f384a740 1 --2- 172.21.15.77:0/3599410628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ba34d4d50 0x563ba34e72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.945+0000 7f59f384a740 1 --2- 172.21.15.77:0/3599410628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ba34d4360 0x563ba34e6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.945+0000 7f59f384a740 1 -- 172.21.15.77:0/3599410628 >> 172.21.15.77:0/3599410628 conn(0x563ba34cf8d0 msgr2=0x563ba34e1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.945+0000 7f59d1ffb640 0 client.16479 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:19:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.945+0000 7f59f384a740 1 -- 172.21.15.77:0/3599410628 shutdown_connections 2024-03-16T10:19:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.946+0000 7f59f384a740 1 -- 172.21.15.77:0/3599410628 wait complete. 2024-03-16T10:19:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.946+0000 7f59f384a740 20 client.16479 trim_cache size 0 max 16384 2024-03-16T10:19:56.948 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:19:56.948 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:19:56.948 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:19:56.948 INFO:teuthology.orchestra.run.smithi077.stdout: "b9f7336e-f2e2-4b8a-9fb4-bff0ee5d1be5": { 2024-03-16T10:19:56.948 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:19:56.948 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "b9f7336e-f2e2-4b8a-9fb4-bff0ee5d1be5", 2024-03-16T10:19:56.948 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:19:56.948 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:19:56.948 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:19:56.948 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:19:56.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.954+0000 7f59f0c63640 1 -- 172.21.15.77:0/669597031 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f59d40a2a10 msgr2=0x7f59d40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.954+0000 7f59f0c63640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f59d40a2a10 0x7f59d40a4ed0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f59dc004650 tx=0x7f59dc034a80 comp rx=0 tx=0).stop 2024-03-16T10:19:56.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.954+0000 7f59f0c63640 1 -- 172.21.15.77:0/669597031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59ec0735f0 msgr2=0x7f59ec1a0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:19:56.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.954+0000 7f59f0c63640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59ec0735f0 0x7f59ec1a0040 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f59e000d6d0 tx=0x7f59e000db90 comp rx=0 tx=0).stop 2024-03-16T10:19:56.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.954+0000 7f59f0c63640 1 -- 172.21.15.77:0/669597031 shutdown_connections 2024-03-16T10:19:56.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.954+0000 7f59f0c63640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59ec10e990 0x7f59ec1a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.954+0000 7f59f0c63640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59ec10bed0 0x7f59ec1a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.954+0000 7f59f0c63640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f59d40a2a10 0x7f59d40a4ed0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.954+0000 7f59f0c63640 1 --2- 172.21.15.77:0/669597031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59ec0735f0 0x7f59ec1a0040 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:19:56.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.954+0000 7f59f0c63640 1 -- 172.21.15.77:0/669597031 >> 172.21.15.77:0/669597031 conn(0x7f59ec06e600 msgr2=0x7f59ec10b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:19:56.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.954+0000 7f59f0c63640 1 -- 172.21.15.77:0/669597031 shutdown_connections 2024-03-16T10:19:56.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:19:56.955+0000 7f59f0c63640 1 -- 172.21.15.77:0/669597031 wait complete. 2024-03-16T10:19:57.019 INFO:tasks.cephfs.filesystem:scrub status for tag:b9f7336e-f2e2-4b8a-9fb4-bff0ee5d1be5 - {'path': '/', 'tag': 'b9f7336e-f2e2-4b8a-9fb4-bff0ee5d1be5', 'options': 'recursive,force'} 2024-03-16T10:19:57.103 INFO:tasks.workunit.client.0.smithi077.stdout: 56 1484538 514 972611 567 844121 1032 2024-03-16T10:19:57.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:56 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:19:57.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:56 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:19:57.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:56 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:19:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:57 smithi192 ceph-mon[36354]: scrub queued for path: / 2024-03-16T10:19:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:57 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:57 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:19:58.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:57 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:58.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:57 smithi192 ceph-mon[36354]: pgmap v436: 97 pgs: 97 active+clean; 225 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-03-16T10:19:58.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:57 smithi077 ceph-mon[32400]: scrub queued for path: / 2024-03-16T10:19:58.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:57 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:58.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:57 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:19:58.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:57 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:58.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:57 smithi077 ceph-mon[32400]: pgmap v436: 97 pgs: 97 active+clean; 225 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-03-16T10:19:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:57 smithi193 ceph-mon[36311]: scrub queued for path: / 2024-03-16T10:19:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:57 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:57 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:19:58.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:57 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:58.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:57 smithi193 ceph-mon[36311]: pgmap v436: 97 pgs: 97 active+clean; 225 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-03-16T10:19:59.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:58 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:19:59.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:58 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:59.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:58 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:19:59.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:58 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:19:59.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:58 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:59.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:58 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:19:59.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:58 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:59.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:58 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:19:59.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:58 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:19:59.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:58 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:59.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:58 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:19:59.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:58 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:19:59.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:58 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:19:59.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:58 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:19:59.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:58 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:20:00.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:59 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:20:00.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:19:59 smithi192 ceph-mon[36354]: pgmap v437: 97 pgs: 97 active+clean; 228 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 148 op/s 2024-03-16T10:20:00.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:59 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:20:00.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:19:59 smithi077 ceph-mon[32400]: pgmap v437: 97 pgs: 97 active+clean; 228 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 148 op/s 2024-03-16T10:20:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:59 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:20:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:19:59 smithi193 ceph-mon[36311]: pgmap v437: 97 pgs: 97 active+clean; 228 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 148 op/s 2024-03-16T10:20:01.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:00 smithi192 ceph-mon[36354]: overall HEALTH_OK 2024-03-16T10:20:01.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:00 smithi077 ceph-mon[32400]: overall HEALTH_OK 2024-03-16T10:20:01.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:00 smithi193 ceph-mon[36311]: overall HEALTH_OK 2024-03-16T10:20:02.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:01 smithi192 ceph-mon[36354]: pgmap v438: 97 pgs: 97 active+clean; 227 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-03-16T10:20:02.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:01 smithi077 ceph-mon[32400]: pgmap v438: 97 pgs: 97 active+clean; 227 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-03-16T10:20:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:01 smithi193 ceph-mon[36311]: pgmap v438: 97 pgs: 97 active+clean; 227 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-03-16T10:20:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.127+0000 7f4aca4fe640 1 -- 172.21.15.77:0/4275981418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac41094d0 msgr2=0x7f4ac410e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.127+0000 7f4aca4fe640 1 --2- 172.21.15.77:0/4275981418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac41094d0 0x7f4ac410e140 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4abc01acb0 tx=0x7f4abc03d170 comp rx=0 tx=0).stop 2024-03-16T10:20:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.127+0000 7f4aca4fe640 1 -- 172.21.15.77:0/4275981418 shutdown_connections 2024-03-16T10:20:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.127+0000 7f4aca4fe640 1 --2- 172.21.15.77:0/4275981418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac41094d0 0x7f4ac410e140 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.127+0000 7f4aca4fe640 1 --2- 172.21.15.77:0/4275981418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac40733f0 0x7f4ac4108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.127+0000 7f4aca4fe640 1 --2- 172.21.15.77:0/4275981418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ac4072a90 0x7f4ac4072eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.127+0000 7f4aca4fe640 1 -- 172.21.15.77:0/4275981418 >> 172.21.15.77:0/4275981418 conn(0x7f4ac406e650 msgr2=0x7f4ac4070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:04.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.132+0000 7f4aca4fe640 1 -- 172.21.15.77:0/4275981418 shutdown_connections 2024-03-16T10:20:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.139+0000 7f4aca4fe640 1 -- 172.21.15.77:0/4275981418 wait complete. 2024-03-16T10:20:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.140+0000 7f4aca4fe640 1 Processor -- start 2024-03-16T10:20:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.140+0000 7f4aca4fe640 1 -- start start 2024-03-16T10:20:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.140+0000 7f4aca4fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ac4072a90 0x7f4ac419df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.140+0000 7f4aca4fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac40733f0 0x7f4ac419e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.140+0000 7f4aca4fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac41094d0 0x7f4ac41c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.140+0000 7f4aca4fe640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac4111900 con 0x7f4ac4072a90 2024-03-16T10:20:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.140+0000 7f4aca4fe640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac4111780 con 0x7f4ac41094d0 2024-03-16T10:20:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.140+0000 7f4aca4fe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac4111a80 con 0x7f4ac40733f0 2024-03-16T10:20:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.141+0000 7f4ac94fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ac4072a90 0x7f4ac419df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.141+0000 7f4ac94fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ac4072a90 0x7f4ac419df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52584/0 (socket says 172.21.15.77:52584) 2024-03-16T10:20:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.141+0000 7f4ac94fc640 1 -- 172.21.15.77:0/3359221351 learned_addr learned my addr 172.21.15.77:0/3359221351 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.142+0000 7f4ac9cfd640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac41094d0 0x7f4ac41c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.143+0000 7f4ac94fc640 1 -- 172.21.15.77:0/3359221351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac40733f0 msgr2=0x7f4ac419e490 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T10:20:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.143+0000 7f4ac94fc640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac40733f0 0x7f4ac419e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.143+0000 7f4ac94fc640 1 -- 172.21.15.77:0/3359221351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac41094d0 msgr2=0x7f4ac41c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.143+0000 7f4ac94fc640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac41094d0 0x7f4ac41c7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.143+0000 7f4ac94fc640 1 -- 172.21.15.77:0/3359221351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac41c7bf0 con 0x7f4ac4072a90 2024-03-16T10:20:04.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.145+0000 7f4ac94fc640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ac4072a90 0x7f4ac419df50 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000d950 tx=0x7f4ac000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:04.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.146+0000 7f4aba7fc640 1 -- 172.21.15.77:0/3359221351 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0017070 con 0x7f4ac4072a90 2024-03-16T10:20:04.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.146+0000 7f4aba7fc640 1 -- 172.21.15.77:0/3359221351 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ac0009c30 con 0x7f4ac4072a90 2024-03-16T10:20:04.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.146+0000 7f4aca4fe640 1 -- 172.21.15.77:0/3359221351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac41c7ee0 con 0x7f4ac4072a90 2024-03-16T10:20:04.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.146+0000 7f4aba7fc640 1 -- 172.21.15.77:0/3359221351 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac000f690 con 0x7f4ac4072a90 2024-03-16T10:20:04.149 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.147+0000 7f4aca4fe640 1 -- 172.21.15.77:0/3359221351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac41c8420 con 0x7f4ac4072a90 2024-03-16T10:20:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.153+0000 7f4aba7fc640 1 -- 172.21.15.77:0/3359221351 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f4ac00109d0 con 0x7f4ac4072a90 2024-03-16T10:20:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.154+0000 7f4aba7fc640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4ab00a2930 0x7f4ab00a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.154+0000 7f4aba7fc640 1 -- 172.21.15.77:0/3359221351 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f4ac00ce000 con 0x7f4ac4072a90 2024-03-16T10:20:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.156+0000 7f4ace420740 1 Processor -- start 2024-03-16T10:20:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.156+0000 7f4ace420740 1 -- start start 2024-03-16T10:20:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.156+0000 7f4ace420740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d989327500 0x55d989327920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.156+0000 7f4ace420740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327ef0 0x55d98932c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.156+0000 7f4ace420740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d98932d060 0x55d98932f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.156+0000 7f4ace420740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d98928ca40 con 0x55d98932d060 2024-03-16T10:20:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.156+0000 7f4ace420740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d989235920 con 0x55d989327ef0 2024-03-16T10:20:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.156+0000 7f4ace420740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d989288ae0 con 0x55d989327500 2024-03-16T10:20:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.157+0000 7f4ac94fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d989327500 0x55d989327920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.157+0000 7f4ac94fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d989327500 0x55d989327920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:40000/0 (socket says 172.21.15.77:40000) 2024-03-16T10:20:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.157+0000 7f4ac94fc640 1 -- 172.21.15.77:0/2381482734 learned_addr learned my addr 172.21.15.77:0/2381482734 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.157+0000 7f4ac8cfb640 1 --2- 172.21.15.77:0/2381482734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327ef0 0x55d98932c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.157+0000 7f4ac8cfb640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4ab00a2930 0x7f4ab00a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.157+0000 7f4ac94fc640 1 -- 172.21.15.77:0/2381482734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327ef0 msgr2=0x55d98932c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.157+0000 7f4ac94fc640 1 --2- 172.21.15.77:0/2381482734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327ef0 0x55d98932c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.157+0000 7f4ac94fc640 1 -- 172.21.15.77:0/2381482734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d98932d060 msgr2=0x55d98932f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.157+0000 7f4ac94fc640 1 --2- 172.21.15.77:0/2381482734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d98932d060 0x55d98932f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.157+0000 7f4ac94fc640 1 -- 172.21.15.77:0/2381482734 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d98924d460 con 0x55d989327500 2024-03-16T10:20:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.159+0000 7f4ac8cfb640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4ab00a2930 0x7f4ab00a4df0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x55d9893282f0 tx=0x7f4ab40068b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.164+0000 7f4ac94fc640 1 --2- 172.21.15.77:0/2381482734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d989327500 0x55d989327920 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0097360 tx=0x7f4ac009b090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1334241a3b8c5483 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:04.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:03 smithi192 ceph-mon[36354]: pgmap v439: 97 pgs: 97 active+clean; 232 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-03-16T10:20:04.170 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.170+0000 7f4aaf7fe640 1 -- 172.21.15.77:0/2381482734 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac00b3070 con 0x55d989327500 2024-03-16T10:20:04.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.170+0000 7f4aaf7fe640 1 -- 172.21.15.77:0/2381482734 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ac00a3130 con 0x55d989327500 2024-03-16T10:20:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.174+0000 7f4aaf7fe640 1 -- 172.21.15.77:0/2381482734 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0020040 con 0x55d989327500 2024-03-16T10:20:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.175+0000 7f4ace420740 1 -- 172.21.15.77:0/2381482734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d989327500 msgr2=0x55d989327920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.175+0000 7f4ace420740 1 --2- 172.21.15.77:0/2381482734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d989327500 0x55d989327920 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0097360 tx=0x7f4ac009b090 comp rx=0 tx=0).stop 2024-03-16T10:20:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.175+0000 7f4ace420740 1 -- 172.21.15.77:0/2381482734 shutdown_connections 2024-03-16T10:20:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.175+0000 7f4ace420740 1 --2- 172.21.15.77:0/2381482734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d98932d060 0x55d98932f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.175+0000 7f4ace420740 1 --2- 172.21.15.77:0/2381482734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327ef0 0x55d98932c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.175+0000 7f4ace420740 1 --2- 172.21.15.77:0/2381482734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d989327500 0x55d989327920 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.176 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.175+0000 7f4ace420740 1 -- 172.21.15.77:0/2381482734 >> 172.21.15.77:0/2381482734 conn(0x55d989322a70 msgr2=0x55d989322e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:04.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.175+0000 7f4ace420740 1 -- 172.21.15.77:0/2381482734 shutdown_connections 2024-03-16T10:20:04.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.175+0000 7f4ace420740 1 -- 172.21.15.77:0/2381482734 wait complete. 2024-03-16T10:20:04.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.176+0000 7f4ace420740 1 -- start start 2024-03-16T10:20:04.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.176+0000 7f4ace420740 1 Processor -- start 2024-03-16T10:20:04.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.176+0000 7f4ace420740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327500 0x55d9893538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.176+0000 7f4ace420740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327ef0 0x55d989353df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.176+0000 7f4ace420740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d98932d060 0x55d98935adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.176+0000 7f4ace420740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d989352020 con 0x55d989327ef0 2024-03-16T10:20:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.176+0000 7f4ace420740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d989248d70 con 0x55d989327500 2024-03-16T10:20:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.176+0000 7f4ace420740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d9893521a0 con 0x55d98932d060 2024-03-16T10:20:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.177+0000 7f4ace420740 10 client.0 _ll_register_callbacks cb 0x55d988fa0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.177+0000 7f4ac94fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327500 0x55d9893538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.177+0000 7f4ac94fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327500 0x55d9893538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:59924/0 (socket says 172.21.15.77:59924) 2024-03-16T10:20:04.181 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.177+0000 7f4ac94fc640 1 -- 172.21.15.77:0/2704031190 learned_addr learned my addr 172.21.15.77:0/2704031190 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:04.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.177+0000 7f4ac94fc640 1 -- 172.21.15.77:0/2704031190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d98932d060 msgr2=0x55d98935adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:04.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.177+0000 7f4ac94fc640 1 --2- 172.21.15.77:0/2704031190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d98932d060 0x55d98935adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.177+0000 7f4ac94fc640 1 -- 172.21.15.77:0/2704031190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327ef0 msgr2=0x55d989353df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:04.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.177+0000 7f4ac94fc640 1 --2- 172.21.15.77:0/2704031190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327ef0 0x55d989353df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.177+0000 7f4ac94fc640 1 -- 172.21.15.77:0/2704031190 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d988fa1210 con 0x55d989327500 2024-03-16T10:20:04.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.178+0000 7f4ac94fc640 1 --2- 172.21.15.77:0/2704031190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327500 0x55d9893538b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ac009a690 tx=0x7f4ac00cbb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b851a23d0da0e6e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:04.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.178+0000 7f4aacff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:20:04.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.179+0000 7f4aacff9640 1 -- 172.21.15.77:0/2704031190 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0020050 con 0x55d989327500 2024-03-16T10:20:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.179+0000 7f4ace420740 1 -- 172.21.15.77:0/2704031190 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ac00a3130 con 0x55d989327500 2024-03-16T10:20:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.179+0000 7f4aacff9640 1 -- 172.21.15.77:0/2704031190 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f4ac00966e0 con 0x55d989327500 2024-03-16T10:20:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.179+0000 7f4aacff9640 1 -- 172.21.15.77:0/2704031190 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4ac00b3070 con 0x55d989327500 2024-03-16T10:20:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.180+0000 7f4ace420740 10 client.26249 fetch_fsmap learned FSMap version 32 2024-03-16T10:20:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.180+0000 7f4ace420740 1 -- 172.21.15.77:0/2704031190 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d98906d620 con 0x55d989327500 2024-03-16T10:20:04.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.180+0000 7f4aacff9640 1 -- 172.21.15.77:0/2704031190 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ac00a3130 con 0x55d989327500 2024-03-16T10:20:04.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.181+0000 7f4aacff9640 1 -- 172.21.15.77:0/2704031190 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f4ac0010570 con 0x55d989327500 2024-03-16T10:20:04.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.181+0000 7f4ace420740 10 client.26249 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:20:04.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.181+0000 7f4ace420740 10 client.26249 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:04.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.181+0000 7f4ace420740 20 client.26249 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:04.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.181+0000 7f4ace420740 1 --2- 172.21.15.77:0/2704031190 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d98935f890 0x55d989361d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.182+0000 7f4ace420740 4 client.26249 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:20:04.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.182+0000 7f4ace420740 1 -- 172.21.15.77:0/2704031190 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d989299f80 con 0x55d98935f890 2024-03-16T10:20:04.185 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.184+0000 7f4ac8cfb640 1 --2- 172.21.15.77:0/2704031190 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d98935f890 0x55d989361d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:04.189 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.188+0000 7f4ac8cfb640 1 --2- 172.21.15.77:0/2704031190 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d98935f890 0x55d989361d30 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a00331c16364d61c server_cookie=2796a43387927d4b in_seq=0 out_seq=0 2024-03-16T10:20:04.189 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.188+0000 7f4aacff9640 10 client.26249 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.194+0000 7f4aacff9640 1 -- 172.21.15.77:0/2704031190 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d989299f80 con 0x55d98935f890 2024-03-16T10:20:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.194+0000 7f4aacff9640 10 client.26249 handle_command_reply: tid=0 2024-03-16T10:20:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.194+0000 7f4ace420740 1 client.26249 shutdown 2024-03-16T10:20:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.194+0000 7f4ace420740 1 -- 172.21.15.77:0/2704031190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327500 msgr2=0x55d9893538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.194+0000 7f4ace420740 1 --2- 172.21.15.77:0/2704031190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327500 0x55d9893538b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ac009a690 tx=0x7f4ac00cbb00 comp rx=0 tx=0).stop 2024-03-16T10:20:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.194+0000 7f4ace420740 1 -- 172.21.15.77:0/2704031190 shutdown_connections 2024-03-16T10:20:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.194+0000 7f4ace420740 1 --2- 172.21.15.77:0/2704031190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d98932d060 0x55d98935adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.194+0000 7f4ace420740 1 --2- 172.21.15.77:0/2704031190 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d98935f890 0x55d989361d30 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.194+0000 7f4ace420740 1 --2- 172.21.15.77:0/2704031190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327ef0 0x55d989353df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.194+0000 7f4ace420740 1 --2- 172.21.15.77:0/2704031190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327500 0x55d9893538b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.194+0000 7f4ace420740 1 -- 172.21.15.77:0/2704031190 >> 172.21.15.77:0/2704031190 conn(0x55d989322a70 msgr2=0x55d989321c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.194+0000 7f4aacff9640 0 client.26249 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.195+0000 7f4ace420740 1 -- 172.21.15.77:0/2704031190 shutdown_connections 2024-03-16T10:20:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.195+0000 7f4ace420740 1 -- 172.21.15.77:0/2704031190 wait complete. 2024-03-16T10:20:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.195+0000 7f4ace420740 20 client.26249 trim_cache size 0 max 16384 2024-03-16T10:20:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.246+0000 7f4ace420740 1 Processor -- start 2024-03-16T10:20:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.247+0000 7f4ace420740 1 -- start start 2024-03-16T10:20:04.248 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:03 smithi077 ceph-mon[32400]: pgmap v439: 97 pgs: 97 active+clean; 232 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-03-16T10:20:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.247+0000 7f4ace420740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327500 0x55d9893534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.248+0000 7f4ace420740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327ef0 0x55d98935ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.248+0000 7f4ace420740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d98932d060 0x55d989353a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.248+0000 7f4ace420740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac407c4d0 con 0x55d989327ef0 2024-03-16T10:20:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.248+0000 7f4ace420740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac407e190 con 0x55d989327500 2024-03-16T10:20:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.248+0000 7f4ace420740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac407cba0 con 0x55d98932d060 2024-03-16T10:20:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.248+0000 7f4ac9cfd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d98932d060 0x55d989353a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.248+0000 7f4ac9cfd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d98932d060 0x55d989353a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:40020/0 (socket says 172.21.15.77:40020) 2024-03-16T10:20:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.248+0000 7f4ac9cfd640 1 -- 172.21.15.77:0/73354758 learned_addr learned my addr 172.21.15.77:0/73354758 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.249+0000 7f4ac8cfb640 1 --2- 172.21.15.77:0/73354758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327ef0 0x55d98935ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.249+0000 7f4ac9cfd640 1 -- 172.21.15.77:0/73354758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327500 msgr2=0x55d9893534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.249+0000 7f4ac9cfd640 1 --2- 172.21.15.77:0/73354758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327500 0x55d9893534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.249+0000 7f4ac9cfd640 1 -- 172.21.15.77:0/73354758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327ef0 msgr2=0x55d98935ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.249+0000 7f4ac9cfd640 1 --2- 172.21.15.77:0/73354758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327ef0 0x55d98935ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.249+0000 7f4ac9cfd640 1 -- 172.21.15.77:0/73354758 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9893237e0 con 0x55d98932d060 2024-03-16T10:20:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.250+0000 7f4ac9cfd640 1 --2- 172.21.15.77:0/73354758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d98932d060 0x55d989353a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4abc045fa0 tx=0x7f4abc005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa87a16eb2d2f18d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:03 smithi193 ceph-mon[36311]: pgmap v439: 97 pgs: 97 active+clean; 232 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-03-16T10:20:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.253+0000 7f4aaeffd640 1 -- 172.21.15.77:0/73354758 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc013020 con 0x55d98932d060 2024-03-16T10:20:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.253+0000 7f4aaeffd640 1 -- 172.21.15.77:0/73354758 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4abc043ab0 con 0x55d98932d060 2024-03-16T10:20:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.259+0000 7f4ace420740 1 -- 172.21.15.77:0/73354758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d98932d060 msgr2=0x55d989353a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.259+0000 7f4ace420740 1 --2- 172.21.15.77:0/73354758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d98932d060 0x55d989353a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4abc045fa0 tx=0x7f4abc005ef0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.259+0000 7f4ace420740 1 -- 172.21.15.77:0/73354758 shutdown_connections 2024-03-16T10:20:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.259+0000 7f4ace420740 1 --2- 172.21.15.77:0/73354758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d98932d060 0x55d989353a00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.259+0000 7f4ace420740 1 --2- 172.21.15.77:0/73354758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327ef0 0x55d98935ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.259+0000 7f4ace420740 1 --2- 172.21.15.77:0/73354758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d989327500 0x55d9893534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.259+0000 7f4ace420740 1 -- 172.21.15.77:0/73354758 >> 172.21.15.77:0/73354758 conn(0x55d989322a70 msgr2=0x55d9893292b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.267+0000 7f4ace420740 1 -- 172.21.15.77:0/73354758 shutdown_connections 2024-03-16T10:20:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.269+0000 7f4ace420740 1 -- 172.21.15.77:0/73354758 wait complete. 2024-03-16T10:20:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.270+0000 7f4ace420740 1 -- start start 2024-03-16T10:20:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.270+0000 7f4ace420740 1 Processor -- start 2024-03-16T10:20:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.273+0000 7f4ace420740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327500 0x55d989357bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.273+0000 7f4ace420740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d989327ef0 0x55d989358110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.273+0000 7f4ace420740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d98932d060 0x55d9893c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.273+0000 7f4ace420740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac407da00 con 0x55d989327500 2024-03-16T10:20:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.273+0000 7f4ace420740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac407d330 con 0x55d98932d060 2024-03-16T10:20:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.273+0000 7f4ace420740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac407a5c0 con 0x55d989327ef0 2024-03-16T10:20:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.273+0000 7f4ace420740 10 client.0 _ll_register_callbacks cb 0x7f4ac009a5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.274+0000 7f4ac9cfd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d98932d060 0x55d9893c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.274+0000 7f4ac9cfd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d98932d060 0x55d9893c3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:59934/0 (socket says 172.21.15.77:59934) 2024-03-16T10:20:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.274+0000 7f4ac9cfd640 1 -- 172.21.15.77:0/3715703999 learned_addr learned my addr 172.21.15.77:0/3715703999 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.274+0000 7f4ac9cfd640 1 -- 172.21.15.77:0/3715703999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d989327ef0 msgr2=0x55d989358110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.274+0000 7f4ac9cfd640 1 --2- 172.21.15.77:0/3715703999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d989327ef0 0x55d989358110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.274+0000 7f4ac9cfd640 1 -- 172.21.15.77:0/3715703999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327500 msgr2=0x55d989357bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.274+0000 7f4ac9cfd640 1 --2- 172.21.15.77:0/3715703999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327500 0x55d989357bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.274+0000 7f4ac9cfd640 1 -- 172.21.15.77:0/3715703999 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9893c4160 con 0x55d98932d060 2024-03-16T10:20:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.275+0000 7f4ac9cfd640 1 --2- 172.21.15.77:0/3715703999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d98932d060 0x55d9893c3c20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4abc03f010 tx=0x7f4abc0457d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef8ff5046a01e8ab server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.275+0000 7f4aacff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:20:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.276+0000 7f4aacff9640 1 -- 172.21.15.77:0/3715703999 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc04a070 con 0x55d98932d060 2024-03-16T10:20:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.276+0000 7f4aacff9640 1 -- 172.21.15.77:0/3715703999 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f4abc00f470 con 0x55d98932d060 2024-03-16T10:20:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.276+0000 7f4ace420740 1 -- 172.21.15.77:0/3715703999 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4abc043ab0 con 0x55d98932d060 2024-03-16T10:20:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.280+0000 7f4aacff9640 1 -- 172.21.15.77:0/3715703999 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4abc00f670 con 0x55d98932d060 2024-03-16T10:20:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.283+0000 7f4ace420740 10 client.26252 fetch_fsmap learned FSMap version 32 2024-03-16T10:20:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.283+0000 7f4ace420740 1 -- 172.21.15.77:0/3715703999 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d9893c5370 con 0x55d98932d060 2024-03-16T10:20:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.285+0000 7f4aacff9640 1 -- 172.21.15.77:0/3715703999 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4abc043ab0 con 0x55d98932d060 2024-03-16T10:20:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.285+0000 7f4aacff9640 1 -- 172.21.15.77:0/3715703999 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f4abc0188e0 con 0x55d98932d060 2024-03-16T10:20:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.285+0000 7f4ace420740 10 client.26252 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:20:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.285+0000 7f4ace420740 10 client.26252 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.285+0000 7f4ace420740 20 client.26252 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.285+0000 7f4ace420740 1 --2- 172.21.15.77:0/3715703999 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d9893c5d50 0x55d9893c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.285+0000 7f4ace420740 4 client.26252 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:20:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.285+0000 7f4ace420740 1 -- 172.21.15.77:0/3715703999 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d9892942d0 con 0x55d9893c5d50 2024-03-16T10:20:04.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.289+0000 7f4ac94fc640 1 --2- 172.21.15.77:0/3715703999 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d9893c5d50 0x55d9893c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:04.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.290+0000 7f4ac94fc640 1 --2- 172.21.15.77:0/3715703999 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d9893c5d50 0x55d9893c8170 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7df2ec63373108b3 server_cookie=bf04d1cbaec2da19 in_seq=0 out_seq=0 2024-03-16T10:20:04.291 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:04.291+0000 7f4aacff9640 10 client.26252 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4aacff9640 1 -- 172.21.15.77:0/3715703999 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d9892942d0 con 0x55d9893c5d50 2024-03-16T10:20:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4aacff9640 10 client.26252 handle_command_reply: tid=0 2024-03-16T10:20:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4ace420740 1 client.26252 shutdown 2024-03-16T10:20:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4ace420740 1 -- 172.21.15.77:0/3715703999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d98932d060 msgr2=0x55d9893c3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4ace420740 1 --2- 172.21.15.77:0/3715703999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d98932d060 0x55d9893c3c20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4abc03f010 tx=0x7f4abc0457d0 comp rx=0 tx=0).stop 2024-03-16T10:20:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4ace420740 1 -- 172.21.15.77:0/3715703999 shutdown_connections 2024-03-16T10:20:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4ace420740 1 --2- 172.21.15.77:0/3715703999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d98932d060 0x55d9893c3c20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4ace420740 1 --2- 172.21.15.77:0/3715703999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d989327ef0 0x55d989358110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4ace420740 1 --2- 172.21.15.77:0/3715703999 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d9893c5d50 0x55d9893c8170 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4ace420740 1 --2- 172.21.15.77:0/3715703999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d989327500 0x55d989357bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4aacff9640 0 client.26252 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4ace420740 1 -- 172.21.15.77:0/3715703999 >> 172.21.15.77:0/3715703999 conn(0x55d989322a70 msgr2=0x55d9893534a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4ace420740 1 -- 172.21.15.77:0/3715703999 shutdown_connections 2024-03-16T10:20:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.452+0000 7f4ace420740 1 -- 172.21.15.77:0/3715703999 wait complete. 2024-03-16T10:20:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.453+0000 7f4ace420740 20 client.26252 trim_cache size 0 max 16384 2024-03-16T10:20:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:20:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:20:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:20:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:20:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.453+0000 7f4aca4fe640 1 -- 172.21.15.77:0/3359221351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4ab00a2930 msgr2=0x7f4ab00a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.453+0000 7f4aca4fe640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4ab00a2930 0x7f4ab00a4df0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x55d9893282f0 tx=0x7f4ab40068b0 comp rx=0 tx=0).stop 2024-03-16T10:20:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.453+0000 7f4aca4fe640 1 -- 172.21.15.77:0/3359221351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ac4072a90 msgr2=0x7f4ac419df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.453+0000 7f4aca4fe640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ac4072a90 0x7f4ac419df50 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000d950 tx=0x7f4ac000de10 comp rx=0 tx=0).stop 2024-03-16T10:20:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.453+0000 7f4aca4fe640 1 -- 172.21.15.77:0/3359221351 shutdown_connections 2024-03-16T10:20:05.457 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.453+0000 7f4aca4fe640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac41094d0 0x7f4ac41c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:05.457 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.453+0000 7f4aca4fe640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4ab00a2930 0x7f4ab00a4df0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:05.457 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.453+0000 7f4aca4fe640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac40733f0 0x7f4ac419e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:05.457 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.453+0000 7f4aca4fe640 1 --2- 172.21.15.77:0/3359221351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ac4072a90 0x7f4ac419df50 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:05.457 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.453+0000 7f4aca4fe640 1 -- 172.21.15.77:0/3359221351 >> 172.21.15.77:0/3359221351 conn(0x7f4ac406e650 msgr2=0x7f4ac410c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:05.457 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.454+0000 7f4aca4fe640 1 -- 172.21.15.77:0/3359221351 shutdown_connections 2024-03-16T10:20:05.457 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:05.454+0000 7f4aca4fe640 1 -- 172.21.15.77:0/3359221351 wait complete. 2024-03-16T10:20:06.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:05 smithi192 ceph-mon[36354]: pgmap v440: 97 pgs: 97 active+clean; 238 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 151 op/s 2024-03-16T10:20:06.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:05 smithi077 ceph-mon[32400]: pgmap v440: 97 pgs: 97 active+clean; 238 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 151 op/s 2024-03-16T10:20:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:05 smithi193 ceph-mon[36311]: pgmap v440: 97 pgs: 97 active+clean; 238 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 151 op/s 2024-03-16T10:20:07.103 INFO:tasks.workunit.client.0.smithi077.stdout: 60 1188307 425 807546 228 653549 765 2024-03-16T10:20:08.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:07 smithi192 ceph-mon[36354]: pgmap v441: 97 pgs: 97 active+clean; 277 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 324 op/s 2024-03-16T10:20:08.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:07 smithi077 ceph-mon[32400]: pgmap v441: 97 pgs: 97 active+clean; 277 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 324 op/s 2024-03-16T10:20:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:07 smithi193 ceph-mon[36311]: pgmap v441: 97 pgs: 97 active+clean; 277 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 324 op/s 2024-03-16T10:20:10.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:09 smithi192 ceph-mon[36354]: pgmap v442: 97 pgs: 97 active+clean; 302 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.9 MiB/s wr, 428 op/s 2024-03-16T10:20:10.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:09 smithi077 ceph-mon[32400]: pgmap v442: 97 pgs: 97 active+clean; 302 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.9 MiB/s wr, 428 op/s 2024-03-16T10:20:10.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:09 smithi193 ceph-mon[36311]: pgmap v442: 97 pgs: 97 active+clean; 302 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.9 MiB/s wr, 428 op/s 2024-03-16T10:20:11.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:10 smithi077 ceph-mon[32400]: pgmap v443: 97 pgs: 97 active+clean; 307 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.1 MiB/s wr, 464 op/s 2024-03-16T10:20:11.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:10 smithi193 ceph-mon[36311]: pgmap v443: 97 pgs: 97 active+clean; 307 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.1 MiB/s wr, 464 op/s 2024-03-16T10:20:11.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:10 smithi192 ceph-mon[36354]: pgmap v443: 97 pgs: 97 active+clean; 307 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.1 MiB/s wr, 464 op/s 2024-03-16T10:20:14.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:13 smithi192 ceph-mon[36354]: pgmap v444: 97 pgs: 97 active+clean; 307 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.3 MiB/s wr, 476 op/s 2024-03-16T10:20:14.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:13 smithi077 ceph-mon[32400]: pgmap v444: 97 pgs: 97 active+clean; 307 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.3 MiB/s wr, 476 op/s 2024-03-16T10:20:14.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:13 smithi193 ceph-mon[36311]: pgmap v444: 97 pgs: 97 active+clean; 307 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.3 MiB/s wr, 476 op/s 2024-03-16T10:20:16.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:15 smithi192 ceph-mon[36354]: pgmap v445: 97 pgs: 97 active+clean; 310 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.1 MiB/s wr, 458 op/s 2024-03-16T10:20:16.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:15 smithi077 ceph-mon[32400]: pgmap v445: 97 pgs: 97 active+clean; 310 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.1 MiB/s wr, 458 op/s 2024-03-16T10:20:16.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:15 smithi193 ceph-mon[36311]: pgmap v445: 97 pgs: 97 active+clean; 310 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.1 MiB/s wr, 458 op/s 2024-03-16T10:20:17.103 INFO:tasks.workunit.client.0.smithi077.stdout: 68 1513188 494 1016011 352 827958 914 2024-03-16T10:20:18.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:17 smithi192 ceph-mon[36354]: pgmap v446: 97 pgs: 97 active+clean; 313 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.0 MiB/s wr, 434 op/s 2024-03-16T10:20:18.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:17 smithi077 ceph-mon[32400]: pgmap v446: 97 pgs: 97 active+clean; 313 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.0 MiB/s wr, 434 op/s 2024-03-16T10:20:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:17 smithi193 ceph-mon[36311]: pgmap v446: 97 pgs: 97 active+clean; 313 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.0 MiB/s wr, 434 op/s 2024-03-16T10:20:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.978+0000 7f544587a640 1 -- 172.21.15.77:0/3851539454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54400763b0 msgr2=0x7f5440078820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.978+0000 7f544587a640 1 --2- 172.21.15.77:0/3851539454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54400763b0 0x7f5440078820 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f542c00b9f0 tx=0x7f542c02d8e0 comp rx=0 tx=0).stop 2024-03-16T10:20:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.978+0000 7f544587a640 1 -- 172.21.15.77:0/3851539454 shutdown_connections 2024-03-16T10:20:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.978+0000 7f544587a640 1 --2- 172.21.15.77:0/3851539454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5440078d60 0x7f544007b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.978+0000 7f544587a640 1 --2- 172.21.15.77:0/3851539454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54400763b0 0x7f5440078820 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.978+0000 7f544587a640 1 --2- 172.21.15.77:0/3851539454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54400734d0 0x7f54400738f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.978+0000 7f544587a640 1 -- 172.21.15.77:0/3851539454 >> 172.21.15.77:0/3851539454 conn(0x7f544006e650 msgr2=0x7f5440070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.985+0000 7f544587a640 1 -- 172.21.15.77:0/3851539454 shutdown_connections 2024-03-16T10:20:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.985+0000 7f544587a640 1 -- 172.21.15.77:0/3851539454 wait complete. 2024-03-16T10:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.985+0000 7f544587a640 1 Processor -- start 2024-03-16T10:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f544587a640 1 -- start start 2024-03-16T10:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f544587a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54400734d0 0x7f544012ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f544587a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54400763b0 0x7f544012ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f544587a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5440078d60 0x7f5440131540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f544587a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f544007eae0 con 0x7f5440078d60 2024-03-16T10:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f544587a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f544007e960 con 0x7f54400763b0 2024-03-16T10:20:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f544587a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f544007ec60 con 0x7f54400734d0 2024-03-16T10:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f543ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54400763b0 0x7f544012ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f543ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54400763b0 0x7f544012ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:58810/0 (socket says 172.21.15.77:58810) 2024-03-16T10:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f543ffff640 1 -- 172.21.15.77:0/1139238228 learned_addr learned my addr 172.21.15.77:0/1139238228 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f5445079640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5440078d60 0x7f5440131540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f543ffff640 1 -- 172.21.15.77:0/1139238228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54400734d0 msgr2=0x7f544012ca20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f543ffff640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54400734d0 0x7f544012ca20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.986+0000 7f543ffff640 1 -- 172.21.15.77:0/1139238228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5440078d60 msgr2=0x7f5440131540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.987+0000 7f543ffff640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5440078d60 0x7f5440131540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.987+0000 7f543ffff640 1 -- 172.21.15.77:0/1139238228 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5440131e00 con 0x7f54400763b0 2024-03-16T10:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.987+0000 7f5445079640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5440078d60 0x7f5440131540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.987+0000 7f543ffff640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54400763b0 0x7f544012ef70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f542c005e00 tx=0x7f542c034f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.987+0000 7f543dffb640 1 -- 172.21.15.77:0/1139238228 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c037040 con 0x7f54400763b0 2024-03-16T10:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.987+0000 7f543dffb640 1 -- 172.21.15.77:0/1139238228 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f542c009db0 con 0x7f54400763b0 2024-03-16T10:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.987+0000 7f543dffb640 1 -- 172.21.15.77:0/1139238228 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c004900 con 0x7f54400763b0 2024-03-16T10:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.987+0000 7f544587a640 1 -- 172.21.15.77:0/1139238228 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5440132090 con 0x7f54400763b0 2024-03-16T10:20:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.988+0000 7f544587a640 1 -- 172.21.15.77:0/1139238228 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5440132530 con 0x7f54400763b0 2024-03-16T10:20:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.991+0000 7f543dffb640 1 -- 172.21.15.77:0/1139238228 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f542c03a020 con 0x7f54400763b0 2024-03-16T10:20:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.992+0000 7f543dffb640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f54300a28f0 0x7f54300a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.993+0000 7f543dffb640 1 -- 172.21.15.77:0/1139238228 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f542c0f1550 con 0x7f54400763b0 2024-03-16T10:20:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.993+0000 7f5444878640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f54300a28f0 0x7f54300a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.993+0000 7f544979f740 1 Processor -- start 2024-03-16T10:20:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.993+0000 7f5444878640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f54300a28f0 0x7f54300a4db0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5434005fd0 tx=0x7f543400c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.993+0000 7f544979f740 1 -- start start 2024-03-16T10:20:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.994+0000 7f544979f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9500 0x561cb58f9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.994+0000 7f544979f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 0x561cb58fe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.994+0000 7f544979f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb59014f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.994+0000 7f544979f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561cb585ea40 con 0x561cb58f9500 2024-03-16T10:20:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.994+0000 7f544979f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561cb5807920 con 0x561cb58f9ef0 2024-03-16T10:20:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.994+0000 7f544979f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561cb585aae0 con 0x561cb58ff060 2024-03-16T10:20:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.994+0000 7f543ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 0x561cb58fe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.994+0000 7f543ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 0x561cb58fe9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:58822/0 (socket says 172.21.15.77:58822) 2024-03-16T10:20:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.994+0000 7f543ffff640 1 -- 172.21.15.77:0/3519555785 learned_addr learned my addr 172.21.15.77:0/3519555785 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.994+0000 7f5445079640 1 --2- 172.21.15.77:0/3519555785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb59014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.994+0000 7f5444878640 1 --2- 172.21.15.77:0/3519555785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9500 0x561cb58f9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.995+0000 7f543ffff640 1 -- 172.21.15.77:0/3519555785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 msgr2=0x561cb59014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.995+0000 7f543ffff640 1 --2- 172.21.15.77:0/3519555785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb59014f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.995+0000 7f543ffff640 1 -- 172.21.15.77:0/3519555785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9500 msgr2=0x561cb58f9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.995+0000 7f543ffff640 1 --2- 172.21.15.77:0/3519555785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9500 0x561cb58f9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.995+0000 7f543ffff640 1 -- 172.21.15.77:0/3519555785 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cb581f460 con 0x561cb58f9ef0 2024-03-16T10:20:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.995+0000 7f5444878640 1 --2- 172.21.15.77:0/3519555785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9500 0x561cb58f9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.995+0000 7f5445079640 1 --2- 172.21.15.77:0/3519555785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb59014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.995+0000 7f543ffff640 1 --2- 172.21.15.77:0/3519555785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 0x561cb58fe9f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f542c00e090 tx=0x7f542c02d8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=179bae227d1c5757 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.995+0000 7f542affd640 1 -- 172.21.15.77:0/3519555785 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c0bdee0 con 0x561cb58f9ef0 2024-03-16T10:20:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.995+0000 7f542affd640 1 -- 172.21.15.77:0/3519555785 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f542c0c2030 con 0x561cb58f9ef0 2024-03-16T10:20:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.995+0000 7f542affd640 1 -- 172.21.15.77:0/3519555785 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c0f1340 con 0x561cb58f9ef0 2024-03-16T10:20:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.996+0000 7f544979f740 1 -- 172.21.15.77:0/3519555785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 msgr2=0x561cb58fe9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.996+0000 7f544979f740 1 --2- 172.21.15.77:0/3519555785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 0x561cb58fe9f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f542c00e090 tx=0x7f542c02d8e0 comp rx=0 tx=0).stop 2024-03-16T10:20:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.997+0000 7f544979f740 1 -- 172.21.15.77:0/3519555785 shutdown_connections 2024-03-16T10:20:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.997+0000 7f544979f740 1 --2- 172.21.15.77:0/3519555785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb59014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.997+0000 7f544979f740 1 --2- 172.21.15.77:0/3519555785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 0x561cb58fe9f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.997+0000 7f544979f740 1 --2- 172.21.15.77:0/3519555785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9500 0x561cb58f9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.997+0000 7f544979f740 1 -- 172.21.15.77:0/3519555785 >> 172.21.15.77:0/3519555785 conn(0x561cb58f4a70 msgr2=0x561cb58f4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.997+0000 7f544979f740 1 -- 172.21.15.77:0/3519555785 shutdown_connections 2024-03-16T10:20:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.997+0000 7f544979f740 1 -- 172.21.15.77:0/3519555785 wait complete. 2024-03-16T10:20:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.997+0000 7f544979f740 1 -- start start 2024-03-16T10:20:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.997+0000 7f544979f740 1 Processor -- start 2024-03-16T10:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f544979f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9500 0x561cb59035b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f544979f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 0x561cb5903af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f544979f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb590ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f544979f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561cb5901dd0 con 0x561cb58f9500 2024-03-16T10:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f544979f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561cb581ad70 con 0x561cb58f9ef0 2024-03-16T10:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f544979f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561cb5901f50 con 0x561cb58ff060 2024-03-16T10:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f544979f740 10 client.0 _ll_register_callbacks cb 0x561cb5572a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f543ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 0x561cb5903af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f543ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 0x561cb5903af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:58828/0 (socket says 172.21.15.77:58828) 2024-03-16T10:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f543ffff640 1 -- 172.21.15.77:0/1854299560 learned_addr learned my addr 172.21.15.77:0/1854299560 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f5444878640 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9500 0x561cb59035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f543ffff640 1 -- 172.21.15.77:0/1854299560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 msgr2=0x561cb590ab30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f543ffff640 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb590ab30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f543ffff640 1 -- 172.21.15.77:0/1854299560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9500 msgr2=0x561cb59035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f543ffff640 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9500 0x561cb59035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f543ffff640 1 -- 172.21.15.77:0/1854299560 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cb5573210 con 0x561cb58f9ef0 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.998+0000 7f5444878640 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9500 0x561cb59035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.999+0000 7f543ffff640 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 0x561cb5903af0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f542c005560 tx=0x7f542c03f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b208e8a885a43e04 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.999+0000 7f541ffff640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.999+0000 7f541ffff640 1 -- 172.21.15.77:0/1854299560 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c0ba180 con 0x561cb58f9ef0 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.999+0000 7f541ffff640 1 -- 172.21.15.77:0/1854299560 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f542c0bd330 con 0x561cb58f9ef0 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:18.999+0000 7f544979f740 1 -- 172.21.15.77:0/1854299560 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f542c0c2030 con 0x561cb58f9ef0 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.001+0000 7f541ffff640 1 -- 172.21.15.77:0/1854299560 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f542c0c6680 con 0x561cb58f9ef0 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.001+0000 7f544979f740 10 client.26264 fetch_fsmap learned FSMap version 32 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.001+0000 7f544979f740 1 -- 172.21.15.77:0/1854299560 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561cb563f620 con 0x561cb58f9ef0 2024-03-16T10:20:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.001+0000 7f541ffff640 1 -- 172.21.15.77:0/1854299560 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f542c0c2030 con 0x561cb58f9ef0 2024-03-16T10:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.001+0000 7f541ffff640 1 -- 172.21.15.77:0/1854299560 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f542c04b020 con 0x561cb58f9ef0 2024-03-16T10:20:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.002+0000 7f544979f740 10 client.26264 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.002+0000 7f544979f740 10 client.26264 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.002+0000 7f544979f740 20 client.26264 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.002+0000 7f544979f740 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cb590f360 0x561cb59117e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.002+0000 7f544979f740 4 client.26264 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.002+0000 7f544979f740 1 -- 172.21.15.77:0/1854299560 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cb586bf80 con 0x561cb590f360 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.002+0000 7f5444878640 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cb590f360 0x561cb59117e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.002+0000 7f541ffff640 10 client.26264 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.003+0000 7f5444878640 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cb590f360 0x561cb59117e0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a061aa795679e3ad server_cookie=f413517bf04bce57 in_seq=0 out_seq=0 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.004+0000 7f541ffff640 1 -- 172.21.15.77:0/1854299560 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561cb586bf80 con 0x561cb590f360 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.004+0000 7f541ffff640 10 client.26264 handle_command_reply: tid=0 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.004+0000 7f544979f740 1 client.26264 shutdown 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.004+0000 7f544979f740 1 -- 172.21.15.77:0/1854299560 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 msgr2=0x561cb5903af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.004+0000 7f544979f740 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 0x561cb5903af0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f542c005560 tx=0x7f542c03f000 comp rx=0 tx=0).stop 2024-03-16T10:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.004+0000 7f544979f740 1 -- 172.21.15.77:0/1854299560 shutdown_connections 2024-03-16T10:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.004+0000 7f544979f740 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb590ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.004+0000 7f544979f740 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9ef0 0x561cb5903af0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.004+0000 7f544979f740 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cb590f360 0x561cb59117e0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.004+0000 7f544979f740 1 --2- 172.21.15.77:0/1854299560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9500 0x561cb59035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.004+0000 7f544979f740 1 -- 172.21.15.77:0/1854299560 >> 172.21.15.77:0/1854299560 conn(0x561cb58f4a70 msgr2=0x561cb58f3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.004+0000 7f541ffff640 0 client.26264 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.005+0000 7f544979f740 1 -- 172.21.15.77:0/1854299560 shutdown_connections 2024-03-16T10:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.005+0000 7f544979f740 1 -- 172.21.15.77:0/1854299560 wait complete. 2024-03-16T10:20:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.005+0000 7f544979f740 20 client.26264 trim_cache size 0 max 16384 2024-03-16T10:20:19.035 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.033+0000 7f544979f740 1 Processor -- start 2024-03-16T10:20:19.035 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.033+0000 7f544979f740 1 -- start start 2024-03-16T10:20:19.035 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.033+0000 7f544979f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 0x561cb5911370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:19.035 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.033+0000 7f544979f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9ef0 0x561cb590aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.033+0000 7f544979f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb58f83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.033+0000 7f544979f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f542400e2e0 con 0x561cb58f9ef0 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.033+0000 7f544979f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f542400ffa0 con 0x561cb58f9500 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.033+0000 7f544979f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f542400e9b0 con 0x561cb58ff060 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f5444878640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 0x561cb5911370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f5444878640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 0x561cb5911370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:58834/0 (socket says 172.21.15.77:58834) 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f5444878640 1 -- 172.21.15.77:0/361313703 learned_addr learned my addr 172.21.15.77:0/361313703 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f543ffff640 1 --2- 172.21.15.77:0/361313703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9ef0 0x561cb590aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f5445079640 1 --2- 172.21.15.77:0/361313703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb58f83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f5444878640 1 -- 172.21.15.77:0/361313703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 msgr2=0x561cb58f83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f5444878640 1 --2- 172.21.15.77:0/361313703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb58f83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f5444878640 1 -- 172.21.15.77:0/361313703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9ef0 msgr2=0x561cb590aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f5444878640 1 --2- 172.21.15.77:0/361313703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9ef0 0x561cb590aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f5444878640 1 -- 172.21.15.77:0/361313703 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cb58f57e0 con 0x561cb58f9500 2024-03-16T10:20:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f543ffff640 1 --2- 172.21.15.77:0/361313703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9ef0 0x561cb590aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f5444878640 1 --2- 172.21.15.77:0/361313703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 0x561cb5911370 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5434008330 tx=0x7f5434011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c58797a1fbcd15ef server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.034+0000 7f542a7fc640 1 -- 172.21.15.77:0/361313703 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54340248f0 con 0x561cb58f9500 2024-03-16T10:20:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.035+0000 7f542a7fc640 1 -- 172.21.15.77:0/361313703 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5434005020 con 0x561cb58f9500 2024-03-16T10:20:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.035+0000 7f542a7fc640 1 -- 172.21.15.77:0/361313703 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54340049c0 con 0x561cb58f9500 2024-03-16T10:20:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.035+0000 7f544979f740 1 -- 172.21.15.77:0/361313703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 msgr2=0x561cb5911370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.035+0000 7f544979f740 1 --2- 172.21.15.77:0/361313703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 0x561cb5911370 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5434008330 tx=0x7f5434011000 comp rx=0 tx=0).stop 2024-03-16T10:20:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.035+0000 7f544979f740 1 -- 172.21.15.77:0/361313703 shutdown_connections 2024-03-16T10:20:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.035+0000 7f544979f740 1 --2- 172.21.15.77:0/361313703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb58f83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.035+0000 7f544979f740 1 --2- 172.21.15.77:0/361313703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9ef0 0x561cb590aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.035+0000 7f544979f740 1 --2- 172.21.15.77:0/361313703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 0x561cb5911370 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.035+0000 7f544979f740 1 -- 172.21.15.77:0/361313703 >> 172.21.15.77:0/361313703 conn(0x561cb58f4a70 msgr2=0x561cb58f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.035+0000 7f544979f740 1 -- 172.21.15.77:0/361313703 shutdown_connections 2024-03-16T10:20:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.035+0000 7f544979f740 1 -- 172.21.15.77:0/361313703 wait complete. 2024-03-16T10:20:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.036+0000 7f544979f740 1 -- start start 2024-03-16T10:20:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.036+0000 7f544979f740 1 Processor -- start 2024-03-16T10:20:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.036+0000 7f544979f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 0x561cb5908890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.036+0000 7f544979f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9ef0 0x561cb5908dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.036+0000 7f544979f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb5909d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.036+0000 7f544979f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f542400f810 con 0x561cb58f9ef0 2024-03-16T10:20:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.036+0000 7f544979f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f542400f140 con 0x561cb58f9500 2024-03-16T10:20:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.036+0000 7f544979f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f542400c2c0 con 0x561cb58ff060 2024-03-16T10:20:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.036+0000 7f544979f740 10 client.0 _ll_register_callbacks cb 0x7f5434008230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f5445079640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb5909d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f543ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9ef0 0x561cb5908dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f5445079640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb5909d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:33978/0 (socket says 172.21.15.77:33978) 2024-03-16T10:20:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f5445079640 1 -- 172.21.15.77:0/982656094 learned_addr learned my addr 172.21.15.77:0/982656094 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f5444878640 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 0x561cb5908890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f543ffff640 1 -- 172.21.15.77:0/982656094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 msgr2=0x561cb5909d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f543ffff640 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb5909d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f543ffff640 1 -- 172.21.15.77:0/982656094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 msgr2=0x561cb5908890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f543ffff640 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 0x561cb5908890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f543ffff640 1 -- 172.21.15.77:0/982656094 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cb590a380 con 0x561cb58f9ef0 2024-03-16T10:20:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f5445079640 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb5909d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f5444878640 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 0x561cb5908890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f543ffff640 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9ef0 0x561cb5908dd0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f542c005600 tx=0x7f542c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470a69425ce0828b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f541ffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:20:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f541ffff640 1 -- 172.21.15.77:0/982656094 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c005350 con 0x561cb58f9ef0 2024-03-16T10:20:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.037+0000 7f541ffff640 1 -- 172.21.15.77:0/982656094 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f542c0bcda0 con 0x561cb58f9ef0 2024-03-16T10:20:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.038+0000 7f544979f740 1 -- 172.21.15.77:0/982656094 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5434005020 con 0x561cb58f9ef0 2024-03-16T10:20:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.038+0000 7f541ffff640 1 -- 172.21.15.77:0/982656094 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f542c0cb050 con 0x561cb58f9ef0 2024-03-16T10:20:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.038+0000 7f544979f740 10 client.16503 fetch_fsmap learned FSMap version 32 2024-03-16T10:20:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.038+0000 7f544979f740 1 -- 172.21.15.77:0/982656094 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561cb590a380 con 0x561cb58f9ef0 2024-03-16T10:20:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.038+0000 7f541ffff640 1 -- 172.21.15.77:0/982656094 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5434005020 con 0x561cb58f9ef0 2024-03-16T10:20:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.038+0000 7f541ffff640 1 -- 172.21.15.77:0/982656094 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f542c04b040 con 0x561cb58f9ef0 2024-03-16T10:20:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.038+0000 7f544979f740 10 client.16503 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:20:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.038+0000 7f544979f740 10 client.16503 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.039+0000 7f544979f740 20 client.16503 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.039+0000 7f544979f740 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cb59763e0 0x561cb5978800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.039+0000 7f544979f740 4 client.16503 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:20:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.039+0000 7f544979f740 1 -- 172.21.15.77:0/982656094 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561cb58662d0 con 0x561cb59763e0 2024-03-16T10:20:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.039+0000 7f5444878640 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cb59763e0 0x561cb5978800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.039+0000 7f541ffff640 10 client.16503 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.039+0000 7f5444878640 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cb59763e0 0x561cb5978800 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=958e497b0abe588 server_cookie=2b22531d08748ced in_seq=0 out_seq=0 2024-03-16T10:20:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.367+0000 7f541ffff640 1 -- 172.21.15.77:0/982656094 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561cb58662d0 con 0x561cb59763e0 2024-03-16T10:20:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.367+0000 7f541ffff640 10 client.16503 handle_command_reply: tid=0 2024-03-16T10:20:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.367+0000 7f544979f740 1 client.16503 shutdown 2024-03-16T10:20:19.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.371+0000 7f544979f740 1 -- 172.21.15.77:0/982656094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9ef0 msgr2=0x561cb5908dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:19.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.371+0000 7f544979f740 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9ef0 0x561cb5908dd0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f542c005600 tx=0x7f542c009920 comp rx=0 tx=0).stop 2024-03-16T10:20:19.373 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.373+0000 7f544979f740 1 -- 172.21.15.77:0/982656094 shutdown_connections 2024-03-16T10:20:19.373 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.373+0000 7f544979f740 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561cb58ff060 0x561cb5909d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.373+0000 7f544979f740 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561cb59763e0 0x561cb5978800 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.373+0000 7f544979f740 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561cb58f9ef0 0x561cb5908dd0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.373+0000 7f544979f740 1 --2- 172.21.15.77:0/982656094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561cb58f9500 0x561cb5908890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.373+0000 7f541ffff640 0 client.16503 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.373+0000 7f544979f740 1 -- 172.21.15.77:0/982656094 >> 172.21.15.77:0/982656094 conn(0x561cb58f4a70 msgr2=0x561cb58f8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:19.375 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.374+0000 7f544979f740 1 -- 172.21.15.77:0/982656094 shutdown_connections 2024-03-16T10:20:19.378 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.378+0000 7f544979f740 1 -- 172.21.15.77:0/982656094 wait complete. 2024-03-16T10:20:19.378 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.378+0000 7f544979f740 20 client.16503 trim_cache size 0 max 16384 2024-03-16T10:20:19.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:20:19.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:20:19.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:20:19.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:20:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.459+0000 7f544587a640 1 -- 172.21.15.77:0/1139238228 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f54300a28f0 msgr2=0x7f54300a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.459+0000 7f544587a640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f54300a28f0 0x7f54300a4db0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5434005fd0 tx=0x7f543400c040 comp rx=0 tx=0).stop 2024-03-16T10:20:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.459+0000 7f544587a640 1 -- 172.21.15.77:0/1139238228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54400763b0 msgr2=0x7f544012ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.459+0000 7f544587a640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54400763b0 0x7f544012ef70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f542c005e00 tx=0x7f542c034f50 comp rx=0 tx=0).stop 2024-03-16T10:20:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.460+0000 7f544587a640 1 -- 172.21.15.77:0/1139238228 shutdown_connections 2024-03-16T10:20:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.460+0000 7f544587a640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5440078d60 0x7f5440131540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.460+0000 7f544587a640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54400763b0 0x7f544012ef70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.463+0000 7f544587a640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f54300a28f0 0x7f54300a4db0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.464+0000 7f544587a640 1 --2- 172.21.15.77:0/1139238228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54400734d0 0x7f544012ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.464+0000 7f544587a640 1 -- 172.21.15.77:0/1139238228 >> 172.21.15.77:0/1139238228 conn(0x7f544006e650 msgr2=0x7f54400791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.469+0000 7f544587a640 1 -- 172.21.15.77:0/1139238228 shutdown_connections 2024-03-16T10:20:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:19.469+0000 7f544587a640 1 -- 172.21.15.77:0/1139238228 wait complete. 2024-03-16T10:20:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:19 smithi192 ceph-mon[36354]: pgmap v447: 97 pgs: 97 active+clean; 315 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 249 op/s 2024-03-16T10:20:20.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:19 smithi077 ceph-mon[32400]: pgmap v447: 97 pgs: 97 active+clean; 315 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 249 op/s 2024-03-16T10:20:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:19 smithi193 ceph-mon[36311]: pgmap v447: 97 pgs: 97 active+clean; 315 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 249 op/s 2024-03-16T10:20:22.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:21 smithi192 ceph-mon[36354]: pgmap v448: 97 pgs: 97 active+clean; 315 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 139 op/s 2024-03-16T10:20:22.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:21 smithi077 ceph-mon[32400]: pgmap v448: 97 pgs: 97 active+clean; 315 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 139 op/s 2024-03-16T10:20:22.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:21 smithi193 ceph-mon[36311]: pgmap v448: 97 pgs: 97 active+clean; 315 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 139 op/s 2024-03-16T10:20:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:23 smithi192 ceph-mon[36354]: pgmap v449: 97 pgs: 97 active+clean; 317 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 125 op/s 2024-03-16T10:20:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:23 smithi077 ceph-mon[32400]: pgmap v449: 97 pgs: 97 active+clean; 317 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 125 op/s 2024-03-16T10:20:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:23 smithi193 ceph-mon[36311]: pgmap v449: 97 pgs: 97 active+clean; 317 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 125 op/s 2024-03-16T10:20:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:25 smithi192 ceph-mon[36354]: pgmap v450: 97 pgs: 97 active+clean; 317 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-03-16T10:20:26.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:25 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:20:26.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:25 smithi077 ceph-mon[32400]: pgmap v450: 97 pgs: 97 active+clean; 317 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-03-16T10:20:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:25 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:20:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:25 smithi193 ceph-mon[36311]: pgmap v450: 97 pgs: 97 active+clean; 317 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-03-16T10:20:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:25 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:20:27.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:20:27.020 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:20:27.103 INFO:tasks.workunit.client.0.smithi077.stdout: 73 736838 185 496927 379 401075 592 2024-03-16T10:20:27.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.129+0000 7f929ad90640 1 -- 172.21.15.77:0/3149648864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9294101800 msgr2=0x7f9294101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:27.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.129+0000 7f929ad90640 1 --2- 172.21.15.77:0/3149648864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9294101800 0x7f9294101c20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9288009900 tx=0x7f928802bc30 comp rx=0 tx=0).stop 2024-03-16T10:20:27.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.129+0000 7f929ad90640 1 -- 172.21.15.77:0/3149648864 shutdown_connections 2024-03-16T10:20:27.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.129+0000 7f929ad90640 1 --2- 172.21.15.77:0/3149648864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f92941033b0 0x7f9294109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.129+0000 7f929ad90640 1 --2- 172.21.15.77:0/3149648864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92941029f0 0x7f9294102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.129+0000 7f929ad90640 1 --2- 172.21.15.77:0/3149648864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9294101800 0x7f9294101c20 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.129+0000 7f929ad90640 1 -- 172.21.15.77:0/3149648864 >> 172.21.15.77:0/3149648864 conn(0x7f92940fcd50 msgr2=0x7f92940ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:27.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.129+0000 7f929ad90640 1 -- 172.21.15.77:0/3149648864 shutdown_connections 2024-03-16T10:20:27.133 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.129+0000 7f929ad90640 1 -- 172.21.15.77:0/3149648864 wait complete. 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.129+0000 7f929ad90640 1 Processor -- start 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.129+0000 7f929ad90640 1 -- start start 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929ad90640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9294101800 0x7f9294197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929ad90640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f92941029f0 0x7f9294198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929ad90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92941033b0 0x7f929419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929ad90640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f929410d480 con 0x7f92941033b0 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929ad90640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f929410d300 con 0x7f9294101800 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929ad90640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f929410d600 con 0x7f92941029f0 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929932d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92941033b0 0x7f929419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f9293fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f92941029f0 0x7f9294198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929932d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92941033b0 0x7f929419f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51904/0 (socket says 172.21.15.77:51904) 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929932d640 1 -- 172.21.15.77:0/496535750 learned_addr learned my addr 172.21.15.77:0/496535750 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:27.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f9298b2c640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9294101800 0x7f9294197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929932d640 1 -- 172.21.15.77:0/496535750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f92941029f0 msgr2=0x7f9294198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:27.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929932d640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f92941029f0 0x7f9294198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929932d640 1 -- 172.21.15.77:0/496535750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9294101800 msgr2=0x7f9294197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:27.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929932d640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9294101800 0x7f9294197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929932d640 1 -- 172.21.15.77:0/496535750 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929419f740 con 0x7f92941033b0 2024-03-16T10:20:27.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929932d640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92941033b0 0x7f929419f0d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f928c00eb10 tx=0x7f928c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:27.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f9291ffb640 1 -- 172.21.15.77:0/496535750 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c004b20 con 0x7f92941033b0 2024-03-16T10:20:27.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f9291ffb640 1 -- 172.21.15.77:0/496535750 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f928c004cc0 con 0x7f92941033b0 2024-03-16T10:20:27.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f929ad90640 1 -- 172.21.15.77:0/496535750 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929419fa30 con 0x7f92941033b0 2024-03-16T10:20:27.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.130+0000 7f9291ffb640 1 -- 172.21.15.77:0/496535750 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c010760 con 0x7f92941033b0 2024-03-16T10:20:27.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.131+0000 7f929ad90640 1 -- 172.21.15.77:0/496535750 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929419ffd0 con 0x7f92941033b0 2024-03-16T10:20:27.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.133+0000 7f9291ffb640 1 -- 172.21.15.77:0/496535750 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f928c005020 con 0x7f92941033b0 2024-03-16T10:20:27.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.133+0000 7f929d97f740 1 Processor -- start 2024-03-16T10:20:27.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f929d97f740 1 -- start start 2024-03-16T10:20:27.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f929d97f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87170 0x55d7aba87590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f929d97f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87b60 0x55d7aba8c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f929d97f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba8ccd0 0x55d7aba8f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f929d97f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d7ab9ec500 con 0x55d7aba8ccd0 2024-03-16T10:20:27.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f929d97f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d7ab994be0 con 0x55d7aba87170 2024-03-16T10:20:27.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f929d97f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d7ab9e85a0 con 0x55d7aba87b60 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9291ffb640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f92840a2880 0x7f92840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9293fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87170 0x55d7aba87590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9298b2c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87b60 0x55d7aba8c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9293fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87170 0x55d7aba87590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:60896/0 (socket says 172.21.15.77:60896) 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9298b2c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87b60 0x55d7aba8c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52266/0 (socket says 172.21.15.77:52266) 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9293fff640 1 -- 172.21.15.77:0/669872121 learned_addr learned my addr 172.21.15.77:0/669872121 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f929932d640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f92840a2880 0x7f92840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9293fff640 1 --2- 172.21.15.77:0/669872121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba8ccd0 0x55d7aba8f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9291ffb640 1 -- 172.21.15.77:0/496535750 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f928c0cf020 con 0x7f92941033b0 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9293fff640 1 -- 172.21.15.77:0/669872121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87b60 msgr2=0x55d7aba8c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9293fff640 1 --2- 172.21.15.77:0/669872121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87b60 0x55d7aba8c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9293fff640 1 -- 172.21.15.77:0/669872121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba8ccd0 msgr2=0x55d7aba8f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9293fff640 1 --2- 172.21.15.77:0/669872121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba8ccd0 0x55d7aba8f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9293fff640 1 -- 172.21.15.77:0/669872121 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7ab9a84b0 con 0x55d7aba87170 2024-03-16T10:20:27.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9298b2c640 1 --2- 172.21.15.77:0/669872121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87b60 0x55d7aba8c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f929932d640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f92840a2880 0x7f92840a4d40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f928c06e750 tx=0x7f928c06e780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.134+0000 7f9293fff640 1 --2- 172.21.15.77:0/669872121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87170 0x55d7aba87590 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f92800058c0 tx=0x7f9280005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80d2fabc334cfe07 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.135+0000 7f927effd640 1 -- 172.21.15.77:0/669872121 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280004030 con 0x55d7aba87170 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.135+0000 7f927effd640 1 -- 172.21.15.77:0/669872121 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9280005500 con 0x55d7aba87170 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.135+0000 7f927effd640 1 -- 172.21.15.77:0/669872121 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280004c70 con 0x55d7aba87170 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.135+0000 7f929d97f740 1 -- 172.21.15.77:0/669872121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87170 msgr2=0x55d7aba87590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.135+0000 7f929d97f740 1 --2- 172.21.15.77:0/669872121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87170 0x55d7aba87590 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f92800058c0 tx=0x7f9280005850 comp rx=0 tx=0).stop 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.135+0000 7f929d97f740 1 -- 172.21.15.77:0/669872121 shutdown_connections 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.135+0000 7f929d97f740 1 --2- 172.21.15.77:0/669872121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba8ccd0 0x55d7aba8f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.135+0000 7f929d97f740 1 --2- 172.21.15.77:0/669872121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87b60 0x55d7aba8c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.135+0000 7f929d97f740 1 --2- 172.21.15.77:0/669872121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87170 0x55d7aba87590 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.135+0000 7f929d97f740 1 -- 172.21.15.77:0/669872121 >> 172.21.15.77:0/669872121 conn(0x55d7aba826e0 msgr2=0x55d7aba82ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.135+0000 7f929d97f740 1 -- 172.21.15.77:0/669872121 shutdown_connections 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f929d97f740 1 -- 172.21.15.77:0/669872121 wait complete. 2024-03-16T10:20:27.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f929d97f740 1 -- start start 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f929d97f740 1 Processor -- start 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f929d97f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87170 0x55d7aba91670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f929d97f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87b60 0x55d7aba91bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f929d97f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba8ccd0 0x55d7aba98bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f929d97f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d7aba8fc80 con 0x55d7aba87b60 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f929d97f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d7ab9a8160 con 0x55d7aba87170 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f929d97f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d7aba8fe00 con 0x55d7aba8ccd0 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f929d97f740 10 client.0 _ll_register_callbacks cb 0x55d7ab98c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f9298b2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87b60 0x55d7aba91bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f9298b2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87b60 0x55d7aba91bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51922/0 (socket says 172.21.15.77:51922) 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f9298b2c640 1 -- 172.21.15.77:0/133294241 learned_addr learned my addr 172.21.15.77:0/133294241 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f9293fff640 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87170 0x55d7aba91670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.136+0000 7f9293fff640 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba8ccd0 0x55d7aba98bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f9298b2c640 1 -- 172.21.15.77:0/133294241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba8ccd0 msgr2=0x55d7aba98bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f9298b2c640 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba8ccd0 0x55d7aba98bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f9298b2c640 1 -- 172.21.15.77:0/133294241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87170 msgr2=0x55d7aba91670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f9298b2c640 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87170 0x55d7aba91670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f9298b2c640 1 -- 172.21.15.77:0/133294241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7ab7ca7d0 con 0x55d7aba87b60 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f9298b2c640 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87b60 0x55d7aba91bb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f928802cfd0 tx=0x7f9288004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd8d76976ed69a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f9273fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f9273fff640 1 -- 172.21.15.77:0/133294241 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288005d50 con 0x55d7aba87b60 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f9273fff640 1 -- 172.21.15.77:0/133294241 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f9288005230 con 0x55d7aba87b60 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f929d97f740 1 -- 172.21.15.77:0/133294241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9280005500 con 0x55d7aba87b60 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f9273fff640 1 -- 172.21.15.77:0/133294241 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9288005450 con 0x55d7aba87b60 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f929d97f740 10 client.16515 fetch_fsmap learned FSMap version 32 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f929d97f740 1 -- 172.21.15.77:0/133294241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d7ab7003f0 con 0x55d7aba87b60 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.137+0000 7f9273fff640 1 -- 172.21.15.77:0/133294241 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9280005500 con 0x55d7aba87b60 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.138+0000 7f9273fff640 1 -- 172.21.15.77:0/133294241 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f9288003820 con 0x55d7aba87b60 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.138+0000 7f929d97f740 10 client.16515 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.138+0000 7f929d97f740 10 client.16515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:27.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.138+0000 7f929d97f740 20 client.16515 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.138+0000 7f929d97f740 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d7aba9d140 0x55d7aba9f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.138+0000 7f929d97f740 4 client.16515 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.138+0000 7f929d97f740 1 -- 172.21.15.77:0/133294241 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7ab9f9a40 con 0x55d7aba9d140 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.138+0000 7f9293fff640 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d7aba9d140 0x55d7aba9f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.138+0000 7f9273fff640 10 client.16515 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.138+0000 7f9293fff640 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d7aba9d140 0x55d7aba9f5c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e783178100ef8de6 server_cookie=2afa40157b223619 in_seq=0 out_seq=0 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.139+0000 7f9273fff640 1 -- 172.21.15.77:0/133294241 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d7ab9f9a40 con 0x55d7aba9d140 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.139+0000 7f9273fff640 10 client.16515 handle_command_reply: tid=0 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.139+0000 7f929d97f740 1 client.16515 shutdown 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.139+0000 7f929d97f740 1 -- 172.21.15.77:0/133294241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87b60 msgr2=0x55d7aba91bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.139+0000 7f929d97f740 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87b60 0x55d7aba91bb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f928802cfd0 tx=0x7f9288004270 comp rx=0 tx=0).stop 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.139+0000 7f929d97f740 1 -- 172.21.15.77:0/133294241 shutdown_connections 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.139+0000 7f929d97f740 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba8ccd0 0x55d7aba98bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.139+0000 7f929d97f740 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d7aba9d140 0x55d7aba9f5c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.139+0000 7f929d97f740 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87b60 0x55d7aba91bb0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.139+0000 7f929d97f740 1 --2- 172.21.15.77:0/133294241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87170 0x55d7aba91670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.139+0000 7f9273fff640 0 client.16515 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.139+0000 7f929d97f740 1 -- 172.21.15.77:0/133294241 >> 172.21.15.77:0/133294241 conn(0x55d7aba826e0 msgr2=0x55d7aba81900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.140+0000 7f929d97f740 1 -- 172.21.15.77:0/133294241 shutdown_connections 2024-03-16T10:20:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.140+0000 7f929d97f740 1 -- 172.21.15.77:0/133294241 wait complete. 2024-03-16T10:20:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.140+0000 7f929d97f740 20 client.16515 trim_cache size 0 max 16384 2024-03-16T10:20:27.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.157+0000 7f929d97f740 1 Processor -- start 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.157+0000 7f929d97f740 1 -- start start 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.157+0000 7f929d97f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87170 0x55d7aba82ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.157+0000 7f929d97f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87b60 0x55d7aba98bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.157+0000 7f929d97f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba8ccd0 0x55d7aba86050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.157+0000 7f929d97f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f92941a8be0 con 0x55d7aba87170 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.157+0000 7f929d97f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f92941aa8a0 con 0x55d7aba8ccd0 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.157+0000 7f929d97f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f92941a92b0 con 0x55d7aba87b60 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.157+0000 7f9293fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87170 0x55d7aba82ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.157+0000 7f9293fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87170 0x55d7aba82ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51930/0 (socket says 172.21.15.77:51930) 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.157+0000 7f9293fff640 1 -- 172.21.15.77:0/2875637163 learned_addr learned my addr 172.21.15.77:0/2875637163 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.157+0000 7f9298b2c640 1 --2- 172.21.15.77:0/2875637163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87b60 0x55d7aba98bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.158+0000 7f9293fff640 1 --2- 172.21.15.77:0/2875637163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba8ccd0 0x55d7aba86050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.158+0000 7f9293fff640 1 -- 172.21.15.77:0/2875637163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87b60 msgr2=0x55d7aba98bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:27.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.158+0000 7f9293fff640 1 --2- 172.21.15.77:0/2875637163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87b60 0x55d7aba98bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.158+0000 7f9293fff640 1 -- 172.21.15.77:0/2875637163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba8ccd0 msgr2=0x55d7aba86050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:27.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.158+0000 7f9293fff640 1 --2- 172.21.15.77:0/2875637163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba8ccd0 0x55d7aba86050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.158+0000 7f9293fff640 1 -- 172.21.15.77:0/2875637163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7ab6c00f0 con 0x55d7aba87170 2024-03-16T10:20:27.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.158+0000 7f9293fff640 1 --2- 172.21.15.77:0/2875637163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87170 0x55d7aba82ac0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f9280009fd0 tx=0x7f9280009cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1edab511d5e0ee81 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:27.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.158+0000 7f927e7fc640 1 -- 172.21.15.77:0/2875637163 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280028040 con 0x55d7aba87170 2024-03-16T10:20:27.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.158+0000 7f927e7fc640 1 -- 172.21.15.77:0/2875637163 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9280004a10 con 0x55d7aba87170 2024-03-16T10:20:27.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.158+0000 7f927e7fc640 1 -- 172.21.15.77:0/2875637163 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928000fd70 con 0x55d7aba87170 2024-03-16T10:20:27.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 -- 172.21.15.77:0/2875637163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87170 msgr2=0x55d7aba82ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:27.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 --2- 172.21.15.77:0/2875637163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87170 0x55d7aba82ac0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f9280009fd0 tx=0x7f9280009cf0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 -- 172.21.15.77:0/2875637163 shutdown_connections 2024-03-16T10:20:27.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 --2- 172.21.15.77:0/2875637163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba8ccd0 0x55d7aba86050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 --2- 172.21.15.77:0/2875637163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87b60 0x55d7aba98bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 --2- 172.21.15.77:0/2875637163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba87170 0x55d7aba82ac0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 -- 172.21.15.77:0/2875637163 >> 172.21.15.77:0/2875637163 conn(0x55d7aba826e0 msgr2=0x55d7aba85940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 -- 172.21.15.77:0/2875637163 shutdown_connections 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 -- 172.21.15.77:0/2875637163 wait complete. 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 -- start start 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 Processor -- start 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87170 0x55d7aba966c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87b60 0x55d7aba96c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba8ccd0 0x55d7aba97c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f92941aa110 con 0x55d7aba8ccd0 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f92941a9a40 con 0x55d7aba87b60 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f92941a6bc0 con 0x55d7aba87170 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.159+0000 7f929d97f740 10 client.0 _ll_register_callbacks cb 0x7f9288009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:27.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9293fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba8ccd0 0x55d7aba97c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9293fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba8ccd0 0x55d7aba97c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51932/0 (socket says 172.21.15.77:51932) 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9293fff640 1 -- 172.21.15.77:0/1044002986 learned_addr learned my addr 172.21.15.77:0/1044002986 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9298b2c640 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87b60 0x55d7aba96c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9293fff640 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87170 0x55d7aba966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9293fff640 1 -- 172.21.15.77:0/1044002986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87170 msgr2=0x55d7aba966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9293fff640 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87170 0x55d7aba966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9293fff640 1 -- 172.21.15.77:0/1044002986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87b60 msgr2=0x55d7aba96c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9293fff640 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87b60 0x55d7aba96c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9293fff640 1 -- 172.21.15.77:0/1044002986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7ab9b39d0 con 0x55d7aba8ccd0 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9293fff640 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba8ccd0 0x55d7aba97c10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9280009c40 tx=0x7f92800048e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6700d9953597d5f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9273fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9273fff640 1 -- 172.21.15.77:0/1044002986 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928000fcf0 con 0x55d7aba8ccd0 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f929d97f740 1 -- 172.21.15.77:0/1044002986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9280004a10 con 0x55d7aba8ccd0 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9273fff640 1 -- 172.21.15.77:0/1044002986 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f9280004bc0 con 0x55d7aba8ccd0 2024-03-16T10:20:27.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f9273fff640 1 -- 172.21.15.77:0/1044002986 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9280005580 con 0x55d7aba8ccd0 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f929d97f740 10 client.16521 fetch_fsmap learned FSMap version 32 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.160+0000 7f929d97f740 1 -- 172.21.15.77:0/1044002986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d7abb037b0 con 0x55d7aba8ccd0 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.161+0000 7f9273fff640 1 -- 172.21.15.77:0/1044002986 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9280004a10 con 0x55d7aba8ccd0 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.161+0000 7f9273fff640 1 -- 172.21.15.77:0/1044002986 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f9280033ce0 con 0x55d7aba8ccd0 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.161+0000 7f929d97f740 10 client.16521 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.161+0000 7f929d97f740 10 client.16521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.161+0000 7f929d97f740 20 client.16521 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.161+0000 7f929d97f740 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d7abb03f40 0x55d7abb06360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.161+0000 7f929d97f740 4 client.16521 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.161+0000 7f929d97f740 1 -- 172.21.15.77:0/1044002986 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d7ab9f3d90 con 0x55d7abb03f40 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.161+0000 7f9298b2c640 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d7abb03f40 0x55d7abb06360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.161+0000 7f9298b2c640 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d7abb03f40 0x55d7abb06360 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95a72b86cd9b0f13 server_cookie=f1be4cecc2634d9d in_seq=0 out_seq=0 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.161+0000 7f9273fff640 10 client.16521 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f9273fff640 1 -- 172.21.15.77:0/1044002986 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d7ab9f3d90 con 0x55d7abb03f40 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f9273fff640 10 client.16521 handle_command_reply: tid=0 2024-03-16T10:20:27.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f929d97f740 1 client.16521 shutdown 2024-03-16T10:20:27.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f929d97f740 1 -- 172.21.15.77:0/1044002986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba8ccd0 msgr2=0x55d7aba97c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:27.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f929d97f740 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba8ccd0 0x55d7aba97c10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9280009c40 tx=0x7f92800048e0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f929d97f740 1 -- 172.21.15.77:0/1044002986 shutdown_connections 2024-03-16T10:20:27.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f929d97f740 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d7abb03f40 0x55d7abb06360 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f929d97f740 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d7aba8ccd0 0x55d7aba97c10 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f9273fff640 0 client.16521 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:27.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f929d97f740 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7aba87b60 0x55d7aba96c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f929d97f740 1 --2- 172.21.15.77:0/1044002986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7aba87170 0x55d7aba966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f929d97f740 1 -- 172.21.15.77:0/1044002986 >> 172.21.15.77:0/1044002986 conn(0x55d7aba826e0 msgr2=0x55d7aba861d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:27.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f929d97f740 1 -- 172.21.15.77:0/1044002986 shutdown_connections 2024-03-16T10:20:27.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f929d97f740 1 -- 172.21.15.77:0/1044002986 wait complete. 2024-03-16T10:20:27.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.162+0000 7f929d97f740 20 client.16521 trim_cache size 0 max 16384 2024-03-16T10:20:27.171 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:20:27.171 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:20:27.171 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:20:27.171 INFO:teuthology.orchestra.run.smithi077.stdout: "b9f7336e-f2e2-4b8a-9fb4-bff0ee5d1be5": { 2024-03-16T10:20:27.171 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:20:27.171 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "b9f7336e-f2e2-4b8a-9fb4-bff0ee5d1be5", 2024-03-16T10:20:27.171 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:20:27.171 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:20:27.171 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:20:27.171 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:20:27.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.163+0000 7f929ad90640 1 -- 172.21.15.77:0/496535750 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f92840a2880 msgr2=0x7f92840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:27.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.163+0000 7f929ad90640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f92840a2880 0x7f92840a4d40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f928c06e750 tx=0x7f928c06e780 comp rx=0 tx=0).stop 2024-03-16T10:20:27.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.163+0000 7f929ad90640 1 -- 172.21.15.77:0/496535750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92941033b0 msgr2=0x7f929419f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:27.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.163+0000 7f929ad90640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92941033b0 0x7f929419f0d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f928c00eb10 tx=0x7f928c00c2d0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.163+0000 7f929ad90640 1 -- 172.21.15.77:0/496535750 shutdown_connections 2024-03-16T10:20:27.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.163+0000 7f929ad90640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92941033b0 0x7f929419f0d0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.163+0000 7f929ad90640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f92840a2880 0x7f92840a4d40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.163+0000 7f929ad90640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f92941029f0 0x7f9294198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.163+0000 7f929ad90640 1 --2- 172.21.15.77:0/496535750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9294101800 0x7f9294197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:27.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.163+0000 7f929ad90640 1 -- 172.21.15.77:0/496535750 >> 172.21.15.77:0/496535750 conn(0x7f92940fcd50 msgr2=0x7f9294105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:27.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.163+0000 7f929ad90640 1 -- 172.21.15.77:0/496535750 shutdown_connections 2024-03-16T10:20:27.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:27.163+0000 7f929ad90640 1 -- 172.21.15.77:0/496535750 wait complete. 2024-03-16T10:20:27.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:26 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:20:27.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:26 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:20:27.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:26 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:20:27.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:26 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:20:27.175 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:26 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:20:27.175 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:26 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:20:27.176 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:26 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:20:27.176 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:26 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:20:27.177 INFO:tasks.cephfs.filesystem:scrub status for tag:b9f7336e-f2e2-4b8a-9fb4-bff0ee5d1be5 - {'path': '/', 'tag': 'b9f7336e-f2e2-4b8a-9fb4-bff0ee5d1be5', 'options': 'recursive,force'} 2024-03-16T10:20:27.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:26 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:20:27.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:26 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:20:27.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:26 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:20:27.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:26 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:20:28.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:27 smithi192 ceph-mon[36354]: pgmap v451: 97 pgs: 97 active+clean; 317 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-16T10:20:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:27 smithi077 ceph-mon[32400]: pgmap v451: 97 pgs: 97 active+clean; 317 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-16T10:20:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:27 smithi193 ceph-mon[36311]: pgmap v451: 97 pgs: 97 active+clean; 317 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-16T10:20:30.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:29 smithi192 ceph-mon[36354]: pgmap v452: 97 pgs: 97 active+clean; 317 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-16T10:20:30.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:29 smithi077 ceph-mon[32400]: pgmap v452: 97 pgs: 97 active+clean; 317 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-16T10:20:30.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:29 smithi193 ceph-mon[36311]: pgmap v452: 97 pgs: 97 active+clean; 317 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-16T10:20:32.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:31 smithi192 ceph-mon[36354]: pgmap v453: 97 pgs: 97 active+clean; 325 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-03-16T10:20:32.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:31 smithi077 ceph-mon[32400]: pgmap v453: 97 pgs: 97 active+clean; 325 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-03-16T10:20:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:31 smithi193 ceph-mon[36311]: pgmap v453: 97 pgs: 97 active+clean; 325 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-03-16T10:20:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.630+0000 7ff194d77640 1 -- 172.21.15.77:0/3545261318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1901022b0 msgr2=0x7ff190108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.630+0000 7ff194d77640 1 --2- 172.21.15.77:0/3545261318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1901022b0 0x7ff190108b40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff188009ca0 tx=0x7ff18802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.630+0000 7ff194d77640 1 -- 172.21.15.77:0/3545261318 shutdown_connections 2024-03-16T10:20:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.630+0000 7ff194d77640 1 --2- 172.21.15.77:0/3545261318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1901022b0 0x7ff190108b40 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.630+0000 7ff194d77640 1 --2- 172.21.15.77:0/3545261318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1901018f0 0x7ff190101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.630+0000 7ff194d77640 1 --2- 172.21.15.77:0/3545261318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff190100700 0x7ff190100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.630+0000 7ff194d77640 1 -- 172.21.15.77:0/3545261318 >> 172.21.15.77:0/3545261318 conn(0x7ff1900fbc70 msgr2=0x7ff1900fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.630+0000 7ff194d77640 1 -- 172.21.15.77:0/3545261318 shutdown_connections 2024-03-16T10:20:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.630+0000 7ff194d77640 1 -- 172.21.15.77:0/3545261318 wait complete. 2024-03-16T10:20:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff194d77640 1 Processor -- start 2024-03-16T10:20:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff194d77640 1 -- start start 2024-03-16T10:20:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff194d77640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff190100700 0x7ff190195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff194d77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1901018f0 0x7ff190195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff194d77640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1901022b0 0x7ff19019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff194d77640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff19010b270 con 0x7ff1901018f0 2024-03-16T10:20:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff194d77640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff19010b0f0 con 0x7ff190100700 2024-03-16T10:20:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff194d77640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff19010b3f0 con 0x7ff1901022b0 2024-03-16T10:20:32.635 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff187fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1901018f0 0x7ff190195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff187fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1901018f0 0x7ff190195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56146/0 (socket says 172.21.15.77:56146) 2024-03-16T10:20:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff187fff640 1 -- 172.21.15.77:0/2927649419 learned_addr learned my addr 172.21.15.77:0/2927649419 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff18e59c640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff190100700 0x7ff190195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.631+0000 7ff18ed9d640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1901022b0 0x7ff19019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.632+0000 7ff187fff640 1 -- 172.21.15.77:0/2927649419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1901022b0 msgr2=0x7ff19019cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.632+0000 7ff187fff640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1901022b0 0x7ff19019cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.632+0000 7ff187fff640 1 -- 172.21.15.77:0/2927649419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff190100700 msgr2=0x7ff190195820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.632+0000 7ff187fff640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff190100700 0x7ff190195820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.632+0000 7ff187fff640 1 -- 172.21.15.77:0/2927649419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19019d440 con 0x7ff1901018f0 2024-03-16T10:20:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.632+0000 7ff187fff640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1901018f0 0x7ff190195d60 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff17800e9f0 tx=0x7ff17800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.632+0000 7ff1877fe640 1 -- 172.21.15.77:0/2927649419 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17800cd80 con 0x7ff1901018f0 2024-03-16T10:20:32.636 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.632+0000 7ff194d77640 1 -- 172.21.15.77:0/2927649419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19019d730 con 0x7ff1901018f0 2024-03-16T10:20:32.637 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.632+0000 7ff1877fe640 1 -- 172.21.15.77:0/2927649419 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff178005500 con 0x7ff1901018f0 2024-03-16T10:20:32.637 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.632+0000 7ff1877fe640 1 -- 172.21.15.77:0/2927649419 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff178010690 con 0x7ff1901018f0 2024-03-16T10:20:32.637 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.632+0000 7ff194d77640 1 -- 172.21.15.77:0/2927649419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19019dc80 con 0x7ff1901018f0 2024-03-16T10:20:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.634+0000 7ff1877fe640 1 -- 172.21.15.77:0/2927649419 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7ff1780046c0 con 0x7ff1901018f0 2024-03-16T10:20:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.634+0000 7ff197a3a740 1 Processor -- start 2024-03-16T10:20:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff197a3a740 1 -- start start 2024-03-16T10:20:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff197a3a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 0x5563c7d6a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff197a3a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6ad30 0x5563c7d6f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff197a3a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d6fea0 0x5563c7d72330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff197a3a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5563c7ccfa40 con 0x5563c7d6a340 2024-03-16T10:20:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff197a3a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5563c7c78920 con 0x5563c7d6ad30 2024-03-16T10:20:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff197a3a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5563c7ccbae0 con 0x5563c7d6fea0 2024-03-16T10:20:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff18ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 0x5563c7d6a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff18e59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6ad30 0x5563c7d6f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff18ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 0x5563c7d6a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56150/0 (socket says 172.21.15.77:56150) 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff18e59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6ad30 0x5563c7d6f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:46382/0 (socket says 172.21.15.77:46382) 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff18ed9d640 1 -- 172.21.15.77:0/247430000 learned_addr learned my addr 172.21.15.77:0/247430000 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff187fff640 1 --2- 172.21.15.77:0/247430000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d6fea0 0x5563c7d72330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff1877fe640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff17c0a4ab0 0x7ff17c0a6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff18e59c640 1 -- 172.21.15.77:0/247430000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d6fea0 msgr2=0x5563c7d72330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff18e59c640 1 --2- 172.21.15.77:0/247430000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d6fea0 0x5563c7d72330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff18e59c640 1 -- 172.21.15.77:0/247430000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 msgr2=0x5563c7d6a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff18e59c640 1 --2- 172.21.15.77:0/247430000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 0x5563c7d6a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff18e59c640 1 -- 172.21.15.77:0/247430000 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563c7c90460 con 0x5563c7d6ad30 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff18ed9d640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff17c0a4ab0 0x7ff17c0a6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.635+0000 7ff1877fe640 1 -- 172.21.15.77:0/2927649419 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ff1780cedc0 con 0x7ff1901018f0 2024-03-16T10:20:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.636+0000 7ff18e59c640 1 --2- 172.21.15.77:0/247430000 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6ad30 0x5563c7d6f830 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff180009bf0 tx=0x7ff180011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1801c33dd76b8433 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.636+0000 7ff184ff9640 1 -- 172.21.15.77:0/247430000 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180009da0 con 0x5563c7d6ad30 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.636+0000 7ff18ed9d640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff17c0a4ab0 0x7ff17c0a6f70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff188006010 tx=0x7ff18802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.636+0000 7ff184ff9640 1 -- 172.21.15.77:0/247430000 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff180005020 con 0x5563c7d6ad30 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.636+0000 7ff184ff9640 1 -- 172.21.15.77:0/247430000 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180004b20 con 0x5563c7d6ad30 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.636+0000 7ff197a3a740 1 -- 172.21.15.77:0/247430000 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6ad30 msgr2=0x5563c7d6f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.636+0000 7ff197a3a740 1 --2- 172.21.15.77:0/247430000 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6ad30 0x5563c7d6f830 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff180009bf0 tx=0x7ff180011810 comp rx=0 tx=0).stop 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 -- 172.21.15.77:0/247430000 shutdown_connections 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 --2- 172.21.15.77:0/247430000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d6fea0 0x5563c7d72330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 --2- 172.21.15.77:0/247430000 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6ad30 0x5563c7d6f830 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 --2- 172.21.15.77:0/247430000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 0x5563c7d6a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 -- 172.21.15.77:0/247430000 >> 172.21.15.77:0/247430000 conn(0x5563c7d658b0 msgr2=0x5563c7d65cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 -- 172.21.15.77:0/247430000 shutdown_connections 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 -- 172.21.15.77:0/247430000 wait complete. 2024-03-16T10:20:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 -- start start 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 Processor -- start 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 0x5563c7d74640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6ad30 0x5563c7d74b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d6fea0 0x5563c7d7bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5563c7d72e50 con 0x5563c7d6a340 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5563c7c8bd70 con 0x5563c7d6ad30 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5563c7d72fd0 con 0x5563c7d6fea0 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff18e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 0x5563c7d74640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff197a3a740 10 client.0 _ll_register_callbacks cb 0x5563c79e3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff18e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 0x5563c7d74640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56164/0 (socket says 172.21.15.77:56164) 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff18e59c640 1 -- 172.21.15.77:0/723884824 learned_addr learned my addr 172.21.15.77:0/723884824 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff187fff640 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6ad30 0x5563c7d74b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.637+0000 7ff18ed9d640 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d6fea0 0x5563c7d7bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff18e59c640 1 -- 172.21.15.77:0/723884824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d6fea0 msgr2=0x5563c7d7bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff18e59c640 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d6fea0 0x5563c7d7bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff18e59c640 1 -- 172.21.15.77:0/723884824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6ad30 msgr2=0x5563c7d74b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff18e59c640 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6ad30 0x5563c7d74b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff18e59c640 1 -- 172.21.15.77:0/723884824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563c79e4210 con 0x5563c7d6a340 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff18e59c640 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 0x5563c7d74640 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff1800096e0 tx=0x7ff180009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d18eb79e9b81a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff175ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff175ffb640 1 -- 172.21.15.77:0/723884824 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180029020 con 0x5563c7d6a340 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff175ffb640 1 -- 172.21.15.77:0/723884824 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ff180004a90 con 0x5563c7d6a340 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff197a3a740 1 -- 172.21.15.77:0/723884824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff180005020 con 0x5563c7d6a340 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff175ffb640 1 -- 172.21.15.77:0/723884824 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff180034050 con 0x5563c7d6a340 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff197a3a740 10 client.16533 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff197a3a740 1 -- 172.21.15.77:0/723884824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5563c7ab0620 con 0x5563c7d6a340 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.638+0000 7ff175ffb640 1 -- 172.21.15.77:0/723884824 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff180005020 con 0x5563c7d6a340 2024-03-16T10:20:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.639+0000 7ff175ffb640 1 -- 172.21.15.77:0/723884824 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7ff180019450 con 0x5563c7d6a340 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.639+0000 7ff197a3a740 10 client.16533 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.639+0000 7ff197a3a740 10 client.16533 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.639+0000 7ff197a3a740 20 client.16533 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.639+0000 7ff197a3a740 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5563c7d802f0 0x5563c7d82700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.639+0000 7ff197a3a740 4 client.16533 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.639+0000 7ff197a3a740 1 -- 172.21.15.77:0/723884824 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563c7cdcf80 con 0x5563c7d802f0 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.639+0000 7ff187fff640 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5563c7d802f0 0x5563c7d82700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.639+0000 7ff187fff640 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5563c7d802f0 0x5563c7d82700 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c939e46c4aa2aa1d server_cookie=2e0545c8003c3651 in_seq=0 out_seq=0 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.639+0000 7ff175ffb640 10 client.16533 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff175ffb640 1 -- 172.21.15.77:0/723884824 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5563c7cdcf80 con 0x5563c7d802f0 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff175ffb640 10 client.16533 handle_command_reply: tid=0 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff197a3a740 1 client.16533 shutdown 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff197a3a740 1 -- 172.21.15.77:0/723884824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 msgr2=0x5563c7d74640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff197a3a740 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 0x5563c7d74640 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff1800096e0 tx=0x7ff180009fd0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff197a3a740 1 -- 172.21.15.77:0/723884824 shutdown_connections 2024-03-16T10:20:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff197a3a740 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d6fea0 0x5563c7d7bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff197a3a740 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6ad30 0x5563c7d74b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff197a3a740 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5563c7d802f0 0x5563c7d82700 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff197a3a740 1 --2- 172.21.15.77:0/723884824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6a340 0x5563c7d74640 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff197a3a740 1 -- 172.21.15.77:0/723884824 >> 172.21.15.77:0/723884824 conn(0x5563c7d658b0 msgr2=0x5563c7d64bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff175ffb640 0 client.16533 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff197a3a740 1 -- 172.21.15.77:0/723884824 shutdown_connections 2024-03-16T10:20:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff197a3a740 1 -- 172.21.15.77:0/723884824 wait complete. 2024-03-16T10:20:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.640+0000 7ff197a3a740 20 client.16533 trim_cache size 0 max 16384 2024-03-16T10:20:32.668 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.666+0000 7ff197a3a740 1 Processor -- start 2024-03-16T10:20:32.668 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.666+0000 7ff197a3a740 1 -- start start 2024-03-16T10:20:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff197a3a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 0x5563c7d69220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff197a3a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6a340 0x5563c7d73c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff197a3a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6ad30 0x5563c7d741a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff197a3a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901a6950 con 0x5563c7d6ad30 2024-03-16T10:20:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff197a3a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901a8580 con 0x5563c7d6a340 2024-03-16T10:20:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff197a3a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901a7020 con 0x5563c7d658b0 2024-03-16T10:20:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff18ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6ad30 0x5563c7d741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff18e59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 0x5563c7d69220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff18ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6ad30 0x5563c7d741a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56180/0 (socket says 172.21.15.77:56180) 2024-03-16T10:20:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff18e59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 0x5563c7d69220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:49652/0 (socket says 172.21.15.77:49652) 2024-03-16T10:20:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff187fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6a340 0x5563c7d73c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff18ed9d640 1 -- 172.21.15.77:0/2489801740 learned_addr learned my addr 172.21.15.77:0/2489801740 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff18ed9d640 1 -- 172.21.15.77:0/2489801740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 msgr2=0x5563c7d69220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff18ed9d640 1 --2- 172.21.15.77:0/2489801740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 0x5563c7d69220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff18ed9d640 1 -- 172.21.15.77:0/2489801740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6a340 msgr2=0x5563c7d73c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff18ed9d640 1 --2- 172.21.15.77:0/2489801740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6a340 0x5563c7d73c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff18ed9d640 1 -- 172.21.15.77:0/2489801740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563c7d61310 con 0x5563c7d6ad30 2024-03-16T10:20:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.667+0000 7ff18ed9d640 1 --2- 172.21.15.77:0/2489801740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6ad30 0x5563c7d741a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff188037040 tx=0x7ff188041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a8269fb0608675 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.668+0000 7ff177fff640 1 -- 172.21.15.77:0/2489801740 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188052070 con 0x5563c7d6ad30 2024-03-16T10:20:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.668+0000 7ff177fff640 1 -- 172.21.15.77:0/2489801740 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff188046030 con 0x5563c7d6ad30 2024-03-16T10:20:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.668+0000 7ff177fff640 1 -- 172.21.15.77:0/2489801740 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188039040 con 0x5563c7d6ad30 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 -- 172.21.15.77:0/2489801740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6ad30 msgr2=0x5563c7d741a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 --2- 172.21.15.77:0/2489801740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6ad30 0x5563c7d741a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff188037040 tx=0x7ff188041000 comp rx=0 tx=0).stop 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 -- 172.21.15.77:0/2489801740 shutdown_connections 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 --2- 172.21.15.77:0/2489801740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6ad30 0x5563c7d741a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 --2- 172.21.15.77:0/2489801740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6a340 0x5563c7d73c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 --2- 172.21.15.77:0/2489801740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 0x5563c7d69220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 -- 172.21.15.77:0/2489801740 >> 172.21.15.77:0/2489801740 conn(0x5563c7d6f9a0 msgr2=0x5563c7d68b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 -- 172.21.15.77:0/2489801740 shutdown_connections 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 -- 172.21.15.77:0/2489801740 wait complete. 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 -- start start 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 Processor -- start 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 0x5563c7d78fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6a340 0x5563c7d794e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6ad30 0x5563c7d7a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901a7e80 con 0x5563c7d6ad30 2024-03-16T10:20:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901a77b0 con 0x5563c7d6a340 2024-03-16T10:20:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.669+0000 7ff197a3a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff1901a4930 con 0x5563c7d658b0 2024-03-16T10:20:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff197a3a740 10 client.0 _ll_register_callbacks cb 0x7ff180005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff18e59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 0x5563c7d78fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff187fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6a340 0x5563c7d794e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff18e59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 0x5563c7d78fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:49656/0 (socket says 172.21.15.77:49656) 2024-03-16T10:20:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff18e59c640 1 -- 172.21.15.77:0/3479727778 learned_addr learned my addr 172.21.15.77:0/3479727778 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff18ed9d640 1 --2- 172.21.15.77:0/3479727778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6ad30 0x5563c7d7a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff18e59c640 1 -- 172.21.15.77:0/3479727778 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6a340 msgr2=0x5563c7d794e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff18e59c640 1 --2- 172.21.15.77:0/3479727778 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6a340 0x5563c7d794e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff18e59c640 1 -- 172.21.15.77:0/3479727778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6ad30 msgr2=0x5563c7d7a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff18e59c640 1 --2- 172.21.15.77:0/3479727778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6ad30 0x5563c7d7a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff18e59c640 1 -- 172.21.15.77:0/3479727778 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563c7d7aa90 con 0x5563c7d658b0 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff18e59c640 1 --2- 172.21.15.77:0/3479727778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 0x5563c7d78fa0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff180019ab0 tx=0x7ff1800055b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f45d880c7fd0735 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff175ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff175ffb640 1 -- 172.21.15.77:0/3479727778 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180029040 con 0x5563c7d658b0 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff175ffb640 1 -- 172.21.15.77:0/3479727778 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ff180019b30 con 0x5563c7d658b0 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.670+0000 7ff197a3a740 1 -- 172.21.15.77:0/3479727778 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff188046030 con 0x5563c7d658b0 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.671+0000 7ff175ffb640 1 -- 172.21.15.77:0/3479727778 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff180019d30 con 0x5563c7d658b0 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.671+0000 7ff197a3a740 10 client.26296 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.671+0000 7ff197a3a740 1 -- 172.21.15.77:0/3479727778 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5563c7de6640 con 0x5563c7d658b0 2024-03-16T10:20:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.671+0000 7ff175ffb640 1 -- 172.21.15.77:0/3479727778 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff188046030 con 0x5563c7d658b0 2024-03-16T10:20:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.671+0000 7ff175ffb640 1 -- 172.21.15.77:0/3479727778 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7ff180041050 con 0x5563c7d658b0 2024-03-16T10:20:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.671+0000 7ff197a3a740 10 client.26296 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.671+0000 7ff197a3a740 10 client.26296 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.671+0000 7ff197a3a740 20 client.26296 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.671+0000 7ff197a3a740 1 --2- 172.21.15.77:0/3479727778 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5563c7d6fea0 0x5563c7de9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.671+0000 7ff197a3a740 4 client.26296 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:20:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.671+0000 7ff197a3a740 1 -- 172.21.15.77:0/3479727778 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563c7d73a80 con 0x5563c7d6fea0 2024-03-16T10:20:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.671+0000 7ff187fff640 1 --2- 172.21.15.77:0/3479727778 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5563c7d6fea0 0x5563c7de9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.672+0000 7ff175ffb640 10 client.26296 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:32.672+0000 7ff187fff640 1 --2- 172.21.15.77:0/3479727778 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5563c7d6fea0 0x5563c7de9090 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29e46a617e7752f4 server_cookie=f2d0315f3975ef4f in_seq=0 out_seq=0 2024-03-16T10:20:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.132+0000 7ff175ffb640 1 -- 172.21.15.77:0/3479727778 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5563c7d73a80 con 0x5563c7d6fea0 2024-03-16T10:20:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.132+0000 7ff175ffb640 10 client.26296 handle_command_reply: tid=0 2024-03-16T10:20:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.132+0000 7ff197a3a740 1 client.26296 shutdown 2024-03-16T10:20:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff197a3a740 1 -- 172.21.15.77:0/3479727778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 msgr2=0x5563c7d78fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff197a3a740 1 --2- 172.21.15.77:0/3479727778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 0x5563c7d78fa0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff180019ab0 tx=0x7ff1800055b0 comp rx=0 tx=0).stop 2024-03-16T10:20:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff197a3a740 1 -- 172.21.15.77:0/3479727778 shutdown_connections 2024-03-16T10:20:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff197a3a740 1 --2- 172.21.15.77:0/3479727778 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5563c7d6fea0 0x5563c7de9090 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff197a3a740 1 --2- 172.21.15.77:0/3479727778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5563c7d6ad30 0x5563c7d7a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff197a3a740 1 --2- 172.21.15.77:0/3479727778 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5563c7d6a340 0x5563c7d794e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff175ffb640 0 client.26296 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff197a3a740 1 --2- 172.21.15.77:0/3479727778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563c7d658b0 0x5563c7d78fa0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff197a3a740 1 -- 172.21.15.77:0/3479727778 >> 172.21.15.77:0/3479727778 conn(0x5563c7d6f9a0 msgr2=0x5563c7d693a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff197a3a740 1 -- 172.21.15.77:0/3479727778 shutdown_connections 2024-03-16T10:20:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff197a3a740 1 -- 172.21.15.77:0/3479727778 wait complete. 2024-03-16T10:20:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff197a3a740 20 client.26296 trim_cache size 0 max 16384 2024-03-16T10:20:33.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:20:33.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:20:33.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:20:33.136 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:20:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff194d77640 1 -- 172.21.15.77:0/2927649419 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff17c0a4ab0 msgr2=0x7ff17c0a6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff194d77640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff17c0a4ab0 0x7ff17c0a6f70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff188006010 tx=0x7ff18802e040 comp rx=0 tx=0).stop 2024-03-16T10:20:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff194d77640 1 -- 172.21.15.77:0/2927649419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1901018f0 msgr2=0x7ff190195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff194d77640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1901018f0 0x7ff190195d60 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff17800e9f0 tx=0x7ff17800edf0 comp rx=0 tx=0).stop 2024-03-16T10:20:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff194d77640 1 -- 172.21.15.77:0/2927649419 shutdown_connections 2024-03-16T10:20:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff194d77640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff17c0a4ab0 0x7ff17c0a6f70 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff194d77640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1901022b0 0x7ff19019cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff194d77640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1901018f0 0x7ff190195d60 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff194d77640 1 --2- 172.21.15.77:0/2927649419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff190100700 0x7ff190195820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.133+0000 7ff194d77640 1 -- 172.21.15.77:0/2927649419 >> 172.21.15.77:0/2927649419 conn(0x7ff1900fbc70 msgr2=0x7ff190104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:33.137 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.134+0000 7ff194d77640 1 -- 172.21.15.77:0/2927649419 shutdown_connections 2024-03-16T10:20:33.138 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:33.134+0000 7ff194d77640 1 -- 172.21.15.77:0/2927649419 wait complete. 2024-03-16T10:20:33.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:32 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:20:33.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:32 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:20:33.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:32 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:20:34.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:33 smithi192 ceph-mon[36354]: pgmap v454: 97 pgs: 97 active+clean; 351 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 3.8 MiB/s wr, 301 op/s 2024-03-16T10:20:34.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:33 smithi077 ceph-mon[32400]: pgmap v454: 97 pgs: 97 active+clean; 351 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 3.8 MiB/s wr, 301 op/s 2024-03-16T10:20:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:33 smithi193 ceph-mon[36311]: pgmap v454: 97 pgs: 97 active+clean; 351 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 3.8 MiB/s wr, 301 op/s 2024-03-16T10:20:36.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:35 smithi192 ceph-mon[36354]: pgmap v455: 97 pgs: 97 active+clean; 365 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 4.7 MiB/s wr, 346 op/s 2024-03-16T10:20:36.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:35 smithi077 ceph-mon[32400]: pgmap v455: 97 pgs: 97 active+clean; 365 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 4.7 MiB/s wr, 346 op/s 2024-03-16T10:20:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:35 smithi193 ceph-mon[36311]: pgmap v455: 97 pgs: 97 active+clean; 365 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 4.7 MiB/s wr, 346 op/s 2024-03-16T10:20:37.103 INFO:tasks.workunit.client.0.smithi077.stdout: 75 500392 354 349435 124 275916 584 2024-03-16T10:20:37.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:36 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:20:37.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:36 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:20:37.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:36 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:20:37.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:36 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:20:37.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:36 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:20:37.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:36 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:20:37.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:36 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:20:37.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:36 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:20:37.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:36 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:20:38.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:37 smithi192 ceph-mon[36354]: pgmap v456: 97 pgs: 97 active+clean; 368 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 5.1 MiB/s wr, 362 op/s 2024-03-16T10:20:38.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:37 smithi077 ceph-mon[32400]: pgmap v456: 97 pgs: 97 active+clean; 368 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 5.1 MiB/s wr, 362 op/s 2024-03-16T10:20:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:37 smithi193 ceph-mon[36311]: pgmap v456: 97 pgs: 97 active+clean; 368 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 5.1 MiB/s wr, 362 op/s 2024-03-16T10:20:39.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:38 smithi192 ceph-mon[36354]: pgmap v457: 97 pgs: 97 active+clean; 371 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.2 MiB/s wr, 355 op/s 2024-03-16T10:20:39.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:38 smithi077 ceph-mon[32400]: pgmap v457: 97 pgs: 97 active+clean; 371 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.2 MiB/s wr, 355 op/s 2024-03-16T10:20:39.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:38 smithi193 ceph-mon[36311]: pgmap v457: 97 pgs: 97 active+clean; 371 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.2 MiB/s wr, 355 op/s 2024-03-16T10:20:42.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:41 smithi192 ceph-mon[36354]: pgmap v458: 97 pgs: 97 active+clean; 370 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 5.1 MiB/s wr, 353 op/s 2024-03-16T10:20:42.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:41 smithi077 ceph-mon[32400]: pgmap v458: 97 pgs: 97 active+clean; 370 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 5.1 MiB/s wr, 353 op/s 2024-03-16T10:20:42.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:41 smithi193 ceph-mon[36311]: pgmap v458: 97 pgs: 97 active+clean; 370 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 5.1 MiB/s wr, 353 op/s 2024-03-16T10:20:44.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:43 smithi192 ceph-mon[36354]: pgmap v459: 97 pgs: 97 active+clean; 376 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 321 op/s 2024-03-16T10:20:44.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:43 smithi077 ceph-mon[32400]: pgmap v459: 97 pgs: 97 active+clean; 376 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 321 op/s 2024-03-16T10:20:44.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:43 smithi193 ceph-mon[36311]: pgmap v459: 97 pgs: 97 active+clean; 376 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 321 op/s 2024-03-16T10:20:46.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:45 smithi192 ceph-mon[36354]: pgmap v460: 97 pgs: 97 active+clean; 376 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 195 op/s 2024-03-16T10:20:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.225+0000 7fcf519e6640 1 -- 172.21.15.77:0/3750320298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf4c1022f0 msgr2=0x7fcf4c108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.225+0000 7fcf519e6640 1 --2- 172.21.15.77:0/3750320298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf4c1022f0 0x7fcf4c108b80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009ca0 tx=0x7fcf4402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.225+0000 7fcf519e6640 1 -- 172.21.15.77:0/3750320298 shutdown_connections 2024-03-16T10:20:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.225+0000 7fcf519e6640 1 --2- 172.21.15.77:0/3750320298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf4c1022f0 0x7fcf4c108b80 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.225+0000 7fcf519e6640 1 --2- 172.21.15.77:0/3750320298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf4c101930 0x7fcf4c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.225+0000 7fcf519e6640 1 --2- 172.21.15.77:0/3750320298 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf4c100740 0x7fcf4c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.225+0000 7fcf519e6640 1 -- 172.21.15.77:0/3750320298 >> 172.21.15.77:0/3750320298 conn(0x7fcf4c0fbcb0 msgr2=0x7fcf4c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.225+0000 7fcf519e6640 1 -- 172.21.15.77:0/3750320298 shutdown_connections 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.225+0000 7fcf519e6640 1 -- 172.21.15.77:0/3750320298 wait complete. 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf519e6640 1 Processor -- start 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf519e6640 1 -- start start 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf519e6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf4c100740 0x7fcf4c199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf519e6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf4c101930 0x7fcf4c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf519e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf4c1022f0 0x7fcf4c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf519e6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c10c3a0 con 0x7fcf4c1022f0 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf519e6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c10c220 con 0x7fcf4c100740 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf519e6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c10c520 con 0x7fcf4c101930 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf4b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf4c1022f0 0x7fcf4c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf4b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf4c1022f0 0x7fcf4c1a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55008/0 (socket says 172.21.15.77:55008) 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf427fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf4c101930 0x7fcf4c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf4b7fe640 1 -- 172.21.15.77:0/4247597051 learned_addr learned my addr 172.21.15.77:0/4247597051 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf4affd640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf4c100740 0x7fcf4c199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf4b7fe640 1 -- 172.21.15.77:0/4247597051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf4c101930 msgr2=0x7fcf4c19a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf4b7fe640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf4c101930 0x7fcf4c19a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf4b7fe640 1 -- 172.21.15.77:0/4247597051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf4c100740 msgr2=0x7fcf4c199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf4b7fe640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf4c100740 0x7fcf4c199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.226+0000 7fcf4b7fe640 1 -- 172.21.15.77:0/4247597051 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4c1a1840 con 0x7fcf4c1022f0 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.227+0000 7fcf4b7fe640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf4c1022f0 0x7fcf4c1a11a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fcf44037040 tx=0x7fcf44032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.227+0000 7fcf509e4640 1 -- 172.21.15.77:0/4247597051 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44041020 con 0x7fcf4c1022f0 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.227+0000 7fcf519e6640 1 -- 172.21.15.77:0/4247597051 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf4c1a1ad0 con 0x7fcf4c1022f0 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.227+0000 7fcf519e6640 1 -- 172.21.15.77:0/4247597051 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf4c1a2010 con 0x7fcf4c1022f0 2024-03-16T10:20:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.227+0000 7fcf509e4640 1 -- 172.21.15.77:0/4247597051 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf44034920 con 0x7fcf4c1022f0 2024-03-16T10:20:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.227+0000 7fcf509e4640 1 -- 172.21.15.77:0/4247597051 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44039040 con 0x7fcf4c1022f0 2024-03-16T10:20:46.237 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.229+0000 7fcf545d1740 1 Processor -- start 2024-03-16T10:20:46.237 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf509e4640 1 -- 172.21.15.77:0/4247597051 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fcf44044020 con 0x7fcf4c1022f0 2024-03-16T10:20:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf545d1740 1 -- start start 2024-03-16T10:20:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf545d1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9a340 0x55914bc9a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf545d1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 0x55914bc9f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf545d1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bca2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf545d1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55914bbffa40 con 0x55914bc9fea0 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf545d1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55914bba8920 con 0x55914bc9ad30 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf545d1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55914bbfbae0 con 0x55914bc9a340 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf427fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bca2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf4affd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 0x55914bc9f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf427fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bca2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55020/0 (socket says 172.21.15.77:55020) 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf4affd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 0x55914bc9f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:55958/0 (socket says 172.21.15.77:55958) 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf427fc640 1 -- 172.21.15.77:0/3220705177 learned_addr learned my addr 172.21.15.77:0/3220705177 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf427fc640 1 --2- 172.21.15.77:0/3220705177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9a340 0x55914bc9a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf509e4640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf380a28a0 0x7fcf380a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf4affd640 1 -- 172.21.15.77:0/3220705177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9a340 msgr2=0x55914bc9a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf4affd640 1 --2- 172.21.15.77:0/3220705177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9a340 0x55914bc9a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf4affd640 1 -- 172.21.15.77:0/3220705177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 msgr2=0x55914bca2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf4affd640 1 --2- 172.21.15.77:0/3220705177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bca2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf4affd640 1 -- 172.21.15.77:0/3220705177 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55914bbc0460 con 0x55914bc9ad30 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf427fc640 1 --2- 172.21.15.77:0/3220705177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bca2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.230+0000 7fcf4b7fe640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf380a28a0 0x7fcf380a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.231+0000 7fcf509e4640 1 -- 172.21.15.77:0/4247597051 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fcf440f0270 con 0x7fcf4c1022f0 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.231+0000 7fcf4affd640 1 --2- 172.21.15.77:0/3220705177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 0x55914bc9f830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c101790 tx=0x7fcf3c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94c9d1adc8a50a76 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.231+0000 7fcf41ffb640 1 -- 172.21.15.77:0/3220705177 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c009d70 con 0x55914bc9ad30 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.231+0000 7fcf41ffb640 1 -- 172.21.15.77:0/3220705177 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf3c0035f0 con 0x55914bc9ad30 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.231+0000 7fcf41ffb640 1 -- 172.21.15.77:0/3220705177 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c019670 con 0x55914bc9ad30 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.231+0000 7fcf4b7fe640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf380a28a0 0x7fcf380a4d60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcf44005e30 tx=0x7fcf44087f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 -- 172.21.15.77:0/3220705177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 msgr2=0x55914bc9f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 --2- 172.21.15.77:0/3220705177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 0x55914bc9f830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c101790 tx=0x7fcf3c011810 comp rx=0 tx=0).stop 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 -- 172.21.15.77:0/3220705177 shutdown_connections 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 --2- 172.21.15.77:0/3220705177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bca2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 --2- 172.21.15.77:0/3220705177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 0x55914bc9f830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 --2- 172.21.15.77:0/3220705177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9a340 0x55914bc9a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 -- 172.21.15.77:0/3220705177 >> 172.21.15.77:0/3220705177 conn(0x55914bc958b0 msgr2=0x55914bc95cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 -- 172.21.15.77:0/3220705177 shutdown_connections 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 -- 172.21.15.77:0/3220705177 wait complete. 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 -- start start 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 Processor -- start 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9a340 0x55914bca4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.232+0000 7fcf545d1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 0x55914bca4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf545d1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9fea0 0x55914bcabbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf545d1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55914bca2e50 con 0x55914bc9a340 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf545d1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55914bbbbd70 con 0x55914bc9ad30 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf545d1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55914bca2fd0 con 0x55914bc9fea0 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf545d1740 10 client.0 _ll_register_callbacks cb 0x55914b913a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf427fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9a340 0x55914bca4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf4affd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 0x55914bca4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf4affd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 0x55914bca4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:55974/0 (socket says 172.21.15.77:55974) 2024-03-16T10:20:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf427fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9a340 0x55914bca4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55034/0 (socket says 172.21.15.77:55034) 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf4affd640 1 -- 172.21.15.77:0/3641016763 learned_addr learned my addr 172.21.15.77:0/3641016763 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf427fc640 1 --2- 172.21.15.77:0/3641016763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9fea0 0x55914bcabbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf4affd640 1 -- 172.21.15.77:0/3641016763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9fea0 msgr2=0x55914bcabbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf4affd640 1 --2- 172.21.15.77:0/3641016763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9fea0 0x55914bcabbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf4affd640 1 -- 172.21.15.77:0/3641016763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9a340 msgr2=0x55914bca4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf4affd640 1 --2- 172.21.15.77:0/3641016763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9a340 0x55914bca4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf4affd640 1 -- 172.21.15.77:0/3641016763 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55914b914210 con 0x55914bc9ad30 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf4affd640 1 --2- 172.21.15.77:0/3641016763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 0x55914bca4b80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c01b040 tx=0x7fcf3c009df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cefa174a4e393fd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.233+0000 7fcf2affd640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf2affd640 1 -- 172.21.15.77:0/3641016763 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c028020 con 0x55914bc9ad30 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf2affd640 1 -- 172.21.15.77:0/3641016763 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fcf3c016040 con 0x55914bc9ad30 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf545d1740 1 -- 172.21.15.77:0/3641016763 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf3c0035f0 con 0x55914bc9ad30 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf2affd640 1 -- 172.21.15.77:0/3641016763 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fcf3c033050 con 0x55914bc9ad30 2024-03-16T10:20:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf545d1740 10 client.26318 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf545d1740 1 -- 172.21.15.77:0/3641016763 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55914b9e0620 con 0x55914bc9ad30 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf2affd640 1 -- 172.21.15.77:0/3641016763 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf3c005520 con 0x55914bc9ad30 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf2affd640 1 -- 172.21.15.77:0/3641016763 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7fcf3c004910 con 0x55914bc9ad30 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf545d1740 10 client.26318 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf545d1740 10 client.26318 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf545d1740 20 client.26318 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf545d1740 1 --2- 172.21.15.77:0/3641016763 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55914bcb0390 0x55914bcb27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf545d1740 4 client.26318 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.234+0000 7fcf545d1740 1 -- 172.21.15.77:0/3641016763 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcf3c0035f0 con 0x55914bcb0390 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.235+0000 7fcf427fc640 1 --2- 172.21.15.77:0/3641016763 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55914bcb0390 0x55914bcb27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.235+0000 7fcf2affd640 10 client.26318 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.235+0000 7fcf427fc640 1 --2- 172.21.15.77:0/3641016763 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55914bcb0390 0x55914bcb27a0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=680ae96c289b702f server_cookie=6aba3254a1957fa5 in_seq=0 out_seq=0 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf2affd640 1 -- 172.21.15.77:0/3641016763 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fcf3c0035f0 con 0x55914bcb0390 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf2affd640 10 client.26318 handle_command_reply: tid=0 2024-03-16T10:20:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf545d1740 1 client.26318 shutdown 2024-03-16T10:20:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf545d1740 1 -- 172.21.15.77:0/3641016763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 msgr2=0x55914bca4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf545d1740 1 --2- 172.21.15.77:0/3641016763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 0x55914bca4b80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c01b040 tx=0x7fcf3c009df0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf545d1740 1 -- 172.21.15.77:0/3641016763 shutdown_connections 2024-03-16T10:20:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf545d1740 1 --2- 172.21.15.77:0/3641016763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9fea0 0x55914bcabbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf545d1740 1 --2- 172.21.15.77:0/3641016763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9ad30 0x55914bca4b80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf545d1740 1 --2- 172.21.15.77:0/3641016763 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55914bcb0390 0x55914bcb27a0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf545d1740 1 --2- 172.21.15.77:0/3641016763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9a340 0x55914bca4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf545d1740 1 -- 172.21.15.77:0/3641016763 >> 172.21.15.77:0/3641016763 conn(0x55914bc958b0 msgr2=0x55914bc94bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf2affd640 0 client.26318 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf545d1740 1 -- 172.21.15.77:0/3641016763 shutdown_connections 2024-03-16T10:20:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf545d1740 1 -- 172.21.15.77:0/3641016763 wait complete. 2024-03-16T10:20:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.236+0000 7fcf545d1740 20 client.26318 trim_cache size 0 max 16384 2024-03-16T10:20:46.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:45 smithi077 ceph-mon[32400]: pgmap v460: 97 pgs: 97 active+clean; 376 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 195 op/s 2024-03-16T10:20:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:45 smithi193 ceph-mon[36311]: pgmap v460: 97 pgs: 97 active+clean; 376 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 195 op/s 2024-03-16T10:20:46.255 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf545d1740 1 Processor -- start 2024-03-16T10:20:46.255 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf545d1740 1 -- start start 2024-03-16T10:20:46.255 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf545d1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9a340 0x55914bc90a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.255 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf545d1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9ad30 0x55914bcabb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.255 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf545d1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bc94320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.255 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf545d1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c00e330 con 0x55914bc9fea0 2024-03-16T10:20:46.255 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf545d1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c00ff60 con 0x55914bc9a340 2024-03-16T10:20:46.255 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf545d1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c00ea00 con 0x55914bc9ad30 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf427fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bc94320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf4affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9ad30 0x55914bcabb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf427fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bc94320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55042/0 (socket says 172.21.15.77:55042) 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf427fc640 1 -- 172.21.15.77:0/2795402342 learned_addr learned my addr 172.21.15.77:0/2795402342 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.253+0000 7fcf427fc640 1 --2- 172.21.15.77:0/2795402342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9a340 0x55914bc90a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.254+0000 7fcf427fc640 1 -- 172.21.15.77:0/2795402342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9ad30 msgr2=0x55914bcabb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.254+0000 7fcf427fc640 1 --2- 172.21.15.77:0/2795402342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9ad30 0x55914bcabb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.254+0000 7fcf427fc640 1 -- 172.21.15.77:0/2795402342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9a340 msgr2=0x55914bc90a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.254+0000 7fcf427fc640 1 --2- 172.21.15.77:0/2795402342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9a340 0x55914bc90a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.254+0000 7fcf427fc640 1 -- 172.21.15.77:0/2795402342 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55914bc91310 con 0x55914bc9fea0 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.254+0000 7fcf4affd640 1 --2- 172.21.15.77:0/2795402342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9ad30 0x55914bcabb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.254+0000 7fcf427fc640 1 --2- 172.21.15.77:0/2795402342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bc94320 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fcf340041c0 tx=0x7fcf34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7546e1602bd347f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:46.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.254+0000 7fcf417fa640 1 -- 172.21.15.77:0/2795402342 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3401a040 con 0x55914bc9fea0 2024-03-16T10:20:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.254+0000 7fcf417fa640 1 -- 172.21.15.77:0/2795402342 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf34013d30 con 0x55914bc9fea0 2024-03-16T10:20:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.254+0000 7fcf417fa640 1 -- 172.21.15.77:0/2795402342 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf34005020 con 0x55914bc9fea0 2024-03-16T10:20:46.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.255+0000 7fcf545d1740 1 -- 172.21.15.77:0/2795402342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 msgr2=0x55914bc94320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.255+0000 7fcf545d1740 1 --2- 172.21.15.77:0/2795402342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bc94320 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fcf340041c0 tx=0x7fcf34004730 comp rx=0 tx=0).stop 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.255+0000 7fcf545d1740 1 -- 172.21.15.77:0/2795402342 shutdown_connections 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.255+0000 7fcf545d1740 1 --2- 172.21.15.77:0/2795402342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bc94320 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.255+0000 7fcf545d1740 1 --2- 172.21.15.77:0/2795402342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9ad30 0x55914bcabb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.255+0000 7fcf545d1740 1 --2- 172.21.15.77:0/2795402342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9a340 0x55914bc90a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.255+0000 7fcf545d1740 1 -- 172.21.15.77:0/2795402342 >> 172.21.15.77:0/2795402342 conn(0x55914bc958b0 msgr2=0x55914bca3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.255+0000 7fcf545d1740 1 -- 172.21.15.77:0/2795402342 shutdown_connections 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.255+0000 7fcf545d1740 1 -- 172.21.15.77:0/2795402342 wait complete. 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf545d1740 1 -- start start 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf545d1740 1 Processor -- start 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf545d1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9a340 0x55914bca9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf545d1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9ad30 0x55914bca9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf545d1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bcaaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf545d1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c00f860 con 0x55914bc9fea0 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf545d1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c00f190 con 0x55914bc9a340 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf545d1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c00c310 con 0x55914bc9ad30 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf545d1740 10 client.0 _ll_register_callbacks cb 0x7fcf34009b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf427fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bcaaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf427fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bcaaa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55044/0 (socket says 172.21.15.77:55044) 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf427fc640 1 -- 172.21.15.77:0/2056388748 learned_addr learned my addr 172.21.15.77:0/2056388748 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf4affd640 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9ad30 0x55914bca9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.256+0000 7fcf427fc640 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9a340 0x55914bca9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf427fc640 1 -- 172.21.15.77:0/2056388748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9ad30 msgr2=0x55914bca9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf427fc640 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9ad30 0x55914bca9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf427fc640 1 -- 172.21.15.77:0/2056388748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9a340 msgr2=0x55914bca9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf427fc640 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9a340 0x55914bca9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf427fc640 1 -- 172.21.15.77:0/2056388748 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55914bcab000 con 0x55914bc9fea0 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf4affd640 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9ad30 0x55914bca9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf427fc640 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bcaaa10 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcf340135a0 tx=0x7fcf340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37919934117ca154 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf2affd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf2affd640 1 -- 172.21.15.77:0/2056388748 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf34011980 con 0x55914bc9fea0 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf545d1740 1 -- 172.21.15.77:0/2056388748 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf34013d30 con 0x55914bc9fea0 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf2affd640 1 -- 172.21.15.77:0/2056388748 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fcf34011b20 con 0x55914bc9fea0 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf2affd640 1 -- 172.21.15.77:0/2056388748 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fcf34011d20 con 0x55914bc9fea0 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf545d1740 10 client.16557 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf545d1740 1 -- 172.21.15.77:0/2056388748 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55914bd16930 con 0x55914bc9fea0 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf2affd640 1 -- 172.21.15.77:0/2056388748 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf34013d30 con 0x55914bc9fea0 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.257+0000 7fcf2affd640 1 -- 172.21.15.77:0/2056388748 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7fcf34003920 con 0x55914bc9fea0 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.258+0000 7fcf545d1740 10 client.16557 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.258+0000 7fcf545d1740 10 client.16557 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.258+0000 7fcf545d1740 20 client.16557 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.258+0000 7fcf545d1740 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55914bd17370 0x55914bd19790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.258+0000 7fcf545d1740 4 client.16557 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:20:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.258+0000 7fcf545d1740 1 -- 172.21.15.77:0/2056388748 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55914bc0cf80 con 0x55914bd17370 2024-03-16T10:20:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.258+0000 7fcf4affd640 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55914bd17370 0x55914bd19790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.258+0000 7fcf2affd640 10 client.16557 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.258+0000 7fcf4affd640 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55914bd17370 0x55914bd19790 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37cb752bbdc61f40 server_cookie=4fd9aa2776febe6c in_seq=0 out_seq=0 2024-03-16T10:20:46.546 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.545+0000 7fcf2affd640 1 -- 172.21.15.77:0/2056388748 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55914bc0cf80 con 0x55914bd17370 2024-03-16T10:20:46.546 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.545+0000 7fcf2affd640 10 client.16557 handle_command_reply: tid=0 2024-03-16T10:20:46.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.547+0000 7fcf545d1740 1 client.16557 shutdown 2024-03-16T10:20:46.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.547+0000 7fcf545d1740 1 -- 172.21.15.77:0/2056388748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 msgr2=0x55914bcaaa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:46.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.547+0000 7fcf545d1740 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bcaaa10 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcf340135a0 tx=0x7fcf340046c0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.547+0000 7fcf545d1740 1 -- 172.21.15.77:0/2056388748 shutdown_connections 2024-03-16T10:20:46.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.547+0000 7fcf545d1740 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55914bd17370 0x55914bd19790 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.547+0000 7fcf545d1740 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55914bc9fea0 0x55914bcaaa10 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.547+0000 7fcf545d1740 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55914bc9ad30 0x55914bca9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.547+0000 7fcf545d1740 1 --2- 172.21.15.77:0/2056388748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55914bc9a340 0x55914bca9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.547+0000 7fcf545d1740 1 -- 172.21.15.77:0/2056388748 >> 172.21.15.77:0/2056388748 conn(0x55914bc958b0 msgr2=0x55914bc90a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.547+0000 7fcf2affd640 0 client.16557 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.547+0000 7fcf545d1740 1 -- 172.21.15.77:0/2056388748 shutdown_connections 2024-03-16T10:20:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.548+0000 7fcf545d1740 1 -- 172.21.15.77:0/2056388748 wait complete. 2024-03-16T10:20:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.548+0000 7fcf545d1740 20 client.16557 trim_cache size 0 max 16384 2024-03-16T10:20:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:20:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:20:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:20:46.550 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:20:46.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.549+0000 7fcf519e6640 1 -- 172.21.15.77:0/4247597051 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf380a28a0 msgr2=0x7fcf380a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:46.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.549+0000 7fcf519e6640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf380a28a0 0x7fcf380a4d60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcf44005e30 tx=0x7fcf44087f00 comp rx=0 tx=0).stop 2024-03-16T10:20:46.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.549+0000 7fcf519e6640 1 -- 172.21.15.77:0/4247597051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf4c1022f0 msgr2=0x7fcf4c1a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:46.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.549+0000 7fcf519e6640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf4c1022f0 0x7fcf4c1a11a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fcf44037040 tx=0x7fcf44032f40 comp rx=0 tx=0).stop 2024-03-16T10:20:46.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.550+0000 7fcf519e6640 1 -- 172.21.15.77:0/4247597051 shutdown_connections 2024-03-16T10:20:46.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.550+0000 7fcf519e6640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf4c1022f0 0x7fcf4c1a11a0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.550+0000 7fcf519e6640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf380a28a0 0x7fcf380a4d60 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.550+0000 7fcf519e6640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf4c101930 0x7fcf4c19a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.550+0000 7fcf519e6640 1 --2- 172.21.15.77:0/4247597051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf4c100740 0x7fcf4c199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:46.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.550+0000 7fcf519e6640 1 -- 172.21.15.77:0/4247597051 >> 172.21.15.77:0/4247597051 conn(0x7fcf4c0fbcb0 msgr2=0x7fcf4c069a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:46.551 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.550+0000 7fcf519e6640 1 -- 172.21.15.77:0/4247597051 shutdown_connections 2024-03-16T10:20:46.552 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:46.550+0000 7fcf519e6640 1 -- 172.21.15.77:0/4247597051 wait complete. 2024-03-16T10:20:47.103 INFO:tasks.workunit.client.0.smithi077.stdout: 81 684771 319 470511 311 368444 1166 2024-03-16T10:20:48.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:47 smithi192 ceph-mon[36354]: pgmap v461: 97 pgs: 97 active+clean; 377 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 153 op/s 2024-03-16T10:20:48.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:47 smithi077 ceph-mon[32400]: pgmap v461: 97 pgs: 97 active+clean; 377 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 153 op/s 2024-03-16T10:20:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:47 smithi193 ceph-mon[36311]: pgmap v461: 97 pgs: 97 active+clean; 377 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 153 op/s 2024-03-16T10:20:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:49 smithi192 ceph-mon[36354]: pgmap v462: 97 pgs: 97 active+clean; 377 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-03-16T10:20:50.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:49 smithi077 ceph-mon[32400]: pgmap v462: 97 pgs: 97 active+clean; 377 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-03-16T10:20:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:49 smithi193 ceph-mon[36311]: pgmap v462: 97 pgs: 97 active+clean; 377 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-03-16T10:20:52.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:51 smithi192 ceph-mon[36354]: pgmap v463: 97 pgs: 97 active+clean; 380 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-03-16T10:20:52.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:51 smithi077 ceph-mon[32400]: pgmap v463: 97 pgs: 97 active+clean; 380 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-03-16T10:20:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:51 smithi193 ceph-mon[36311]: pgmap v463: 97 pgs: 97 active+clean; 380 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-03-16T10:20:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:53 smithi192 ceph-mon[36354]: pgmap v464: 97 pgs: 97 active+clean; 379 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 159 op/s 2024-03-16T10:20:54.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:53 smithi077 ceph-mon[32400]: pgmap v464: 97 pgs: 97 active+clean; 379 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 159 op/s 2024-03-16T10:20:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:53 smithi193 ceph-mon[36311]: pgmap v464: 97 pgs: 97 active+clean; 379 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 159 op/s 2024-03-16T10:20:56.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:55 smithi192 ceph-mon[36354]: pgmap v465: 97 pgs: 97 active+clean; 383 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-03-16T10:20:56.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:55 smithi077 ceph-mon[32400]: pgmap v465: 97 pgs: 97 active+clean; 383 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-03-16T10:20:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:55 smithi193 ceph-mon[36311]: pgmap v465: 97 pgs: 97 active+clean; 383 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-03-16T10:20:57.104 INFO:tasks.workunit.client.0.smithi077.stdout: 85 1149065 348 787845 267 637658 813 2024-03-16T10:20:57.178 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:20:57.178 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:20:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.471+0000 7f94f5777640 1 -- 172.21.15.77:0/939010052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f0100fe0 msgr2=0x7f94f0101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.471+0000 7f94f5777640 1 --2- 172.21.15.77:0/939010052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f0100fe0 0x7f94f0101490 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f94e8009cd0 tx=0x7f94e802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.471+0000 7f94f5777640 1 -- 172.21.15.77:0/939010052 shutdown_connections 2024-03-16T10:20:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.471+0000 7f94f5777640 1 --2- 172.21.15.77:0/939010052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f0100fe0 0x7f94f0101490 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.471+0000 7f94f5777640 1 --2- 172.21.15.77:0/939010052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94f00698e0 0x7f94f01003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.471+0000 7f94f5777640 1 --2- 172.21.15.77:0/939010052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94f00694c0 0x7f94f00ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.471+0000 7f94f5777640 1 -- 172.21.15.77:0/939010052 >> 172.21.15.77:0/939010052 conn(0x7f94f00fb400 msgr2=0x7f94f00fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:57.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.471+0000 7f94f5777640 1 -- 172.21.15.77:0/939010052 shutdown_connections 2024-03-16T10:20:57.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.471+0000 7f94f5777640 1 -- 172.21.15.77:0/939010052 wait complete. 2024-03-16T10:20:57.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.471+0000 7f94f5777640 1 Processor -- start 2024-03-16T10:20:57.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94f5777640 1 -- start start 2024-03-16T10:20:57.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94f5777640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94f00694c0 0x7f94f0102e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94f5777640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f00698e0 0x7f94f019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94f5777640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94f0100fe0 0x7f94f019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94f5777640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f94f010c410 con 0x7f94f00698e0 2024-03-16T10:20:57.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94f5777640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f94f010c290 con 0x7f94f00694c0 2024-03-16T10:20:57.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94f5777640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f94f010c590 con 0x7f94f0100fe0 2024-03-16T10:20:57.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94ee7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f00698e0 0x7f94f019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94eeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94f00694c0 0x7f94f0102e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94ee7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f00698e0 0x7f94f019c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54380/0 (socket says 172.21.15.77:54380) 2024-03-16T10:20:57.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94ef7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94f0100fe0 0x7f94f019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94ee7fc640 1 -- 172.21.15.77:0/2903833777 learned_addr learned my addr 172.21.15.77:0/2903833777 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:57.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94ee7fc640 1 -- 172.21.15.77:0/2903833777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94f0100fe0 msgr2=0x7f94f019eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94ee7fc640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94f0100fe0 0x7f94f019eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94ee7fc640 1 -- 172.21.15.77:0/2903833777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94f00694c0 msgr2=0x7f94f0102e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94ee7fc640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94f00694c0 0x7f94f0102e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.476 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94ee7fc640 1 -- 172.21.15.77:0/2903833777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f019f760 con 0x7f94f00698e0 2024-03-16T10:20:57.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.472+0000 7f94eeffd640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94f00694c0 0x7f94f0102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:57.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.473+0000 7f94ee7fc640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f00698e0 0x7f94f019c860 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f94e400e9c0 tx=0x7f94e400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:57.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.473+0000 7f94d7fff640 1 -- 172.21.15.77:0/2903833777 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e400ccc0 con 0x7f94f00698e0 2024-03-16T10:20:57.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.473+0000 7f94d7fff640 1 -- 172.21.15.77:0/2903833777 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e4005500 con 0x7f94f00698e0 2024-03-16T10:20:57.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.473+0000 7f94f5777640 1 -- 172.21.15.77:0/2903833777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94f019fa50 con 0x7f94f00698e0 2024-03-16T10:20:57.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.473+0000 7f94d7fff640 1 -- 172.21.15.77:0/2903833777 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e4010690 con 0x7f94f00698e0 2024-03-16T10:20:57.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.473+0000 7f94f5777640 1 -- 172.21.15.77:0/2903833777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94f019ff50 con 0x7f94f00698e0 2024-03-16T10:20:57.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.475+0000 7f94f7ffd740 1 Processor -- start 2024-03-16T10:20:57.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.475+0000 7f94d7fff640 1 -- 172.21.15.77:0/2903833777 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f94e4004e80 con 0x7f94f00698e0 2024-03-16T10:20:57.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.475+0000 7f94f7ffd740 1 -- start start 2024-03-16T10:20:57.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94f7ffd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15170 0x55ab3ab15590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94f7ffd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab15b60 0x55ab3ab1a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94f7ffd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab1d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94f7ffd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ab3aa7a500 con 0x55ab3ab15b60 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94f7ffd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ab3aa22be0 con 0x55ab3ab1acd0 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94f7ffd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ab3aa765a0 con 0x55ab3ab15170 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94ee7fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab1d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94eeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab15b60 0x55ab3ab1a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94ef7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15170 0x55ab3ab15590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94eeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab15b60 0x55ab3ab1a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54388/0 (socket says 172.21.15.77:54388) 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94eeffd640 1 -- 172.21.15.77:0/1824832014 learned_addr learned my addr 172.21.15.77:0/1824832014 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94ef7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15170 0x55ab3ab15590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52444/0 (socket says 172.21.15.77:52444) 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94eeffd640 1 -- 172.21.15.77:0/1824832014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15170 msgr2=0x55ab3ab15590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94eeffd640 1 --2- 172.21.15.77:0/1824832014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15170 0x55ab3ab15590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94eeffd640 1 -- 172.21.15.77:0/1824832014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab1acd0 msgr2=0x55ab3ab1d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94eeffd640 1 --2- 172.21.15.77:0/1824832014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab1d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94eeffd640 1 -- 172.21.15.77:0/1824832014 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab3aa364b0 con 0x55ab3ab15b60 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94d7fff640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94d80a2800 0x7f94d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94ee7fc640 1 --2- 172.21.15.77:0/1824832014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab1d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94eeffd640 1 --2- 172.21.15.77:0/1824832014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab15b60 0x55ab3ab1a660 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f94e00115f0 tx=0x7f94e0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2d98c1845bfbac server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94d7fff640 1 -- 172.21.15.77:0/2903833777 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f94e40cedc0 con 0x7f94f00698e0 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.476+0000 7f94ef7fe640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94d80a2800 0x7f94d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.477+0000 7f94d57fa640 1 -- 172.21.15.77:0/1824832014 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e001e070 con 0x55ab3ab15b60 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.477+0000 7f94d57fa640 1 -- 172.21.15.77:0/1824832014 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e000fb20 con 0x55ab3ab15b60 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.477+0000 7f94d57fa640 1 -- 172.21.15.77:0/1824832014 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e0019c40 con 0x55ab3ab15b60 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.478+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1824832014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab15b60 msgr2=0x55ab3ab1a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.478+0000 7f94ef7fe640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94d80a2800 0x7f94d80a4cc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f94e8009ca0 tx=0x7f94e8032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.478+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/1824832014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab15b60 0x55ab3ab1a660 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f94e00115f0 tx=0x7f94e0011ab0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.478+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1824832014 shutdown_connections 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.478+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/1824832014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab1d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.478+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/1824832014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab15b60 0x55ab3ab1a660 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.478+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/1824832014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15170 0x55ab3ab15590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.478+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1824832014 >> 172.21.15.77:0/1824832014 conn(0x55ab3ab106e0 msgr2=0x55ab3ab10ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.478+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1824832014 shutdown_connections 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.478+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1824832014 wait complete. 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.478+0000 7f94f7ffd740 1 -- start start 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.478+0000 7f94f7ffd740 1 Processor -- start 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94f7ffd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 0x55ab3ab1f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94f7ffd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15b60 0x55ab3ab1fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94f7ffd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab26c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94f7ffd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ab3ab1dc80 con 0x55ab3ab1acd0 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94f7ffd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ab3aa36160 con 0x55ab3ab15170 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94f7ffd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ab3ab1de00 con 0x55ab3ab15b60 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94f7ffd740 10 client.0 _ll_register_callbacks cb 0x55ab3aa1a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94eeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 0x55ab3ab1f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94eeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 0x55ab3ab1f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41960/0 (socket says 172.21.15.77:41960) 2024-03-16T10:20:57.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94eeffd640 1 -- 172.21.15.77:0/1381823018 learned_addr learned my addr 172.21.15.77:0/1381823018 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94ee7fc640 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15b60 0x55ab3ab1fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94ef7fe640 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab26c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94eeffd640 1 -- 172.21.15.77:0/1381823018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15b60 msgr2=0x55ab3ab1fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94eeffd640 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15b60 0x55ab3ab1fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94eeffd640 1 -- 172.21.15.77:0/1381823018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 msgr2=0x55ab3ab26c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94eeffd640 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab26c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94eeffd640 1 -- 172.21.15.77:0/1381823018 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab3a8587d0 con 0x55ab3ab15170 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94eeffd640 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 0x55ab3ab1f6e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f94e00115c0 tx=0x7f94e000f870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37c4a4323ede0a9a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.479+0000 7f94d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94d2ff5640 1 -- 172.21.15.77:0/1381823018 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e0027060 con 0x55ab3ab15170 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94d2ff5640 1 -- 172.21.15.77:0/1381823018 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f94e00199c0 con 0x55ab3ab15170 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1381823018 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94e000fb20 con 0x55ab3ab15170 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94d2ff5640 1 -- 172.21.15.77:0/1381823018 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f94e001e040 con 0x55ab3ab15170 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94f7ffd740 10 client.26336 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:57.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1381823018 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ab3a78e3f0 con 0x55ab3ab15170 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94d2ff5640 1 -- 172.21.15.77:0/1381823018 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e000fb20 con 0x55ab3ab15170 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94d2ff5640 1 -- 172.21.15.77:0/1381823018 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f94e0021070 con 0x55ab3ab15170 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94f7ffd740 10 client.26336 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94f7ffd740 10 client.26336 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94f7ffd740 20 client.26336 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ab3ab2ae60 0x55ab3ab2d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94f7ffd740 4 client.26336 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.480+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1381823018 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab3aa87a40 con 0x55ab3ab2ae60 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.481+0000 7f94ee7fc640 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ab3ab2ae60 0x55ab3ab2d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.481+0000 7f94d2ff5640 10 client.26336 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.481+0000 7f94ee7fc640 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ab3ab2ae60 0x55ab3ab2d2b0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2732880cb03a649e server_cookie=9dd573cd59f80f58 in_seq=0 out_seq=0 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94d2ff5640 1 -- 172.21.15.77:0/1381823018 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ab3aa87a40 con 0x55ab3ab2ae60 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94d2ff5640 10 client.26336 handle_command_reply: tid=0 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94f7ffd740 1 client.26336 shutdown 2024-03-16T10:20:57.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1381823018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 msgr2=0x55ab3ab1f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 0x55ab3ab1f6e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f94e00115c0 tx=0x7f94e000f870 comp rx=0 tx=0).stop 2024-03-16T10:20:57.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1381823018 shutdown_connections 2024-03-16T10:20:57.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ab3ab2ae60 0x55ab3ab2d2b0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab26c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15b60 0x55ab3ab1fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/1381823018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 0x55ab3ab1f6e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1381823018 >> 172.21.15.77:0/1381823018 conn(0x55ab3ab106e0 msgr2=0x55ab3ab0f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:57.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94d2ff5640 0 client.26336 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:57.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1381823018 shutdown_connections 2024-03-16T10:20:57.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94f7ffd740 1 -- 172.21.15.77:0/1381823018 wait complete. 2024-03-16T10:20:57.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.482+0000 7f94f7ffd740 20 client.26336 trim_cache size 0 max 16384 2024-03-16T10:20:57.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.517+0000 7f94f7ffd740 1 Processor -- start 2024-03-16T10:20:57.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94f7ffd740 1 -- start start 2024-03-16T10:20:57.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94f7ffd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15170 0x55ab3ab14050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94f7ffd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15b60 0x55ab3ab26c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94f7ffd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab1ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94f7ffd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f94f01a8c20 con 0x55ab3ab1acd0 2024-03-16T10:20:57.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94f7ffd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f94f01aa850 con 0x55ab3ab15b60 2024-03-16T10:20:57.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94f7ffd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f94f01a92f0 con 0x55ab3ab15170 2024-03-16T10:20:57.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94ef7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab1ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94ee7fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15b60 0x55ab3ab26c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94eeffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15170 0x55ab3ab14050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94ef7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab1ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54410/0 (socket says 172.21.15.77:54410) 2024-03-16T10:20:57.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94ee7fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15b60 0x55ab3ab26c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41970/0 (socket says 172.21.15.77:41970) 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94ef7fe640 1 -- 172.21.15.77:0/951691626 learned_addr learned my addr 172.21.15.77:0/951691626 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94ef7fe640 1 -- 172.21.15.77:0/951691626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15170 msgr2=0x55ab3ab14050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94ef7fe640 1 --2- 172.21.15.77:0/951691626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15170 0x55ab3ab14050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94ef7fe640 1 -- 172.21.15.77:0/951691626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15b60 msgr2=0x55ab3ab26c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94ef7fe640 1 --2- 172.21.15.77:0/951691626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15b60 0x55ab3ab26c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94ef7fe640 1 -- 172.21.15.77:0/951691626 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab3a74e0f0 con 0x55ab3ab1acd0 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94eeffd640 1 --2- 172.21.15.77:0/951691626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15170 0x55ab3ab14050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.518+0000 7f94ef7fe640 1 --2- 172.21.15.77:0/951691626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab1ed00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f94e8036000 tx=0x7f94e8041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd711c40e42ae16c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.519+0000 7f94d4ff9640 1 -- 172.21.15.77:0/951691626 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e8039040 con 0x55ab3ab1acd0 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.519+0000 7f94d4ff9640 1 -- 172.21.15.77:0/951691626 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e8053070 con 0x55ab3ab1acd0 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.519+0000 7f94d4ff9640 1 -- 172.21.15.77:0/951691626 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e8004800 con 0x55ab3ab1acd0 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.519+0000 7f94f7ffd740 1 -- 172.21.15.77:0/951691626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 msgr2=0x55ab3ab1ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.519+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/951691626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab1ed00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f94e8036000 tx=0x7f94e8041000 comp rx=0 tx=0).stop 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.519+0000 7f94f7ffd740 1 -- 172.21.15.77:0/951691626 shutdown_connections 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.519+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/951691626 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab1ed00 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.519+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/951691626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15b60 0x55ab3ab26c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.519+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/951691626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15170 0x55ab3ab14050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.519+0000 7f94f7ffd740 1 -- 172.21.15.77:0/951691626 >> 172.21.15.77:0/951691626 conn(0x55ab3ab106e0 msgr2=0x55ab3ab13940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.519+0000 7f94f7ffd740 1 -- 172.21.15.77:0/951691626 shutdown_connections 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.519+0000 7f94f7ffd740 1 -- 172.21.15.77:0/951691626 wait complete. 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94f7ffd740 1 -- start start 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94f7ffd740 1 Processor -- start 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94f7ffd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 0x55ab3ab23b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94f7ffd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15b60 0x55ab3ab240c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94f7ffd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab8f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94f7ffd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f94f01aa150 con 0x55ab3ab1acd0 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94f7ffd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f94f01a9a80 con 0x55ab3ab15170 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94f7ffd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f94f01a6c20 con 0x55ab3ab15b60 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94f7ffd740 10 client.0 _ll_register_callbacks cb 0x7f94e001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94ef7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab8f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94eeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 0x55ab3ab23b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94ee7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15b60 0x55ab3ab240c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94eeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 0x55ab3ab23b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41984/0 (socket says 172.21.15.77:41984) 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94ef7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab8f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54426/0 (socket says 172.21.15.77:54426) 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94eeffd640 1 -- 172.21.15.77:0/296335224 learned_addr learned my addr 172.21.15.77:0/296335224 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94ee7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15b60 0x55ab3ab240c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52470/0 (socket says 172.21.15.77:52470) 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94ee7fc640 1 -- 172.21.15.77:0/296335224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 msgr2=0x55ab3ab23b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94ee7fc640 1 --2- 172.21.15.77:0/296335224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 0x55ab3ab23b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94ee7fc640 1 -- 172.21.15.77:0/296335224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 msgr2=0x55ab3ab8f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:57.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94ee7fc640 1 --2- 172.21.15.77:0/296335224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab8f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94ee7fc640 1 -- 172.21.15.77:0/296335224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab3aa419d0 con 0x55ab3ab15b60 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.520+0000 7f94ef7fe640 1 --2- 172.21.15.77:0/296335224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab8f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94ee7fc640 1 --2- 172.21.15.77:0/296335224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15b60 0x55ab3ab240c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f94e40cc9a0 tx=0x7f94e4086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc7a0f7c3138a222 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94d2ff5640 1 -- 172.21.15.77:0/296335224 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e40b3070 con 0x55ab3ab15b60 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94f7ffd740 1 -- 172.21.15.77:0/296335224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94e8053070 con 0x55ab3ab15b60 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94d2ff5640 1 -- 172.21.15.77:0/296335224 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f94e4099af0 con 0x55ab3ab15b60 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94d2ff5640 1 -- 172.21.15.77:0/296335224 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f94e409b150 con 0x55ab3ab15b60 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94f7ffd740 10 client.26332 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94f7ffd740 1 -- 172.21.15.77:0/296335224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ab3ab90870 con 0x55ab3ab15b60 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94d2ff5640 1 -- 172.21.15.77:0/296335224 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e8053070 con 0x55ab3ab15b60 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94d2ff5640 1 -- 172.21.15.77:0/296335224 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f94e40bb040 con 0x55ab3ab15b60 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94f7ffd740 10 client.26332 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94f7ffd740 10 client.26332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:57.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.521+0000 7f94f7ffd740 20 client.26332 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/296335224 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ab3ab91200 0x55ab3ab93620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94f7ffd740 4 client.26332 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94f7ffd740 1 -- 172.21.15.77:0/296335224 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ab3aa81d90 con 0x55ab3ab91200 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94eeffd640 1 --2- 172.21.15.77:0/296335224 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ab3ab91200 0x55ab3ab93620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94d2ff5640 10 client.26332 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94eeffd640 1 --2- 172.21.15.77:0/296335224 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ab3ab91200 0x55ab3ab93620 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=426100325324ef23 server_cookie=d5692b4e0192d9f1 in_seq=0 out_seq=0 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94d2ff5640 1 -- 172.21.15.77:0/296335224 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ab3aa81d90 con 0x55ab3ab91200 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94d2ff5640 10 client.26332 handle_command_reply: tid=0 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94f7ffd740 1 client.26332 shutdown 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94f7ffd740 1 -- 172.21.15.77:0/296335224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15b60 msgr2=0x55ab3ab240c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/296335224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15b60 0x55ab3ab240c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f94e40cc9a0 tx=0x7f94e4086040 comp rx=0 tx=0).stop 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94f7ffd740 1 -- 172.21.15.77:0/296335224 shutdown_connections 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/296335224 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ab3ab91200 0x55ab3ab93620 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/296335224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3ab1acd0 0x55ab3ab8f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/296335224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab3ab15b60 0x55ab3ab240c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94d2ff5640 0 client.26332 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:57.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94f7ffd740 1 --2- 172.21.15.77:0/296335224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab3ab15170 0x55ab3ab23b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.522+0000 7f94f7ffd740 1 -- 172.21.15.77:0/296335224 >> 172.21.15.77:0/296335224 conn(0x55ab3ab106e0 msgr2=0x55ab3ab141d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:57.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f7ffd740 1 -- 172.21.15.77:0/296335224 shutdown_connections 2024-03-16T10:20:57.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f7ffd740 1 -- 172.21.15.77:0/296335224 wait complete. 2024-03-16T10:20:57.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f7ffd740 20 client.26332 trim_cache size 0 max 16384 2024-03-16T10:20:57.530 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:20:57.530 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T10:20:57.531 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T10:20:57.531 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:20:57.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f5777640 1 -- 172.21.15.77:0/2903833777 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94d80a2800 msgr2=0x7f94d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f5777640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94d80a2800 0x7f94d80a4cc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f94e8009ca0 tx=0x7f94e8032590 comp rx=0 tx=0).stop 2024-03-16T10:20:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f5777640 1 -- 172.21.15.77:0/2903833777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f00698e0 msgr2=0x7f94f019c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f5777640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f00698e0 0x7f94f019c860 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f94e400e9c0 tx=0x7f94e400ee80 comp rx=0 tx=0).stop 2024-03-16T10:20:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f5777640 1 -- 172.21.15.77:0/2903833777 shutdown_connections 2024-03-16T10:20:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f5777640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94d80a2800 0x7f94d80a4cc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f5777640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94f0100fe0 0x7f94f019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f5777640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f00698e0 0x7f94f019c860 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f5777640 1 --2- 172.21.15.77:0/2903833777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94f00694c0 0x7f94f0102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f5777640 1 -- 172.21.15.77:0/2903833777 >> 172.21.15.77:0/2903833777 conn(0x7f94f00fb400 msgr2=0x7f94f00fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f5777640 1 -- 172.21.15.77:0/2903833777 shutdown_connections 2024-03-16T10:20:57.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.523+0000 7f94f5777640 1 -- 172.21.15.77:0/2903833777 wait complete. 2024-03-16T10:20:57.536 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T10:20:57.536 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:20:57.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.678+0000 7f188c5b3640 1 -- 172.21.15.77:0/2492181115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18841018f0 msgr2=0x7f1884101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.678+0000 7f188c5b3640 1 --2- 172.21.15.77:0/2492181115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18841018f0 0x7f1884101d70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1874009900 tx=0x7f187402bc30 comp rx=0 tx=0).stop 2024-03-16T10:20:57.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 -- 172.21.15.77:0/2492181115 shutdown_connections 2024-03-16T10:20:57.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 --2- 172.21.15.77:0/2492181115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18841022b0 0x7f1884108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 --2- 172.21.15.77:0/2492181115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18841018f0 0x7f1884101d70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 --2- 172.21.15.77:0/2492181115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1884100700 0x7f1884100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 -- 172.21.15.77:0/2492181115 >> 172.21.15.77:0/2492181115 conn(0x7f18840fbc70 msgr2=0x7f18840fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:57.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 -- 172.21.15.77:0/2492181115 shutdown_connections 2024-03-16T10:20:57.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 -- 172.21.15.77:0/2492181115 wait complete. 2024-03-16T10:20:57.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 Processor -- start 2024-03-16T10:20:57.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 -- start start 2024-03-16T10:20:57.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1884100700 0x7f1884197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18841018f0 0x7f18841980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18841022b0 0x7f188419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f188410c380 con 0x7f1884100700 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f188410c200 con 0x7f18841022b0 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.679+0000 7f188c5b3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f188410c500 con 0x7f18841018f0 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f188a34f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1884100700 0x7f1884197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f188a34f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1884100700 0x7f1884197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54446/0 (socket says 172.21.15.77:54446) 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f188a34f640 1 -- 172.21.15.77:0/72740269 learned_addr learned my addr 172.21.15.77:0/72740269 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f1889b4e640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18841018f0 0x7f18841980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f188ab50640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18841022b0 0x7f188419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f188a34f640 1 -- 172.21.15.77:0/72740269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18841018f0 msgr2=0x7f18841980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f188a34f640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18841018f0 0x7f18841980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f188a34f640 1 -- 172.21.15.77:0/72740269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18841022b0 msgr2=0x7f188419f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f188a34f640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18841022b0 0x7f188419f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f188a34f640 1 -- 172.21.15.77:0/72740269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f188419f750 con 0x7f1884100700 2024-03-16T10:20:57.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f188a34f640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1884100700 0x7f1884197b80 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f188000e9c0 tx=0x7f188000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:57.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f187b7fe640 1 -- 172.21.15.77:0/72740269 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188000cd80 con 0x7f1884100700 2024-03-16T10:20:57.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f188c5b3640 1 -- 172.21.15.77:0/72740269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f188419fa40 con 0x7f1884100700 2024-03-16T10:20:57.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f187b7fe640 1 -- 172.21.15.77:0/72740269 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18800035f0 con 0x7f1884100700 2024-03-16T10:20:57.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f187b7fe640 1 -- 172.21.15.77:0/72740269 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1880010690 con 0x7f1884100700 2024-03-16T10:20:57.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.680+0000 7f188c5b3640 1 -- 172.21.15.77:0/72740269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f188419ffb0 con 0x7f1884100700 2024-03-16T10:20:57.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.681+0000 7f188c5b3640 1 -- 172.21.15.77:0/72740269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18841a01e0 con 0x7f1884100700 2024-03-16T10:20:57.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.683+0000 7f187b7fe640 1 -- 172.21.15.77:0/72740269 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f18800037a0 con 0x7f1884100700 2024-03-16T10:20:57.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.683+0000 7f187b7fe640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f18700a2800 0x7f18700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:57.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.683+0000 7f187b7fe640 1 -- 172.21.15.77:0/72740269 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f18800cee60 con 0x7f1884100700 2024-03-16T10:20:57.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.683+0000 7f1889b4e640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f18700a2800 0x7f18700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:57.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.684+0000 7f1889b4e640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f18700a2800 0x7f18700a4cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f187402c7b0 tx=0x7f1874005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:57.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.687+0000 7f187b7fe640 1 -- 172.21.15.77:0/72740269 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f188009c020 con 0x7f1884100700 2024-03-16T10:20:57.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.960+0000 7f188c5b3640 1 -- 172.21.15.77:0/72740269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18841a03b0 con 0x7f1884100700 2024-03-16T10:20:57.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.962+0000 7f187b7fe640 1 -- 172.21.15.77:0/72740269 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11946 (secure 0 0 0) 0x7f188009b920 con 0x7f1884100700 2024-03-16T10:20:57.966 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:20:57.966 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:20:31.042870+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24454,24475,24482,24455,14658]},"id":1}]} 2024-03-16T10:20:57.966 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2024-03-16T10:20:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.967+0000 7f18797fa640 1 -- 172.21.15.77:0/72740269 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f18700a2800 msgr2=0x7f18700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.967+0000 7f18797fa640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f18700a2800 0x7f18700a4cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f187402c7b0 tx=0x7f1874005fd0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.967+0000 7f18797fa640 1 -- 172.21.15.77:0/72740269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1884100700 msgr2=0x7f1884197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:57.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.967+0000 7f18797fa640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1884100700 0x7f1884197b80 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f188000e9c0 tx=0x7f188000ee80 comp rx=0 tx=0).stop 2024-03-16T10:20:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.968+0000 7f18797fa640 1 -- 172.21.15.77:0/72740269 shutdown_connections 2024-03-16T10:20:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.968+0000 7f18797fa640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18841022b0 0x7f188419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.968+0000 7f18797fa640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f18700a2800 0x7f18700a4cc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.968+0000 7f18797fa640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18841018f0 0x7f18841980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.968+0000 7f18797fa640 1 --2- 172.21.15.77:0/72740269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1884100700 0x7f1884197b80 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.968+0000 7f18797fa640 1 -- 172.21.15.77:0/72740269 >> 172.21.15.77:0/72740269 conn(0x7f18840fbc70 msgr2=0x7f18840fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.968+0000 7f18797fa640 1 -- 172.21.15.77:0/72740269 shutdown_connections 2024-03-16T10:20:57.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:57.968+0000 7f18797fa640 1 -- 172.21.15.77:0/72740269 wait complete. 2024-03-16T10:20:57.992 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-16T10:20:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:57 smithi192 ceph-mon[36354]: pgmap v466: 97 pgs: 97 active+clean; 397 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 230 op/s 2024-03-16T10:20:58.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:57 smithi077 ceph-mon[32400]: pgmap v466: 97 pgs: 97 active+clean; 397 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 230 op/s 2024-03-16T10:20:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:57 smithi193 ceph-mon[36311]: pgmap v466: 97 pgs: 97 active+clean; 397 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 230 op/s 2024-03-16T10:20:58.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.256+0000 7f3775e61640 1 -- 172.21.15.77:0/2267154606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3770101170 msgr2=0x7f3770105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.256+0000 7f3775e61640 1 --2- 172.21.15.77:0/2267154606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3770101170 0x7f3770105a30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3760009860 tx=0x7f376002bb90 comp rx=0 tx=0).stop 2024-03-16T10:20:58.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.256+0000 7f3775e61640 1 -- 172.21.15.77:0/2267154606 shutdown_connections 2024-03-16T10:20:58.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.256+0000 7f3775e61640 1 --2- 172.21.15.77:0/2267154606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3770101170 0x7f3770105a30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.256+0000 7f3775e61640 1 --2- 172.21.15.77:0/2267154606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37701007d0 0x7f3770100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.256+0000 7f3775e61640 1 --2- 172.21.15.77:0/2267154606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37700ff6a0 0x7f37700ffac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.256+0000 7f3775e61640 1 -- 172.21.15.77:0/2267154606 >> 172.21.15.77:0/2267154606 conn(0x7f37700fafa0 msgr2=0x7f37700fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.256+0000 7f3775e61640 1 -- 172.21.15.77:0/2267154606 shutdown_connections 2024-03-16T10:20:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.256+0000 7f3775e61640 1 -- 172.21.15.77:0/2267154606 wait complete. 2024-03-16T10:20:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.257+0000 7f3775e61640 1 Processor -- start 2024-03-16T10:20:58.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.257+0000 7f3775e61640 1 -- start start 2024-03-16T10:20:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.257+0000 7f3775e61640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37700ff6a0 0x7f377010c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.257+0000 7f3775e61640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37701007d0 0x7f377010c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.257+0000 7f3775e61640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377010f640 0x7f37701ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.257+0000 7f3775e61640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3770113ae0 con 0x7f377010f640 2024-03-16T10:20:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.257+0000 7f3775e61640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3770113960 con 0x7f37700ff6a0 2024-03-16T10:20:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.257+0000 7f3775e61640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3770113c60 con 0x7f37701007d0 2024-03-16T10:20:58.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.257+0000 7f376effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37701007d0 0x7f377010c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.258+0000 7f376ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377010f640 0x7f37701ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.258+0000 7f376ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377010f640 0x7f37701ae190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54460/0 (socket says 172.21.15.77:54460) 2024-03-16T10:20:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.258+0000 7f376effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37701007d0 0x7f377010c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52524/0 (socket says 172.21.15.77:52524) 2024-03-16T10:20:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.258+0000 7f376ffff640 1 -- 172.21.15.77:0/2145232396 learned_addr learned my addr 172.21.15.77:0/2145232396 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.258+0000 7f376f7fe640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37700ff6a0 0x7f377010c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.258+0000 7f376ffff640 1 -- 172.21.15.77:0/2145232396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37701007d0 msgr2=0x7f377010c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.258+0000 7f376ffff640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37701007d0 0x7f377010c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.258+0000 7f376ffff640 1 -- 172.21.15.77:0/2145232396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37700ff6a0 msgr2=0x7f377010c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.258+0000 7f376ffff640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37700ff6a0 0x7f377010c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.258+0000 7f376ffff640 1 -- 172.21.15.77:0/2145232396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f377006b330 con 0x7f377010f640 2024-03-16T10:20:58.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.259+0000 7f376ffff640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377010f640 0x7f37701ae190 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f37600349e0 tx=0x7f3760034a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.287+0000 7f376cff9640 1 -- 172.21.15.77:0/2145232396 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760039040 con 0x7f377010f640 2024-03-16T10:20:58.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.287+0000 7f376cff9640 1 -- 172.21.15.77:0/2145232396 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3760034e50 con 0x7f377010f640 2024-03-16T10:20:58.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.287+0000 7f3775e61640 1 -- 172.21.15.77:0/2145232396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f377006b5c0 con 0x7f377010f640 2024-03-16T10:20:58.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.287+0000 7f3775e61640 1 -- 172.21.15.77:0/2145232396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f377006bab0 con 0x7f377010f640 2024-03-16T10:20:58.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.287+0000 7f376cff9640 1 -- 172.21.15.77:0/2145232396 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760003850 con 0x7f377010f640 2024-03-16T10:20:58.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.291+0000 7f376cff9640 1 -- 172.21.15.77:0/2145232396 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f37600039f0 con 0x7f377010f640 2024-03-16T10:20:58.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f3778a54740 1 Processor -- start 2024-03-16T10:20:58.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f3778a54740 1 -- start start 2024-03-16T10:20:58.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f3778a54740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69758850 0x55ee69758c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f3778a54740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee69759240 0x55ee6975dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f3778a54740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee6975e3b0 0x55ee69760840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f3778a54740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ee696bd490 con 0x55ee69759240 2024-03-16T10:20:58.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f3778a54740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ee69665c90 con 0x55ee69758850 2024-03-16T10:20:58.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f3778a54740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ee696b9530 con 0x55ee6975e3b0 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f376effd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69758850 0x55ee69758c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f376effd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69758850 0x55ee69758c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42030/0 (socket says 172.21.15.77:42030) 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f376effd640 1 -- 172.21.15.77:0/1577899641 learned_addr learned my addr 172.21.15.77:0/1577899641 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f376effd640 1 --2- 172.21.15.77:0/1577899641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee6975e3b0 0x55ee69760840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.292+0000 7f376f7fe640 1 --2- 172.21.15.77:0/1577899641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee69759240 0x55ee6975dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.293+0000 7f376effd640 1 -- 172.21.15.77:0/1577899641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee6975e3b0 msgr2=0x55ee69760840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.293+0000 7f376effd640 1 --2- 172.21.15.77:0/1577899641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee6975e3b0 0x55ee69760840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.293+0000 7f376effd640 1 -- 172.21.15.77:0/1577899641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee69759240 msgr2=0x55ee6975dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.293+0000 7f376effd640 1 --2- 172.21.15.77:0/1577899641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee69759240 0x55ee6975dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.293+0000 7f376effd640 1 -- 172.21.15.77:0/1577899641 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee69720500 con 0x55ee69758850 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.293+0000 7f376f7fe640 1 --2- 172.21.15.77:0/1577899641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee69759240 0x55ee6975dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.293+0000 7f376cff9640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f37580a2990 0x7f37580a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.293+0000 7f376effd640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f37580a2990 0x7f37580a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.293+0000 7f376cff9640 1 -- 172.21.15.77:0/2145232396 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f37600f0ab0 con 0x7f377010f640 2024-03-16T10:20:58.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.294+0000 7f376effd640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f37580a2990 0x7f37580a4e50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f376800efd0 tx=0x7f3768011000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.294+0000 7f376effd640 1 --2- 172.21.15.77:0/1577899641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69758850 0x55ee69758c70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f376800ecc0 tx=0x7f3768010040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0a209c219d86e29 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.304+0000 7f3755ffb640 1 -- 172.21.15.77:0/1577899641 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3768004490 con 0x55ee69758850 2024-03-16T10:20:58.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.304+0000 7f3755ffb640 1 -- 172.21.15.77:0/1577899641 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3768004630 con 0x55ee69758850 2024-03-16T10:20:58.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.304+0000 7f3755ffb640 1 -- 172.21.15.77:0/1577899641 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376801f3e0 con 0x55ee69758850 2024-03-16T10:20:58.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.305+0000 7f3778a54740 1 -- 172.21.15.77:0/1577899641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69758850 msgr2=0x55ee69758c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.305+0000 7f3778a54740 1 --2- 172.21.15.77:0/1577899641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69758850 0x55ee69758c70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f376800ecc0 tx=0x7f3768010040 comp rx=0 tx=0).stop 2024-03-16T10:20:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.323+0000 7f3778a54740 1 -- 172.21.15.77:0/1577899641 shutdown_connections 2024-03-16T10:20:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.323+0000 7f3778a54740 1 --2- 172.21.15.77:0/1577899641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee6975e3b0 0x55ee69760840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.323+0000 7f3778a54740 1 --2- 172.21.15.77:0/1577899641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee69759240 0x55ee6975dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.323+0000 7f3778a54740 1 --2- 172.21.15.77:0/1577899641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69758850 0x55ee69758c70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.323+0000 7f3778a54740 1 -- 172.21.15.77:0/1577899641 >> 172.21.15.77:0/1577899641 conn(0x55ee69753dc0 msgr2=0x55ee697541c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.323+0000 7f3778a54740 1 -- 172.21.15.77:0/1577899641 shutdown_connections 2024-03-16T10:20:58.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.323+0000 7f3778a54740 1 -- 172.21.15.77:0/1577899641 wait complete. 2024-03-16T10:20:58.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.324+0000 7f3778a54740 1 -- start start 2024-03-16T10:20:58.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.324+0000 7f3778a54740 1 Processor -- start 2024-03-16T10:20:58.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.328+0000 7f3778a54740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee69758850 0x55ee69762e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.328+0000 7f3778a54740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 0x55ee697633a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.328+0000 7f3778a54740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee6975e3b0 0x55ee6976a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.328+0000 7f3778a54740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ee69761360 con 0x55ee6975e3b0 2024-03-16T10:20:58.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.328+0000 7f3778a54740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ee69679230 con 0x55ee69759240 2024-03-16T10:20:58.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.328+0000 7f3778a54740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ee697614e0 con 0x55ee69758850 2024-03-16T10:20:58.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.328+0000 7f3778a54740 10 client.0 _ll_register_callbacks cb 0x55ee696b5010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:58.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.328+0000 7f376f7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee69758850 0x55ee69762e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.328+0000 7f376f7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee69758850 0x55ee69762e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52538/0 (socket says 172.21.15.77:52538) 2024-03-16T10:20:58.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.328+0000 7f376f7fe640 1 -- 172.21.15.77:0/2781318667 learned_addr learned my addr 172.21.15.77:0/2781318667 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.329+0000 7f376ffff640 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee6975e3b0 0x55ee6976a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.329+0000 7f376effd640 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 0x55ee697633a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.329+0000 7f376effd640 1 -- 172.21.15.77:0/2781318667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee69758850 msgr2=0x55ee69762e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.329+0000 7f376effd640 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee69758850 0x55ee69762e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.329+0000 7f376effd640 1 -- 172.21.15.77:0/2781318667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee6975e3b0 msgr2=0x55ee6976a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.329+0000 7f376effd640 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee6975e3b0 0x55ee6976a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.329+0000 7f376effd640 1 -- 172.21.15.77:0/2781318667 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee69688100 con 0x55ee69759240 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.329+0000 7f376ffff640 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee6975e3b0 0x55ee6976a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.330+0000 7f376effd640 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 0x55ee697633a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3768009250 tx=0x7f376800e470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23c2dc740d23bd83 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.330+0000 7f37537f6640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.330+0000 7f37537f6640 1 -- 172.21.15.77:0/2781318667 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3768004490 con 0x55ee69759240 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.330+0000 7f3778a54740 1 -- 172.21.15.77:0/2781318667 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3768004630 con 0x55ee69759240 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.330+0000 7f37537f6640 1 -- 172.21.15.77:0/2781318667 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f3768020650 con 0x55ee69759240 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.330+0000 7f37537f6640 1 -- 172.21.15.77:0/2781318667 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f376800e980 con 0x55ee69759240 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.330+0000 7f3778a54740 10 client.26360 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:58.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.331+0000 7f3778a54740 1 -- 172.21.15.77:0/2781318667 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ee693d13f0 con 0x55ee69759240 2024-03-16T10:20:58.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.331+0000 7f37537f6640 1 -- 172.21.15.77:0/2781318667 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3768004630 con 0x55ee69759240 2024-03-16T10:20:58.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.331+0000 7f37537f6640 1 -- 172.21.15.77:0/2781318667 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f376803d020 con 0x55ee69759240 2024-03-16T10:20:58.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.331+0000 7f3778a54740 10 client.26360 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:58.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.331+0000 7f3778a54740 10 client.26360 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:20:58.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.331+0000 7f3778a54740 20 client.26360 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:58.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.331+0000 7f3778a54740 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55ee6976f100 0x55ee69771520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.331+0000 7f3778a54740 4 client.26360 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:20:58.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.331+0000 7f3778a54740 1 -- 172.21.15.77:0/2781318667 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee696ca9d0 con 0x55ee6976f100 2024-03-16T10:20:58.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.332+0000 7f376f7fe640 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55ee6976f100 0x55ee69771520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.332+0000 7f37537f6640 10 client.26360 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:20:58.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.332+0000 7f376f7fe640 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55ee6976f100 0x55ee69771520 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=218f68f41629d841 server_cookie=7cbc26ead64f491f in_seq=0 out_seq=0 2024-03-16T10:20:58.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.333+0000 7f37537f6640 1 -- 172.21.15.77:0/2781318667 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ee696ca9d0 con 0x55ee6976f100 2024-03-16T10:20:58.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.333+0000 7f37537f6640 10 client.26360 handle_command_reply: tid=0 2024-03-16T10:20:58.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.333+0000 7f3778a54740 1 client.26360 shutdown 2024-03-16T10:20:58.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.342+0000 7f3778a54740 1 -- 172.21.15.77:0/2781318667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 msgr2=0x55ee697633a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.342+0000 7f3778a54740 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 0x55ee697633a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3768009250 tx=0x7f376800e470 comp rx=0 tx=0).stop 2024-03-16T10:20:58.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.342+0000 7f3778a54740 1 -- 172.21.15.77:0/2781318667 shutdown_connections 2024-03-16T10:20:58.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.342+0000 7f3778a54740 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee6975e3b0 0x55ee6976a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.342+0000 7f3778a54740 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 0x55ee697633a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.342+0000 7f3778a54740 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55ee6976f100 0x55ee69771520 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.342+0000 7f3778a54740 1 --2- 172.21.15.77:0/2781318667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee69758850 0x55ee69762e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.342+0000 7f37537f6640 0 client.26360 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:20:58.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.342+0000 7f3778a54740 1 -- 172.21.15.77:0/2781318667 >> 172.21.15.77:0/2781318667 conn(0x55ee69753dc0 msgr2=0x55ee69752fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.353+0000 7f3778a54740 1 -- 172.21.15.77:0/2781318667 shutdown_connections 2024-03-16T10:20:58.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.353+0000 7f3778a54740 1 -- 172.21.15.77:0/2781318667 wait complete. 2024-03-16T10:20:58.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.353+0000 7f3778a54740 20 client.26360 trim_cache size 0 max 16384 2024-03-16T10:20:58.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.406+0000 7f3778a54740 1 Processor -- start 2024-03-16T10:20:58.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.407+0000 7f3778a54740 1 -- start start 2024-03-16T10:20:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.410+0000 7f3778a54740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee69758850 0x55ee69762150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.410+0000 7f3778a54740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 0x55ee6976a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.410+0000 7f3778a54740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee6975e3b0 0x55ee69762690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.410+0000 7f3778a54740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f375c00e400 con 0x55ee6975e3b0 2024-03-16T10:20:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.410+0000 7f376effd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 0x55ee6976a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.410+0000 7f3778a54740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f375c010030 con 0x55ee69759240 2024-03-16T10:20:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.410+0000 7f376f7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee69758850 0x55ee69762150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.410+0000 7f376effd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 0x55ee6976a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42052/0 (socket says 172.21.15.77:42052) 2024-03-16T10:20:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.410+0000 7f376effd640 1 -- 172.21.15.77:0/2612109077 learned_addr learned my addr 172.21.15.77:0/2612109077 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.410+0000 7f376ffff640 1 --2- 172.21.15.77:0/2612109077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee6975e3b0 0x55ee69762690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.410+0000 7f3778a54740 1 -- 172.21.15.77:0/2612109077 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f375c00ead0 con 0x55ee69758850 2024-03-16T10:20:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.411+0000 7f376effd640 1 -- 172.21.15.77:0/2612109077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee69758850 msgr2=0x55ee69762150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.411+0000 7f376effd640 1 --2- 172.21.15.77:0/2612109077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee69758850 0x55ee69762150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.411+0000 7f376effd640 1 -- 172.21.15.77:0/2612109077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee6975e3b0 msgr2=0x55ee69762690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.411+0000 7f376effd640 1 --2- 172.21.15.77:0/2612109077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee6975e3b0 0x55ee69762690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.411+0000 7f376effd640 1 -- 172.21.15.77:0/2612109077 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee6949b7d0 con 0x55ee69759240 2024-03-16T10:20:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.411+0000 7f376ffff640 1 --2- 172.21.15.77:0/2612109077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee6975e3b0 0x55ee69762690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:58.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.411+0000 7f376effd640 1 --2- 172.21.15.77:0/2612109077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 0x55ee6976a3a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3768005940 tx=0x7f376802b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e8a67e0a72b919d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.424+0000 7f37557fa640 1 -- 172.21.15.77:0/2612109077 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376802b5d0 con 0x55ee69759240 2024-03-16T10:20:58.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.431+0000 7f37557fa640 1 -- 172.21.15.77:0/2612109077 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f376802bdd0 con 0x55ee69759240 2024-03-16T10:20:58.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.431+0000 7f37557fa640 1 -- 172.21.15.77:0/2612109077 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37680204b0 con 0x55ee69759240 2024-03-16T10:20:58.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.432+0000 7f3778a54740 1 -- 172.21.15.77:0/2612109077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 msgr2=0x55ee6976a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.432+0000 7f3778a54740 1 --2- 172.21.15.77:0/2612109077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 0x55ee6976a3a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3768005940 tx=0x7f376802b480 comp rx=0 tx=0).stop 2024-03-16T10:20:58.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.432+0000 7f3778a54740 1 -- 172.21.15.77:0/2612109077 shutdown_connections 2024-03-16T10:20:58.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.432+0000 7f3778a54740 1 --2- 172.21.15.77:0/2612109077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee6975e3b0 0x55ee69762690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.432+0000 7f3778a54740 1 --2- 172.21.15.77:0/2612109077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69759240 0x55ee6976a3a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.432+0000 7f3778a54740 1 --2- 172.21.15.77:0/2612109077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee69758850 0x55ee69762150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.432+0000 7f3778a54740 1 -- 172.21.15.77:0/2612109077 >> 172.21.15.77:0/2612109077 conn(0x55ee69753dc0 msgr2=0x55ee6975a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.432+0000 7f3778a54740 1 -- 172.21.15.77:0/2612109077 shutdown_connections 2024-03-16T10:20:58.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.432+0000 7f3778a54740 1 -- 172.21.15.77:0/2612109077 wait complete. 2024-03-16T10:20:58.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.432+0000 7f3778a54740 1 -- start start 2024-03-16T10:20:58.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f3778a54740 1 Processor -- start 2024-03-16T10:20:58.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f3778a54740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee69758850 0x55ee6976a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f3778a54740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee6975e3b0 0x55ee69766e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f3778a54740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69767370 0x55ee697d3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f3778a54740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f375c00f930 con 0x55ee69758850 2024-03-16T10:20:58.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f3778a54740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f375c00f260 con 0x55ee69767370 2024-03-16T10:20:58.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f3778a54740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f375c00c3e0 con 0x55ee6975e3b0 2024-03-16T10:20:58.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f3778a54740 10 client.0 _ll_register_callbacks cb 0x7f376400b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:58.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f376effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee6975e3b0 0x55ee69766e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f376effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee6975e3b0 0x55ee69766e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52552/0 (socket says 172.21.15.77:52552) 2024-03-16T10:20:58.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f376effd640 1 -- 172.21.15.77:0/1530585841 learned_addr learned my addr 172.21.15.77:0/1530585841 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f376f7fe640 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee69758850 0x55ee6976a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.433+0000 7f376ffff640 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69767370 0x55ee697d3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.434+0000 7f376effd640 1 -- 172.21.15.77:0/1530585841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69767370 msgr2=0x55ee697d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.434+0000 7f376effd640 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69767370 0x55ee697d3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.434+0000 7f376effd640 1 -- 172.21.15.77:0/1530585841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee69758850 msgr2=0x55ee6976a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.434+0000 7f376effd640 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee69758850 0x55ee6976a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.434+0000 7f376effd640 1 -- 172.21.15.77:0/1530585841 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee697d3990 con 0x55ee6975e3b0 2024-03-16T10:20:58.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.434+0000 7f376f7fe640 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee69758850 0x55ee6976a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:58.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.434+0000 7f376effd640 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee6975e3b0 0x55ee69766e30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3768006010 tx=0x7f376800e980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=509dce238e1b1a0d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.434+0000 7f37537f6640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:20:58.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.435+0000 7f37537f6640 1 -- 172.21.15.77:0/1530585841 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3768002030 con 0x55ee6975e3b0 2024-03-16T10:20:58.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.435+0000 7f3778a54740 1 -- 172.21.15.77:0/1530585841 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f376802bdd0 con 0x55ee6975e3b0 2024-03-16T10:20:58.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.435+0000 7f37537f6640 1 -- 172.21.15.77:0/1530585841 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f376800e070 con 0x55ee6975e3b0 2024-03-16T10:20:58.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.437+0000 7f37537f6640 1 -- 172.21.15.77:0/1530585841 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f376802b4f0 con 0x55ee6975e3b0 2024-03-16T10:20:58.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.439+0000 7f3778a54740 10 client.26356 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:58.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.439+0000 7f3778a54740 1 -- 172.21.15.77:0/1530585841 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ee697d4b70 con 0x55ee6975e3b0 2024-03-16T10:20:58.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.444+0000 7f37537f6640 1 -- 172.21.15.77:0/1530585841 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f376802bdd0 con 0x55ee6975e3b0 2024-03-16T10:20:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.444+0000 7f37537f6640 1 -- 172.21.15.77:0/1530585841 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f3768033070 con 0x55ee6975e3b0 2024-03-16T10:20:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.444+0000 7f3778a54740 10 client.26356 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.444+0000 7f3778a54740 10 client.26356 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:20:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.444+0000 7f3778a54740 20 client.26356 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.444+0000 7f3778a54740 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55ee697d53c0 0x55ee697d77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.445+0000 7f3778a54740 4 client.26356 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:20:58.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.445+0000 7f3778a54740 1 -- 172.21.15.77:0/1530585841 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ee696c4d20 con 0x55ee697d53c0 2024-03-16T10:20:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.445+0000 7f376f7fe640 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55ee697d53c0 0x55ee697d77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.446+0000 7f376f7fe640 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55ee697d53c0 0x55ee697d77e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2081cc7251241398 server_cookie=27ab2bf30ef8b93b in_seq=0 out_seq=0 2024-03-16T10:20:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.446+0000 7f37537f6640 10 client.26356 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:20:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.446+0000 7f37537f6640 1 -- 172.21.15.77:0/1530585841 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ee696c4d20 con 0x55ee697d53c0 2024-03-16T10:20:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.446+0000 7f37537f6640 10 client.26356 handle_command_reply: tid=0 2024-03-16T10:20:58.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.446+0000 7f3778a54740 1 client.26356 shutdown 2024-03-16T10:20:58.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.458+0000 7f3778a54740 1 -- 172.21.15.77:0/1530585841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee6975e3b0 msgr2=0x55ee69766e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.458+0000 7f3778a54740 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee6975e3b0 0x55ee69766e30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3768006010 tx=0x7f376800e980 comp rx=0 tx=0).stop 2024-03-16T10:20:58.469 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:20:58.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.467+0000 7f3778a54740 1 -- 172.21.15.77:0/1530585841 shutdown_connections 2024-03-16T10:20:58.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.467+0000 7f3778a54740 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ee69767370 0x55ee697d3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.467+0000 7f3778a54740 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55ee697d53c0 0x55ee697d77e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.467+0000 7f3778a54740 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee6975e3b0 0x55ee69766e30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.467+0000 7f3778a54740 1 --2- 172.21.15.77:0/1530585841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ee69758850 0x55ee6976a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.467+0000 7f37537f6640 0 client.26356 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:20:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.467+0000 7f3778a54740 1 -- 172.21.15.77:0/1530585841 >> 172.21.15.77:0/1530585841 conn(0x55ee69753dc0 msgr2=0x55ee697578b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.467+0000 7f3778a54740 1 -- 172.21.15.77:0/1530585841 shutdown_connections 2024-03-16T10:20:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.467+0000 7f3778a54740 1 -- 172.21.15.77:0/1530585841 wait complete. 2024-03-16T10:20:58.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.467+0000 7f3778a54740 20 client.26356 trim_cache size 0 max 16384 2024-03-16T10:20:58.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.472+0000 7f3775e61640 1 -- 172.21.15.77:0/2145232396 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f37580a2990 msgr2=0x7f37580a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.472+0000 7f3775e61640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f37580a2990 0x7f37580a4e50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f376800efd0 tx=0x7f3768011000 comp rx=0 tx=0).stop 2024-03-16T10:20:58.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.472+0000 7f3775e61640 1 -- 172.21.15.77:0/2145232396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377010f640 msgr2=0x7f37701ae190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.472+0000 7f3775e61640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377010f640 0x7f37701ae190 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f37600349e0 tx=0x7f3760034a10 comp rx=0 tx=0).stop 2024-03-16T10:20:58.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.473+0000 7f3775e61640 1 -- 172.21.15.77:0/2145232396 shutdown_connections 2024-03-16T10:20:58.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.473+0000 7f3775e61640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f377010f640 0x7f37701ae190 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.473+0000 7f3775e61640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f37580a2990 0x7f37580a4e50 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.473+0000 7f3775e61640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37701007d0 0x7f377010c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.473+0000 7f3775e61640 1 --2- 172.21.15.77:0/2145232396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37700ff6a0 0x7f377010c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.473+0000 7f3775e61640 1 -- 172.21.15.77:0/2145232396 >> 172.21.15.77:0/2145232396 conn(0x7f37700fafa0 msgr2=0x7f3770101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.477+0000 7f3775e61640 1 -- 172.21.15.77:0/2145232396 shutdown_connections 2024-03-16T10:20:58.477 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.477+0000 7f3775e61640 1 -- 172.21.15.77:0/2145232396 wait complete. 2024-03-16T10:20:58.512 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-16T10:20:58.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.778+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1584488023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b58101390 msgr2=0x7f7b58101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.778+0000 7f7b5ca0b640 1 --2- 172.21.15.77:0/1584488023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b58101390 0x7f7b58101810 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009be0 tx=0x7f7b5002bef0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.778+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1584488023 shutdown_connections 2024-03-16T10:20:58.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.778+0000 7f7b5ca0b640 1 --2- 172.21.15.77:0/1584488023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b58101390 0x7f7b58101810 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.778+0000 7f7b5ca0b640 1 --2- 172.21.15.77:0/1584488023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b581009f0 0x7f7b58100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.778+0000 7f7b5ca0b640 1 --2- 172.21.15.77:0/1584488023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58100090 0x7f7b581004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.778+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1584488023 >> 172.21.15.77:0/1584488023 conn(0x7f7b580fbc70 msgr2=0x7f7b580fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.778+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1584488023 shutdown_connections 2024-03-16T10:20:58.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.778+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1584488023 wait complete. 2024-03-16T10:20:58.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.779+0000 7f7b5ca0b640 1 Processor -- start 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.779+0000 7f7b5ca0b640 1 -- start start 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.779+0000 7f7b5ca0b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58100090 0x7f7b58199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.779+0000 7f7b5ca0b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b581009f0 0x7f7b5819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.779+0000 7f7b5ca0b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b58101390 0x7f7b581a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.779+0000 7f7b5ca0b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5810f5d0 con 0x7f7b581009f0 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.779+0000 7f7b5ca0b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5810f450 con 0x7f7b58100090 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.779+0000 7f7b5ca0b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5810f750 con 0x7f7b58101390 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b5659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58100090 0x7f7b58199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b5659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58100090 0x7f7b58199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42084/0 (socket says 172.21.15.77:42084) 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b5659c640 1 -- 172.21.15.77:0/1335467637 learned_addr learned my addr 172.21.15.77:0/1335467637 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b55d9b640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b581009f0 0x7f7b5819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b56d9d640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b58101390 0x7f7b581a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b5659c640 1 -- 172.21.15.77:0/1335467637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b58101390 msgr2=0x7f7b581a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b5659c640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b58101390 0x7f7b581a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b5659c640 1 -- 172.21.15.77:0/1335467637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b581009f0 msgr2=0x7f7b5819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b5659c640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b581009f0 0x7f7b5819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b5659c640 1 -- 172.21.15.77:0/1335467637 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b581a18c0 con 0x7f7b58100090 2024-03-16T10:20:58.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b55d9b640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b581009f0 0x7f7b5819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:20:58.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b5659c640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58100090 0x7f7b58199c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800e9f0 tx=0x7f7b4800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.780+0000 7f7b477fe640 1 -- 172.21.15.77:0/1335467637 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4800cd80 con 0x7f7b58100090 2024-03-16T10:20:58.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.781+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1335467637 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b581a1bb0 con 0x7f7b58100090 2024-03-16T10:20:58.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.781+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1335467637 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b581a2150 con 0x7f7b58100090 2024-03-16T10:20:58.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.781+0000 7f7b477fe640 1 -- 172.21.15.77:0/1335467637 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b48010910 con 0x7f7b58100090 2024-03-16T10:20:58.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.781+0000 7f7b477fe640 1 -- 172.21.15.77:0/1335467637 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48018990 con 0x7f7b58100090 2024-03-16T10:20:58.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.783+0000 7f7b5f5f3740 1 Processor -- start 2024-03-16T10:20:58.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b5f5f3740 1 -- start start 2024-03-16T10:20:58.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b5f5f3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 0x55a1ddd24c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b5f5f3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 0x55a1ddd29d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b5f5f3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd2c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b5f5f3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a1ddc89490 con 0x55a1ddd24850 2024-03-16T10:20:58.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b5f5f3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a1ddc31c90 con 0x55a1ddd25240 2024-03-16T10:20:58.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b5f5f3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a1ddc85530 con 0x55a1ddd2a3b0 2024-03-16T10:20:58.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b477fe640 1 -- 172.21.15.77:0/1335467637 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f7b48018b30 con 0x7f7b58100090 2024-03-16T10:20:58.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b55d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd2c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b55d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd2c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52592/0 (socket says 172.21.15.77:52592) 2024-03-16T10:20:58.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b55d9b640 1 -- 172.21.15.77:0/2432122165 learned_addr learned my addr 172.21.15.77:0/2432122165 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b5659c640 1 --2- 172.21.15.77:0/2432122165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 0x55a1ddd29d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.784+0000 7f7b56d9d640 1 --2- 172.21.15.77:0/2432122165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 0x55a1ddd24c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.785+0000 7f7b477fe640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7b400a2880 0x7f7b400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.785+0000 7f7b55d9b640 1 -- 172.21.15.77:0/2432122165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 msgr2=0x55a1ddd29d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.785+0000 7f7b55d9b640 1 --2- 172.21.15.77:0/2432122165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 0x55a1ddd29d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.785+0000 7f7b55d9b640 1 -- 172.21.15.77:0/2432122165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 msgr2=0x55a1ddd24c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.786+0000 7f7b55d9b640 1 --2- 172.21.15.77:0/2432122165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 0x55a1ddd24c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.786+0000 7f7b55d9b640 1 -- 172.21.15.77:0/2432122165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1ddcec500 con 0x55a1ddd2a3b0 2024-03-16T10:20:58.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.786+0000 7f7b477fe640 1 -- 172.21.15.77:0/1335467637 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7b480cecd0 con 0x7f7b58100090 2024-03-16T10:20:58.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.786+0000 7f7b55d9b640 1 --2- 172.21.15.77:0/2432122165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd2c840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c009ce0 tx=0x7f7b4c0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da9fa063e13af9a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.786+0000 7f7b5659c640 1 --2- 172.21.15.77:0/2432122165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 0x55a1ddd29d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:20:58.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.786+0000 7f7b56d9d640 1 --2- 172.21.15.77:0/2432122165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 0x55a1ddd24c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:20:58.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.787+0000 7f7b44ff9640 1 -- 172.21.15.77:0/2432122165 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c00f440 con 0x55a1ddd2a3b0 2024-03-16T10:20:58.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.787+0000 7f7b44ff9640 1 -- 172.21.15.77:0/2432122165 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b4c00fac0 con 0x55a1ddd2a3b0 2024-03-16T10:20:58.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.787+0000 7f7b44ff9640 1 -- 172.21.15.77:0/2432122165 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c022c10 con 0x55a1ddd2a3b0 2024-03-16T10:20:58.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.787+0000 7f7b56d9d640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7b400a2880 0x7f7b400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.787+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/2432122165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd2a3b0 msgr2=0x55a1ddd2c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.787+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/2432122165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd2c840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c009ce0 tx=0x7f7b4c0117b0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.792+0000 7f7b56d9d640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7b400a2880 0x7f7b400a4d40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009370 tx=0x7f7b5002bef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.793+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/2432122165 shutdown_connections 2024-03-16T10:20:58.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.793+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/2432122165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd2c840 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.793+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/2432122165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 0x55a1ddd29d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.793+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/2432122165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 0x55a1ddd24c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.793+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/2432122165 >> 172.21.15.77:0/2432122165 conn(0x55a1ddd1fdc0 msgr2=0x55a1ddd201c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.794+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/2432122165 shutdown_connections 2024-03-16T10:20:58.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.794+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/2432122165 wait complete. 2024-03-16T10:20:58.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.794+0000 7f7b5f5f3740 1 -- start start 2024-03-16T10:20:58.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.794+0000 7f7b5f5f3740 1 Processor -- start 2024-03-16T10:20:58.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.794+0000 7f7b5f5f3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 0x55a1ddd2ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.794+0000 7f7b5f5f3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 0x55a1ddd2f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.794+0000 7f7b5f5f3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd36240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.794+0000 7f7b5f5f3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a1ddd2d360 con 0x55a1ddd2a3b0 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.794+0000 7f7b5f5f3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a1ddc45230 con 0x55a1ddd25240 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.794+0000 7f7b5f5f3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a1ddd2d4e0 con 0x55a1ddd24850 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.794+0000 7f7b5f5f3740 10 client.0 _ll_register_callbacks cb 0x55a1ddc81010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b5659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 0x55a1ddd2ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b55d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 0x55a1ddd2f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b5659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 0x55a1ddd2ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52606/0 (socket says 172.21.15.77:52606) 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b55d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 0x55a1ddd2f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42110/0 (socket says 172.21.15.77:42110) 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b5659c640 1 -- 172.21.15.77:0/1269798760 learned_addr learned my addr 172.21.15.77:0/1269798760 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b56d9d640 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd36240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b5659c640 1 -- 172.21.15.77:0/1269798760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 msgr2=0x55a1ddd2f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b5659c640 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 0x55a1ddd2f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b5659c640 1 -- 172.21.15.77:0/1269798760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd2a3b0 msgr2=0x55a1ddd36240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b5659c640 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd36240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b5659c640 1 -- 172.21.15.77:0/1269798760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1ddc54100 con 0x55a1ddd24850 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b55d9b640 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 0x55a1ddd2f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b5659c640 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 0x55a1ddd2ecd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800c6a0 tx=0x7f7b4809b2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f98ec3a43189740 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b3dffb640 1 -- 172.21.15.77:0/1269798760 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4809f7b0 con 0x55a1ddd24850 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b3dffb640 1 -- 172.21.15.77:0/1269798760 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7b480ccbe0 con 0x55a1ddd24850 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.795+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1269798760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b4c00fac0 con 0x55a1ddd24850 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.796+0000 7f7b3dffb640 1 -- 172.21.15.77:0/1269798760 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7b480a3e00 con 0x55a1ddd24850 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.796+0000 7f7b5f5f3740 10 client.26368 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.796+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1269798760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a1dd99d3f0 con 0x55a1ddd24850 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.796+0000 7f7b3dffb640 1 -- 172.21.15.77:0/1269798760 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b4c00fac0 con 0x55a1ddd24850 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.796+0000 7f7b3dffb640 1 -- 172.21.15.77:0/1269798760 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f7b48097440 con 0x55a1ddd24850 2024-03-16T10:20:58.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.796+0000 7f7b5f5f3740 10 client.26368 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:58.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.796+0000 7f7b5f5f3740 10 client.26368 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:20:58.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.796+0000 7f7b5f5f3740 20 client.26368 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:58.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.796+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55a1ddd3aa80 0x55a1ddd3cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.796+0000 7f7b5f5f3740 4 client.26368 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:20:58.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.796+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1269798760 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1ddc969d0 con 0x55a1ddd3aa80 2024-03-16T10:20:58.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.796+0000 7f7b55d9b640 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55a1ddd3aa80 0x55a1ddd3cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.797+0000 7f7b55d9b640 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55a1ddd3aa80 0x55a1ddd3cf20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9f5c4b96cb5e8da8 server_cookie=2c08217de37dd8e3 in_seq=0 out_seq=0 2024-03-16T10:20:58.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.797+0000 7f7b3dffb640 10 client.26368 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:20:58.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.797+0000 7f7b3dffb640 1 -- 172.21.15.77:0/1269798760 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a1ddc969d0 con 0x55a1ddd3aa80 2024-03-16T10:20:58.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.797+0000 7f7b3dffb640 10 client.26368 handle_command_reply: tid=0 2024-03-16T10:20:58.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.797+0000 7f7b5f5f3740 1 client.26368 shutdown 2024-03-16T10:20:58.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.797+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1269798760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 msgr2=0x55a1ddd2ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.797+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 0x55a1ddd2ecd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800c6a0 tx=0x7f7b4809b2a0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.797+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1269798760 shutdown_connections 2024-03-16T10:20:58.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.797+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55a1ddd3aa80 0x55a1ddd3cf20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.797+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd36240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.798+0000 7f7b3dffb640 0 client.26368 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:20:58.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.798+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd25240 0x55a1ddd2f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.798+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/1269798760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 0x55a1ddd2ecd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.798+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1269798760 >> 172.21.15.77:0/1269798760 conn(0x55a1ddd1fdc0 msgr2=0x55a1ddd1efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.798+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1269798760 shutdown_connections 2024-03-16T10:20:58.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.798+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1269798760 wait complete. 2024-03-16T10:20:58.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.798+0000 7f7b5f5f3740 20 client.26368 trim_cache size 0 max 16384 2024-03-16T10:20:58.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.814+0000 7f7b5f5f3740 1 Processor -- start 2024-03-16T10:20:58.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5f5f3740 1 -- start start 2024-03-16T10:20:58.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5f5f3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 0x55a1ddd266a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5f5f3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd25240 0x55a1ddd36200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5f5f3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd26be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5f5f3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3801f2a0 con 0x55a1ddd24850 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5f5f3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7b38020ed0 con 0x55a1ddd2a3b0 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5f5f3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3801f970 con 0x55a1ddd25240 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 0x55a1ddd266a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 0x55a1ddd266a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54512/0 (socket says 172.21.15.77:54512) 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5659c640 1 -- 172.21.15.77:0/3996456615 learned_addr learned my addr 172.21.15.77:0/3996456615 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b56d9d640 1 --2- 172.21.15.77:0/3996456615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd26be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b55d9b640 1 --2- 172.21.15.77:0/3996456615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd25240 0x55a1ddd36200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5659c640 1 -- 172.21.15.77:0/3996456615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd25240 msgr2=0x55a1ddd36200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5659c640 1 --2- 172.21.15.77:0/3996456615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd25240 0x55a1ddd36200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5659c640 1 -- 172.21.15.77:0/3996456615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd2a3b0 msgr2=0x55a1ddd26be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5659c640 1 --2- 172.21.15.77:0/3996456615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd26be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b5659c640 1 -- 172.21.15.77:0/3996456615 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1dda677d0 con 0x55a1ddd24850 2024-03-16T10:20:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.815+0000 7f7b56d9d640 1 --2- 172.21.15.77:0/3996456615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd26be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:58.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.816+0000 7f7b5659c640 1 --2- 172.21.15.77:0/3996456615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 0x55a1ddd266a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7b4809b310 tx=0x7f7b48097860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62fa4507adac7870 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.816+0000 7f7b3ffff640 1 -- 172.21.15.77:0/3996456615 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4809f7b0 con 0x55a1ddd24850 2024-03-16T10:20:58.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.816+0000 7f7b3ffff640 1 -- 172.21.15.77:0/3996456615 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b480973b0 con 0x55a1ddd24850 2024-03-16T10:20:58.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.816+0000 7f7b3ffff640 1 -- 172.21.15.77:0/3996456615 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4809ad10 con 0x55a1ddd24850 2024-03-16T10:20:58.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.817+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/3996456615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 msgr2=0x55a1ddd266a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.817+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/3996456615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 0x55a1ddd266a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7b4809b310 tx=0x7f7b48097860 comp rx=0 tx=0).stop 2024-03-16T10:20:58.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.817+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/3996456615 shutdown_connections 2024-03-16T10:20:58.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.817+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/3996456615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd26be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.817+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/3996456615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd25240 0x55a1ddd36200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.817+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/3996456615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd24850 0x55a1ddd266a0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.817+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/3996456615 >> 172.21.15.77:0/3996456615 conn(0x55a1ddd1fdc0 msgr2=0x55a1ddd201a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.817+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/3996456615 shutdown_connections 2024-03-16T10:20:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.817+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/3996456615 wait complete. 2024-03-16T10:20:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.818+0000 7f7b5f5f3740 1 -- start start 2024-03-16T10:20:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.818+0000 7f7b5f5f3740 1 Processor -- start 2024-03-16T10:20:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.818+0000 7f7b5f5f3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 0x55a1ddd33260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.818+0000 7f7b5f5f3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd25240 0x55a1ddd337a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.818+0000 7f7b5f5f3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd9ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.818+0000 7f7b5f5f3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b380207d0 con 0x55a1ddd25240 2024-03-16T10:20:58.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.818+0000 7f7b5f5f3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7b38020100 con 0x55a1ddd2a3b0 2024-03-16T10:20:58.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.818+0000 7f7b5f5f3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3801d280 con 0x55a1ddd24850 2024-03-16T10:20:58.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.818+0000 7f7b5659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 0x55a1ddd33260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.818+0000 7f7b55d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd25240 0x55a1ddd337a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.818+0000 7f7b5659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 0x55a1ddd33260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52630/0 (socket says 172.21.15.77:52630) 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.818+0000 7f7b5659c640 1 -- 172.21.15.77:0/1892936031 learned_addr learned my addr 172.21.15.77:0/1892936031 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b56d9d640 1 --2- 172.21.15.77:0/1892936031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd9ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b5f5f3740 10 client.0 _ll_register_callbacks cb 0x7f7b48009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b5659c640 1 -- 172.21.15.77:0/1892936031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd2a3b0 msgr2=0x55a1ddd9ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b5659c640 1 --2- 172.21.15.77:0/1892936031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd9ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b5659c640 1 -- 172.21.15.77:0/1892936031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd25240 msgr2=0x55a1ddd337a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b5659c640 1 --2- 172.21.15.77:0/1892936031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd25240 0x55a1ddd337a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b5659c640 1 -- 172.21.15.77:0/1892936031 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1ddd9f400 con 0x55a1ddd24850 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b5659c640 1 --2- 172.21.15.77:0/1892936031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 0x55a1ddd33260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7b4809b5f0 tx=0x7f7b480106e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1e11bc17c49c444 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b3dffb640 1 -- 172.21.15.77:0/1892936031 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4809ade0 con 0x55a1ddd24850 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b3dffb640 1 -- 172.21.15.77:0/1892936031 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f7b4809a520 con 0x55a1ddd24850 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1892936031 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b480973b0 con 0x55a1ddd24850 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b3dffb640 1 -- 172.21.15.77:0/1892936031 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7b4809f7b0 con 0x55a1ddd24850 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b5f5f3740 10 client.26374 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:58.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.819+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1892936031 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a1dddb1690 con 0x55a1ddd24850 2024-03-16T10:20:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.820+0000 7f7b3dffb640 1 -- 172.21.15.77:0/1892936031 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b4809a720 con 0x55a1ddd24850 2024-03-16T10:20:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.820+0000 7f7b3dffb640 1 -- 172.21.15.77:0/1892936031 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f7b480b4390 con 0x55a1ddd24850 2024-03-16T10:20:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.820+0000 7f7b5f5f3740 10 client.26374 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.820+0000 7f7b5f5f3740 10 client.26374 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:20:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.820+0000 7f7b5f5f3740 20 client.26374 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.820+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/1892936031 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55a1ddd3aa80 0x55a1dddb3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.820+0000 7f7b5f5f3740 4 client.26374 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:20:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.820+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1892936031 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f7b480973b0 con 0x55a1ddd3aa80 2024-03-16T10:20:58.823 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.820+0000 7f7b55d9b640 1 --2- 172.21.15.77:0/1892936031 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55a1ddd3aa80 0x55a1dddb3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.822+0000 7f7b3dffb640 10 client.26374 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:20:58.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.822+0000 7f7b55d9b640 1 --2- 172.21.15.77:0/1892936031 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55a1ddd3aa80 0x55a1dddb3f00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5857043e905496c7 server_cookie=3c7400da55f8cff8 in_seq=0 out_seq=0 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.822+0000 7f7b3dffb640 1 -- 172.21.15.77:0/1892936031 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f7b480973b0 con 0x55a1ddd3aa80 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.822+0000 7f7b3dffb640 10 client.26374 handle_command_reply: tid=0 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.822+0000 7f7b5f5f3740 1 client.26374 shutdown 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.823+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1892936031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 msgr2=0x55a1ddd33260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.823+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/1892936031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 0x55a1ddd33260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7b4809b5f0 tx=0x7f7b480106e0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.823+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1892936031 shutdown_connections 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.823+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/1892936031 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55a1ddd3aa80 0x55a1dddb3f00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.823+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/1892936031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a1ddd2a3b0 0x55a1ddd9ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.823+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/1892936031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1ddd25240 0x55a1ddd337a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.823+0000 7f7b5f5f3740 1 --2- 172.21.15.77:0/1892936031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1ddd24850 0x55a1ddd33260 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.823+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1892936031 >> 172.21.15.77:0/1892936031 conn(0x55a1ddd1fdc0 msgr2=0x55a1ddd266a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.823+0000 7f7b3dffb640 0 client.26374 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.823+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1892936031 shutdown_connections 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.823+0000 7f7b5f5f3740 1 -- 172.21.15.77:0/1892936031 wait complete. 2024-03-16T10:20:58.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.823+0000 7f7b5f5f3740 20 client.26374 trim_cache size 0 max 16384 2024-03-16T10:20:58.826 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:20:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.824+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1335467637 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7b400a2880 msgr2=0x7f7b400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.824+0000 7f7b5ca0b640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7b400a2880 0x7f7b400a4d40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009370 tx=0x7f7b5002bef0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.824+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1335467637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58100090 msgr2=0x7f7b58199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.824+0000 7f7b5ca0b640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58100090 0x7f7b58199c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800e9f0 tx=0x7f7b4800eeb0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.824+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1335467637 shutdown_connections 2024-03-16T10:20:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.824+0000 7f7b5ca0b640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7b400a2880 0x7f7b400a4d40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.824+0000 7f7b5ca0b640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b58101390 0x7f7b581a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.824+0000 7f7b5ca0b640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b581009f0 0x7f7b5819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.824+0000 7f7b5ca0b640 1 --2- 172.21.15.77:0/1335467637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b58100090 0x7f7b58199c80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.824+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1335467637 >> 172.21.15.77:0/1335467637 conn(0x7f7b580fbc70 msgr2=0x7f7b580fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.825+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1335467637 shutdown_connections 2024-03-16T10:20:58.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.825+0000 7f7b5ca0b640 1 -- 172.21.15.77:0/1335467637 wait complete. 2024-03-16T10:20:58.838 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T10:20:58.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.973+0000 7fe610c0b640 1 -- 172.21.15.77:0/4065567291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c1022a0 msgr2=0x7fe60c104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.973+0000 7fe610c0b640 1 --2- 172.21.15.77:0/4065567291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c1022a0 0x7fe60c104690 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009900 tx=0x7fe5fc02bc30 comp rx=0 tx=0).stop 2024-03-16T10:20:58.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.973+0000 7fe610c0b640 1 -- 172.21.15.77:0/4065567291 shutdown_connections 2024-03-16T10:20:58.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.973+0000 7fe610c0b640 1 --2- 172.21.15.77:0/4065567291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe60c107500 0x7fe60c109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.973+0000 7fe610c0b640 1 --2- 172.21.15.77:0/4065567291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe60c104bd0 0x7fe60c106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.973+0000 7fe610c0b640 1 --2- 172.21.15.77:0/4065567291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c1022a0 0x7fe60c104690 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.973+0000 7fe610c0b640 1 -- 172.21.15.77:0/4065567291 >> 172.21.15.77:0/4065567291 conn(0x7fe60c0fbc70 msgr2=0x7fe60c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.973+0000 7fe610c0b640 1 -- 172.21.15.77:0/4065567291 shutdown_connections 2024-03-16T10:20:58.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.973+0000 7fe610c0b640 1 -- 172.21.15.77:0/4065567291 wait complete. 2024-03-16T10:20:58.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe610c0b640 1 Processor -- start 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe610c0b640 1 -- start start 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe610c0b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c1022a0 0x7fe60c101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe610c0b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe60c104bd0 0x7fe60c100250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe610c0b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe60c107500 0x7fe60c100790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe610c0b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c10d480 con 0x7fe60c1022a0 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe610c0b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c10d300 con 0x7fe60c107500 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe610c0b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c10d600 con 0x7fe60c104bd0 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe60a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c1022a0 0x7fe60c101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe60a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c1022a0 0x7fe60c101c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54534/0 (socket says 172.21.15.77:54534) 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe60a59c640 1 -- 172.21.15.77:0/3290817175 learned_addr learned my addr 172.21.15.77:0/3290817175 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe609d9b640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe60c104bd0 0x7fe60c100250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.974+0000 7fe60ad9d640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe60c107500 0x7fe60c100790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.975+0000 7fe60a59c640 1 -- 172.21.15.77:0/3290817175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe60c104bd0 msgr2=0x7fe60c100250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.975+0000 7fe60a59c640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe60c104bd0 0x7fe60c100250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.975+0000 7fe60a59c640 1 -- 172.21.15.77:0/3290817175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe60c107500 msgr2=0x7fe60c100790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.975+0000 7fe60a59c640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe60c107500 0x7fe60c100790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.975+0000 7fe60a59c640 1 -- 172.21.15.77:0/3290817175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60c101020 con 0x7fe60c1022a0 2024-03-16T10:20:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.975+0000 7fe60a59c640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c1022a0 0x7fe60c101c10 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009230 tx=0x7fe5fc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.975+0000 7fe5fb7fe640 1 -- 172.21.15.77:0/3290817175 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc041070 con 0x7fe60c1022a0 2024-03-16T10:20:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.975+0000 7fe610c0b640 1 -- 172.21.15.77:0/3290817175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe60c1a1c40 con 0x7fe60c1022a0 2024-03-16T10:20:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.975+0000 7fe5fb7fe640 1 -- 172.21.15.77:0/3290817175 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5fc004450 con 0x7fe60c1022a0 2024-03-16T10:20:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.975+0000 7fe610c0b640 1 -- 172.21.15.77:0/3290817175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe60c1a2190 con 0x7fe60c1022a0 2024-03-16T10:20:58.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.975+0000 7fe5fb7fe640 1 -- 172.21.15.77:0/3290817175 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc003e20 con 0x7fe60c1022a0 2024-03-16T10:20:58.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.978+0000 7fe5fb7fe640 1 -- 172.21.15.77:0/3290817175 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fe5fc044070 con 0x7fe60c1022a0 2024-03-16T10:20:58.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.978+0000 7fe5fb7fe640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5f40a2800 0x7fe5f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.978+0000 7fe5fb7fe640 1 -- 172.21.15.77:0/3290817175 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe5fc0f0580 con 0x7fe60c1022a0 2024-03-16T10:20:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.978+0000 7fe609d9b640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5f40a2800 0x7fe5f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe6137f9740 1 Processor -- start 2024-03-16T10:20:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe6137f9740 1 -- start start 2024-03-16T10:20:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe6137f9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff951480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe6137f9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff951a50 0x55ffff956550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe6137f9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff956bc0 0x55ffff959050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe6137f9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ffff8ad490 con 0x55ffff951a50 2024-03-16T10:20:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe6137f9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ffff855c90 con 0x55ffff956bc0 2024-03-16T10:20:58.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe6137f9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ffff8a9530 con 0x55ffff951060 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe60ad9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff956bc0 0x55ffff959050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe60a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff951480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe60ad9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff956bc0 0x55ffff959050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42150/0 (socket says 172.21.15.77:42150) 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe60ad9d640 1 -- 172.21.15.77:0/2351146742 learned_addr learned my addr 172.21.15.77:0/2351146742 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.979+0000 7fe609d9b640 1 --2- 172.21.15.77:0/2351146742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff951a50 0x55ffff956550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.980+0000 7fe60ad9d640 1 -- 172.21.15.77:0/2351146742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 msgr2=0x55ffff951480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.980+0000 7fe60ad9d640 1 --2- 172.21.15.77:0/2351146742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff951480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.980+0000 7fe60ad9d640 1 -- 172.21.15.77:0/2351146742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff951a50 msgr2=0x55ffff956550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.980+0000 7fe60ad9d640 1 --2- 172.21.15.77:0/2351146742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff951a50 0x55ffff956550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.980+0000 7fe60ad9d640 1 -- 172.21.15.77:0/2351146742 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffff910500 con 0x55ffff956bc0 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.980+0000 7fe60a59c640 1 --2- 172.21.15.77:0/2351146742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff951480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.980+0000 7fe609d9b640 1 --2- 172.21.15.77:0/2351146742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff951a50 0x55ffff956550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.980+0000 7fe609d9b640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5f40a2800 0x7fe5f40a4cc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe60c101980 tx=0x7fe600005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.980+0000 7fe60ad9d640 1 --2- 172.21.15.77:0/2351146742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff956bc0 0x55ffff959050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe604009fd0 tx=0x7fe6040118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0779bf55764c9bb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.980+0000 7fe5f8ff9640 1 -- 172.21.15.77:0/2351146742 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604004d20 con 0x55ffff956bc0 2024-03-16T10:20:58.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.980+0000 7fe5f8ff9640 1 -- 172.21.15.77:0/2351146742 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe604004050 con 0x55ffff956bc0 2024-03-16T10:20:58.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.980+0000 7fe5f8ff9640 1 -- 172.21.15.77:0/2351146742 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604005240 con 0x55ffff956bc0 2024-03-16T10:20:58.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.981+0000 7fe6137f9740 1 -- 172.21.15.77:0/2351146742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff956bc0 msgr2=0x55ffff959050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.981+0000 7fe6137f9740 1 --2- 172.21.15.77:0/2351146742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff956bc0 0x55ffff959050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe604009fd0 tx=0x7fe6040118d0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.981+0000 7fe6137f9740 1 -- 172.21.15.77:0/2351146742 shutdown_connections 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.981+0000 7fe6137f9740 1 --2- 172.21.15.77:0/2351146742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff956bc0 0x55ffff959050 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.981+0000 7fe6137f9740 1 --2- 172.21.15.77:0/2351146742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff951a50 0x55ffff956550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.981+0000 7fe6137f9740 1 --2- 172.21.15.77:0/2351146742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff951480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.981+0000 7fe6137f9740 1 -- 172.21.15.77:0/2351146742 >> 172.21.15.77:0/2351146742 conn(0x55ffff943dc0 msgr2=0x55ffff9441c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.981+0000 7fe6137f9740 1 -- 172.21.15.77:0/2351146742 shutdown_connections 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.981+0000 7fe6137f9740 1 -- 172.21.15.77:0/2351146742 wait complete. 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe6137f9740 1 -- start start 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe6137f9740 1 Processor -- start 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe6137f9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951060 0x55ffff95b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe6137f9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff951a50 0x55ffff95ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe6137f9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff956bc0 0x55ffff962aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe6137f9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ffff959b70 con 0x55ffff951a50 2024-03-16T10:20:58.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe6137f9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ffff869230 con 0x55ffff951060 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe6137f9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ffff959cf0 con 0x55ffff956bc0 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe609d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff951a50 0x55ffff95ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe60a59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951060 0x55ffff95b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe60a59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951060 0x55ffff95b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42156/0 (socket says 172.21.15.77:42156) 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe60ad9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff956bc0 0x55ffff962aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe60a59c640 1 -- 172.21.15.77:0/4162334742 learned_addr learned my addr 172.21.15.77:0/4162334742 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe609d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff951a50 0x55ffff95ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54556/0 (socket says 172.21.15.77:54556) 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.982+0000 7fe6137f9740 10 client.0 _ll_register_callbacks cb 0x55ffff8a5010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe60a59c640 1 -- 172.21.15.77:0/4162334742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff956bc0 msgr2=0x55ffff962aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe60a59c640 1 --2- 172.21.15.77:0/4162334742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff956bc0 0x55ffff962aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe60a59c640 1 -- 172.21.15.77:0/4162334742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff951a50 msgr2=0x55ffff95ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe60a59c640 1 --2- 172.21.15.77:0/4162334742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff951a50 0x55ffff95ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe60a59c640 1 -- 172.21.15.77:0/4162334742 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffff878100 con 0x55ffff951060 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe60ad9d640 1 --2- 172.21.15.77:0/4162334742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff956bc0 0x55ffff962aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe60a59c640 1 --2- 172.21.15.77:0/4162334742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951060 0x55ffff95b520 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc02c7b0 tx=0x7fe5fc0bca80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=362e457156e9e22f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:58.988 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe5f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe5f1ffb640 1 -- 172.21.15.77:0/4162334742 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc04b040 con 0x55ffff951060 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe5f1ffb640 1 -- 172.21.15.77:0/4162334742 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe5fc0b7bd0 con 0x55ffff951060 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe6137f9740 1 -- 172.21.15.77:0/4162334742 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe604004050 con 0x55ffff951060 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe5f1ffb640 1 -- 172.21.15.77:0/4162334742 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe5fc0a0050 con 0x55ffff951060 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe6137f9740 10 client.26393 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.983+0000 7fe6137f9740 1 -- 172.21.15.77:0/4162334742 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ffff5c13f0 con 0x55ffff951060 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.984+0000 7fe5f1ffb640 1 -- 172.21.15.77:0/4162334742 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe604004050 con 0x55ffff951060 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.984+0000 7fe5f1ffb640 1 -- 172.21.15.77:0/4162334742 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7fe5fc0f00f0 con 0x55ffff951060 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.985+0000 7fe6137f9740 10 client.26393 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.985+0000 7fe6137f9740 10 client.26393 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.985+0000 7fe6137f9740 20 client.26393 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.985+0000 7fe6137f9740 1 --2- 172.21.15.77:0/4162334742 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ffff9674d0 0x55ffff969990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.985+0000 7fe6137f9740 4 client.26393 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.985+0000 7fe6137f9740 1 -- 172.21.15.77:0/4162334742 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffff8ba9d0 con 0x55ffff9674d0 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.985+0000 7fe609d9b640 1 --2- 172.21.15.77:0/4162334742 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ffff9674d0 0x55ffff969990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:58.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.985+0000 7fe5f1ffb640 10 client.26393 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:20:58.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.985+0000 7fe609d9b640 1 --2- 172.21.15.77:0/4162334742 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ffff9674d0 0x55ffff969990 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17eb4bb051bf8cb5 server_cookie=e7e7196b4ad2d9f0 in_seq=0 out_seq=0 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.987+0000 7fe5f1ffb640 1 -- 172.21.15.77:0/4162334742 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ffff8ba9d0 con 0x55ffff9674d0 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.987+0000 7fe5f1ffb640 10 client.26393 handle_command_reply: tid=0 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.987+0000 7fe6137f9740 1 client.26393 shutdown 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.987+0000 7fe6137f9740 1 -- 172.21.15.77:0/4162334742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951060 msgr2=0x55ffff95b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.987+0000 7fe6137f9740 1 --2- 172.21.15.77:0/4162334742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951060 0x55ffff95b520 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc02c7b0 tx=0x7fe5fc0bca80 comp rx=0 tx=0).stop 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.987+0000 7fe6137f9740 1 -- 172.21.15.77:0/4162334742 shutdown_connections 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.987+0000 7fe6137f9740 1 --2- 172.21.15.77:0/4162334742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff956bc0 0x55ffff962aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.987+0000 7fe6137f9740 1 --2- 172.21.15.77:0/4162334742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff951a50 0x55ffff95ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.988+0000 7fe6137f9740 1 --2- 172.21.15.77:0/4162334742 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ffff9674d0 0x55ffff969990 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.988+0000 7fe6137f9740 1 --2- 172.21.15.77:0/4162334742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951060 0x55ffff95b520 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.988+0000 7fe5f1ffb640 0 client.26393 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.988+0000 7fe6137f9740 1 -- 172.21.15.77:0/4162334742 >> 172.21.15.77:0/4162334742 conn(0x55ffff943dc0 msgr2=0x55ffff942fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.988+0000 7fe6137f9740 1 -- 172.21.15.77:0/4162334742 shutdown_connections 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.988+0000 7fe6137f9740 1 -- 172.21.15.77:0/4162334742 wait complete. 2024-03-16T10:20:58.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:58.988+0000 7fe6137f9740 20 client.26393 trim_cache size 0 max 16384 2024-03-16T10:20:59.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.013+0000 7fe6137f9740 1 Processor -- start 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.013+0000 7fe6137f9740 1 -- start start 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe6137f9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff95b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe6137f9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951a50 0x55ffff962a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe6137f9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff956bc0 0x55ffff942750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe6137f9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1a6a20 con 0x55ffff956bc0 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe6137f9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1a8650 con 0x55ffff951a50 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe6137f9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1a70f0 con 0x55ffff951060 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe609d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951a50 0x55ffff962a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe609d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951a50 0x55ffff962a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42162/0 (socket says 172.21.15.77:42162) 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe609d9b640 1 -- 172.21.15.77:0/3223012088 learned_addr learned my addr 172.21.15.77:0/3223012088 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe60a59c640 1 --2- 172.21.15.77:0/3223012088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff95b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe60ad9d640 1 --2- 172.21.15.77:0/3223012088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff956bc0 0x55ffff942750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe60a59c640 1 -- 172.21.15.77:0/3223012088 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951a50 msgr2=0x55ffff962a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe60a59c640 1 --2- 172.21.15.77:0/3223012088 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951a50 0x55ffff962a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.014+0000 7fe60a59c640 1 -- 172.21.15.77:0/3223012088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff956bc0 msgr2=0x55ffff942750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.015+0000 7fe60a59c640 1 --2- 172.21.15.77:0/3223012088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff956bc0 0x55ffff942750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.015+0000 7fe60a59c640 1 -- 172.21.15.77:0/3223012088 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffff68b7d0 con 0x55ffff951060 2024-03-16T10:20:59.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.015+0000 7fe609d9b640 1 --2- 172.21.15.77:0/3223012088 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951a50 0x55ffff962a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:59.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.015+0000 7fe60ad9d640 1 --2- 172.21.15.77:0/3223012088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff956bc0 0x55ffff942750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:59.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.015+0000 7fe60a59c640 1 --2- 172.21.15.77:0/3223012088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff95b840 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0bb310 tx=0x7fe5fc0bcaf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd8e77c0727992e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.015+0000 7fe5f3fff640 1 -- 172.21.15.77:0/3223012088 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc04b070 con 0x55ffff951060 2024-03-16T10:20:59.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.015+0000 7fe5f3fff640 1 -- 172.21.15.77:0/3223012088 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5fc0c5520 con 0x55ffff951060 2024-03-16T10:20:59.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.015+0000 7fe5f3fff640 1 -- 172.21.15.77:0/3223012088 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0f00f0 con 0x55ffff951060 2024-03-16T10:20:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.016+0000 7fe6137f9740 1 -- 172.21.15.77:0/3223012088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 msgr2=0x55ffff95b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.016+0000 7fe6137f9740 1 --2- 172.21.15.77:0/3223012088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff95b840 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0bb310 tx=0x7fe5fc0bcaf0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.016+0000 7fe6137f9740 1 -- 172.21.15.77:0/3223012088 shutdown_connections 2024-03-16T10:20:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.016+0000 7fe6137f9740 1 --2- 172.21.15.77:0/3223012088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff956bc0 0x55ffff942750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.016+0000 7fe6137f9740 1 --2- 172.21.15.77:0/3223012088 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951a50 0x55ffff962a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.016+0000 7fe6137f9740 1 --2- 172.21.15.77:0/3223012088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff95b840 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.016+0000 7fe6137f9740 1 -- 172.21.15.77:0/3223012088 >> 172.21.15.77:0/3223012088 conn(0x55ffff943dc0 msgr2=0x55ffff95b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.016+0000 7fe6137f9740 1 -- 172.21.15.77:0/3223012088 shutdown_connections 2024-03-16T10:20:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.016+0000 7fe6137f9740 1 -- 172.21.15.77:0/3223012088 wait complete. 2024-03-16T10:20:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe6137f9740 1 -- start start 2024-03-16T10:20:59.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe6137f9740 1 Processor -- start 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe6137f9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff9605d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe6137f9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951a50 0x55ffff960b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe6137f9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff956bc0 0x55ffff961b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe6137f9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1a7f50 con 0x55ffff956bc0 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe6137f9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1a7880 con 0x55ffff951a50 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe6137f9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1a4a00 con 0x55ffff951060 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe60a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff9605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe6137f9740 10 client.0 _ll_register_callbacks cb 0x7fe5fc0bcf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe609d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951a50 0x55ffff960b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe60a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff9605d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52682/0 (socket says 172.21.15.77:52682) 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe60a59c640 1 -- 172.21.15.77:0/1986230859 learned_addr learned my addr 172.21.15.77:0/1986230859 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe60ad9d640 1 --2- 172.21.15.77:0/1986230859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff956bc0 0x55ffff961b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe60a59c640 1 -- 172.21.15.77:0/1986230859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951a50 msgr2=0x55ffff960b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe60a59c640 1 --2- 172.21.15.77:0/1986230859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951a50 0x55ffff960b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe60a59c640 1 -- 172.21.15.77:0/1986230859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff956bc0 msgr2=0x55ffff961b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe60a59c640 1 --2- 172.21.15.77:0/1986230859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff956bc0 0x55ffff961b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.017+0000 7fe60a59c640 1 -- 172.21.15.77:0/1986230859 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffff962110 con 0x55ffff951060 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.018+0000 7fe60a59c640 1 --2- 172.21.15.77:0/1986230859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff9605d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0bcff0 tx=0x7fe5fc0bca80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c26fc8d7e819acb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.018+0000 7fe5f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.018+0000 7fe5f1ffb640 1 -- 172.21.15.77:0/1986230859 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc04b050 con 0x55ffff951060 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.018+0000 7fe5f1ffb640 1 -- 172.21.15.77:0/1986230859 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe5fc0b7dd0 con 0x55ffff951060 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.018+0000 7fe6137f9740 1 -- 172.21.15.77:0/1986230859 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5fc0c5520 con 0x55ffff951060 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.018+0000 7fe5f1ffb640 1 -- 172.21.15.77:0/1986230859 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe5fc0bc4b0 con 0x55ffff951060 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.019+0000 7fe6137f9740 10 client.26392 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.019+0000 7fe6137f9740 1 -- 172.21.15.77:0/1986230859 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ffff962110 con 0x55ffff951060 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.019+0000 7fe5f1ffb640 1 -- 172.21.15.77:0/1986230859 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5fc0cc070 con 0x55ffff951060 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.019+0000 7fe5f1ffb640 1 -- 172.21.15.77:0/1986230859 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7fe5fc0dbab0 con 0x55ffff951060 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.019+0000 7fe6137f9740 10 client.26392 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.019+0000 7fe6137f9740 10 client.26392 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:20:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.019+0000 7fe6137f9740 20 client.26392 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.019+0000 7fe6137f9740 1 --2- 172.21.15.77:0/1986230859 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ffff9ce290 0x55ffff9d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.019+0000 7fe6137f9740 4 client.26392 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:20:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.019+0000 7fe6137f9740 1 -- 172.21.15.77:0/1986230859 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ffff8b4d20 con 0x55ffff9ce290 2024-03-16T10:20:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.019+0000 7fe609d9b640 1 --2- 172.21.15.77:0/1986230859 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ffff9ce290 0x55ffff9d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.020+0000 7fe5f1ffb640 10 client.26392 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:20:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.020+0000 7fe609d9b640 1 --2- 172.21.15.77:0/1986230859 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ffff9ce290 0x55ffff9d06b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fd2d15df413d86f0 server_cookie=66a1befcef7e5da7 in_seq=0 out_seq=0 2024-03-16T10:20:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe5f1ffb640 1 -- 172.21.15.77:0/1986230859 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ffff8b4d20 con 0x55ffff9ce290 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe5f1ffb640 10 client.26392 handle_command_reply: tid=0 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe6137f9740 1 client.26392 shutdown 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe6137f9740 1 -- 172.21.15.77:0/1986230859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 msgr2=0x55ffff9605d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe6137f9740 1 --2- 172.21.15.77:0/1986230859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff9605d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0bcff0 tx=0x7fe5fc0bca80 comp rx=0 tx=0).stop 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe6137f9740 1 -- 172.21.15.77:0/1986230859 shutdown_connections 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe6137f9740 1 --2- 172.21.15.77:0/1986230859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ffff956bc0 0x55ffff961b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe6137f9740 1 --2- 172.21.15.77:0/1986230859 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55ffff9ce290 0x55ffff9d06b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe6137f9740 1 --2- 172.21.15.77:0/1986230859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ffff951a50 0x55ffff960b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe5f1ffb640 0 client.26392 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe6137f9740 1 --2- 172.21.15.77:0/1986230859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffff951060 0x55ffff9605d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe6137f9740 1 -- 172.21.15.77:0/1986230859 >> 172.21.15.77:0/1986230859 conn(0x55ffff943dc0 msgr2=0x55ffff947020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe6137f9740 1 -- 172.21.15.77:0/1986230859 shutdown_connections 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.022+0000 7fe6137f9740 1 -- 172.21.15.77:0/1986230859 wait complete. 2024-03-16T10:20:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.023+0000 7fe6137f9740 20 client.26392 trim_cache size 0 max 16384 2024-03-16T10:20:59.026 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:20:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.023+0000 7fe610c0b640 1 -- 172.21.15.77:0/3290817175 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5f40a2800 msgr2=0x7fe5f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.023+0000 7fe610c0b640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5f40a2800 0x7fe5f40a4cc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe60c101980 tx=0x7fe600005f90 comp rx=0 tx=0).stop 2024-03-16T10:20:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.023+0000 7fe610c0b640 1 -- 172.21.15.77:0/3290817175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c1022a0 msgr2=0x7fe60c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.023+0000 7fe610c0b640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c1022a0 0x7fe60c101c10 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009230 tx=0x7fe5fc02c650 comp rx=0 tx=0).stop 2024-03-16T10:20:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.024+0000 7fe610c0b640 1 -- 172.21.15.77:0/3290817175 shutdown_connections 2024-03-16T10:20:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.024+0000 7fe610c0b640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe60c107500 0x7fe60c100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.024+0000 7fe610c0b640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe5f40a2800 0x7fe5f40a4cc0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.024+0000 7fe610c0b640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe60c104bd0 0x7fe60c100250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.024+0000 7fe610c0b640 1 --2- 172.21.15.77:0/3290817175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe60c1022a0 0x7fe60c101c10 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.024+0000 7fe610c0b640 1 -- 172.21.15.77:0/3290817175 >> 172.21.15.77:0/3290817175 conn(0x7fe60c0fbc70 msgr2=0x7fe60c1040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.024+0000 7fe610c0b640 1 -- 172.21.15.77:0/3290817175 shutdown_connections 2024-03-16T10:20:59.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.024+0000 7fe610c0b640 1 -- 172.21.15.77:0/3290817175 wait complete. 2024-03-16T10:20:59.037 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T10:20:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.152+0000 7f3ea47f4640 1 -- 172.21.15.77:0/624192628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e9c1006e0 msgr2=0x7f3e9c100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.152+0000 7f3ea47f4640 1 --2- 172.21.15.77:0/624192628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e9c1006e0 0x7f3e9c100b00 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009900 tx=0x7f3e9802bc30 comp rx=0 tx=0).stop 2024-03-16T10:20:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 -- 172.21.15.77:0/624192628 shutdown_connections 2024-03-16T10:20:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 --2- 172.21.15.77:0/624192628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c102290 0x7f3e9c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 --2- 172.21.15.77:0/624192628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e9c1018d0 0x7f3e9c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 --2- 172.21.15.77:0/624192628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e9c1006e0 0x7f3e9c100b00 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 -- 172.21.15.77:0/624192628 >> 172.21.15.77:0/624192628 conn(0x7f3e9c0fbc70 msgr2=0x7f3e9c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 -- 172.21.15.77:0/624192628 shutdown_connections 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 -- 172.21.15.77:0/624192628 wait complete. 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 Processor -- start 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 -- start start 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c1006e0 0x7f3e9c0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e9c1018d0 0x7f3e9c19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e9c102290 0x7f3e9c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c10c330 con 0x7f3e9c102290 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c10c1b0 con 0x7f3e9c1006e0 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea47f4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c10c4b0 con 0x7f3e9c1018d0 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea2d91640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e9c102290 0x7f3e9c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.153+0000 7f3ea2590640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c1006e0 0x7f3e9c0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea2590640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c1006e0 0x7f3e9c0694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42202/0 (socket says 172.21.15.77:42202) 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea2d91640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e9c102290 0x7f3e9c19eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54598/0 (socket says 172.21.15.77:54598) 2024-03-16T10:20:59.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea2590640 1 -- 172.21.15.77:0/3379839696 learned_addr learned my addr 172.21.15.77:0/3379839696 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea1d8f640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e9c1018d0 0x7f3e9c19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea2590640 1 -- 172.21.15.77:0/3379839696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e9c1018d0 msgr2=0x7f3e9c19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea2590640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e9c1018d0 0x7f3e9c19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea2590640 1 -- 172.21.15.77:0/3379839696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e9c102290 msgr2=0x7f3e9c19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea2590640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e9c102290 0x7f3e9c19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea2590640 1 -- 172.21.15.77:0/3379839696 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9c19f7d0 con 0x7f3e9c1006e0 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea1d8f640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e9c1018d0 0x7f3e9c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea2590640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c1006e0 0x7f3e9c0694b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3e98000c00 tx=0x7f3e9802c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3e937fe640 1 -- 172.21.15.77:0/3379839696 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e98041070 con 0x7f3e9c1006e0 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea47f4640 1 -- 172.21.15.77:0/3379839696 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9c19fa00 con 0x7f3e9c1006e0 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3e937fe640 1 -- 172.21.15.77:0/3379839696 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e98004450 con 0x7f3e9c1006e0 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3e937fe640 1 -- 172.21.15.77:0/3379839696 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e98034440 con 0x7f3e9c1006e0 2024-03-16T10:20:59.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.154+0000 7f3ea47f4640 1 -- 172.21.15.77:0/3379839696 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e9c19ff40 con 0x7f3e9c1006e0 2024-03-16T10:20:59.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.157+0000 7f3ea73e6740 1 Processor -- start 2024-03-16T10:20:59.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.157+0000 7f3ea73e6740 1 -- start start 2024-03-16T10:20:59.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3e937fe640 1 -- 172.21.15.77:0/3379839696 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f3e98044070 con 0x7f3e9c1006e0 2024-03-16T10:20:59.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea73e6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fd850 0x55a6276fdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea73e6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6276fe240 0x55a627702d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea73e6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6277033b0 0x55a627705840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea73e6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a627662490 con 0x55a6277033b0 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea73e6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a62760ac90 con 0x55a6276fd850 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea73e6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a62765e530 con 0x55a6276fe240 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea2d91640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fd850 0x55a6276fdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea2d91640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fd850 0x55a6276fdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42214/0 (socket says 172.21.15.77:42214) 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea2d91640 1 -- 172.21.15.77:0/2623350745 learned_addr learned my addr 172.21.15.77:0/2623350745 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea2590640 1 --2- 172.21.15.77:0/2623350745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6276fe240 0x55a627702d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea1d8f640 1 --2- 172.21.15.77:0/2623350745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6277033b0 0x55a627705840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea2d91640 1 -- 172.21.15.77:0/2623350745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6276fe240 msgr2=0x55a627702d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/2623350745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6276fe240 0x55a627702d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea2d91640 1 -- 172.21.15.77:0/2623350745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6277033b0 msgr2=0x55a627705840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/2623350745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6277033b0 0x55a627705840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea2d91640 1 -- 172.21.15.77:0/2623350745 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6276c5500 con 0x55a6276fd850 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea2590640 1 --2- 172.21.15.77:0/2623350745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6276fe240 0x55a627702d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3e937fe640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e880a2800 0x7f3e880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.158+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/2623350745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fd850 0x55a6276fdc70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3e94009fd0 tx=0x7f3e940118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1ec3cccb168ebe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.159+0000 7f3ea2590640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e880a2800 0x7f3e880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.159+0000 7f3e90ff9640 1 -- 172.21.15.77:0/2623350745 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94004d20 con 0x55a6276fd850 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.159+0000 7f3e937fe640 1 -- 172.21.15.77:0/3379839696 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f3e980f0490 con 0x7f3e9c1006e0 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.159+0000 7f3e90ff9640 1 -- 172.21.15.77:0/2623350745 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e94004050 con 0x55a6276fd850 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.159+0000 7f3e90ff9640 1 -- 172.21.15.77:0/2623350745 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94005260 con 0x55a6276fd850 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.159+0000 7f3ea2590640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e880a2800 0x7f3e880a4cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x55a6276fecb0 tx=0x7f3e980c3040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.159+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2623350745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fd850 msgr2=0x55a6276fdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/2623350745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fd850 0x55a6276fdc70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3e94009fd0 tx=0x7f3e940118d0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2623350745 shutdown_connections 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/2623350745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6277033b0 0x55a627705840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/2623350745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6276fe240 0x55a627702d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/2623350745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fd850 0x55a6276fdc70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2623350745 >> 172.21.15.77:0/2623350745 conn(0x55a6276f8dc0 msgr2=0x55a6276f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2623350745 shutdown_connections 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2623350745 wait complete. 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 -- start start 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 Processor -- start 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a627707d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6276fe240 0x55a6277082c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6277033b0 0x55a62770f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a627706400 con 0x55a6276fd850 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a62761e230 con 0x55a6277033b0 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a627706580 con 0x55a6276fe240 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea73e6740 10 client.0 _ll_register_callbacks cb 0x55a62765a010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:59.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.160+0000 7f3ea2d91640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a627707d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea2d91640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a627707d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54608/0 (socket says 172.21.15.77:54608) 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea2d91640 1 -- 172.21.15.77:0/397923215 learned_addr learned my addr 172.21.15.77:0/397923215 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea1d8f640 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6276fe240 0x55a6277082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6277033b0 0x55a62770f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea2d91640 1 -- 172.21.15.77:0/397923215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6276fe240 msgr2=0x55a6277082c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6276fe240 0x55a6277082c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea2d91640 1 -- 172.21.15.77:0/397923215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6277033b0 msgr2=0x55a62770f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6277033b0 0x55a62770f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea2d91640 1 -- 172.21.15.77:0/397923215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a62762d100 con 0x55a6276fd850 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a627707d80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3e94011dd0 tx=0x7f3e94009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f6a1840df19456 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3e85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3e85ffb640 1 -- 172.21.15.77:0/397923215 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94004710 con 0x55a6276fd850 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea73e6740 1 -- 172.21.15.77:0/397923215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e94004050 con 0x55a6276fd850 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3e85ffb640 1 -- 172.21.15.77:0/397923215 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f3e9400f4e0 con 0x55a6276fd850 2024-03-16T10:20:59.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3e85ffb640 1 -- 172.21.15.77:0/397923215 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3e9400f6e0 con 0x55a6276fd850 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea73e6740 10 client.16647 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.161+0000 7f3ea73e6740 1 -- 172.21.15.77:0/397923215 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a6273763f0 con 0x55a6276fd850 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.162+0000 7f3e85ffb640 1 -- 172.21.15.77:0/397923215 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e94004050 con 0x55a6276fd850 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.162+0000 7f3e85ffb640 1 -- 172.21.15.77:0/397923215 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f3e940054c0 con 0x55a6276fd850 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.162+0000 7f3ea73e6740 10 client.16647 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.162+0000 7f3ea73e6740 10 client.16647 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.162+0000 7f3ea73e6740 20 client.16647 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.162+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a6277139f0 0x55a627715e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.162+0000 7f3ea73e6740 4 client.16647 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.162+0000 7f3ea73e6740 1 -- 172.21.15.77:0/397923215 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a62766f9d0 con 0x55a6277139f0 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.162+0000 7f3ea1d8f640 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a6277139f0 0x55a627715e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.162+0000 7f3e85ffb640 10 client.16647 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:20:59.168 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.162+0000 7f3ea1d8f640 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a6277139f0 0x55a627715e00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa38b24e57a588fe server_cookie=b6f63c0ae3e9c50b in_seq=0 out_seq=0 2024-03-16T10:20:59.169 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:58 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/72740269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:20:59.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3e85ffb640 1 -- 172.21.15.77:0/397923215 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a62766f9d0 con 0x55a6277139f0 2024-03-16T10:20:59.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3e85ffb640 10 client.16647 handle_command_reply: tid=0 2024-03-16T10:20:59.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3ea73e6740 1 client.16647 shutdown 2024-03-16T10:20:59.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3ea73e6740 1 -- 172.21.15.77:0/397923215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 msgr2=0x55a627707d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a627707d80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3e94011dd0 tx=0x7f3e94009ca0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3ea73e6740 1 -- 172.21.15.77:0/397923215 shutdown_connections 2024-03-16T10:20:59.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a6277139f0 0x55a627715e00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6277033b0 0x55a62770f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3e85ffb640 0 client.16647 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:20:59.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6276fe240 0x55a6277082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/397923215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a627707d80 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3ea73e6740 1 -- 172.21.15.77:0/397923215 >> 172.21.15.77:0/397923215 conn(0x55a6276f8dc0 msgr2=0x55a6276f7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.163+0000 7f3ea73e6740 1 -- 172.21.15.77:0/397923215 shutdown_connections 2024-03-16T10:20:59.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.164+0000 7f3ea73e6740 1 -- 172.21.15.77:0/397923215 wait complete. 2024-03-16T10:20:59.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.164+0000 7f3ea73e6740 20 client.16647 trim_cache size 0 max 16384 2024-03-16T10:20:59.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.177+0000 7f3ea73e6740 1 Processor -- start 2024-03-16T10:20:59.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.177+0000 7f3ea73e6740 1 -- start start 2024-03-16T10:20:59.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.177+0000 7f3ea73e6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a627707990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.177+0000 7f3ea73e6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fe240 0x55a62770f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.182 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.177+0000 7f3ea73e6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6277033b0 0x55a627707ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.177+0000 7f3ea73e6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c1a69f0 con 0x55a6276fd850 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.177+0000 7f3ea73e6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c1a8620 con 0x55a6276fe240 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.177+0000 7f3ea73e6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c1a70c0 con 0x55a6277033b0 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.177+0000 7f3ea2d91640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a627707990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.177+0000 7f3ea2d91640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a627707990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54612/0 (socket says 172.21.15.77:54612) 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.178+0000 7f3ea2d91640 1 -- 172.21.15.77:0/1219229205 learned_addr learned my addr 172.21.15.77:0/1219229205 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.178+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/1219229205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6277033b0 0x55a627707ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.178+0000 7f3ea1d8f640 1 --2- 172.21.15.77:0/1219229205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fe240 0x55a62770f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.178+0000 7f3ea2d91640 1 -- 172.21.15.77:0/1219229205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6277033b0 msgr2=0x55a627707ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.178+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/1219229205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6277033b0 0x55a627707ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.178+0000 7f3ea2d91640 1 -- 172.21.15.77:0/1219229205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fe240 msgr2=0x55a62770f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.178+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/1219229205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fe240 0x55a62770f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.178+0000 7f3ea2d91640 1 -- 172.21.15.77:0/1219229205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6274407d0 con 0x55a6276fd850 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.178+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/1219229205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a627707990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3e94005d70 tx=0x7f3e94009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2324ed4a110146 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.178+0000 7f3e87fff640 1 -- 172.21.15.77:0/1219229205 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94009ac0 con 0x55a6276fd850 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.178+0000 7f3e87fff640 1 -- 172.21.15.77:0/1219229205 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e9402c070 con 0x55a6276fd850 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.178+0000 7f3e87fff640 1 -- 172.21.15.77:0/1219229205 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e9400fb70 con 0x55a6276fd850 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 -- 172.21.15.77:0/1219229205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 msgr2=0x55a627707990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/1219229205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a627707990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3e94005d70 tx=0x7f3e94009520 comp rx=0 tx=0).stop 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 -- 172.21.15.77:0/1219229205 shutdown_connections 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/1219229205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6277033b0 0x55a627707ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/1219229205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fe240 0x55a62770f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/1219229205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a627707990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 -- 172.21.15.77:0/1219229205 >> 172.21.15.77:0/1219229205 conn(0x55a6276f8dc0 msgr2=0x55a6276ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 -- 172.21.15.77:0/1219229205 shutdown_connections 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 -- 172.21.15.77:0/1219229205 wait complete. 2024-03-16T10:20:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 -- start start 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 Processor -- start 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a62770c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.179+0000 7f3ea73e6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fe240 0x55a62770c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea73e6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6277033b0 0x55a627777cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea73e6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c1a7f20 con 0x55a6276fd850 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea73e6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c1a7850 con 0x55a6276fe240 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea73e6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c1a49d0 con 0x55a6277033b0 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea73e6740 10 client.0 _ll_register_callbacks cb 0x7f3e8c009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea2d91640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a62770c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea2d91640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a62770c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54618/0 (socket says 172.21.15.77:54618) 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea2d91640 1 -- 172.21.15.77:0/2853096902 learned_addr learned my addr 172.21.15.77:0/2853096902 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea1d8f640 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fe240 0x55a62770c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6277033b0 0x55a627777cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea2d91640 1 -- 172.21.15.77:0/2853096902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6277033b0 msgr2=0x55a627777cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6277033b0 0x55a627777cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea2d91640 1 -- 172.21.15.77:0/2853096902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fe240 msgr2=0x55a62770c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fe240 0x55a62770c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea2d91640 1 -- 172.21.15.77:0/2853096902 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6277782b0 con 0x55a6276fd850 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea2d91640 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a62770c280 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3e94005c40 tx=0x7f3e940021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9552b8d47e808fb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3e85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3e85ffb640 1 -- 172.21.15.77:0/2853096902 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94004720 con 0x55a6276fd850 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3e85ffb640 1 -- 172.21.15.77:0/2853096902 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f3e9400f8f0 con 0x55a6276fd850 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.180+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2853096902 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e9402c070 con 0x55a6276fd850 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.181+0000 7f3e85ffb640 1 -- 172.21.15.77:0/2853096902 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3e9400faf0 con 0x55a6276fd850 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.181+0000 7f3ea73e6740 10 client.16653 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.181+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2853096902 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a6277795b0 con 0x55a6276fd850 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.181+0000 7f3e85ffb640 1 -- 172.21.15.77:0/2853096902 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e9400fc90 con 0x55a6276fd850 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.181+0000 7f3e85ffb640 1 -- 172.21.15.77:0/2853096902 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f3e94004ed0 con 0x55a6276fd850 2024-03-16T10:20:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.181+0000 7f3ea73e6740 10 client.16653 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.181+0000 7f3ea73e6740 10 client.16653 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:20:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.181+0000 7f3ea73e6740 20 client.16653 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.181+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a6277139f0 0x55a62777be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.181+0000 7f3ea73e6740 4 client.16653 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:20:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.181+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2853096902 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f3e9402c070 con 0x55a6277139f0 2024-03-16T10:20:59.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.182+0000 7f3ea1d8f640 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a6277139f0 0x55a62777be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea1d8f640 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a6277139f0 0x55a62777be50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=58e8a6f7c0b28d7c server_cookie=200b34dd7f4b030c in_seq=0 out_seq=0 2024-03-16T10:20:59.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3e85ffb640 10 client.16653 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3e85ffb640 1 -- 172.21.15.77:0/2853096902 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f3e9402c070 con 0x55a6277139f0 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3e85ffb640 10 client.16653 handle_command_reply: tid=0 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea73e6740 1 client.16653 shutdown 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2853096902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 msgr2=0x55a62770c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a62770c280 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3e94005c40 tx=0x7f3e940021d0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2853096902 shutdown_connections 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55a6277139f0 0x55a62777be50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6277033b0 0x55a627777cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6276fe240 0x55a62770c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea73e6740 1 --2- 172.21.15.77:0/2853096902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a6276fd850 0x55a62770c280 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3e85ffb640 0 client.16653 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2853096902 >> 172.21.15.77:0/2853096902 conn(0x55a6276f8dc0 msgr2=0x55a627707990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2853096902 shutdown_connections 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea73e6740 1 -- 172.21.15.77:0/2853096902 wait complete. 2024-03-16T10:20:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.183+0000 7f3ea73e6740 20 client.16653 trim_cache size 0 max 16384 2024-03-16T10:20:59.190 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:20:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.184+0000 7f3ea47f4640 1 -- 172.21.15.77:0/3379839696 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e880a2800 msgr2=0x7f3e880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.184+0000 7f3ea47f4640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e880a2800 0x7f3e880a4cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x55a6276fecb0 tx=0x7f3e980c3040 comp rx=0 tx=0).stop 2024-03-16T10:20:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.184+0000 7f3ea47f4640 1 -- 172.21.15.77:0/3379839696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c1006e0 msgr2=0x7f3e9c0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.184+0000 7f3ea47f4640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c1006e0 0x7f3e9c0694b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3e98000c00 tx=0x7f3e9802c640 comp rx=0 tx=0).stop 2024-03-16T10:20:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.184+0000 7f3ea47f4640 1 -- 172.21.15.77:0/3379839696 shutdown_connections 2024-03-16T10:20:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.184+0000 7f3ea47f4640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e9c102290 0x7f3e9c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.184+0000 7f3ea47f4640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3e880a2800 0x7f3e880a4cc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.184+0000 7f3ea47f4640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e9c1018d0 0x7f3e9c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.184+0000 7f3ea47f4640 1 --2- 172.21.15.77:0/3379839696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c1006e0 0x7f3e9c0694b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.184+0000 7f3ea47f4640 1 -- 172.21.15.77:0/3379839696 >> 172.21.15.77:0/3379839696 conn(0x7f3e9c0fbc70 msgr2=0x7f3e9c104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.184+0000 7f3ea47f4640 1 -- 172.21.15.77:0/3379839696 shutdown_connections 2024-03-16T10:20:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.184+0000 7f3ea47f4640 1 -- 172.21.15.77:0/3379839696 wait complete. 2024-03-16T10:20:59.194 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:58 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/72740269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:20:59.196 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T10:20:59.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:58 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/72740269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:20:59.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.311+0000 7ff47f71b640 1 -- 172.21.15.77:0/497069150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4781022b0 msgr2=0x7ff478108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.311+0000 7ff47f71b640 1 --2- 172.21.15.77:0/497069150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4781022b0 0x7ff478108b40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff470009ca0 tx=0x7ff47002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.311+0000 7ff47f71b640 1 -- 172.21.15.77:0/497069150 shutdown_connections 2024-03-16T10:20:59.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.311+0000 7ff47f71b640 1 --2- 172.21.15.77:0/497069150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff4781022b0 0x7ff478108b40 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.311+0000 7ff47f71b640 1 --2- 172.21.15.77:0/497069150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4781018f0 0x7ff478101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.311+0000 7ff47f71b640 1 --2- 172.21.15.77:0/497069150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff478100700 0x7ff478100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.311+0000 7ff47f71b640 1 -- 172.21.15.77:0/497069150 >> 172.21.15.77:0/497069150 conn(0x7ff4780fbc90 msgr2=0x7ff4780fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.311+0000 7ff47f71b640 1 -- 172.21.15.77:0/497069150 shutdown_connections 2024-03-16T10:20:59.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.311+0000 7ff47f71b640 1 -- 172.21.15.77:0/497069150 wait complete. 2024-03-16T10:20:59.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.311+0000 7ff47f71b640 1 Processor -- start 2024-03-16T10:20:59.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47f71b640 1 -- start start 2024-03-16T10:20:59.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47f71b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478100700 0x7ff47806a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47f71b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4781018f0 0x7ff47819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47f71b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4781022b0 0x7ff47819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47f71b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff47810c380 con 0x7ff478100700 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47f71b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff47810c200 con 0x7ff4781022b0 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47f71b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff47810c500 con 0x7ff4781018f0 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47d4b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478100700 0x7ff47806a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47d4b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478100700 0x7ff47806a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54646/0 (socket says 172.21.15.77:54646) 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47d4b7640 1 -- 172.21.15.77:0/1466533941 learned_addr learned my addr 172.21.15.77:0/1466533941 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47dcb8640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4781022b0 0x7ff47819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47ccb6640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4781018f0 0x7ff47819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47d4b7640 1 -- 172.21.15.77:0/1466533941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4781018f0 msgr2=0x7ff47819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47d4b7640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4781018f0 0x7ff47819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47d4b7640 1 -- 172.21.15.77:0/1466533941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4781022b0 msgr2=0x7ff47819ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47d4b7640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4781022b0 0x7ff47819ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47d4b7640 1 -- 172.21.15.77:0/1466533941 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff47819f770 con 0x7ff478100700 2024-03-16T10:20:59.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.312+0000 7ff47d4b7640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478100700 0x7ff47806a940 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff47400e9c0 tx=0x7ff47400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.313+0000 7ff46e7fc640 1 -- 172.21.15.77:0/1466533941 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47400cd80 con 0x7ff478100700 2024-03-16T10:20:59.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.313+0000 7ff46e7fc640 1 -- 172.21.15.77:0/1466533941 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4740035f0 con 0x7ff478100700 2024-03-16T10:20:59.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.313+0000 7ff47f71b640 1 -- 172.21.15.77:0/1466533941 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff47819fa00 con 0x7ff478100700 2024-03-16T10:20:59.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.313+0000 7ff46e7fc640 1 -- 172.21.15.77:0/1466533941 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff474010690 con 0x7ff478100700 2024-03-16T10:20:59.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.313+0000 7ff47f71b640 1 -- 172.21.15.77:0/1466533941 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff47819ff50 con 0x7ff478100700 2024-03-16T10:20:59.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.316+0000 7ff46e7fc640 1 -- 172.21.15.77:0/1466533941 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7ff4740037a0 con 0x7ff478100700 2024-03-16T10:20:59.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.316+0000 7ff4823e2740 1 Processor -- start 2024-03-16T10:20:59.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.316+0000 7ff4823e2740 1 -- start start 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.316+0000 7ff4823e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869bf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.316+0000 7ff4823e2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869bf650 0x55d5869c4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff4823e2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 0x55d5869c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff4823e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d58691f490 con 0x55d5869bec60 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff4823e2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d5868c7c90 con 0x55d5869c47c0 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff4823e2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d58691b530 con 0x55d5869bf650 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47dcb8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869bf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47ccb6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 0x55d5869c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff46e7fc640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff4640a2800 0x7ff4640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47ccb6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 0x55d5869c6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42268/0 (socket says 172.21.15.77:42268) 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47ccb6640 1 -- 172.21.15.77:0/1840513441 learned_addr learned my addr 172.21.15.77:0/1840513441 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47d4b7640 1 --2- 172.21.15.77:0/1840513441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869bf650 0x55d5869c4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47dcb8640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff4640a2800 0x7ff4640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47ccb6640 1 -- 172.21.15.77:0/1840513441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869bf650 msgr2=0x55d5869c4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47ccb6640 1 --2- 172.21.15.77:0/1840513441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869bf650 0x55d5869c4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47ccb6640 1 -- 172.21.15.77:0/1840513441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 msgr2=0x55d5869bf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47ccb6640 1 --2- 172.21.15.77:0/1840513441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869bf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47ccb6640 1 -- 172.21.15.77:0/1840513441 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d586982500 con 0x55d5869c47c0 2024-03-16T10:20:59.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47dcb8640 1 --2- 172.21.15.77:0/1840513441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869bf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:20:59.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff46e7fc640 1 -- 172.21.15.77:0/1466533941 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ff4740ced80 con 0x7ff478100700 2024-03-16T10:20:59.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.317+0000 7ff47ccb6640 1 --2- 172.21.15.77:0/1840513441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 0x55d5869c6c50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff468009ca0 tx=0x7ff468011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86a807f468a6bfd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.318+0000 7ff47dcb8640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff4640a2800 0x7ff4640a4cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff470005e30 tx=0x7ff47002c0f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.318+0000 7ff4637fe640 1 -- 172.21.15.77:0/1840513441 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46801e070 con 0x55d5869c47c0 2024-03-16T10:20:59.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.318+0000 7ff4637fe640 1 -- 172.21.15.77:0/1840513441 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff468019b10 con 0x55d5869c47c0 2024-03-16T10:20:59.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.318+0000 7ff4637fe640 1 -- 172.21.15.77:0/1840513441 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46800eae0 con 0x55d5869c47c0 2024-03-16T10:20:59.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.318+0000 7ff4823e2740 1 -- 172.21.15.77:0/1840513441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 msgr2=0x55d5869c6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.318+0000 7ff4823e2740 1 --2- 172.21.15.77:0/1840513441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 0x55d5869c6c50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff468009ca0 tx=0x7ff468011810 comp rx=0 tx=0).stop 2024-03-16T10:20:59.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.318+0000 7ff4823e2740 1 -- 172.21.15.77:0/1840513441 shutdown_connections 2024-03-16T10:20:59.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.318+0000 7ff4823e2740 1 --2- 172.21.15.77:0/1840513441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 0x55d5869c6c50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.318+0000 7ff4823e2740 1 --2- 172.21.15.77:0/1840513441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869bf650 0x55d5869c4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.318+0000 7ff4823e2740 1 --2- 172.21.15.77:0/1840513441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869bf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.318+0000 7ff4823e2740 1 -- 172.21.15.77:0/1840513441 >> 172.21.15.77:0/1840513441 conn(0x55d5869b5dc0 msgr2=0x55d5869b61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.319+0000 7ff4823e2740 1 -- 172.21.15.77:0/1840513441 shutdown_connections 2024-03-16T10:20:59.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.319+0000 7ff4823e2740 1 -- 172.21.15.77:0/1840513441 wait complete. 2024-03-16T10:20:59.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.319+0000 7ff4823e2740 1 -- start start 2024-03-16T10:20:59.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.319+0000 7ff4823e2740 1 Processor -- start 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.319+0000 7ff4823e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869c9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.319+0000 7ff4823e2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 0x55d5869c9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.319+0000 7ff4823e2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869c47c0 0x55d5869d06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.319+0000 7ff4823e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d5869c7770 con 0x55d5869bec60 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.319+0000 7ff4823e2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d5868db230 con 0x55d5869bf650 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.319+0000 7ff4823e2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d5869c78f0 con 0x55d5869c47c0 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff4823e2740 10 client.0 _ll_register_callbacks cb 0x55d586917010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff47d4b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff47d4b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869c9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54668/0 (socket says 172.21.15.77:54668) 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff47d4b7640 1 -- 172.21.15.77:0/648233029 learned_addr learned my addr 172.21.15.77:0/648233029 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff47ccb6640 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 0x55d5869c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff47dcb8640 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869c47c0 0x55d5869d06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff47ccb6640 1 -- 172.21.15.77:0/648233029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869c47c0 msgr2=0x55d5869d06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff47ccb6640 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869c47c0 0x55d5869d06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff47ccb6640 1 -- 172.21.15.77:0/648233029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 msgr2=0x55d5869c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff47ccb6640 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff47ccb6640 1 -- 172.21.15.77:0/648233029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5868ea100 con 0x55d5869bf650 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff47d4b7640 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff47ccb6640 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 0x55d5869c9680 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff468011d10 tx=0x7ff468019640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f75b617fc3527edd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.320+0000 7ff460ff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.321+0000 7ff460ff9640 1 -- 172.21.15.77:0/648233029 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff468027030 con 0x55d5869bf650 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.321+0000 7ff4823e2740 1 -- 172.21.15.77:0/648233029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff468019b10 con 0x55d5869bf650 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.321+0000 7ff460ff9640 1 -- 172.21.15.77:0/648233029 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ff468019830 con 0x55d5869bf650 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.321+0000 7ff460ff9640 1 -- 172.21.15.77:0/648233029 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff46801e040 con 0x55d5869bf650 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.321+0000 7ff4823e2740 10 client.26429 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.321+0000 7ff4823e2740 1 -- 172.21.15.77:0/648233029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d5866333f0 con 0x55d5869bf650 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.321+0000 7ff460ff9640 1 -- 172.21.15.77:0/648233029 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff468019b10 con 0x55d5869bf650 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.322+0000 7ff460ff9640 1 -- 172.21.15.77:0/648233029 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7ff46800e7e0 con 0x55d5869bf650 2024-03-16T10:20:59.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.322+0000 7ff4823e2740 10 client.26429 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.322+0000 7ff4823e2740 10 client.26429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.322+0000 7ff4823e2740 20 client.26429 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.322+0000 7ff4823e2740 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d5869d4ca0 0x55d5869d7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.322+0000 7ff4823e2740 4 client.26429 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.322+0000 7ff4823e2740 1 -- 172.21.15.77:0/648233029 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d58692c9d0 con 0x55d5869d4ca0 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.322+0000 7ff47d4b7640 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d5869d4ca0 0x55d5869d7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.323+0000 7ff460ff9640 10 client.26429 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.323+0000 7ff47d4b7640 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d5869d4ca0 0x55d5869d7110 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e7d6037987bf01f server_cookie=15f61861c79d8a82 in_seq=0 out_seq=0 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.323+0000 7ff460ff9640 1 -- 172.21.15.77:0/648233029 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d58692c9d0 con 0x55d5869d4ca0 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.323+0000 7ff460ff9640 10 client.26429 handle_command_reply: tid=0 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff4823e2740 1 client.26429 shutdown 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff4823e2740 1 -- 172.21.15.77:0/648233029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 msgr2=0x55d5869c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff4823e2740 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 0x55d5869c9680 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff468011d10 tx=0x7ff468019640 comp rx=0 tx=0).stop 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff4823e2740 1 -- 172.21.15.77:0/648233029 shutdown_connections 2024-03-16T10:20:59.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff4823e2740 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869c47c0 0x55d5869d06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff4823e2740 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 0x55d5869c9680 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff4823e2740 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d5869d4ca0 0x55d5869d7110 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff4823e2740 1 --2- 172.21.15.77:0/648233029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff460ff9640 0 client.26429 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff4823e2740 1 -- 172.21.15.77:0/648233029 >> 172.21.15.77:0/648233029 conn(0x55d5869b5dc0 msgr2=0x55d5869b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff4823e2740 1 -- 172.21.15.77:0/648233029 shutdown_connections 2024-03-16T10:20:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff4823e2740 1 -- 172.21.15.77:0/648233029 wait complete. 2024-03-16T10:20:59.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.324+0000 7ff4823e2740 20 client.26429 trim_cache size 0 max 16384 2024-03-16T10:20:59.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff4823e2740 1 Processor -- start 2024-03-16T10:20:59.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff4823e2740 1 -- start start 2024-03-16T10:20:59.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff4823e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869b9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff4823e2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 0x55d5869d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff4823e2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869c47c0 0x55d5869db9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff4823e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff4781a6a50 con 0x55d5869bec60 2024-03-16T10:20:59.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff4823e2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff4781a8680 con 0x55d5869bf650 2024-03-16T10:20:59.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff4823e2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff4781a7120 con 0x55d5869c47c0 2024-03-16T10:20:59.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff47ccb6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 0x55d5869d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff47d4b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869b9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff47ccb6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 0x55d5869d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42288/0 (socket says 172.21.15.77:42288) 2024-03-16T10:20:59.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff47ccb6640 1 -- 172.21.15.77:0/2006951070 learned_addr learned my addr 172.21.15.77:0/2006951070 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff47dcb8640 1 --2- 172.21.15.77:0/2006951070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869c47c0 0x55d5869db9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.343+0000 7ff47ccb6640 1 -- 172.21.15.77:0/2006951070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869c47c0 msgr2=0x55d5869db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.344+0000 7ff47ccb6640 1 --2- 172.21.15.77:0/2006951070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869c47c0 0x55d5869db9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.344+0000 7ff47ccb6640 1 -- 172.21.15.77:0/2006951070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 msgr2=0x55d5869b9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.344+0000 7ff47ccb6640 1 --2- 172.21.15.77:0/2006951070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869b9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.344+0000 7ff47ccb6640 1 -- 172.21.15.77:0/2006951070 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5866fd7d0 con 0x55d5869bf650 2024-03-16T10:20:59.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.344+0000 7ff47d4b7640 1 --2- 172.21.15.77:0/2006951070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869b9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:59.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.344+0000 7ff47ccb6640 1 --2- 172.21.15.77:0/2006951070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 0x55d5869d0680 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff468009800 tx=0x7ff468005970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69ba1bb14bf948db server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.344+0000 7ff462ffd640 1 -- 172.21.15.77:0/2006951070 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff468027040 con 0x55d5869bf650 2024-03-16T10:20:59.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.344+0000 7ff462ffd640 1 -- 172.21.15.77:0/2006951070 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff46800e420 con 0x55d5869bf650 2024-03-16T10:20:59.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.344+0000 7ff462ffd640 1 -- 172.21.15.77:0/2006951070 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff468019910 con 0x55d5869bf650 2024-03-16T10:20:59.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.345+0000 7ff4823e2740 1 -- 172.21.15.77:0/2006951070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 msgr2=0x55d5869d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.345+0000 7ff4823e2740 1 --2- 172.21.15.77:0/2006951070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 0x55d5869d0680 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff468009800 tx=0x7ff468005970 comp rx=0 tx=0).stop 2024-03-16T10:20:59.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.345+0000 7ff4823e2740 1 -- 172.21.15.77:0/2006951070 shutdown_connections 2024-03-16T10:20:59.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.345+0000 7ff4823e2740 1 --2- 172.21.15.77:0/2006951070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869c47c0 0x55d5869db9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.345+0000 7ff4823e2740 1 --2- 172.21.15.77:0/2006951070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869bf650 0x55d5869d0680 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.345+0000 7ff4823e2740 1 --2- 172.21.15.77:0/2006951070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bec60 0x55d5869b9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.345+0000 7ff4823e2740 1 -- 172.21.15.77:0/2006951070 >> 172.21.15.77:0/2006951070 conn(0x55d5869b5dc0 msgr2=0x55d5869b9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.345+0000 7ff4823e2740 1 -- 172.21.15.77:0/2006951070 shutdown_connections 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.345+0000 7ff4823e2740 1 -- 172.21.15.77:0/2006951070 wait complete. 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.345+0000 7ff4823e2740 1 -- start start 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.345+0000 7ff4823e2740 1 Processor -- start 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff4823e2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869bec60 0x55d5869ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff4823e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bf650 0x55d5869ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff4823e2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 0x55d5869cf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff4823e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff4781a7f80 con 0x55d5869bf650 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff4823e2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff4781a78b0 con 0x55d5869c47c0 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff4823e2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff4781a4a30 con 0x55d5869bec60 2024-03-16T10:20:59.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff47dcb8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 0x55d5869cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff47d4b7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869bec60 0x55d5869ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff4823e2740 10 client.0 _ll_register_callbacks cb 0x7ff46801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff47ccb6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bf650 0x55d5869ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff47ccb6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bf650 0x55d5869ce600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54682/0 (socket says 172.21.15.77:54682) 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff47ccb6640 1 -- 172.21.15.77:0/3408046544 learned_addr learned my addr 172.21.15.77:0/3408046544 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff47dcb8640 1 -- 172.21.15.77:0/3408046544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869bec60 msgr2=0x55d5869ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.346+0000 7ff47dcb8640 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869bec60 0x55d5869ce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff47dcb8640 1 -- 172.21.15.77:0/3408046544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bf650 msgr2=0x55d5869ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff47dcb8640 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bf650 0x55d5869ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff47dcb8640 1 -- 172.21.15.77:0/3408046544 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5869cfc00 con 0x55d5869c47c0 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff47d4b7640 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869bec60 0x55d5869ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff47ccb6640 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bf650 0x55d5869ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff47dcb8640 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 0x55d5869cf610 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff47002c630 tx=0x7ff4700095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84e3a626e7d60e7b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff460ff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff460ff9640 1 -- 172.21.15.77:0/3408046544 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff470052070 con 0x55d5869c47c0 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff4823e2740 1 -- 172.21.15.77:0/3408046544 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff46800e420 con 0x55d5869c47c0 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff460ff9640 1 -- 172.21.15.77:0/3408046544 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ff470004450 con 0x55d5869c47c0 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff460ff9640 1 -- 172.21.15.77:0/3408046544 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff47005c050 con 0x55d5869c47c0 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff4823e2740 10 client.26435 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.347+0000 7ff4823e2740 1 -- 172.21.15.77:0/3408046544 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d586a3b2d0 con 0x55d5869c47c0 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.348+0000 7ff460ff9640 1 -- 172.21.15.77:0/3408046544 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff46800e420 con 0x55d5869c47c0 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.348+0000 7ff460ff9640 1 -- 172.21.15.77:0/3408046544 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7ff470034440 con 0x55d5869c47c0 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.348+0000 7ff4823e2740 10 client.26435 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.348+0000 7ff4823e2740 10 client.26435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.348+0000 7ff4823e2740 20 client.26435 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.348+0000 7ff4823e2740 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d586a3bcf0 0x55d586a3e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.348+0000 7ff4823e2740 4 client.26435 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:20:59.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.348+0000 7ff4823e2740 1 -- 172.21.15.77:0/3408046544 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d586926d20 con 0x55d586a3bcf0 2024-03-16T10:20:59.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.348+0000 7ff47d4b7640 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d586a3bcf0 0x55d586a3e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.349+0000 7ff460ff9640 10 client.26435 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:59.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.349+0000 7ff47d4b7640 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d586a3bcf0 0x55d586a3e110 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d96fca1b0ee07f6 server_cookie=5320335d0198cbbf in_seq=0 out_seq=0 2024-03-16T10:20:59.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.349+0000 7ff460ff9640 1 -- 172.21.15.77:0/3408046544 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d586926d20 con 0x55d586a3bcf0 2024-03-16T10:20:59.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.349+0000 7ff460ff9640 10 client.26435 handle_command_reply: tid=0 2024-03-16T10:20:59.354 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.349+0000 7ff4823e2740 1 client.26435 shutdown 2024-03-16T10:20:59.355 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.350+0000 7ff4823e2740 1 -- 172.21.15.77:0/3408046544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 msgr2=0x55d5869cf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.351+0000 7ff4823e2740 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 0x55d5869cf610 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff47002c630 tx=0x7ff4700095c0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.351+0000 7ff4823e2740 1 -- 172.21.15.77:0/3408046544 shutdown_connections 2024-03-16T10:20:59.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.351+0000 7ff4823e2740 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d5869c47c0 0x55d5869cf610 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.351+0000 7ff4823e2740 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d586a3bcf0 0x55d586a3e110 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.351+0000 7ff460ff9640 0 client.26435 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:59.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.351+0000 7ff4823e2740 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5869bf650 0x55d5869ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.352+0000 7ff4823e2740 1 --2- 172.21.15.77:0/3408046544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5869bec60 0x55d5869ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.352+0000 7ff4823e2740 1 -- 172.21.15.77:0/3408046544 >> 172.21.15.77:0/3408046544 conn(0x55d5869b5dc0 msgr2=0x55d5869b98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.352+0000 7ff4823e2740 1 -- 172.21.15.77:0/3408046544 shutdown_connections 2024-03-16T10:20:59.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.352+0000 7ff4823e2740 1 -- 172.21.15.77:0/3408046544 wait complete. 2024-03-16T10:20:59.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.352+0000 7ff4823e2740 20 client.26435 trim_cache size 0 max 16384 2024-03-16T10:20:59.356 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:20:59.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.353+0000 7ff47f71b640 1 -- 172.21.15.77:0/1466533941 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff4640a2800 msgr2=0x7ff4640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.353+0000 7ff47f71b640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff4640a2800 0x7ff4640a4cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff470005e30 tx=0x7ff47002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.353+0000 7ff47f71b640 1 -- 172.21.15.77:0/1466533941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478100700 msgr2=0x7ff47806a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.353+0000 7ff47f71b640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478100700 0x7ff47806a940 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff47400e9c0 tx=0x7ff47400ee80 comp rx=0 tx=0).stop 2024-03-16T10:20:59.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.353+0000 7ff47f71b640 1 -- 172.21.15.77:0/1466533941 shutdown_connections 2024-03-16T10:20:59.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.353+0000 7ff47f71b640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4781022b0 0x7ff47819ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.353+0000 7ff47f71b640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff4640a2800 0x7ff4640a4cc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.353+0000 7ff47f71b640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4781018f0 0x7ff47819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.353+0000 7ff47f71b640 1 --2- 172.21.15.77:0/1466533941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff478100700 0x7ff47806a940 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.353+0000 7ff47f71b640 1 -- 172.21.15.77:0/1466533941 >> 172.21.15.77:0/1466533941 conn(0x7ff4780fbc90 msgr2=0x7ff478104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.353+0000 7ff47f71b640 1 -- 172.21.15.77:0/1466533941 shutdown_connections 2024-03-16T10:20:59.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:20:59.353+0000 7ff47f71b640 1 -- 172.21.15.77:0/1466533941 wait complete. 2024-03-16T10:20:59.736 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.733+0000 7f0147e61640 1 -- 172.21.15.77:0/3337519234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0140102a90 msgr2=0x7f0140102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.736 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.733+0000 7f0147e61640 1 --2- 172.21.15.77:0/3337519234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0140102a90 0x7f0140102f10 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0130009930 tx=0x7f013002bc30 comp rx=0 tx=0).stop 2024-03-16T10:20:59.736 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.733+0000 7f0147e61640 1 -- 172.21.15.77:0/3337519234 shutdown_connections 2024-03-16T10:20:59.736 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.733+0000 7f0147e61640 1 --2- 172.21.15.77:0/3337519234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0140103450 0x7f0140109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.736 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.733+0000 7f0147e61640 1 --2- 172.21.15.77:0/3337519234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0140102a90 0x7f0140102f10 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.736 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.733+0000 7f0147e61640 1 --2- 172.21.15.77:0/3337519234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01400ffac0 0x7f01400ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.733+0000 7f0147e61640 1 -- 172.21.15.77:0/3337519234 >> 172.21.15.77:0/3337519234 conn(0x7f01400fb430 msgr2=0x7f01400fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.733+0000 7f0147e61640 1 -- 172.21.15.77:0/3337519234 shutdown_connections 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.733+0000 7f0147e61640 1 -- 172.21.15.77:0/3337519234 wait complete. 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f0147e61640 1 Processor -- start 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f0147e61640 1 -- start start 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f0147e61640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01400ffac0 0x7f0140197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f0147e61640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0140102a90 0x7f0140198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f0147e61640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0140103450 0x7f014019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f0147e61640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f014010d4a0 con 0x7f01400ffac0 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f0147e61640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f014010d320 con 0x7f0140103450 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f0147e61640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f014010d620 con 0x7f0140102a90 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f0145bfd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01400ffac0 0x7f0140197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f0145bfd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01400ffac0 0x7f0140197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54710/0 (socket says 172.21.15.77:54710) 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f0145bfd640 1 -- 172.21.15.77:0/1824186950 learned_addr learned my addr 172.21.15.77:0/1824186950 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f01463fe640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0140103450 0x7f014019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.734+0000 7f01453fc640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0140102a90 0x7f0140198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.735+0000 7f0145bfd640 1 -- 172.21.15.77:0/1824186950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0140102a90 msgr2=0x7f0140198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.735+0000 7f0145bfd640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0140102a90 0x7f0140198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.735+0000 7f0145bfd640 1 -- 172.21.15.77:0/1824186950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0140103450 msgr2=0x7f014019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.735+0000 7f0145bfd640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0140103450 0x7f014019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.735+0000 7f0145bfd640 1 -- 172.21.15.77:0/1824186950 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014019f6e0 con 0x7f01400ffac0 2024-03-16T10:20:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.735+0000 7f01453fc640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0140102a90 0x7f0140198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.735+0000 7f0145bfd640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01400ffac0 0x7f0140197b10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f013c00e9c0 tx=0x7f013c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.735+0000 7f0136ffd640 1 -- 172.21.15.77:0/1824186950 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c00cd80 con 0x7f01400ffac0 2024-03-16T10:20:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.735+0000 7f0147e61640 1 -- 172.21.15.77:0/1824186950 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01401008e0 con 0x7f01400ffac0 2024-03-16T10:20:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.735+0000 7f0147e61640 1 -- 172.21.15.77:0/1824186950 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0140100e00 con 0x7f01400ffac0 2024-03-16T10:20:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.735+0000 7f0136ffd640 1 -- 172.21.15.77:0/1824186950 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f013c0035f0 con 0x7f01400ffac0 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.736+0000 7f0136ffd640 1 -- 172.21.15.77:0/1824186950 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c010690 con 0x7f01400ffac0 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.738+0000 7f014aa55740 1 Processor -- start 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.738+0000 7f014aa55740 1 -- start start 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f014aa55740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c1c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f014aa55740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c21830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f014aa55740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c24330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f014aa55740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d6d2b81a40 con 0x55d6d2c1c340 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f014aa55740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d6d2b2a920 con 0x55d6d2c1cd30 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f014aa55740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d6d2b7dae0 con 0x55d6d2c21ea0 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f0136ffd640 1 -- 172.21.15.77:0/1824186950 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f013c0037a0 con 0x7f01400ffac0 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f01463fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c1c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f01463fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c1c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54726/0 (socket says 172.21.15.77:54726) 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f01463fe640 1 -- 172.21.15.77:0/3706793064 learned_addr learned my addr 172.21.15.77:0/3706793064 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f01453fc640 1 --2- 172.21.15.77:0/3706793064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c24330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f0145bfd640 1 --2- 172.21.15.77:0/3706793064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c21830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f01463fe640 1 -- 172.21.15.77:0/3706793064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c21ea0 msgr2=0x55d6d2c24330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f01463fe640 1 --2- 172.21.15.77:0/3706793064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c24330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f01463fe640 1 -- 172.21.15.77:0/3706793064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 msgr2=0x55d6d2c21830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f01463fe640 1 --2- 172.21.15.77:0/3706793064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c21830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f01463fe640 1 -- 172.21.15.77:0/3706793064 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6d2b42460 con 0x55d6d2c1c340 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.739+0000 7f01453fc640 1 --2- 172.21.15.77:0/3706793064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c24330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.740+0000 7f01463fe640 1 --2- 172.21.15.77:0/3706793064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c1c760 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0138009fd0 tx=0x7f01380118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a91d924b0802fd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.740+0000 7f012bfff640 1 -- 172.21.15.77:0/3706793064 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013801e070 con 0x55d6d2c1c340 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.740+0000 7f012bfff640 1 -- 172.21.15.77:0/3706793064 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0138003c30 con 0x55d6d2c1c340 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.740+0000 7f012bfff640 1 -- 172.21.15.77:0/3706793064 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013800f9b0 con 0x55d6d2c1c340 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.740+0000 7f0136ffd640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f012c0a28a0 0x7f012c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.740+0000 7f0145bfd640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f012c0a28a0 0x7f012c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.740+0000 7f0136ffd640 1 -- 172.21.15.77:0/1824186950 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f013c0ced60 con 0x7f01400ffac0 2024-03-16T10:20:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.741+0000 7f0145bfd640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f012c0a28a0 0x7f012c0a4d60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55d6d2c1d130 tx=0x7f013c097a30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.741+0000 7f014aa55740 1 -- 172.21.15.77:0/3706793064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 msgr2=0x55d6d2c1c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.741+0000 7f014aa55740 1 --2- 172.21.15.77:0/3706793064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c1c760 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0138009fd0 tx=0x7f01380118d0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.741+0000 7f014aa55740 1 -- 172.21.15.77:0/3706793064 shutdown_connections 2024-03-16T10:20:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.741+0000 7f014aa55740 1 --2- 172.21.15.77:0/3706793064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c24330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.741+0000 7f014aa55740 1 --2- 172.21.15.77:0/3706793064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c21830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.741+0000 7f014aa55740 1 --2- 172.21.15.77:0/3706793064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c1c760 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.741+0000 7f014aa55740 1 -- 172.21.15.77:0/3706793064 >> 172.21.15.77:0/3706793064 conn(0x55d6d2c178b0 msgr2=0x55d6d2c17cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.741+0000 7f014aa55740 1 -- 172.21.15.77:0/3706793064 shutdown_connections 2024-03-16T10:20:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.741+0000 7f014aa55740 1 -- 172.21.15.77:0/3706793064 wait complete. 2024-03-16T10:20:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.741+0000 7f014aa55740 1 -- start start 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.741+0000 7f014aa55740 1 Processor -- start 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f014aa55740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1c340 0x55d6d2c2ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f014aa55740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c2f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f014aa55740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c36410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f014aa55740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d6d2c2d660 con 0x55d6d2c21ea0 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f014aa55740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d6d2b3dd70 con 0x55d6d2c1cd30 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f014aa55740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d6d2c2d7e0 con 0x55d6d2c1c340 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f014aa55740 10 client.0 _ll_register_callbacks cb 0x55d6d2895a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f01463fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c36410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f01463fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1c340 0x55d6d2c2ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f01453fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c2f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f01463fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1c340 0x55d6d2c2ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52820/0 (socket says 172.21.15.77:52820) 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f01463fe640 1 -- 172.21.15.77:0/1837300929 learned_addr learned my addr 172.21.15.77:0/1837300929 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f01463fe640 1 -- 172.21.15.77:0/1837300929 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 msgr2=0x55d6d2c2f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f01463fe640 1 --2- 172.21.15.77:0/1837300929 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c2f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.748 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f01463fe640 1 -- 172.21.15.77:0/1837300929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c21ea0 msgr2=0x55d6d2c36410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f01463fe640 1 --2- 172.21.15.77:0/1837300929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c36410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.742+0000 7f01463fe640 1 -- 172.21.15.77:0/1837300929 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6d2896210 con 0x55d6d2c1c340 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f01463fe640 1 --2- 172.21.15.77:0/1837300929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1c340 0x55d6d2c2ee90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f013801b000 tx=0x7f0138005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8bb5ff0a84652e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f01297fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f01297fa640 1 -- 172.21.15.77:0/1837300929 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013802d060 con 0x55d6d2c1c340 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f014aa55740 1 -- 172.21.15.77:0/1837300929 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0138003c30 con 0x55d6d2c1c340 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f01297fa640 1 -- 172.21.15.77:0/1837300929 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f013800f710 con 0x55d6d2c1c340 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f01297fa640 1 -- 172.21.15.77:0/1837300929 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f013801e070 con 0x55d6d2c1c340 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f014aa55740 10 client.26440 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f014aa55740 1 -- 172.21.15.77:0/1837300929 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d6d2962620 con 0x55d6d2c1c340 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f01297fa640 1 -- 172.21.15.77:0/1837300929 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0138003c30 con 0x55d6d2c1c340 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f01297fa640 1 -- 172.21.15.77:0/1837300929 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f0138019840 con 0x55d6d2c1c340 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f014aa55740 10 client.26440 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f014aa55740 10 client.26440 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f014aa55740 20 client.26440 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f014aa55740 1 --2- 172.21.15.77:0/1837300929 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d6d2c3b030 0x55d6d2c3d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f014aa55740 4 client.26440 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.743+0000 7f014aa55740 1 -- 172.21.15.77:0/1837300929 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6d2b8ef80 con 0x55d6d2c3b030 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.744+0000 7f01453fc640 1 --2- 172.21.15.77:0/1837300929 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d6d2c3b030 0x55d6d2c3d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.744+0000 7f01297fa640 10 client.26440 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.744+0000 7f01453fc640 1 --2- 172.21.15.77:0/1837300929 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d6d2c3b030 0x55d6d2c3d450 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f47b428dbeb3b3f6 server_cookie=b88b64032c7b7e00 in_seq=0 out_seq=0 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f01297fa640 1 -- 172.21.15.77:0/1837300929 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d6d2b8ef80 con 0x55d6d2c3b030 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f01297fa640 10 client.26440 handle_command_reply: tid=0 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f014aa55740 1 client.26440 shutdown 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f014aa55740 1 -- 172.21.15.77:0/1837300929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1c340 msgr2=0x55d6d2c2ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f014aa55740 1 --2- 172.21.15.77:0/1837300929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1c340 0x55d6d2c2ee90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f013801b000 tx=0x7f0138005fd0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f014aa55740 1 -- 172.21.15.77:0/1837300929 shutdown_connections 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f014aa55740 1 --2- 172.21.15.77:0/1837300929 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d6d2c3b030 0x55d6d2c3d450 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f014aa55740 1 --2- 172.21.15.77:0/1837300929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c36410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f014aa55740 1 --2- 172.21.15.77:0/1837300929 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c2f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f01297fa640 0 client.26440 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f014aa55740 1 --2- 172.21.15.77:0/1837300929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1c340 0x55d6d2c2ee90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f014aa55740 1 -- 172.21.15.77:0/1837300929 >> 172.21.15.77:0/1837300929 conn(0x55d6d2c178b0 msgr2=0x55d6d2c16bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f014aa55740 1 -- 172.21.15.77:0/1837300929 shutdown_connections 2024-03-16T10:20:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f014aa55740 1 -- 172.21.15.77:0/1837300929 wait complete. 2024-03-16T10:20:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.745+0000 7f014aa55740 20 client.26440 trim_cache size 0 max 16384 2024-03-16T10:20:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.760+0000 7f014aa55740 1 Processor -- start 2024-03-16T10:20:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.760+0000 7f014aa55740 1 -- start start 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.760+0000 7f014aa55740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c17330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.760+0000 7f014aa55740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c2f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.760+0000 7f014aa55740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c1b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.760+0000 7f014aa55740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f01401a69a0 con 0x55d6d2c1c340 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.760+0000 7f014aa55740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f01401a85d0 con 0x55d6d2c21ea0 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.760+0000 7f014aa55740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f01401a7070 con 0x55d6d2c1cd30 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f01463fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c17330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f01453fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c2f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f01463fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c17330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54738/0 (socket says 172.21.15.77:54738) 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f01463fe640 1 -- 172.21.15.77:0/2393628921 learned_addr learned my addr 172.21.15.77:0/2393628921 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f01463fe640 1 -- 172.21.15.77:0/2393628921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1cd30 msgr2=0x55d6d2c2f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f01463fe640 1 --2- 172.21.15.77:0/2393628921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c2f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f01463fe640 1 -- 172.21.15.77:0/2393628921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c21ea0 msgr2=0x55d6d2c1b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f01463fe640 1 --2- 172.21.15.77:0/2393628921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c1b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f01463fe640 1 -- 172.21.15.77:0/2393628921 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6d2c1bf80 con 0x55d6d2c1c340 2024-03-16T10:20:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f01453fc640 1 --2- 172.21.15.77:0/2393628921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c2f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f01463fe640 1 --2- 172.21.15.77:0/2393628921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c17330 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f0138019cb0 tx=0x7f0138005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9bacc08666ab35f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f012b7fe640 1 -- 172.21.15.77:0/2393628921 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138003e60 con 0x55d6d2c1c340 2024-03-16T10:20:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f012b7fe640 1 -- 172.21.15.77:0/2393628921 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01380333f0 con 0x55d6d2c1c340 2024-03-16T10:20:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.761+0000 7f012b7fe640 1 -- 172.21.15.77:0/2393628921 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138033ae0 con 0x55d6d2c1c340 2024-03-16T10:20:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.762+0000 7f014aa55740 1 -- 172.21.15.77:0/2393628921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 msgr2=0x55d6d2c17330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.762+0000 7f014aa55740 1 --2- 172.21.15.77:0/2393628921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c17330 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f0138019cb0 tx=0x7f0138005f10 comp rx=0 tx=0).stop 2024-03-16T10:20:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.762+0000 7f014aa55740 1 -- 172.21.15.77:0/2393628921 shutdown_connections 2024-03-16T10:20:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.762+0000 7f014aa55740 1 --2- 172.21.15.77:0/2393628921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c1b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.762+0000 7f014aa55740 1 --2- 172.21.15.77:0/2393628921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c2f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.762+0000 7f014aa55740 1 --2- 172.21.15.77:0/2393628921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c17330 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.762+0000 7f014aa55740 1 -- 172.21.15.77:0/2393628921 >> 172.21.15.77:0/2393628921 conn(0x55d6d2c178b0 msgr2=0x55d6d2c1ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.762+0000 7f014aa55740 1 -- 172.21.15.77:0/2393628921 shutdown_connections 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.762+0000 7f014aa55740 1 -- 172.21.15.77:0/2393628921 wait complete. 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.763+0000 7f014aa55740 1 -- start start 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.763+0000 7f014aa55740 1 Processor -- start 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f014aa55740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c33850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f014aa55740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c33d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f014aa55740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c34da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f014aa55740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f01401a7ed0 con 0x55d6d2c1c340 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f014aa55740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f01401a7800 con 0x55d6d2c1cd30 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f014aa55740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f01401a4980 con 0x55d6d2c21ea0 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f014aa55740 10 client.0 _ll_register_callbacks cb 0x7f013801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01453fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c33d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01463fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c33850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01463fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c33850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54740/0 (socket says 172.21.15.77:54740) 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01453fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c33d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42342/0 (socket says 172.21.15.77:42342) 2024-03-16T10:20:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01463fe640 1 -- 172.21.15.77:0/96923347 learned_addr learned my addr 172.21.15.77:0/96923347 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01463fe640 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c34da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01453fc640 1 -- 172.21.15.77:0/96923347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c21ea0 msgr2=0x55d6d2c34da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01453fc640 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c34da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01453fc640 1 -- 172.21.15.77:0/96923347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 msgr2=0x55d6d2c33850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01453fc640 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c33850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01453fc640 1 -- 172.21.15.77:0/96923347 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6d2c35390 con 0x55d6d2c1cd30 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01463fe640 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c33850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01453fc640 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c33d90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f013002c170 tx=0x7f013002cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2914e1618f92c6d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.764+0000 7f01297fa640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f01297fa640 1 -- 172.21.15.77:0/96923347 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013004a070 con 0x55d6d2c1cd30 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f01297fa640 1 -- 172.21.15.77:0/96923347 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f0130046ca0 con 0x55d6d2c1cd30 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f014aa55740 1 -- 172.21.15.77:0/96923347 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01380333f0 con 0x55d6d2c1cd30 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f01297fa640 1 -- 172.21.15.77:0/96923347 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0130056050 con 0x55d6d2c1cd30 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f014aa55740 10 client.26450 fetch_fsmap learned FSMap version 33 2024-03-16T10:20:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f014aa55740 1 -- 172.21.15.77:0/96923347 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d6d2ca1460 con 0x55d6d2c1cd30 2024-03-16T10:20:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f01297fa640 1 -- 172.21.15.77:0/96923347 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01380333f0 con 0x55d6d2c1cd30 2024-03-16T10:20:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f01297fa640 1 -- 172.21.15.77:0/96923347 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f01300046c0 con 0x55d6d2c1cd30 2024-03-16T10:20:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f014aa55740 10 client.26450 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:20:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f014aa55740 10 client.26450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:20:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f014aa55740 20 client.26450 populate_metadata read hostname 'smithi077' 2024-03-16T10:20:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f014aa55740 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d6d2c3b030 0x55d6d2ca3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:20:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f014aa55740 4 client.26450 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:20:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.765+0000 7f014aa55740 1 -- 172.21.15.77:0/96923347 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d6d2b892d0 con 0x55d6d2c3b030 2024-03-16T10:20:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.766+0000 7f01463fe640 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d6d2c3b030 0x55d6d2ca3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:20:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.766+0000 7f01297fa640 10 client.26450 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.766+0000 7f01463fe640 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d6d2c3b030 0x55d6d2ca3ea0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a2f93bd6a5b09ca server_cookie=610b635fe12643e4 in_seq=0 out_seq=0 2024-03-16T10:20:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f01297fa640 1 -- 172.21.15.77:0/96923347 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d6d2b892d0 con 0x55d6d2c3b030 2024-03-16T10:20:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f01297fa640 10 client.26450 handle_command_reply: tid=0 2024-03-16T10:20:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f014aa55740 1 client.26450 shutdown 2024-03-16T10:20:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f014aa55740 1 -- 172.21.15.77:0/96923347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 msgr2=0x55d6d2c33d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f014aa55740 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c33d90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f013002c170 tx=0x7f013002cf90 comp rx=0 tx=0).stop 2024-03-16T10:20:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f014aa55740 1 -- 172.21.15.77:0/96923347 shutdown_connections 2024-03-16T10:20:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f014aa55740 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6d2c21ea0 0x55d6d2c34da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f014aa55740 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6d2c1cd30 0x55d6d2c33d90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f014aa55740 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d6d2c3b030 0x55d6d2ca3ea0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f014aa55740 1 --2- 172.21.15.77:0/96923347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6d2c1c340 0x55d6d2c33850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f01297fa640 0 client.26450 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:20:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f014aa55740 1 -- 172.21.15.77:0/96923347 >> 172.21.15.77:0/96923347 conn(0x55d6d2c178b0 msgr2=0x55d6d2c2f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f014aa55740 1 -- 172.21.15.77:0/96923347 shutdown_connections 2024-03-16T10:20:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f014aa55740 1 -- 172.21.15.77:0/96923347 wait complete. 2024-03-16T10:20:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.827+0000 7f014aa55740 20 client.26450 trim_cache size 0 max 16384 2024-03-16T10:20:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:20:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:20:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:20:59.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:20:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.828+0000 7f0147e61640 1 -- 172.21.15.77:0/1824186950 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f012c0a28a0 msgr2=0x7f012c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.828+0000 7f0147e61640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f012c0a28a0 0x7f012c0a4d60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55d6d2c1d130 tx=0x7f013c097a30 comp rx=0 tx=0).stop 2024-03-16T10:20:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.828+0000 7f0147e61640 1 -- 172.21.15.77:0/1824186950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01400ffac0 msgr2=0x7f0140197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:20:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.828+0000 7f0147e61640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01400ffac0 0x7f0140197b10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f013c00e9c0 tx=0x7f013c00ee80 comp rx=0 tx=0).stop 2024-03-16T10:20:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.828+0000 7f0147e61640 1 -- 172.21.15.77:0/1824186950 shutdown_connections 2024-03-16T10:20:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.828+0000 7f0147e61640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0140103450 0x7f014019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.828+0000 7f0147e61640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f012c0a28a0 0x7f012c0a4d60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.828+0000 7f0147e61640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0140102a90 0x7f0140198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.828+0000 7f0147e61640 1 --2- 172.21.15.77:0/1824186950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01400ffac0 0x7f0140197b10 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:20:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.828+0000 7f0147e61640 1 -- 172.21.15.77:0/1824186950 >> 172.21.15.77:0/1824186950 conn(0x7f01400fb430 msgr2=0x7f0140105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:20:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.828+0000 7f0147e61640 1 -- 172.21.15.77:0/1824186950 shutdown_connections 2024-03-16T10:20:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:20:59.828+0000 7f0147e61640 1 -- 172.21.15.77:0/1824186950 wait complete. 2024-03-16T10:21:00.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:20:59 smithi192 ceph-mon[36354]: pgmap v467: 97 pgs: 97 active+clean; 421 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 297 op/s 2024-03-16T10:21:00.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:20:59 smithi077 ceph-mon[32400]: pgmap v467: 97 pgs: 97 active+clean; 421 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 297 op/s 2024-03-16T10:21:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:20:59 smithi193 ceph-mon[36311]: pgmap v467: 97 pgs: 97 active+clean; 421 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 297 op/s 2024-03-16T10:21:00.366 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T10:21:00.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:21:00.367 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T10:21:01.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.074+0000 7fd75ec30640 1 -- 172.21.15.77:0/198043351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd758100720 msgr2=0x7fd758100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.074+0000 7fd75ec30640 1 --2- 172.21.15.77:0/198043351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd758100720 0x7fd758100b40 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009900 tx=0x7fd74c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:21:01.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.074+0000 7fd75ec30640 1 -- 172.21.15.77:0/198043351 shutdown_connections 2024-03-16T10:21:01.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.074+0000 7fd75ec30640 1 --2- 172.21.15.77:0/198043351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7581022d0 0x7fd758108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.074+0000 7fd75ec30640 1 --2- 172.21.15.77:0/198043351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd758101910 0x7fd758101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.074+0000 7fd75ec30640 1 --2- 172.21.15.77:0/198043351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd758100720 0x7fd758100b40 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.074+0000 7fd75ec30640 1 -- 172.21.15.77:0/198043351 >> 172.21.15.77:0/198043351 conn(0x7fd7580fbc90 msgr2=0x7fd7580fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:01.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.074+0000 7fd75ec30640 1 -- 172.21.15.77:0/198043351 shutdown_connections 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.074+0000 7fd75ec30640 1 -- 172.21.15.77:0/198043351 wait complete. 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.074+0000 7fd75ec30640 1 Processor -- start 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd75ec30640 1 -- start start 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd75ec30640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd758100720 0x7fd75806a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd75ec30640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd758101910 0x7fd75819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd75ec30640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7581022d0 0x7fd75819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd75ec30640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd75810c3c0 con 0x7fd7581022d0 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd75ec30640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd75810c240 con 0x7fd758100720 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd75ec30640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd75810c540 con 0x7fd758101910 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd75d1cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7581022d0 0x7fd75819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd757fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd758101910 0x7fd75819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd75c9cc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd758100720 0x7fd75806a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd757fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd758101910 0x7fd75819c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52848/0 (socket says 172.21.15.77:52848) 2024-03-16T10:21:01.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd757fff640 1 -- 172.21.15.77:0/1041284401 learned_addr learned my addr 172.21.15.77:0/1041284401 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:01.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd75c9cc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd758100720 0x7fd75806a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42360/0 (socket says 172.21.15.77:42360) 2024-03-16T10:21:01.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd757fff640 1 -- 172.21.15.77:0/1041284401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd758100720 msgr2=0x7fd75806a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd757fff640 1 --2- 172.21.15.77:0/1041284401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd758100720 0x7fd75806a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd757fff640 1 -- 172.21.15.77:0/1041284401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7581022d0 msgr2=0x7fd75819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd757fff640 1 --2- 172.21.15.77:0/1041284401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7581022d0 0x7fd75819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd757fff640 1 -- 172.21.15.77:0/1041284401 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd75819f860 con 0x7fd758101910 2024-03-16T10:21:01.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.075+0000 7fd757fff640 1 --2- 172.21.15.77:0/1041284401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd758101910 0x7fd75819c930 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd74800ea60 tx=0x7fd74800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:01.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.076+0000 7fd755ffb640 1 -- 172.21.15.77:0/1041284401 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74800cde0 con 0x7fd758101910 2024-03-16T10:21:01.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.076+0000 7fd75ec30640 1 -- 172.21.15.77:0/1041284401 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd75819faf0 con 0x7fd758101910 2024-03-16T10:21:01.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.076+0000 7fd755ffb640 1 -- 172.21.15.77:0/1041284401 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7480035f0 con 0x7fd758101910 2024-03-16T10:21:01.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.076+0000 7fd755ffb640 1 -- 172.21.15.77:0/1041284401 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748010670 con 0x7fd758101910 2024-03-16T10:21:01.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.076+0000 7fd75ec30640 1 -- 172.21.15.77:0/1041284401 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7581a0090 con 0x7fd758101910 2024-03-16T10:21:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.078+0000 7fd755ffb640 1 -- 172.21.15.77:0/1041284401 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fd7480037a0 con 0x7fd758101910 2024-03-16T10:21:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.078+0000 7fd755ffb640 1 --2- 172.21.15.77:0/1041284401 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd7400a2800 0x7fd7400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.082 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.078+0000 7fd755ffb640 1 -- 172.21.15.77:0/1041284401 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd7480cecb0 con 0x7fd758101910 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.078+0000 7fd75c9cc640 1 --2- 172.21.15.77:0/1041284401 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd7400a2800 0x7fd7400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.079+0000 7fd75c9cc640 1 --2- 172.21.15.77:0/1041284401 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd7400a2800 0x7fd7400a4cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd74c02c240 tx=0x7fd74c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.079+0000 7fd761817740 1 Processor -- start 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.079+0000 7fd761817740 1 -- start start 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.079+0000 7fd761817740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadbda0 0x55e71fadc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.079+0000 7fd761817740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadc790 0x55e71fae1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.079+0000 7fd761817740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 0x55e71fae3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd761817740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e71fa3c0d0 con 0x55e71fae1900 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd761817740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55e71f9e4c70 con 0x55e71fadc790 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd761817740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e71fa38170 con 0x55e71fadbda0 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd75d1cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 0x55e71fae3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd75c9cc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadbda0 0x55e71fadc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd75d1cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 0x55e71fae3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54760/0 (socket says 172.21.15.77:54760) 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd757fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadc790 0x55e71fae1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd75d1cd640 1 -- 172.21.15.77:0/4051630098 learned_addr learned my addr 172.21.15.77:0/4051630098 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd75d1cd640 1 -- 172.21.15.77:0/4051630098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadbda0 msgr2=0x55e71fadc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd75d1cd640 1 --2- 172.21.15.77:0/4051630098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadbda0 0x55e71fadc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd75d1cd640 1 -- 172.21.15.77:0/4051630098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadc790 msgr2=0x55e71fae1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd75d1cd640 1 --2- 172.21.15.77:0/4051630098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadc790 0x55e71fae1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd75d1cd640 1 -- 172.21.15.77:0/4051630098 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e71f8187e0 con 0x55e71fae1900 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd757fff640 1 --2- 172.21.15.77:0/4051630098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadc790 0x55e71fae1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd75d1cd640 1 --2- 172.21.15.77:0/4051630098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 0x55e71fae3d90 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd750009fd0 tx=0x7fd7500118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf5d06d9eb74a9a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd73effd640 1 -- 172.21.15.77:0/4051630098 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd750004d20 con 0x55e71fae1900 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd73effd640 1 -- 172.21.15.77:0/4051630098 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd750004050 con 0x55e71fae1900 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.080+0000 7fd73effd640 1 -- 172.21.15.77:0/4051630098 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd750005280 con 0x55e71fae1900 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.081+0000 7fd761817740 1 -- 172.21.15.77:0/4051630098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 msgr2=0x55e71fae3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.081+0000 7fd761817740 1 --2- 172.21.15.77:0/4051630098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 0x55e71fae3d90 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd750009fd0 tx=0x7fd7500118d0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.081+0000 7fd761817740 1 -- 172.21.15.77:0/4051630098 shutdown_connections 2024-03-16T10:21:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.081+0000 7fd761817740 1 --2- 172.21.15.77:0/4051630098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 0x55e71fae3d90 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.081+0000 7fd761817740 1 --2- 172.21.15.77:0/4051630098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadc790 0x55e71fae1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.081+0000 7fd761817740 1 --2- 172.21.15.77:0/4051630098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadbda0 0x55e71fadc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.081+0000 7fd761817740 1 -- 172.21.15.77:0/4051630098 >> 172.21.15.77:0/4051630098 conn(0x55e71fa9fc50 msgr2=0x55e71fad3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.081+0000 7fd761817740 1 -- 172.21.15.77:0/4051630098 shutdown_connections 2024-03-16T10:21:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.081+0000 7fd761817740 1 -- 172.21.15.77:0/4051630098 wait complete. 2024-03-16T10:21:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd761817740 1 -- start start 2024-03-16T10:21:01.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd761817740 1 Processor -- start 2024-03-16T10:21:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd761817740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadbda0 0x55e71fae62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd761817740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadc790 0x55e71fae67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd761817740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 0x55e71faed830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd761817740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e71fae48b0 con 0x55e71fae1900 2024-03-16T10:21:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd761817740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55e71f9f80d0 con 0x55e71fadc790 2024-03-16T10:21:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd761817740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e71fae4a30 con 0x55e71fadbda0 2024-03-16T10:21:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd75d1cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 0x55e71faed830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd75c9cc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadbda0 0x55e71fae62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd761817740 10 client.0 _ll_register_callbacks cb 0x55e71f9c1760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:01.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd75d1cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 0x55e71faed830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54774/0 (socket says 172.21.15.77:54774) 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.082+0000 7fd757fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadc790 0x55e71fae67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd75d1cd640 1 -- 172.21.15.77:0/2749432379 learned_addr learned my addr 172.21.15.77:0/2749432379 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd75d1cd640 1 -- 172.21.15.77:0/2749432379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadbda0 msgr2=0x55e71fae62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd75d1cd640 1 --2- 172.21.15.77:0/2749432379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadbda0 0x55e71fae62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd75d1cd640 1 -- 172.21.15.77:0/2749432379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadc790 msgr2=0x55e71fae67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd75d1cd640 1 --2- 172.21.15.77:0/2749432379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadc790 0x55e71fae67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd75d1cd640 1 -- 172.21.15.77:0/2749432379 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e71f878f70 con 0x55e71fae1900 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd75d1cd640 1 --2- 172.21.15.77:0/2749432379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 0x55e71faed830 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd75001b040 tx=0x7fd750003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff44584ac2f3bea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd737fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd737fff640 1 -- 172.21.15.77:0/2749432379 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd750029020 con 0x55e71fae1900 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd737fff640 1 -- 172.21.15.77:0/2749432379 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd750004610 con 0x55e71fae1900 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd761817740 1 -- 172.21.15.77:0/2749432379 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd750004050 con 0x55e71fae1900 2024-03-16T10:21:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd737fff640 1 -- 172.21.15.77:0/2749432379 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd750034050 con 0x55e71fae1900 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd761817740 10 client.16701 fetch_fsmap learned FSMap version 33 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd761817740 1 -- 172.21.15.77:0/2749432379 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e71f710170 con 0x55e71fae1900 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.083+0000 7fd737fff640 1 -- 172.21.15.77:0/2749432379 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd750004050 con 0x55e71fae1900 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.084+0000 7fd737fff640 1 -- 172.21.15.77:0/2749432379 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7fd7500255b0 con 0x55e71fae1900 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.084+0000 7fd761817740 10 client.16701 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.084+0000 7fd761817740 10 client.16701 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.084+0000 7fd761817740 20 client.16701 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.084+0000 7fd761817740 1 --2- 172.21.15.77:0/2749432379 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e71faf1cc0 0x55e71faf4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.084+0000 7fd761817740 4 client.16701 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.084+0000 7fd761817740 1 -- 172.21.15.77:0/2749432379 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e71fa49610 con 0x55e71faf1cc0 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.084+0000 7fd75c9cc640 1 --2- 172.21.15.77:0/2749432379 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e71faf1cc0 0x55e71faf4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.084+0000 7fd737fff640 10 client.16701 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.084+0000 7fd75c9cc640 1 --2- 172.21.15.77:0/2749432379 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e71faf1cc0 0x55e71faf4120 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=306a1b1c4a8c20b3 server_cookie=fdcee1e83b6b1d30 in_seq=0 out_seq=0 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd737fff640 1 -- 172.21.15.77:0/2749432379 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e71fa49610 con 0x55e71faf1cc0 2024-03-16T10:21:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd737fff640 10 client.16701 handle_command_reply: tid=0 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd761817740 1 client.16701 shutdown 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd761817740 1 -- 172.21.15.77:0/2749432379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 msgr2=0x55e71faed830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd761817740 1 --2- 172.21.15.77:0/2749432379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 0x55e71faed830 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd75001b040 tx=0x7fd750003ce0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd761817740 1 -- 172.21.15.77:0/2749432379 shutdown_connections 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd761817740 1 --2- 172.21.15.77:0/2749432379 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e71faf1cc0 0x55e71faf4120 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd737fff640 0 client.16701 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd761817740 1 --2- 172.21.15.77:0/2749432379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fae1900 0x55e71faed830 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd761817740 1 --2- 172.21.15.77:0/2749432379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadc790 0x55e71fae67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd761817740 1 --2- 172.21.15.77:0/2749432379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadbda0 0x55e71fae62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd761817740 1 -- 172.21.15.77:0/2749432379 >> 172.21.15.77:0/2749432379 conn(0x55e71fa9fc50 msgr2=0x55e71fad24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd761817740 1 -- 172.21.15.77:0/2749432379 shutdown_connections 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.085+0000 7fd761817740 1 -- 172.21.15.77:0/2749432379 wait complete. 2024-03-16T10:21:01.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.086+0000 7fd761817740 20 client.16701 trim_cache size 0 max 16384 2024-03-16T10:21:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd761817740 1 Processor -- start 2024-03-16T10:21:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd761817740 1 -- start start 2024-03-16T10:21:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd761817740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadbda0 0x55e71fad1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd761817740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadc790 0x55e71faed7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd761817740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fae1900 0x55e71fad2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd761817740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd7581a6a80 con 0x55e71fadc790 2024-03-16T10:21:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd761817740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd7581a86b0 con 0x55e71fadbda0 2024-03-16T10:21:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd761817740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd7581a7150 con 0x55e71fae1900 2024-03-16T10:21:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd757fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadc790 0x55e71faed7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd757fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadc790 0x55e71faed7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54790/0 (socket says 172.21.15.77:54790) 2024-03-16T10:21:01.115 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd75d1cd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fae1900 0x55e71fad2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd757fff640 1 -- 172.21.15.77:0/592905556 learned_addr learned my addr 172.21.15.77:0/592905556 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd75c9cc640 1 --2- 172.21.15.77:0/592905556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadbda0 0x55e71fad1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd75d1cd640 1 -- 172.21.15.77:0/592905556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadbda0 msgr2=0x55e71fad1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd75d1cd640 1 --2- 172.21.15.77:0/592905556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadbda0 0x55e71fad1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd75d1cd640 1 -- 172.21.15.77:0/592905556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadc790 msgr2=0x55e71faed7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd75d1cd640 1 --2- 172.21.15.77:0/592905556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadc790 0x55e71faed7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.113+0000 7fd75d1cd640 1 -- 172.21.15.77:0/592905556 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e71fade370 con 0x55e71fae1900 2024-03-16T10:21:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.114+0000 7fd75d1cd640 1 --2- 172.21.15.77:0/592905556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fae1900 0x55e71fad2190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd750019450 tx=0x7fd750005d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78612ab604f33e55 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.114+0000 7fd73e7fc640 1 -- 172.21.15.77:0/592905556 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd750029040 con 0x55e71fae1900 2024-03-16T10:21:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.114+0000 7fd73e7fc640 1 -- 172.21.15.77:0/592905556 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd750003d50 con 0x55e71fae1900 2024-03-16T10:21:01.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.114+0000 7fd73e7fc640 1 -- 172.21.15.77:0/592905556 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd750019700 con 0x55e71fae1900 2024-03-16T10:21:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 -- 172.21.15.77:0/592905556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fae1900 msgr2=0x55e71fad2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 --2- 172.21.15.77:0/592905556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fae1900 0x55e71fad2190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd750019450 tx=0x7fd750005d10 comp rx=0 tx=0).stop 2024-03-16T10:21:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 -- 172.21.15.77:0/592905556 shutdown_connections 2024-03-16T10:21:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 --2- 172.21.15.77:0/592905556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fae1900 0x55e71fad2190 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 --2- 172.21.15.77:0/592905556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadc790 0x55e71faed7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 --2- 172.21.15.77:0/592905556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fadbda0 0x55e71fad1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 -- 172.21.15.77:0/592905556 >> 172.21.15.77:0/592905556 conn(0x55e71fa9fc50 msgr2=0x55e71fae62b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:01.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 -- 172.21.15.77:0/592905556 shutdown_connections 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 -- 172.21.15.77:0/592905556 wait complete. 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 -- start start 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 Processor -- start 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadbda0 0x55e71faeaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadc790 0x55e71faeb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fae1900 0x55e71faec470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd7581a7fb0 con 0x55e71fadbda0 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd7581a78e0 con 0x55e71fae1900 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.115+0000 7fd761817740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd7581a4a60 con 0x55e71fadc790 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd761817740 10 client.0 _ll_register_callbacks cb 0x7fd748009510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd75c9cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadbda0 0x55e71faeaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd75c9cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadbda0 0x55e71faeaf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54804/0 (socket says 172.21.15.77:54804) 2024-03-16T10:21:01.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd75c9cc640 1 -- 172.21.15.77:0/2256815744 learned_addr learned my addr 172.21.15.77:0/2256815744 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd757fff640 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadc790 0x55e71faeb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd75d1cd640 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fae1900 0x55e71faec470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd75c9cc640 1 -- 172.21.15.77:0/2256815744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadc790 msgr2=0x55e71faeb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd75c9cc640 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadc790 0x55e71faeb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd75c9cc640 1 -- 172.21.15.77:0/2256815744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fae1900 msgr2=0x55e71faec470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd75c9cc640 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fae1900 0x55e71faec470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd75c9cc640 1 -- 172.21.15.77:0/2256815744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e71faeca60 con 0x55e71fadbda0 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd757fff640 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadc790 0x55e71faeb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd75c9cc640 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadbda0 0x55e71faeaf20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd74c02c8e0 tx=0x7fd74c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da348bd3f6d7656 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd737fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd737fff640 1 -- 172.21.15.77:0/2256815744 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74c004250 con 0x55e71fadbda0 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd737fff640 1 -- 172.21.15.77:0/2256815744 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd74c0043f0 con 0x55e71fadbda0 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd761817740 1 -- 172.21.15.77:0/2256815744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd750003d50 con 0x55e71fadbda0 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd737fff640 1 -- 172.21.15.77:0/2256815744 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd74c055240 con 0x55e71fadbda0 2024-03-16T10:21:01.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd761817740 10 client.16707 fetch_fsmap learned FSMap version 33 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.116+0000 7fd761817740 1 -- 172.21.15.77:0/2256815744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e71fb58090 con 0x55e71fadbda0 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.117+0000 7fd737fff640 1 -- 172.21.15.77:0/2256815744 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd750003d50 con 0x55e71fadbda0 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.117+0000 7fd737fff640 1 -- 172.21.15.77:0/2256815744 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7fd74c04b040 con 0x55e71fadbda0 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.117+0000 7fd761817740 10 client.16707 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.117+0000 7fd761817740 10 client.16707 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.117+0000 7fd761817740 20 client.16707 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.117+0000 7fd761817740 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e71faf1cc0 0x55e71fb5aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.117+0000 7fd761817740 4 client.16707 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.117+0000 7fd761817740 1 -- 172.21.15.77:0/2256815744 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e71fa43960 con 0x55e71faf1cc0 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.117+0000 7fd757fff640 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e71faf1cc0 0x55e71fb5aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.117+0000 7fd737fff640 10 client.16707 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.118+0000 7fd757fff640 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e71faf1cc0 0x55e71fb5aab0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e63bb0eb18dd4b2 server_cookie=e512c8ae5422c967 in_seq=0 out_seq=0 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.119+0000 7fd737fff640 1 -- 172.21.15.77:0/2256815744 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e71fa43960 con 0x55e71faf1cc0 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.119+0000 7fd737fff640 10 client.16707 handle_command_reply: tid=0 2024-03-16T10:21:01.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.119+0000 7fd761817740 1 client.16707 shutdown 2024-03-16T10:21:01.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.119+0000 7fd761817740 1 -- 172.21.15.77:0/2256815744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadbda0 msgr2=0x55e71faeaf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.119+0000 7fd761817740 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadbda0 0x55e71faeaf20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd74c02c8e0 tx=0x7fd74c03f000 comp rx=0 tx=0).stop 2024-03-16T10:21:01.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.119+0000 7fd761817740 1 -- 172.21.15.77:0/2256815744 shutdown_connections 2024-03-16T10:21:01.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.119+0000 7fd761817740 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e71fae1900 0x55e71faec470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.119+0000 7fd761817740 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71fadc790 0x55e71faeb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.119+0000 7fd761817740 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e71faf1cc0 0x55e71fb5aab0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.119+0000 7fd761817740 1 --2- 172.21.15.77:0/2256815744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e71fadbda0 0x55e71faeaf20 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.119+0000 7fd761817740 1 -- 172.21.15.77:0/2256815744 >> 172.21.15.77:0/2256815744 conn(0x55e71fa9fc50 msgr2=0x55e71fad6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:01.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.119+0000 7fd737fff640 0 client.16707 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:01.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd761817740 1 -- 172.21.15.77:0/2256815744 shutdown_connections 2024-03-16T10:21:01.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd761817740 1 -- 172.21.15.77:0/2256815744 wait complete. 2024-03-16T10:21:01.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd761817740 20 client.16707 trim_cache size 0 max 16384 2024-03-16T10:21:01.126 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:21:01.126 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T10:21:01.126 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "127457ea-db1f-4f0c-887d-a265bb630cd6", 2024-03-16T10:21:01.126 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T10:21:01.126 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:21:01.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd75ec30640 1 -- 172.21.15.77:0/1041284401 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd7400a2800 msgr2=0x7fd7400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd75ec30640 1 --2- 172.21.15.77:0/1041284401 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd7400a2800 0x7fd7400a4cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd74c02c240 tx=0x7fd74c02df90 comp rx=0 tx=0).stop 2024-03-16T10:21:01.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd75ec30640 1 -- 172.21.15.77:0/1041284401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd758101910 msgr2=0x7fd75819c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd75ec30640 1 --2- 172.21.15.77:0/1041284401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd758101910 0x7fd75819c930 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd74800ea60 tx=0x7fd74800ef20 comp rx=0 tx=0).stop 2024-03-16T10:21:01.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd75ec30640 1 -- 172.21.15.77:0/1041284401 shutdown_connections 2024-03-16T10:21:01.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd75ec30640 1 --2- 172.21.15.77:0/1041284401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7581022d0 0x7fd75819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd75ec30640 1 --2- 172.21.15.77:0/1041284401 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd7400a2800 0x7fd7400a4cc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd75ec30640 1 --2- 172.21.15.77:0/1041284401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd758101910 0x7fd75819c930 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd75ec30640 1 --2- 172.21.15.77:0/1041284401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd758100720 0x7fd75806a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd75ec30640 1 -- 172.21.15.77:0/1041284401 >> 172.21.15.77:0/1041284401 conn(0x7fd7580fbc90 msgr2=0x7fd758104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:01.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd75ec30640 1 -- 172.21.15.77:0/1041284401 shutdown_connections 2024-03-16T10:21:01.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.120+0000 7fd75ec30640 1 -- 172.21.15.77:0/1041284401 wait complete. 2024-03-16T10:21:01.133 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:21:01.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:21:01.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.265+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/1390805477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ca0100700 msgr2=0x7f6ca0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.265+0000 7f6ca6cbf640 1 --2- 172.21.15.77:0/1390805477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ca0100700 0x7f6ca0100b20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6c94009900 tx=0x7f6c9402bc30 comp rx=0 tx=0).stop 2024-03-16T10:21:01.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.265+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/1390805477 shutdown_connections 2024-03-16T10:21:01.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.265+0000 7f6ca6cbf640 1 --2- 172.21.15.77:0/1390805477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca01022b0 0x7f6ca0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.265+0000 7f6ca6cbf640 1 --2- 172.21.15.77:0/1390805477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ca01018f0 0x7f6ca0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.265+0000 7f6ca6cbf640 1 --2- 172.21.15.77:0/1390805477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ca0100700 0x7f6ca0100b20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.265+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/1390805477 >> 172.21.15.77:0/1390805477 conn(0x7f6ca00fbc90 msgr2=0x7f6ca00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:01.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.265+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/1390805477 shutdown_connections 2024-03-16T10:21:01.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.265+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/1390805477 wait complete. 2024-03-16T10:21:01.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca6cbf640 1 Processor -- start 2024-03-16T10:21:01.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca6cbf640 1 -- start start 2024-03-16T10:21:01.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca6cbf640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ca0100700 0x7f6ca0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca6cbf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ca01018f0 0x7f6ca0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca6cbf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca01022b0 0x7f6ca019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca6cbf640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca010d460 con 0x7f6ca01018f0 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca6cbf640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca010d2e0 con 0x7f6ca01022b0 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca6cbf640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca010d5e0 con 0x7f6ca0100700 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ca01018f0 0x7f6ca0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca4a5b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ca0100700 0x7f6ca0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca4a5b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ca0100700 0x7f6ca0197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52904/0 (socket says 172.21.15.77:52904) 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ca01018f0 0x7f6ca0198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54816/0 (socket says 172.21.15.77:54816) 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/6761689 learned_addr learned my addr 172.21.15.77:0/6761689 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca525c640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca01022b0 0x7f6ca019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/6761689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca01022b0 msgr2=0x7f6ca019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca01022b0 0x7f6ca019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/6761689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ca01018f0 msgr2=0x7f6ca0198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ca01018f0 0x7f6ca0198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.266+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/6761689 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca019f760 con 0x7f6ca0100700 2024-03-16T10:21:01.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.267+0000 7f6c9ffff640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ca01018f0 0x7f6ca0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:01.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.267+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ca0100700 0x7f6ca0197b40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6c94000c00 tx=0x7f6c9402c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:01.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.267+0000 7f6c9dffb640 1 -- 172.21.15.77:0/6761689 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c94041070 con 0x7f6ca0100700 2024-03-16T10:21:01.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.267+0000 7f6c9dffb640 1 -- 172.21.15.77:0/6761689 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c94004450 con 0x7f6ca0100700 2024-03-16T10:21:01.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.267+0000 7f6c9dffb640 1 -- 172.21.15.77:0/6761689 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c94034440 con 0x7f6ca0100700 2024-03-16T10:21:01.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.268+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/6761689 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca019f9f0 con 0x7f6ca0100700 2024-03-16T10:21:01.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.268+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/6761689 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca019fe90 con 0x7f6ca0100700 2024-03-16T10:21:01.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.271+0000 7f6c9dffb640 1 -- 172.21.15.77:0/6761689 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f6c94044070 con 0x7f6ca0100700 2024-03-16T10:21:01.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.271+0000 7f6ca9981740 1 Processor -- start 2024-03-16T10:21:01.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.271+0000 7f6ca9981740 1 -- start start 2024-03-16T10:21:01.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.271+0000 7f6c9dffb640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6c900a2880 0x7f6c900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.271+0000 7f6ca9981740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f54590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.271+0000 7f6ca9981740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f59660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.271+0000 7f6ca9981740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f59cd0 0x55d3c4f5c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.271+0000 7f6ca9981740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d3c4eb9500 con 0x55d3c4f59cd0 2024-03-16T10:21:01.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.271+0000 7f6ca9981740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d3c4e61be0 con 0x55d3c4f54b60 2024-03-16T10:21:01.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.271+0000 7f6ca9981740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d3c4eb55a0 con 0x55d3c4f54170 2024-03-16T10:21:01.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6c9dffb640 1 -- 172.21.15.77:0/6761689 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f6c940f0430 con 0x7f6ca0100700 2024-03-16T10:21:01.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6ca525c640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6c900a2880 0x7f6c900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6ca4a5b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f54590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f59660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6ca4a5b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f54590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52920/0 (socket says 172.21.15.77:52920) 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f59660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42414/0 (socket says 172.21.15.77:42414) 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/1884865721 learned_addr learned my addr 172.21.15.77:0/1884865721 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/1884865721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 msgr2=0x55d3c4f59660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/1884865721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f59660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/1884865721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f59cd0 msgr2=0x55d3c4f5c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/1884865721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f59cd0 0x55d3c4f5c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/1884865721 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3c4e754b0 con 0x55d3c4f54170 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6c9ffff640 1 --2- 172.21.15.77:0/1884865721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f59660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6ca525c640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6c900a2880 0x7f6c900a4d40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6c98005fd0 tx=0x7f6c98011810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/1884865721 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f54590 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6c940c50d0 tx=0x7f6c94005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85cdf39591303e82 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6c8affd640 1 -- 172.21.15.77:0/1884865721 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9404b070 con 0x55d3c4f54170 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6c8affd640 1 -- 172.21.15.77:0/1884865721 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c94004c80 con 0x55d3c4f54170 2024-03-16T10:21:01.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.272+0000 7f6c8affd640 1 -- 172.21.15.77:0/1884865721 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c940b9610 con 0x55d3c4f54170 2024-03-16T10:21:01.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.273+0000 7f6ca9981740 1 -- 172.21.15.77:0/1884865721 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 msgr2=0x55d3c4f54590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.273+0000 7f6ca9981740 1 --2- 172.21.15.77:0/1884865721 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f54590 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6c940c50d0 tx=0x7f6c94005e50 comp rx=0 tx=0).stop 2024-03-16T10:21:01.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.273+0000 7f6ca9981740 1 -- 172.21.15.77:0/1884865721 shutdown_connections 2024-03-16T10:21:01.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.273+0000 7f6ca9981740 1 --2- 172.21.15.77:0/1884865721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f59cd0 0x55d3c4f5c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.273+0000 7f6ca9981740 1 --2- 172.21.15.77:0/1884865721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f59660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.273+0000 7f6ca9981740 1 --2- 172.21.15.77:0/1884865721 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f54590 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.273+0000 7f6ca9981740 1 -- 172.21.15.77:0/1884865721 >> 172.21.15.77:0/1884865721 conn(0x55d3c4f4f6e0 msgr2=0x55d3c4f4fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:01.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.274+0000 7f6ca9981740 1 -- 172.21.15.77:0/1884865721 shutdown_connections 2024-03-16T10:21:01.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.274+0000 7f6ca9981740 1 -- 172.21.15.77:0/1884865721 wait complete. 2024-03-16T10:21:01.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.274+0000 7f6ca9981740 1 -- start start 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.274+0000 7f6ca9981740 1 Processor -- start 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.274+0000 7f6ca9981740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f5e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.274+0000 7f6ca9981740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f5e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca9981740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f59cd0 0x55d3c4f65990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca9981740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d3c4f5ca40 con 0x55d3c4f59cd0 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca9981740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d3c4e75160 con 0x55d3c4f54b60 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca9981740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d3c4f5cbc0 con 0x55d3c4f54170 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca4a5b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f5e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca9981740 10 client.0 _ll_register_callbacks cb 0x55d3c4e59150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca4a5b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f5e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52922/0 (socket says 172.21.15.77:52922) 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/1786929945 learned_addr learned my addr 172.21.15.77:0/1786929945 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca525c640 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f59cd0 0x55d3c4f65990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6c9ffff640 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f5e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/1786929945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 msgr2=0x55d3c4f5e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f5e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/1786929945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f59cd0 msgr2=0x55d3c4f65990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f59cd0 0x55d3c4f65990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/1786929945 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3c4c977d0 con 0x55d3c4f54170 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6c9ffff640 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f5e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca525c640 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f59cd0 0x55d3c4f65990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f5e410 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6c940ee010 tx=0x7f6c9402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=289adf5ffd135eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6c7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6c7ffff640 1 -- 172.21.15.77:0/1786929945 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9404b070 con 0x55d3c4f54170 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6c7ffff640 1 -- 172.21.15.77:0/1786929945 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f6c940c30b0 con 0x55d3c4f54170 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.275+0000 7f6ca9981740 1 -- 172.21.15.77:0/1786929945 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c94004c80 con 0x55d3c4f54170 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.276+0000 7f6c7ffff640 1 -- 172.21.15.77:0/1786929945 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6c940c32b0 con 0x55d3c4f54170 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.276+0000 7f6ca9981740 10 client.26476 fetch_fsmap learned FSMap version 33 2024-03-16T10:21:01.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.276+0000 7f6ca9981740 1 -- 172.21.15.77:0/1786929945 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d3c4bcd3f0 con 0x55d3c4f54170 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.276+0000 7f6c7ffff640 1 -- 172.21.15.77:0/1786929945 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c94004c80 con 0x55d3c4f54170 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.276+0000 7f6c7ffff640 1 -- 172.21.15.77:0/1786929945 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f6c940bc3f0 con 0x55d3c4f54170 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.276+0000 7f6ca9981740 10 client.26476 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.276+0000 7f6ca9981740 10 client.26476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.276+0000 7f6ca9981740 20 client.26476 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.276+0000 7f6ca9981740 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d3c4f69c00 0x55d3c4f6c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.276+0000 7f6ca9981740 4 client.26476 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.276+0000 7f6ca9981740 1 -- 172.21.15.77:0/1786929945 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3c4ec6a40 con 0x55d3c4f69c00 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.276+0000 7f6c9ffff640 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d3c4f69c00 0x55d3c4f6c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.277+0000 7f6c9ffff640 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d3c4f69c00 0x55d3c4f6c030 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9ea428c5dc89221 server_cookie=12c06dfde3e9df5 in_seq=0 out_seq=0 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.277+0000 7f6c7ffff640 10 client.26476 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6c7ffff640 1 -- 172.21.15.77:0/1786929945 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d3c4ec6a40 con 0x55d3c4f69c00 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6c7ffff640 10 client.26476 handle_command_reply: tid=0 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6ca9981740 1 client.26476 shutdown 2024-03-16T10:21:01.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6ca9981740 1 -- 172.21.15.77:0/1786929945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 msgr2=0x55d3c4f5e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6ca9981740 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f5e410 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6c940ee010 tx=0x7f6c9402bc30 comp rx=0 tx=0).stop 2024-03-16T10:21:01.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6ca9981740 1 -- 172.21.15.77:0/1786929945 shutdown_connections 2024-03-16T10:21:01.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6ca9981740 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d3c4f69c00 0x55d3c4f6c030 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6ca9981740 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f59cd0 0x55d3c4f65990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6ca9981740 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f5e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6c7ffff640 0 client.26476 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:01.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6ca9981740 1 --2- 172.21.15.77:0/1786929945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f5e410 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6ca9981740 1 -- 172.21.15.77:0/1786929945 >> 172.21.15.77:0/1786929945 conn(0x55d3c4f4f6e0 msgr2=0x55d3c4f4e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:01.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6ca9981740 1 -- 172.21.15.77:0/1786929945 shutdown_connections 2024-03-16T10:21:01.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6ca9981740 1 -- 172.21.15.77:0/1786929945 wait complete. 2024-03-16T10:21:01.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.278+0000 7f6ca9981740 20 client.26476 trim_cache size 0 max 16384 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6ca9981740 1 Processor -- start 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6ca9981740 1 -- start start 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6ca9981740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f5e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6ca9981740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54b60 0x55d3c4f65950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6ca9981740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f59cd0 0x55d3c4f4fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6ca9981740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca01a69b0 con 0x55d3c4f54b60 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6ca9981740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca01a85e0 con 0x55d3c4f59cd0 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6ca9981740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca01a7080 con 0x55d3c4f54170 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54b60 0x55d3c4f65950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54b60 0x55d3c4f65950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54834/0 (socket says 172.21.15.77:54834) 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6c9ffff640 1 -- 172.21.15.77:0/3312366290 learned_addr learned my addr 172.21.15.77:0/3312366290 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6ca525c640 1 --2- 172.21.15.77:0/3312366290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f59cd0 0x55d3c4f4fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/3312366290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f5e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6c9ffff640 1 -- 172.21.15.77:0/3312366290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 msgr2=0x55d3c4f5e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6c9ffff640 1 --2- 172.21.15.77:0/3312366290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f5e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6c9ffff640 1 -- 172.21.15.77:0/3312366290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f59cd0 msgr2=0x55d3c4f4fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6c9ffff640 1 --2- 172.21.15.77:0/3312366290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f59cd0 0x55d3c4f4fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6c9ffff640 1 -- 172.21.15.77:0/3312366290 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3c4b8d0f0 con 0x55d3c4f54b60 2024-03-16T10:21:01.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.301+0000 7f6c9ffff640 1 --2- 172.21.15.77:0/3312366290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54b60 0x55d3c4f65950 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x55d3c4f54f60 tx=0x7f6c8c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73dbfcb28fe8222f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:01.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.302+0000 7f6c8a7fc640 1 -- 172.21.15.77:0/3312366290 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c028070 con 0x55d3c4f54b60 2024-03-16T10:21:01.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.302+0000 7f6c8a7fc640 1 -- 172.21.15.77:0/3312366290 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c8c012110 con 0x55d3c4f54b60 2024-03-16T10:21:01.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.302+0000 7f6c8a7fc640 1 -- 172.21.15.77:0/3312366290 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0227f0 con 0x55d3c4f54b60 2024-03-16T10:21:01.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.303+0000 7f6ca9981740 1 -- 172.21.15.77:0/3312366290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54b60 msgr2=0x55d3c4f65950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.303+0000 7f6ca9981740 1 --2- 172.21.15.77:0/3312366290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54b60 0x55d3c4f65950 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x55d3c4f54f60 tx=0x7f6c8c005f90 comp rx=0 tx=0).stop 2024-03-16T10:21:01.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.303+0000 7f6ca9981740 1 -- 172.21.15.77:0/3312366290 shutdown_connections 2024-03-16T10:21:01.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.303+0000 7f6ca9981740 1 --2- 172.21.15.77:0/3312366290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f59cd0 0x55d3c4f4fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.303+0000 7f6ca9981740 1 --2- 172.21.15.77:0/3312366290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54b60 0x55d3c4f65950 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.303+0000 7f6ca9981740 1 --2- 172.21.15.77:0/3312366290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f54170 0x55d3c4f5e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.303+0000 7f6ca9981740 1 -- 172.21.15.77:0/3312366290 >> 172.21.15.77:0/3312366290 conn(0x55d3c4f4f6e0 msgr2=0x55d3c4f4a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:01.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.303+0000 7f6ca9981740 1 -- 172.21.15.77:0/3312366290 shutdown_connections 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.303+0000 7f6ca9981740 1 -- 172.21.15.77:0/3312366290 wait complete. 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.303+0000 7f6ca9981740 1 -- start start 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca9981740 1 Processor -- start 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca9981740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54170 0x55d3c4f627b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca9981740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f62cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca9981740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f59cd0 0x55d3c4fcdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca9981740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca01a7ee0 con 0x55d3c4f54170 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca9981740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca01a7810 con 0x55d3c4f54b60 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca9981740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca01a4990 con 0x55d3c4f59cd0 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca9981740 10 client.0 _ll_register_callbacks cb 0x7f6c8c009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca4a5b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54170 0x55d3c4f627b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca4a5b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54170 0x55d3c4f627b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54850/0 (socket says 172.21.15.77:54850) 2024-03-16T10:21:01.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca525c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f59cd0 0x55d3c4fcdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/2683892723 learned_addr learned my addr 172.21.15.77:0/2683892723 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6c9ffff640 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f62cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/2683892723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f59cd0 msgr2=0x55d3c4fcdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f59cd0 0x55d3c4fcdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/2683892723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 msgr2=0x55d3c4f62cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f62cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca4a5b640 1 -- 172.21.15.77:0/2683892723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3c4e809d0 con 0x55d3c4f54170 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.304+0000 7f6ca525c640 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f59cd0 0x55d3c4fcdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6ca4a5b640 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54170 0x55d3c4f627b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6c940b9400 tx=0x7f6c940bce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a61c26e86c21256 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6c7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6c7ffff640 1 -- 172.21.15.77:0/2683892723 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9404b040 con 0x55d3c4f54170 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6c7ffff640 1 -- 172.21.15.77:0/2683892723 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f6c940bd200 con 0x55d3c4f54170 2024-03-16T10:21:01.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6ca9981740 1 -- 172.21.15.77:0/2683892723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c8c012110 con 0x55d3c4f54170 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6c7ffff640 1 -- 172.21.15.77:0/2683892723 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6c940bc900 con 0x55d3c4f54170 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6ca9981740 10 client.16722 fetch_fsmap learned FSMap version 33 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6ca9981740 1 -- 172.21.15.77:0/2683892723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d3c4e809d0 con 0x55d3c4f54170 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6c7ffff640 1 -- 172.21.15.77:0/2683892723 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c8c012110 con 0x55d3c4f54170 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6c7ffff640 1 -- 172.21.15.77:0/2683892723 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f6c940b8cf0 con 0x55d3c4f54170 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6ca9981740 10 client.16722 fetch_fsmap finished waiting for FSMap version 33 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6ca9981740 10 client.16722 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6ca9981740 20 client.16722 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6ca9981740 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d3c4fcfb60 0x55d3c4fd1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6ca9981740 4 client.16722 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.305+0000 7f6ca9981740 1 -- 172.21.15.77:0/2683892723 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d3c4ec0d90 con 0x55d3c4fcfb60 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.306+0000 7f6c9ffff640 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d3c4fcfb60 0x55d3c4fd1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.306+0000 7f6c7ffff640 10 client.16722 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:01.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.306+0000 7f6c9ffff640 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d3c4fcfb60 0x55d3c4fd1f80 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2aabe3064adc60cd server_cookie=7df3078b451e5a43 in_seq=0 out_seq=0 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.306+0000 7f6c7ffff640 1 -- 172.21.15.77:0/2683892723 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55d3c4ec0d90 con 0x55d3c4fcfb60 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.306+0000 7f6c7ffff640 10 client.16722 handle_command_reply: tid=0 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.306+0000 7f6ca9981740 1 client.16722 shutdown 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.307+0000 7f6ca9981740 1 -- 172.21.15.77:0/2683892723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54170 msgr2=0x55d3c4f627b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.307+0000 7f6ca9981740 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54170 0x55d3c4f627b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6c940b9400 tx=0x7f6c940bce70 comp rx=0 tx=0).stop 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.307+0000 7f6ca9981740 1 -- 172.21.15.77:0/2683892723 shutdown_connections 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.307+0000 7f6ca9981740 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3c4f59cd0 0x55d3c4fcdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.307+0000 7f6ca9981740 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3c4f54b60 0x55d3c4f62cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.307+0000 7f6ca9981740 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d3c4fcfb60 0x55d3c4fd1f80 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.307+0000 7f6ca9981740 1 --2- 172.21.15.77:0/2683892723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d3c4f54170 0x55d3c4f627b0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.307+0000 7f6c7ffff640 0 client.16722 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.307+0000 7f6ca9981740 1 -- 172.21.15.77:0/2683892723 >> 172.21.15.77:0/2683892723 conn(0x55d3c4f4f6e0 msgr2=0x55d3c4f4fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.307+0000 7f6ca9981740 1 -- 172.21.15.77:0/2683892723 shutdown_connections 2024-03-16T10:21:01.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.307+0000 7f6ca9981740 1 -- 172.21.15.77:0/2683892723 wait complete. 2024-03-16T10:21:01.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.307+0000 7f6ca9981740 20 client.16722 trim_cache size 0 max 16384 2024-03-16T10:21:01.314 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:21:01.314 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-16T10:21:01.314 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:21:01.314 INFO:teuthology.orchestra.run.smithi077.stdout: "127457ea-db1f-4f0c-887d-a265bb630cd6": { 2024-03-16T10:21:01.314 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:21:01.314 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "127457ea-db1f-4f0c-887d-a265bb630cd6", 2024-03-16T10:21:01.314 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:21:01.314 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:21:01.314 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:21:01.315 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:21:01.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.308+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/6761689 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6c900a2880 msgr2=0x7f6c900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.308+0000 7f6ca6cbf640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6c900a2880 0x7f6c900a4d40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6c98005fd0 tx=0x7f6c98011810 comp rx=0 tx=0).stop 2024-03-16T10:21:01.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.308+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/6761689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ca0100700 msgr2=0x7f6ca0197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:01.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.308+0000 7f6ca6cbf640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ca0100700 0x7f6ca0197b40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6c94000c00 tx=0x7f6c9402c640 comp rx=0 tx=0).stop 2024-03-16T10:21:01.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.308+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/6761689 shutdown_connections 2024-03-16T10:21:01.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.308+0000 7f6ca6cbf640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca01022b0 0x7f6ca019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.308+0000 7f6ca6cbf640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ca01018f0 0x7f6ca0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.308+0000 7f6ca6cbf640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6c900a2880 0x7f6c900a4d40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.308+0000 7f6ca6cbf640 1 --2- 172.21.15.77:0/6761689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ca0100700 0x7f6ca0197b40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:01.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.308+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/6761689 >> 172.21.15.77:0/6761689 conn(0x7f6ca00fbc90 msgr2=0x7f6ca0104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:01.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.308+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/6761689 shutdown_connections 2024-03-16T10:21:01.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:01.308+0000 7f6ca6cbf640 1 -- 172.21.15.77:0/6761689 wait complete. 2024-03-16T10:21:01.321 INFO:tasks.cephfs.filesystem:scrub status for tag:127457ea-db1f-4f0c-887d-a265bb630cd6 - {'path': '/', 'tag': '127457ea-db1f-4f0c-887d-a265bb630cd6', 'options': 'recursive,force'} 2024-03-16T10:21:02.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:01 smithi192 ceph-mon[36354]: pgmap v468: 97 pgs: 97 active+clean; 442 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.9 MiB/s wr, 422 op/s 2024-03-16T10:21:02.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:01 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:01 smithi077 ceph-mon[32400]: pgmap v468: 97 pgs: 97 active+clean; 442 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.9 MiB/s wr, 422 op/s 2024-03-16T10:21:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:01 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:01 smithi193 ceph-mon[36311]: pgmap v468: 97 pgs: 97 active+clean; 442 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.9 MiB/s wr, 422 op/s 2024-03-16T10:21:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:01 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:03.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:02 smithi192 ceph-mon[36354]: scrub queued for path: / 2024-03-16T10:21:03.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:02 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:03.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:02 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:03.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:02 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:03.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:02 smithi077 ceph-mon[32400]: scrub queued for path: / 2024-03-16T10:21:03.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:02 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:03.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:02 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:03.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:02 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:03.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:02 smithi193 ceph-mon[36311]: scrub queued for path: / 2024-03-16T10:21:03.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:02 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:03.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:02 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:03.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:02 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:04.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:03 smithi192 ceph-mon[36354]: pgmap v469: 97 pgs: 97 active+clean; 449 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.3 MiB/s wr, 439 op/s 2024-03-16T10:21:04.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:03 smithi077 ceph-mon[32400]: pgmap v469: 97 pgs: 97 active+clean; 449 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.3 MiB/s wr, 439 op/s 2024-03-16T10:21:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:03 smithi193 ceph-mon[36311]: pgmap v469: 97 pgs: 97 active+clean; 449 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.3 MiB/s wr, 439 op/s 2024-03-16T10:21:05.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:04 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:05.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:04 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:05.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:04 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:05.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:04 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:05.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:04 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:05.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:04 smithi077 ceph-mon[32400]: pgmap v470: 97 pgs: 97 active+clean; 447 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.3 MiB/s wr, 448 op/s 2024-03-16T10:21:05.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:04 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:05.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:04 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:05.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:04 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:05.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:04 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:05.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:04 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:05.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:04 smithi193 ceph-mon[36311]: pgmap v470: 97 pgs: 97 active+clean; 447 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.3 MiB/s wr, 448 op/s 2024-03-16T10:21:05.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:04 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:05.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:04 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:05.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:04 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:05.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:04 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:05.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:04 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:05.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:04 smithi192 ceph-mon[36354]: pgmap v470: 97 pgs: 97 active+clean; 447 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.3 MiB/s wr, 448 op/s 2024-03-16T10:21:06.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:05 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:06.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:05 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:06.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:05 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:06.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:05 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:06.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:05 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:05 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:05 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:05 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:05 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:05 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:06.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:05 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:06.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:05 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:06.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:05 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:06.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:05 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:06.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:05 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:07.104 INFO:tasks.workunit.client.0.smithi077.stdout: 91 1539366 317 1041436 308 866540 780 2024-03-16T10:21:07.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:06 smithi077 ceph-mon[32400]: pgmap v471: 97 pgs: 97 active+clean; 447 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.1 MiB/s wr, 480 op/s 2024-03-16T10:21:07.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:06 smithi193 ceph-mon[36311]: pgmap v471: 97 pgs: 97 active+clean; 447 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.1 MiB/s wr, 480 op/s 2024-03-16T10:21:07.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:06 smithi192 ceph-mon[36354]: pgmap v471: 97 pgs: 97 active+clean; 447 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.1 MiB/s wr, 480 op/s 2024-03-16T10:21:08.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.249 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.250 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:07 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:21:08.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:21:08.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:07 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:21:08.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.420 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:08.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:07 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:21:09.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:08 smithi077 ceph-mon[32400]: pgmap v472: 97 pgs: 97 active+clean; 452 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 403 op/s 2024-03-16T10:21:09.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:08 smithi193 ceph-mon[36311]: pgmap v472: 97 pgs: 97 active+clean; 452 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 403 op/s 2024-03-16T10:21:09.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:08 smithi192 ceph-mon[36354]: pgmap v472: 97 pgs: 97 active+clean; 452 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 403 op/s 2024-03-16T10:21:12.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:11 smithi192 ceph-mon[36354]: pgmap v473: 97 pgs: 97 active+clean; 452 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 320 op/s 2024-03-16T10:21:12.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:11 smithi077 ceph-mon[32400]: pgmap v473: 97 pgs: 97 active+clean; 452 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 320 op/s 2024-03-16T10:21:12.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:11 smithi193 ceph-mon[36311]: pgmap v473: 97 pgs: 97 active+clean; 452 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 320 op/s 2024-03-16T10:21:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.098+0000 7f3ff7fff640 1 -- 172.21.15.77:0/376438985 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff80437c0 con 0x7f4000107e40 2024-03-16T10:21:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.098+0000 7f4007812640 1 -- 172.21.15.77:0/376438985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4000107e40 msgr2=0x7f4000108260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.098+0000 7f4007812640 1 --2- 172.21.15.77:0/376438985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4000107e40 0x7f4000108260 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3ff801acb0 tx=0x7f3ff803d130 comp rx=0 tx=0).stop 2024-03-16T10:21:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.099+0000 7f4007812640 1 -- 172.21.15.77:0/376438985 shutdown_connections 2024-03-16T10:21:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.099+0000 7f4007812640 1 --2- 172.21.15.77:0/376438985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4000109950 0x7f4000110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.099+0000 7f4007812640 1 --2- 172.21.15.77:0/376438985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4000108f90 0x7f4000109410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.099+0000 7f4007812640 1 --2- 172.21.15.77:0/376438985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4000107e40 0x7f4000108260 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.099+0000 7f4007812640 1 -- 172.21.15.77:0/376438985 >> 172.21.15.77:0/376438985 conn(0x7f4000076df0 msgr2=0x7f4000079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.099+0000 7f4007812640 1 -- 172.21.15.77:0/376438985 shutdown_connections 2024-03-16T10:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.099+0000 7f4007812640 1 -- 172.21.15.77:0/376438985 wait complete. 2024-03-16T10:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.100+0000 7f4007812640 1 Processor -- start 2024-03-16T10:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.100+0000 7f4007812640 1 -- start start 2024-03-16T10:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.100+0000 7f4007812640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4000107e40 0x7f4000195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.100+0000 7f4007812640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4000108f90 0x7f4000196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.100+0000 7f4007812640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4000109950 0x7f400019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.100+0000 7f4007812640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f400007ccb0 con 0x7f4000107e40 2024-03-16T10:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.100+0000 7f4007812640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f400007cb30 con 0x7f4000109950 2024-03-16T10:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.100+0000 7f4007812640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f400007ce30 con 0x7f4000108f90 2024-03-16T10:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.101+0000 7f40055ae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4000107e40 0x7f4000195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.101+0000 7f40055ae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4000107e40 0x7f4000195e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42726/0 (socket says 172.21.15.77:42726) 2024-03-16T10:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.101+0000 7f40055ae640 1 -- 172.21.15.77:0/2606221477 learned_addr learned my addr 172.21.15.77:0/2606221477 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.101+0000 7f4005daf640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4000109950 0x7f400019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.101+0000 7f4004dad640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4000108f90 0x7f4000196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.101+0000 7f4004dad640 1 -- 172.21.15.77:0/2606221477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4000109950 msgr2=0x7f400019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.101+0000 7f4004dad640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4000109950 0x7f400019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.101+0000 7f4004dad640 1 -- 172.21.15.77:0/2606221477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4000107e40 msgr2=0x7f4000195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.101+0000 7f4004dad640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4000107e40 0x7f4000195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.102+0000 7f4004dad640 1 -- 172.21.15.77:0/2606221477 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400019d590 con 0x7f4000108f90 2024-03-16T10:21:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.102+0000 7f4004dad640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4000108f90 0x7f4000196360 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000e9f0 tx=0x7f3ff000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.102+0000 7f3ff67fc640 1 -- 172.21.15.77:0/2606221477 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff000ccc0 con 0x7f4000108f90 2024-03-16T10:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.103+0000 7f4007812640 1 -- 172.21.15.77:0/2606221477 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f400019d880 con 0x7f4000108f90 2024-03-16T10:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.103+0000 7f4007812640 1 -- 172.21.15.77:0/2606221477 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f400019de20 con 0x7f4000108f90 2024-03-16T10:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.104+0000 7f3ff67fc640 1 -- 172.21.15.77:0/2606221477 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ff000ce60 con 0x7f4000108f90 2024-03-16T10:21:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.104+0000 7f3ff67fc640 1 -- 172.21.15.77:0/2606221477 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff000ccc0 con 0x7f4000108f90 2024-03-16T10:21:13.109 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.108+0000 7f400a401740 1 Processor -- start 2024-03-16T10:21:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.109+0000 7f400a401740 1 -- start start 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.109+0000 7f400a401740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2a4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.109+0000 7f400a401740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a4d30 0x5645fc2a9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.109+0000 7f400a401740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a9ea0 0x5645fc2ac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.109+0000 7f400a401740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5645fc209a40 con 0x5645fc2a4d30 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.109+0000 7f400a401740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5645fc1b2920 con 0x5645fc2a4340 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.109+0000 7f400a401740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5645fc205ae0 con 0x5645fc2a9ea0 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.110+0000 7f4005daf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.110+0000 7f4005daf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2a4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:43446/0 (socket says 172.21.15.77:43446) 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.110+0000 7f4005daf640 1 -- 172.21.15.77:0/3428840887 learned_addr learned my addr 172.21.15.77:0/3428840887 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.110+0000 7f40055ae640 1 --2- 172.21.15.77:0/3428840887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a4d30 0x5645fc2a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.110+0000 7f4005daf640 1 -- 172.21.15.77:0/3428840887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a9ea0 msgr2=0x5645fc2ac330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.110+0000 7f4005daf640 1 --2- 172.21.15.77:0/3428840887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a9ea0 0x5645fc2ac330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.110+0000 7f4005daf640 1 -- 172.21.15.77:0/3428840887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a4d30 msgr2=0x5645fc2a9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.110+0000 7f4005daf640 1 --2- 172.21.15.77:0/3428840887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a4d30 0x5645fc2a9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.110+0000 7f4005daf640 1 -- 172.21.15.77:0/3428840887 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645fc1ca460 con 0x5645fc2a4340 2024-03-16T10:21:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.110+0000 7f4005daf640 1 --2- 172.21.15.77:0/3428840887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2a4760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00ab00 tx=0x7f3ffc00f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf8255a92d85a331 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.111+0000 7f3ff67fc640 1 -- 172.21.15.77:0/2606221477 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f3ff0004020 con 0x7f4000108f90 2024-03-16T10:21:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.111+0000 7f3feb7fe640 1 -- 172.21.15.77:0/3428840887 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc01d070 con 0x5645fc2a4340 2024-03-16T10:21:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.111+0000 7f3feb7fe640 1 -- 172.21.15.77:0/3428840887 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ffc018920 con 0x5645fc2a4340 2024-03-16T10:21:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.112+0000 7f3feb7fe640 1 -- 172.21.15.77:0/3428840887 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc010ba0 con 0x5645fc2a4340 2024-03-16T10:21:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.112+0000 7f400a401740 1 -- 172.21.15.77:0/3428840887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 msgr2=0x5645fc2a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.112+0000 7f400a401740 1 --2- 172.21.15.77:0/3428840887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2a4760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00ab00 tx=0x7f3ffc00f3c0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.112+0000 7f3ff67fc640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3fec0a2bb0 0x7f3fec0a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.112+0000 7f3ff67fc640 1 -- 172.21.15.77:0/2606221477 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f3ff00ce800 con 0x7f4000108f90 2024-03-16T10:21:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.113+0000 7f4005daf640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3fec0a2bb0 0x7f3fec0a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.113+0000 7f400a401740 1 -- 172.21.15.77:0/3428840887 shutdown_connections 2024-03-16T10:21:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.113+0000 7f400a401740 1 --2- 172.21.15.77:0/3428840887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a9ea0 0x5645fc2ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.113+0000 7f400a401740 1 --2- 172.21.15.77:0/3428840887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a4d30 0x5645fc2a9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.113+0000 7f400a401740 1 --2- 172.21.15.77:0/3428840887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2a4760 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.113+0000 7f400a401740 1 -- 172.21.15.77:0/3428840887 >> 172.21.15.77:0/3428840887 conn(0x5645fc29f8b0 msgr2=0x5645fc29fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.113+0000 7f400a401740 1 -- 172.21.15.77:0/3428840887 shutdown_connections 2024-03-16T10:21:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.114+0000 7f400a401740 1 -- 172.21.15.77:0/3428840887 wait complete. 2024-03-16T10:21:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.114+0000 7f400a401740 1 -- start start 2024-03-16T10:21:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.114+0000 7f400a401740 1 Processor -- start 2024-03-16T10:21:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f400a401740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f400a401740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a4d30 0x5645fc2aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f400a401740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f400a401740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5645fc2acef0 con 0x5645fc2a9ea0 2024-03-16T10:21:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f4005daf640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3fec0a2bb0 0x7f3fec0a5070 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00a890 tx=0x7f3ffc010430 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f400a401740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5645fc1c5d70 con 0x5645fc2a4340 2024-03-16T10:21:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f400a401740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5645fc2ad070 con 0x5645fc2a4d30 2024-03-16T10:21:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f400a401740 10 client.0 _ll_register_callbacks cb 0x5645fbf1da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f40055ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f40055ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2ae6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:43456/0 (socket says 172.21.15.77:43456) 2024-03-16T10:21:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f40055ae640 1 -- 172.21.15.77:0/284103381 learned_addr learned my addr 172.21.15.77:0/284103381 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f4004dad640 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a4d30 0x5645fc2aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.115+0000 7f4005daf640 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.116+0000 7f4005daf640 1 -- 172.21.15.77:0/284103381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a4d30 msgr2=0x5645fc2aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.116+0000 7f4005daf640 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a4d30 0x5645fc2aebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.116+0000 7f4005daf640 1 -- 172.21.15.77:0/284103381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 msgr2=0x5645fc2ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.116+0000 7f4005daf640 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.116+0000 7f4005daf640 1 -- 172.21.15.77:0/284103381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645fbf1e210 con 0x5645fc2a9ea0 2024-03-16T10:21:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.116+0000 7f4005daf640 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b5c20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00aad0 tx=0x7f3ffc005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd9018f2c8a5608 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.116+0000 7f3fe8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:21:13.119 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.117+0000 7f3fe8ff9640 1 -- 172.21.15.77:0/284103381 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc005230 con 0x5645fc2a9ea0 2024-03-16T10:21:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.117+0000 7f3fe8ff9640 1 -- 172.21.15.77:0/284103381 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f3ffc0053d0 con 0x5645fc2a9ea0 2024-03-16T10:21:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.117+0000 7f400a401740 1 -- 172.21.15.77:0/284103381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ffc018920 con 0x5645fc2a9ea0 2024-03-16T10:21:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.118+0000 7f3fe8ff9640 1 -- 172.21.15.77:0/284103381 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3ffc0055d0 con 0x5645fc2a9ea0 2024-03-16T10:21:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.118+0000 7f400a401740 10 client.16734 fetch_fsmap learned FSMap version 34 2024-03-16T10:21:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.118+0000 7f400a401740 1 -- 172.21.15.77:0/284103381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5645fbfea620 con 0x5645fc2a9ea0 2024-03-16T10:21:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.118+0000 7f3fe8ff9640 1 -- 172.21.15.77:0/284103381 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ffc018920 con 0x5645fc2a9ea0 2024-03-16T10:21:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.119+0000 7f3fe8ff9640 1 -- 172.21.15.77:0/284103381 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f3ffc020040 con 0x5645fc2a9ea0 2024-03-16T10:21:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.119+0000 7f400a401740 10 client.16734 fetch_fsmap finished waiting for FSMap version 34 2024-03-16T10:21:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.119+0000 7f400a401740 10 client.16734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.119+0000 7f400a401740 20 client.16734 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.119+0000 7f400a401740 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645fc2ba290 0x5645fc2bc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.119+0000 7f400a401740 4 client.16734 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:21:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.119+0000 7f400a401740 1 -- 172.21.15.77:0/284103381 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645fc216f80 con 0x5645fc2ba290 2024-03-16T10:21:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.120+0000 7f40055ae640 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645fc2ba290 0x5645fc2bc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.120+0000 7f40055ae640 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645fc2ba290 0x5645fc2bc6c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c12fe80d5b9b1168 server_cookie=91c1d08d4fc11275 in_seq=0 out_seq=0 2024-03-16T10:21:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.120+0000 7f3fe8ff9640 10 client.16734 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f3fe8ff9640 1 -- 172.21.15.77:0/284103381 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5645fc216f80 con 0x5645fc2ba290 2024-03-16T10:21:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f3fe8ff9640 10 client.16734 handle_command_reply: tid=0 2024-03-16T10:21:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f400a401740 1 client.16734 shutdown 2024-03-16T10:21:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f400a401740 1 -- 172.21.15.77:0/284103381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a9ea0 msgr2=0x5645fc2b5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f400a401740 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b5c20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00aad0 tx=0x7f3ffc005f60 comp rx=0 tx=0).stop 2024-03-16T10:21:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f400a401740 1 -- 172.21.15.77:0/284103381 shutdown_connections 2024-03-16T10:21:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f400a401740 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645fc2ba290 0x5645fc2bc6c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f400a401740 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b5c20 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f400a401740 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a4d30 0x5645fc2aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f400a401740 1 --2- 172.21.15.77:0/284103381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f400a401740 1 -- 172.21.15.77:0/284103381 >> 172.21.15.77:0/284103381 conn(0x5645fc29f8b0 msgr2=0x5645fc29ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:13.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f3fe8ff9640 0 client.16734 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:13.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.122+0000 7f400a401740 1 -- 172.21.15.77:0/284103381 shutdown_connections 2024-03-16T10:21:13.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.123+0000 7f400a401740 1 -- 172.21.15.77:0/284103381 wait complete. 2024-03-16T10:21:13.124 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.124+0000 7f400a401740 20 client.16734 trim_cache size 0 max 16384 2024-03-16T10:21:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.154+0000 7f400a401740 1 Processor -- start 2024-03-16T10:21:13.156 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.156+0000 7f400a401740 1 -- start start 2024-03-16T10:21:13.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.156+0000 7f400a401740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2a5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.157 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.157+0000 7f400a401740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a4d30 0x5645fc2a6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.157+0000 7f40055ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2a5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.157+0000 7f40055ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2a5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:43458/0 (socket says 172.21.15.77:43458) 2024-03-16T10:21:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.157+0000 7f40055ae640 1 -- 172.21.15.77:0/4111001166 learned_addr learned my addr 172.21.15.77:0/4111001166 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.157+0000 7f400a401740 1 --2- 172.21.15.77:0/4111001166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.157+0000 7f400a401740 1 -- 172.21.15.77:0/4111001166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe400e350 con 0x5645fc2a4d30 2024-03-16T10:21:13.158 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.157+0000 7f400a401740 1 -- 172.21.15.77:0/4111001166 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe400ff50 con 0x5645fc2a4340 2024-03-16T10:21:13.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.157+0000 7f4005daf640 1 --2- 172.21.15.77:0/4111001166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.158+0000 7f4004dad640 1 --2- 172.21.15.77:0/4111001166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a4d30 0x5645fc2a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.158+0000 7f400a401740 1 -- 172.21.15.77:0/4111001166 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe400ea20 con 0x5645fc2a9ea0 2024-03-16T10:21:13.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.158+0000 7f4005daf640 1 -- 172.21.15.77:0/4111001166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 msgr2=0x5645fc2a5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:13.159 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.158+0000 7f4005daf640 1 --2- 172.21.15.77:0/4111001166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2a5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.158+0000 7f4005daf640 1 -- 172.21.15.77:0/4111001166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a4d30 msgr2=0x5645fc2a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:13.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.158+0000 7f4005daf640 1 --2- 172.21.15.77:0/4111001166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a4d30 0x5645fc2a6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.158+0000 7f4005daf640 1 -- 172.21.15.77:0/4111001166 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645fc2a0300 con 0x5645fc2a9ea0 2024-03-16T10:21:13.160 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.158+0000 7f4005daf640 1 --2- 172.21.15.77:0/4111001166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b5be0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc02e8b0 tx=0x7f3ffc018750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b849d09d0b196e9b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.165+0000 7f3feaffd640 1 -- 172.21.15.77:0/4111001166 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc00fe80 con 0x5645fc2a9ea0 2024-03-16T10:21:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.165+0000 7f3feaffd640 1 -- 172.21.15.77:0/4111001166 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ffc046070 con 0x5645fc2a9ea0 2024-03-16T10:21:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.166+0000 7f400a401740 1 -- 172.21.15.77:0/4111001166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a9ea0 msgr2=0x5645fc2b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.166+0000 7f400a401740 1 --2- 172.21.15.77:0/4111001166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b5be0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc02e8b0 tx=0x7f3ffc018750 comp rx=0 tx=0).stop 2024-03-16T10:21:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.166+0000 7f400a401740 1 -- 172.21.15.77:0/4111001166 shutdown_connections 2024-03-16T10:21:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.166+0000 7f400a401740 1 --2- 172.21.15.77:0/4111001166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b5be0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.166+0000 7f400a401740 1 --2- 172.21.15.77:0/4111001166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a4d30 0x5645fc2a6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.166+0000 7f400a401740 1 --2- 172.21.15.77:0/4111001166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2a5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.166+0000 7f400a401740 1 -- 172.21.15.77:0/4111001166 >> 172.21.15.77:0/4111001166 conn(0x5645fc29f8b0 msgr2=0x5645fc2ae6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.167+0000 7f400a401740 1 -- 172.21.15.77:0/4111001166 shutdown_connections 2024-03-16T10:21:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.167+0000 7f400a401740 1 -- 172.21.15.77:0/4111001166 wait complete. 2024-03-16T10:21:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.167+0000 7f400a401740 1 -- start start 2024-03-16T10:21:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.167+0000 7f400a401740 1 Processor -- start 2024-03-16T10:21:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.168+0000 7f400a401740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.168+0000 7f400a401740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a4d30 0x5645fc2b3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.168+0000 7f400a401740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.168+0000 7f400a401740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe400f880 con 0x5645fc2a9ea0 2024-03-16T10:21:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.168+0000 7f400a401740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe400f1b0 con 0x5645fc2a4340 2024-03-16T10:21:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.168+0000 7f400a401740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe400c330 con 0x5645fc2a4d30 2024-03-16T10:21:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.168+0000 7f400a401740 10 client.0 _ll_register_callbacks cb 0x7f3ffc01b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.168+0000 7f4004dad640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a4d30 0x5645fc2b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.168+0000 7f40055ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.169+0000 7f40055ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:43468/0 (socket says 172.21.15.77:43468) 2024-03-16T10:21:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.169+0000 7f40055ae640 1 -- 172.21.15.77:0/623521924 learned_addr learned my addr 172.21.15.77:0/623521924 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.169+0000 7f4005daf640 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.169+0000 7f4004dad640 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a4d30 0x5645fc2b3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44242/0 (socket says 172.21.15.77:44242) 2024-03-16T10:21:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.169+0000 7f4005daf640 1 -- 172.21.15.77:0/623521924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a4d30 msgr2=0x5645fc2b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.170+0000 7f4005daf640 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a4d30 0x5645fc2b3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.170+0000 7f4005daf640 1 -- 172.21.15.77:0/623521924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 msgr2=0x5645fc2b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.170+0000 7f4005daf640 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.170+0000 7f4005daf640 1 -- 172.21.15.77:0/623521924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645fc2b4c70 con 0x5645fc2a9ea0 2024-03-16T10:21:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.170+0000 7f40055ae640 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.170+0000 7f4005daf640 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b4680 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc02e6c0 tx=0x7f3ffc00fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd741bd72dd328e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.170+0000 7f3fe8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:21:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.170+0000 7f3fe8ff9640 1 -- 172.21.15.77:0/623521924 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc041dc0 con 0x5645fc2a9ea0 2024-03-16T10:21:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.170+0000 7f3fe8ff9640 1 -- 172.21.15.77:0/623521924 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f3ffc041480 con 0x5645fc2a9ea0 2024-03-16T10:21:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.171+0000 7f400a401740 1 -- 172.21.15.77:0/623521924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5645fc2112d0 con 0x5645fc2a9ea0 2024-03-16T10:21:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.171+0000 7f3fe8ff9640 1 -- 172.21.15.77:0/623521924 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3ffc018900 con 0x5645fc2a9ea0 2024-03-16T10:21:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.171+0000 7f400a401740 10 client.16740 fetch_fsmap learned FSMap version 34 2024-03-16T10:21:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.171+0000 7f400a401740 1 -- 172.21.15.77:0/623521924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5645fc320610 con 0x5645fc2a9ea0 2024-03-16T10:21:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.171+0000 7f3fe8ff9640 1 -- 172.21.15.77:0/623521924 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5645fc2112d0 con 0x5645fc2a9ea0 2024-03-16T10:21:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.171+0000 7f3fe8ff9640 1 -- 172.21.15.77:0/623521924 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f3ffc020050 con 0x5645fc2a9ea0 2024-03-16T10:21:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.171+0000 7f400a401740 10 client.16740 fetch_fsmap finished waiting for FSMap version 34 2024-03-16T10:21:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.171+0000 7f400a401740 10 client.16740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.171+0000 7f400a401740 20 client.16740 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.171+0000 7f400a401740 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645fc320c00 0x5645fc323020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.171+0000 7f400a401740 4 client.16740 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:21:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.171+0000 7f400a401740 1 -- 172.21.15.77:0/623521924 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5645fc18f770 con 0x5645fc320c00 2024-03-16T10:21:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.172+0000 7f40055ae640 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645fc320c00 0x5645fc323020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.172+0000 7f3fe8ff9640 10 client.16740 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.172+0000 7f40055ae640 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645fc320c00 0x5645fc323020 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53deec0ed13d45f7 server_cookie=ab34bb3cf9111370 in_seq=0 out_seq=0 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f3fe8ff9640 1 -- 172.21.15.77:0/623521924 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5645fc18f770 con 0x5645fc320c00 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f3fe8ff9640 10 client.16740 handle_command_reply: tid=0 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f400a401740 1 client.16740 shutdown 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f400a401740 1 -- 172.21.15.77:0/623521924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a9ea0 msgr2=0x5645fc2b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f400a401740 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b4680 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc02e6c0 tx=0x7f3ffc00fef0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f400a401740 1 -- 172.21.15.77:0/623521924 shutdown_connections 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f400a401740 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645fc320c00 0x5645fc323020 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f400a401740 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645fc2a9ea0 0x5645fc2b4680 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f400a401740 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645fc2a4d30 0x5645fc2b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f400a401740 1 --2- 172.21.15.77:0/623521924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645fc2a4340 0x5645fc2b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f400a401740 1 -- 172.21.15.77:0/623521924 >> 172.21.15.77:0/623521924 conn(0x5645fc29f8b0 msgr2=0x5645fc2ae5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f3fe8ff9640 0 client.16740 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f400a401740 1 -- 172.21.15.77:0/623521924 shutdown_connections 2024-03-16T10:21:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f400a401740 1 -- 172.21.15.77:0/623521924 wait complete. 2024-03-16T10:21:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.189+0000 7f400a401740 20 client.16740 trim_cache size 0 max 16384 2024-03-16T10:21:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:21:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:21:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:21:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:21:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.190+0000 7f4007812640 1 -- 172.21.15.77:0/2606221477 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3fec0a2bb0 msgr2=0x7f3fec0a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.190+0000 7f4007812640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3fec0a2bb0 0x7f3fec0a5070 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00a890 tx=0x7f3ffc010430 comp rx=0 tx=0).stop 2024-03-16T10:21:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.190+0000 7f4007812640 1 -- 172.21.15.77:0/2606221477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4000108f90 msgr2=0x7f4000196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.190+0000 7f4007812640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4000108f90 0x7f4000196360 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000e9f0 tx=0x7f3ff000eeb0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.190+0000 7f4007812640 1 -- 172.21.15.77:0/2606221477 shutdown_connections 2024-03-16T10:21:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.190+0000 7f4007812640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4000109950 0x7f400019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.190+0000 7f4007812640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3fec0a2bb0 0x7f3fec0a5070 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.190+0000 7f4007812640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4000108f90 0x7f4000196360 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.190+0000 7f4007812640 1 --2- 172.21.15.77:0/2606221477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4000107e40 0x7f4000195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.190+0000 7f4007812640 1 -- 172.21.15.77:0/2606221477 >> 172.21.15.77:0/2606221477 conn(0x7f4000076df0 msgr2=0x7f400007f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:13.193 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.190+0000 7f4007812640 1 -- 172.21.15.77:0/2606221477 shutdown_connections 2024-03-16T10:21:13.193 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:13.190+0000 7f4007812640 1 -- 172.21.15.77:0/2606221477 wait complete. 2024-03-16T10:21:14.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:13 smithi192 ceph-mon[36354]: pgmap v474: 97 pgs: 97 active+clean; 451 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 191 op/s 2024-03-16T10:21:14.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:13 smithi077 ceph-mon[32400]: pgmap v474: 97 pgs: 97 active+clean; 451 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 191 op/s 2024-03-16T10:21:14.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:13 smithi193 ceph-mon[36311]: pgmap v474: 97 pgs: 97 active+clean; 451 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 191 op/s 2024-03-16T10:21:16.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:15 smithi192 ceph-mon[36354]: pgmap v475: 97 pgs: 97 active+clean; 455 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 156 op/s 2024-03-16T10:21:16.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:15 smithi077 ceph-mon[32400]: pgmap v475: 97 pgs: 97 active+clean; 455 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 156 op/s 2024-03-16T10:21:16.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:15 smithi193 ceph-mon[36311]: pgmap v475: 97 pgs: 97 active+clean; 455 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 156 op/s 2024-03-16T10:21:17.104 INFO:tasks.workunit.client.0.smithi077.stdout: 96 1266839 357 853831 295 721018 936 2024-03-16T10:21:18.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:17 smithi192 ceph-mon[36354]: pgmap v476: 97 pgs: 97 active+clean; 456 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-03-16T10:21:18.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:17 smithi077 ceph-mon[32400]: pgmap v476: 97 pgs: 97 active+clean; 456 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-03-16T10:21:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:17 smithi193 ceph-mon[36311]: pgmap v476: 97 pgs: 97 active+clean; 456 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-03-16T10:21:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:19 smithi192 ceph-mon[36354]: pgmap v477: 97 pgs: 97 active+clean; 459 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-03-16T10:21:20.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:19 smithi077 ceph-mon[32400]: pgmap v477: 97 pgs: 97 active+clean; 459 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-03-16T10:21:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:19 smithi193 ceph-mon[36311]: pgmap v477: 97 pgs: 97 active+clean; 459 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-03-16T10:21:22.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:21 smithi192 ceph-mon[36354]: pgmap v478: 97 pgs: 97 active+clean; 458 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 869 KiB/s wr, 99 op/s 2024-03-16T10:21:22.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:21 smithi077 ceph-mon[32400]: pgmap v478: 97 pgs: 97 active+clean; 458 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 869 KiB/s wr, 99 op/s 2024-03-16T10:21:22.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:21 smithi193 ceph-mon[36311]: pgmap v478: 97 pgs: 97 active+clean; 458 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 869 KiB/s wr, 99 op/s 2024-03-16T10:21:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:23 smithi192 ceph-mon[36354]: pgmap v479: 97 pgs: 97 active+clean; 483 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 216 op/s 2024-03-16T10:21:24.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:23 smithi077 ceph-mon[32400]: pgmap v479: 97 pgs: 97 active+clean; 483 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 216 op/s 2024-03-16T10:21:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:23 smithi193 ceph-mon[36311]: pgmap v479: 97 pgs: 97 active+clean; 483 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 216 op/s 2024-03-16T10:21:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:25 smithi192 ceph-mon[36354]: pgmap v480: 97 pgs: 97 active+clean; 500 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.5 MiB/s wr, 293 op/s 2024-03-16T10:21:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:25 smithi077 ceph-mon[32400]: pgmap v480: 97 pgs: 97 active+clean; 500 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.5 MiB/s wr, 293 op/s 2024-03-16T10:21:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:25 smithi193 ceph-mon[36311]: pgmap v480: 97 pgs: 97 active+clean; 500 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.5 MiB/s wr, 293 op/s 2024-03-16T10:21:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.312+0000 7fb56adb9640 1 -- 172.21.15.77:0/847149502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5641022b0 msgr2=0x7fb564108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.312+0000 7fb56adb9640 1 --2- 172.21.15.77:0/847149502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5641022b0 0x7fb564108b40 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009ca0 tx=0x7fb55c02c0f0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.312+0000 7fb56adb9640 1 -- 172.21.15.77:0/847149502 shutdown_connections 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.312+0000 7fb56adb9640 1 --2- 172.21.15.77:0/847149502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5641022b0 0x7fb564108b40 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.312+0000 7fb56adb9640 1 --2- 172.21.15.77:0/847149502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5641018f0 0x7fb564101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.312+0000 7fb56adb9640 1 --2- 172.21.15.77:0/847149502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb564100700 0x7fb564100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.312+0000 7fb56adb9640 1 -- 172.21.15.77:0/847149502 >> 172.21.15.77:0/847149502 conn(0x7fb5640fbc70 msgr2=0x7fb5640fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.312+0000 7fb56adb9640 1 -- 172.21.15.77:0/847149502 shutdown_connections 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.312+0000 7fb56adb9640 1 -- 172.21.15.77:0/847149502 wait complete. 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb56adb9640 1 Processor -- start 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb56adb9640 1 -- start start 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb56adb9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb564100700 0x7fb5640699f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb56adb9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5641018f0 0x7fb56419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb56adb9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5641022b0 0x7fb56419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb56adb9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb56410c360 con 0x7fb5641022b0 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb56adb9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb56410c1e0 con 0x7fb5641018f0 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb56adb9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb56410c4e0 con 0x7fb564100700 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb569356640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5641022b0 0x7fb56419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb569356640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5641022b0 0x7fb56419ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59940/0 (socket says 172.21.15.77:59940) 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb568b55640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb564100700 0x7fb5640699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb569356640 1 -- 172.21.15.77:0/154601885 learned_addr learned my addr 172.21.15.77:0/154601885 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.313+0000 7fb563fff640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5641018f0 0x7fb56419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.314+0000 7fb568b55640 1 -- 172.21.15.77:0/154601885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5641018f0 msgr2=0x7fb56419c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.314+0000 7fb568b55640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5641018f0 0x7fb56419c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.314+0000 7fb568b55640 1 -- 172.21.15.77:0/154601885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5641022b0 msgr2=0x7fb56419ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.314+0000 7fb568b55640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5641022b0 0x7fb56419ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.314+0000 7fb568b55640 1 -- 172.21.15.77:0/154601885 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56419f610 con 0x7fb564100700 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.314+0000 7fb569356640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5641022b0 0x7fb56419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.314+0000 7fb568b55640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb564100700 0x7fb5640699f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb55800e9c0 tx=0x7fb55800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.314+0000 7fb561ffb640 1 -- 172.21.15.77:0/154601885 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55800ccc0 con 0x7fb564100700 2024-03-16T10:21:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.314+0000 7fb561ffb640 1 -- 172.21.15.77:0/154601885 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5580035f0 con 0x7fb564100700 2024-03-16T10:21:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.314+0000 7fb56adb9640 1 -- 172.21.15.77:0/154601885 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56419f900 con 0x7fb564100700 2024-03-16T10:21:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.314+0000 7fb561ffb640 1 -- 172.21.15.77:0/154601885 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55800ce60 con 0x7fb564100700 2024-03-16T10:21:26.318 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.315+0000 7fb56adb9640 1 -- 172.21.15.77:0/154601885 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb56419fea0 con 0x7fb564100700 2024-03-16T10:21:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.317+0000 7fb56d9a7740 1 Processor -- start 2024-03-16T10:21:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.317+0000 7fb56d9a7740 1 -- start start 2024-03-16T10:21:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb56d9a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c2920 0x56416c8c2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb56d9a7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c3310 0x56416c8c7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb56d9a7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 0x56416c8ca320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb56d9a7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56416c828a40 con 0x56416c8c3310 2024-03-16T10:21:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb56d9a7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56416c7d1920 con 0x56416c8c7eb0 2024-03-16T10:21:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb56d9a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56416c824ae0 con 0x56416c8c2920 2024-03-16T10:21:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb569356640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c2920 0x56416c8c2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb569356640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c2920 0x56416c8c2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:36696/0 (socket says 172.21.15.77:36696) 2024-03-16T10:21:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb569356640 1 -- 172.21.15.77:0/4285710171 learned_addr learned my addr 172.21.15.77:0/4285710171 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb561ffb640 1 -- 172.21.15.77:0/154601885 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fb5580037a0 con 0x7fb564100700 2024-03-16T10:21:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb569356640 1 -- 172.21.15.77:0/4285710171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 msgr2=0x56416c8ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb569356640 1 --2- 172.21.15.77:0/4285710171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 0x56416c8ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb569356640 1 -- 172.21.15.77:0/4285710171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c3310 msgr2=0x56416c8c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb569356640 1 --2- 172.21.15.77:0/4285710171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c3310 0x56416c8c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.318+0000 7fb569356640 1 -- 172.21.15.77:0/4285710171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56416c7e9460 con 0x56416c8c2920 2024-03-16T10:21:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.319+0000 7fb569356640 1 --2- 172.21.15.77:0/4285710171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c2920 0x56416c8c2d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009430 tx=0x7fb55c005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d3df76489844cf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.319+0000 7fb54effd640 1 -- 172.21.15.77:0/4285710171 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c043070 con 0x56416c8c2920 2024-03-16T10:21:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.319+0000 7fb54effd640 1 -- 172.21.15.77:0/4285710171 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb55c032c90 con 0x56416c8c2920 2024-03-16T10:21:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.319+0000 7fb54effd640 1 -- 172.21.15.77:0/4285710171 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c039040 con 0x56416c8c2920 2024-03-16T10:21:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.319+0000 7fb561ffb640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb5540a2920 0x7fb5540a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb568b55640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb5540a2920 0x7fb5540a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb56d9a7740 1 -- 172.21.15.77:0/4285710171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c2920 msgr2=0x56416c8c2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:26.323 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/4285710171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c2920 0x56416c8c2d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009430 tx=0x7fb55c005ef0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb56d9a7740 1 -- 172.21.15.77:0/4285710171 shutdown_connections 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/4285710171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 0x56416c8ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/4285710171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c3310 0x56416c8c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/4285710171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c2920 0x56416c8c2d40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb56d9a7740 1 -- 172.21.15.77:0/4285710171 >> 172.21.15.77:0/4285710171 conn(0x56416c8be830 msgr2=0x56416c8bec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb56d9a7740 1 -- 172.21.15.77:0/4285710171 shutdown_connections 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb56d9a7740 1 -- 172.21.15.77:0/4285710171 wait complete. 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb561ffb640 1 -- 172.21.15.77:0/154601885 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb5580ced50 con 0x7fb564100700 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb568b55640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb5540a2920 0x7fb5540a4de0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb55809b3e0 tx=0x7fb5580ac040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.320+0000 7fb56d9a7740 1 -- start start 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.321+0000 7fb56d9a7740 1 Processor -- start 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.321+0000 7fb56d9a7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8de5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.321+0000 7fb56d9a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c3310 0x56416c8deae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.321+0000 7fb56d9a7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 0x56416c8e5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.321+0000 7fb56d9a7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56416c8dcf60 con 0x56416c8c2920 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.321+0000 7fb56d9a7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56416c7e4d70 con 0x56416c8c7eb0 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.321+0000 7fb56d9a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56416c8dd0e0 con 0x56416c8c3310 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.321+0000 7fb56d9a7740 10 client.0 _ll_register_callbacks cb 0x56416c53ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.322+0000 7fb563fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c3310 0x56416c8deae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.322+0000 7fb563fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c3310 0x56416c8deae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:36698/0 (socket says 172.21.15.77:36698) 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.322+0000 7fb563fff640 1 -- 172.21.15.77:0/1177128938 learned_addr learned my addr 172.21.15.77:0/1177128938 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.322+0000 7fb569356640 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 0x56416c8e5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.322+0000 7fb569356640 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8de5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.322+0000 7fb563fff640 1 -- 172.21.15.77:0/1177128938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 msgr2=0x56416c8e5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.322+0000 7fb563fff640 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 0x56416c8e5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.322+0000 7fb563fff640 1 -- 172.21.15.77:0/1177128938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 msgr2=0x56416c8de5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.322+0000 7fb563fff640 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8de5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.322+0000 7fb563fff640 1 -- 172.21.15.77:0/1177128938 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56416c53d210 con 0x56416c8c3310 2024-03-16T10:21:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.323+0000 7fb563fff640 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c3310 0x56416c8deae0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb550004700 tx=0x7fb550011b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1657e75971067cd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.323+0000 7fb543fff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:21:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.323+0000 7fb543fff640 1 -- 172.21.15.77:0/1177128938 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55000fe50 con 0x56416c8c3310 2024-03-16T10:21:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.323+0000 7fb543fff640 1 -- 172.21.15.77:0/1177128938 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb550004020 con 0x56416c8c3310 2024-03-16T10:21:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.323+0000 7fb56d9a7740 1 -- 172.21.15.77:0/1177128938 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb55c032c90 con 0x56416c8c3310 2024-03-16T10:21:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.323+0000 7fb543fff640 1 -- 172.21.15.77:0/1177128938 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb550004220 con 0x56416c8c3310 2024-03-16T10:21:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.323+0000 7fb56d9a7740 10 client.26506 fetch_fsmap learned FSMap version 34 2024-03-16T10:21:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.323+0000 7fb56d9a7740 1 -- 172.21.15.77:0/1177128938 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56416c609620 con 0x56416c8c3310 2024-03-16T10:21:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.324+0000 7fb543fff640 1 -- 172.21.15.77:0/1177128938 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb55c032c90 con 0x56416c8c3310 2024-03-16T10:21:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.324+0000 7fb543fff640 1 -- 172.21.15.77:0/1177128938 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fb5500195f0 con 0x56416c8c3310 2024-03-16T10:21:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.324+0000 7fb56d9a7740 10 client.26506 fetch_fsmap finished waiting for FSMap version 34 2024-03-16T10:21:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.324+0000 7fb56d9a7740 10 client.26506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.324+0000 7fb56d9a7740 20 client.26506 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.324+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56416c8ea460 0x56416c8ec890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.324+0000 7fb56d9a7740 4 client.26506 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.324+0000 7fb56d9a7740 1 -- 172.21.15.77:0/1177128938 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56416c835f80 con 0x56416c8ea460 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.325+0000 7fb569356640 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56416c8ea460 0x56416c8ec890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.325+0000 7fb543fff640 10 client.26506 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.325+0000 7fb569356640 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56416c8ea460 0x56416c8ec890 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1d1be041adfac55 server_cookie=4a569f1d5077283b in_seq=0 out_seq=0 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.326+0000 7fb543fff640 1 -- 172.21.15.77:0/1177128938 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56416c835f80 con 0x56416c8ea460 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.326+0000 7fb543fff640 10 client.26506 handle_command_reply: tid=0 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.326+0000 7fb56d9a7740 1 client.26506 shutdown 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.327+0000 7fb56d9a7740 1 -- 172.21.15.77:0/1177128938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c3310 msgr2=0x56416c8deae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.327+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c3310 0x56416c8deae0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb550004700 tx=0x7fb550011b50 comp rx=0 tx=0).stop 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.327+0000 7fb56d9a7740 1 -- 172.21.15.77:0/1177128938 shutdown_connections 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.327+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 0x56416c8e5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.327+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c3310 0x56416c8deae0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.327+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56416c8ea460 0x56416c8ec890 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.327+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/1177128938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8de5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.327+0000 7fb56d9a7740 1 -- 172.21.15.77:0/1177128938 >> 172.21.15.77:0/1177128938 conn(0x56416c8be830 msgr2=0x56416c8cb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.327+0000 7fb543fff640 0 client.26506 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.327+0000 7fb56d9a7740 1 -- 172.21.15.77:0/1177128938 shutdown_connections 2024-03-16T10:21:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.327+0000 7fb56d9a7740 1 -- 172.21.15.77:0/1177128938 wait complete. 2024-03-16T10:21:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.327+0000 7fb56d9a7740 20 client.26506 trim_cache size 0 max 16384 2024-03-16T10:21:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.354+0000 7fb56d9a7740 1 Processor -- start 2024-03-16T10:21:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.354+0000 7fb56d9a7740 1 -- start start 2024-03-16T10:21:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.354+0000 7fb56d9a7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8c4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.354+0000 7fb56d9a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c3310 0x56416c8c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.354+0000 7fb56d9a7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 0x56416c8e5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.354+0000 7fb56d9a7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641a69e0 con 0x56416c8c2920 2024-03-16T10:21:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.354+0000 7fb56d9a7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641a85e0 con 0x56416c8c7eb0 2024-03-16T10:21:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.354+0000 7fb56d9a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641a70b0 con 0x56416c8c3310 2024-03-16T10:21:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.354+0000 7fb569356640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.354+0000 7fb569356640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8c4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59960/0 (socket says 172.21.15.77:59960) 2024-03-16T10:21:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.354+0000 7fb569356640 1 -- 172.21.15.77:0/2423451328 learned_addr learned my addr 172.21.15.77:0/2423451328 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.354+0000 7fb563fff640 1 --2- 172.21.15.77:0/2423451328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c3310 0x56416c8c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.355+0000 7fb569356640 1 --2- 172.21.15.77:0/2423451328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 0x56416c8e5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.355+0000 7fb569356640 1 -- 172.21.15.77:0/2423451328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c3310 msgr2=0x56416c8c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.355+0000 7fb569356640 1 --2- 172.21.15.77:0/2423451328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c3310 0x56416c8c45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.355+0000 7fb569356640 1 -- 172.21.15.77:0/2423451328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 msgr2=0x56416c8e5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.355+0000 7fb569356640 1 --2- 172.21.15.77:0/2423451328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 0x56416c8e5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.355+0000 7fb569356640 1 -- 172.21.15.77:0/2423451328 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56416c8b0140 con 0x56416c8c2920 2024-03-16T10:21:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.356+0000 7fb569356640 1 --2- 172.21.15.77:0/2423451328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8c4060 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb55c02e010 tx=0x7fb55c04b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4dd2351f13fbfb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.356+0000 7fb54e7fc640 1 -- 172.21.15.77:0/2423451328 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c057070 con 0x56416c8c2920 2024-03-16T10:21:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.356+0000 7fb54e7fc640 1 -- 172.21.15.77:0/2423451328 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb55c050a30 con 0x56416c8c2920 2024-03-16T10:21:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.356+0000 7fb54e7fc640 1 -- 172.21.15.77:0/2423451328 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c039050 con 0x56416c8c2920 2024-03-16T10:21:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.357+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2423451328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 msgr2=0x56416c8c4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.357+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/2423451328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8c4060 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fb55c02e010 tx=0x7fb55c04b040 comp rx=0 tx=0).stop 2024-03-16T10:21:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.357+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2423451328 shutdown_connections 2024-03-16T10:21:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.357+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/2423451328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c7eb0 0x56416c8e5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.357+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/2423451328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c3310 0x56416c8c45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.357+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/2423451328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8c4060 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.357+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2423451328 >> 172.21.15.77:0/2423451328 conn(0x56416c8be830 msgr2=0x56416c8b9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.357+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2423451328 shutdown_connections 2024-03-16T10:21:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.357+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2423451328 wait complete. 2024-03-16T10:21:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.357+0000 7fb56d9a7740 1 -- start start 2024-03-16T10:21:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.358+0000 7fb56d9a7740 1 Processor -- start 2024-03-16T10:21:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.359+0000 7fb56d9a7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8e2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.359+0000 7fb56d9a7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c3310 0x56416c8e31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.359+0000 7fb56d9a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c7eb0 0x56416c8e4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.359+0000 7fb56d9a7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641a7f10 con 0x56416c8c2920 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.359+0000 7fb56d9a7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641a7840 con 0x56416c8c3310 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.359+0000 7fb56d9a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641a49c0 con 0x56416c8c7eb0 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.359+0000 7fb569356640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c7eb0 0x56416c8e4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.359+0000 7fb569356640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c7eb0 0x56416c8e4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:36712/0 (socket says 172.21.15.77:36712) 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.359+0000 7fb569356640 1 -- 172.21.15.77:0/2211702485 learned_addr learned my addr 172.21.15.77:0/2211702485 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.359+0000 7fb56d9a7740 10 client.0 _ll_register_callbacks cb 0x7fb550009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb569356640 1 -- 172.21.15.77:0/2211702485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c3310 msgr2=0x56416c8e31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb563fff640 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c3310 0x56416c8e31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb569356640 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c3310 0x56416c8e31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb569356640 1 -- 172.21.15.77:0/2211702485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 msgr2=0x56416c8e2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb569356640 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8e2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb569356640 1 -- 172.21.15.77:0/2211702485 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56416c8e47f0 con 0x56416c8c7eb0 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2211702485 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb55c050a30 con 0x56416c8c7eb0 2024-03-16T10:21:26.362 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb563fff640 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c3310 0x56416c8e31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:21:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb569356640 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c7eb0 0x56416c8e4200 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb55c037040 tx=0x7fb55c02cdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99b2c7ee69ff3fa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb543fff640 10 client.26512 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:21:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb543fff640 1 -- 172.21.15.77:0/2211702485 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c057070 con 0x56416c8c7eb0 2024-03-16T10:21:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb543fff640 1 -- 172.21.15.77:0/2211702485 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb55c050be0 con 0x56416c8c7eb0 2024-03-16T10:21:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb543fff640 1 -- 172.21.15.77:0/2211702485 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb55c039040 con 0x56416c8c7eb0 2024-03-16T10:21:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb56d9a7740 10 client.26512 fetch_fsmap learned FSMap version 34 2024-03-16T10:21:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.360+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2211702485 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=34}) -- 0x56416c8e5ac0 con 0x56416c8c7eb0 2024-03-16T10:21:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.361+0000 7fb543fff640 1 -- 172.21.15.77:0/2211702485 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fb55c050590 con 0x56416c8c7eb0 2024-03-16T10:21:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.361+0000 7fb56d9a7740 10 client.26512 fetch_fsmap finished waiting for FSMap version 34 2024-03-16T10:21:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.361+0000 7fb56d9a7740 10 client.26512 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.361+0000 7fb56d9a7740 20 client.26512 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.361+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56416c9507d0 0x56416c952bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.361+0000 7fb56d9a7740 4 client.26512 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:21:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.361+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2211702485 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56416c8302d0 con 0x56416c9507d0 2024-03-16T10:21:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.361+0000 7fb563fff640 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56416c9507d0 0x56416c952bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.362+0000 7fb543fff640 10 client.26512 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:26.365 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.362+0000 7fb563fff640 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56416c9507d0 0x56416c952bc0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8330532e268ae768 server_cookie=4daa58d61eb4861f in_seq=0 out_seq=0 2024-03-16T10:21:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.885+0000 7fb543fff640 1 -- 172.21.15.77:0/2211702485 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56416c8302d0 con 0x56416c9507d0 2024-03-16T10:21:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.885+0000 7fb543fff640 10 client.26512 handle_command_reply: tid=0 2024-03-16T10:21:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.885+0000 7fb56d9a7740 1 client.26512 shutdown 2024-03-16T10:21:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.886+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2211702485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c7eb0 msgr2=0x56416c8e4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.886+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c7eb0 0x56416c8e4200 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb55c037040 tx=0x7fb55c02cdc0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.886+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2211702485 shutdown_connections 2024-03-16T10:21:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.886+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56416c8c7eb0 0x56416c8e4200 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.886+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56416c8c3310 0x56416c8e31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.886+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56416c9507d0 0x56416c952bc0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.886+0000 7fb56d9a7740 1 --2- 172.21.15.77:0/2211702485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56416c8c2920 0x56416c8e2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.886+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2211702485 >> 172.21.15.77:0/2211702485 conn(0x56416c8be830 msgr2=0x56416c8c4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.886+0000 7fb543fff640 0 client.26512 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.886+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2211702485 shutdown_connections 2024-03-16T10:21:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.886+0000 7fb56d9a7740 1 -- 172.21.15.77:0/2211702485 wait complete. 2024-03-16T10:21:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.886+0000 7fb56d9a7740 20 client.26512 trim_cache size 0 max 16384 2024-03-16T10:21:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:21:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:21:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:21:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:21:26.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.922+0000 7fb56adb9640 1 -- 172.21.15.77:0/154601885 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb5540a2920 msgr2=0x7fb5540a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:26.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.922+0000 7fb56adb9640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb5540a2920 0x7fb5540a4de0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb55809b3e0 tx=0x7fb5580ac040 comp rx=0 tx=0).stop 2024-03-16T10:21:26.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.922+0000 7fb56adb9640 1 -- 172.21.15.77:0/154601885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb564100700 msgr2=0x7fb5640699f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:26.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.922+0000 7fb56adb9640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb564100700 0x7fb5640699f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb55800e9c0 tx=0x7fb55800ee80 comp rx=0 tx=0).stop 2024-03-16T10:21:26.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.923+0000 7fb56adb9640 1 -- 172.21.15.77:0/154601885 shutdown_connections 2024-03-16T10:21:26.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.923+0000 7fb56adb9640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5641022b0 0x7fb56419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.923+0000 7fb56adb9640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5641018f0 0x7fb56419c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.923+0000 7fb56adb9640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb5540a2920 0x7fb5540a4de0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.925+0000 7fb56adb9640 1 --2- 172.21.15.77:0/154601885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb564100700 0x7fb5640699f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:26.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.925+0000 7fb56adb9640 1 -- 172.21.15.77:0/154601885 >> 172.21.15.77:0/154601885 conn(0x7fb5640fbc70 msgr2=0x7fb564104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.925+0000 7fb56adb9640 1 -- 172.21.15.77:0/154601885 shutdown_connections 2024-03-16T10:21:26.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:26.926+0000 7fb56adb9640 1 -- 172.21.15.77:0/154601885 wait complete. 2024-03-16T10:21:27.107 INFO:tasks.workunit.client.0.smithi077.stdout: 100 1181109 291 796335 213 682568 655 2024-03-16T10:21:27.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:26 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:21:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:26 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:21:27.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:26 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:21:28.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:27 smithi192 ceph-mon[36354]: pgmap v481: 97 pgs: 97 active+clean; 517 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.7 MiB/s wr, 391 op/s 2024-03-16T10:21:28.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:21:28.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:21:28.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:27 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:21:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:27 smithi077 ceph-mon[32400]: pgmap v481: 97 pgs: 97 active+clean; 517 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.7 MiB/s wr, 391 op/s 2024-03-16T10:21:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:21:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:21:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:27 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:21:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:27 smithi193 ceph-mon[36311]: pgmap v481: 97 pgs: 97 active+clean; 517 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.7 MiB/s wr, 391 op/s 2024-03-16T10:21:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:21:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:21:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:27 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:21:30.171 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:29 smithi192 ceph-mon[36354]: pgmap v482: 97 pgs: 97 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.2 MiB/s wr, 426 op/s 2024-03-16T10:21:30.248 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:29 smithi077 ceph-mon[32400]: pgmap v482: 97 pgs: 97 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.2 MiB/s wr, 426 op/s 2024-03-16T10:21:30.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:29 smithi193 ceph-mon[36311]: pgmap v482: 97 pgs: 97 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.2 MiB/s wr, 426 op/s 2024-03-16T10:21:31.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:21:31.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:21:31.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.468+0000 7f83d5148640 1 -- 172.21.15.77:0/4288936365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83d01006f0 msgr2=0x7f83d0100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:31.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.468+0000 7f83d5148640 1 --2- 172.21.15.77:0/4288936365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83d01006f0 0x7f83d0100b10 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f83c0009900 tx=0x7f83c002bc30 comp rx=0 tx=0).stop 2024-03-16T10:21:31.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.468+0000 7f83d5148640 1 -- 172.21.15.77:0/4288936365 shutdown_connections 2024-03-16T10:21:31.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.468+0000 7f83d5148640 1 --2- 172.21.15.77:0/4288936365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83d01022a0 0x7f83d0108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.468+0000 7f83d5148640 1 --2- 172.21.15.77:0/4288936365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83d01018e0 0x7f83d0101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.468+0000 7f83d5148640 1 --2- 172.21.15.77:0/4288936365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83d01006f0 0x7f83d0100b10 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.468+0000 7f83d5148640 1 -- 172.21.15.77:0/4288936365 >> 172.21.15.77:0/4288936365 conn(0x7f83d00fbc80 msgr2=0x7f83d00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:31.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.468+0000 7f83d5148640 1 -- 172.21.15.77:0/4288936365 shutdown_connections 2024-03-16T10:21:31.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.468+0000 7f83d5148640 1 -- 172.21.15.77:0/4288936365 wait complete. 2024-03-16T10:21:31.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.469+0000 7f83d5148640 1 Processor -- start 2024-03-16T10:21:31.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.469+0000 7f83d5148640 1 -- start start 2024-03-16T10:21:31.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.469+0000 7f83d5148640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83d01006f0 0x7f83d0196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.469+0000 7f83d5148640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83d01018e0 0x7f83d0196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.469+0000 7f83d5148640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83d01022a0 0x7f83d019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.469+0000 7f83d5148640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f83d010c360 con 0x7f83d01022a0 2024-03-16T10:21:31.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.469+0000 7f83d5148640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f83d010c1e0 con 0x7f83d01018e0 2024-03-16T10:21:31.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.469+0000 7f83d5148640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f83d010c4e0 con 0x7f83d01006f0 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83d01018e0 0x7f83d0196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83cf59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83d01022a0 0x7f83d019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83ced9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83d01006f0 0x7f83d0196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83d01018e0 0x7f83d0196f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:52956/0 (socket says 172.21.15.77:52956) 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83ce59c640 1 -- 172.21.15.77:0/3994608382 learned_addr learned my addr 172.21.15.77:0/3994608382 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83cf59e640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83d01022a0 0x7f83d019dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59990/0 (socket says 172.21.15.77:59990) 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83ced9d640 1 -- 172.21.15.77:0/3994608382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83d01018e0 msgr2=0x7f83d0196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83ced9d640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83d01018e0 0x7f83d0196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83ced9d640 1 -- 172.21.15.77:0/3994608382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83d01022a0 msgr2=0x7f83d019dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83ced9d640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83d01022a0 0x7f83d019dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83ced9d640 1 -- 172.21.15.77:0/3994608382 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d019e650 con 0x7f83d01006f0 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83ce59c640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83d01018e0 0x7f83d0196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83cf59e640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83d01022a0 0x7f83d019dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:31.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83ced9d640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83d01006f0 0x7f83d0196a30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f83c002c170 tx=0x7f83c002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:31.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.470+0000 7f83b7fff640 1 -- 172.21.15.77:0/3994608382 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c0041070 con 0x7f83d01006f0 2024-03-16T10:21:31.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.471+0000 7f83d5148640 1 -- 172.21.15.77:0/3994608382 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d019e8e0 con 0x7f83d01006f0 2024-03-16T10:21:31.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.471+0000 7f83d5148640 1 -- 172.21.15.77:0/3994608382 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d019edf0 con 0x7f83d01006f0 2024-03-16T10:21:31.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.471+0000 7f83b7fff640 1 -- 172.21.15.77:0/3994608382 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c0004c40 con 0x7f83d01006f0 2024-03-16T10:21:31.474 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.471+0000 7f83b7fff640 1 -- 172.21.15.77:0/3994608382 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c0003970 con 0x7f83d01006f0 2024-03-16T10:21:31.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.473+0000 7f83d7e0c740 1 Processor -- start 2024-03-16T10:21:31.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.473+0000 7f83d7e0c740 1 -- start start 2024-03-16T10:21:31.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.473+0000 7f83d7e0c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2a170 0x560956d2a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.473+0000 7f83d7e0c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2ab60 0x560956d2f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.473+0000 7f83d7e0c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2fcd0 0x560956d32160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.473+0000 7f83d7e0c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560956c8f500 con 0x560956d2a170 2024-03-16T10:21:31.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.473+0000 7f83d7e0c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x560956c37be0 con 0x560956d2fcd0 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.473+0000 7f83d7e0c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560956c8b5a0 con 0x560956d2ab60 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.473+0000 7f83cf59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2a170 0x560956d2a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.473+0000 7f83cf59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2a170 0x560956d2a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60002/0 (socket says 172.21.15.77:60002) 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.473+0000 7f83cf59e640 1 -- 172.21.15.77:0/3914165537 learned_addr learned my addr 172.21.15.77:0/3914165537 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.473+0000 7f83ce59c640 1 --2- 172.21.15.77:0/3914165537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2fcd0 0x560956d32160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.474+0000 7f83cf59e640 1 -- 172.21.15.77:0/3914165537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2ab60 msgr2=0x560956d2f660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.474+0000 7f83cf59e640 1 --2- 172.21.15.77:0/3914165537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2ab60 0x560956d2f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.474+0000 7f83cf59e640 1 -- 172.21.15.77:0/3914165537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2fcd0 msgr2=0x560956d32160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.474+0000 7f83cf59e640 1 --2- 172.21.15.77:0/3914165537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2fcd0 0x560956d32160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.474+0000 7f83cf59e640 1 -- 172.21.15.77:0/3914165537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560956c4b4b0 con 0x560956d2a170 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.474+0000 7f83ce59c640 1 --2- 172.21.15.77:0/3914165537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2fcd0 0x560956d32160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.474+0000 7f83cf59e640 1 --2- 172.21.15.77:0/3914165537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2a170 0x560956d2a590 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f83c80117e0 tx=0x7f83c8011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5dd663b7d8143 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.474+0000 7f83b7fff640 1 -- 172.21.15.77:0/3994608382 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f83c0044070 con 0x7f83d01006f0 2024-03-16T10:21:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.474+0000 7f83b57fa640 1 -- 172.21.15.77:0/3914165537 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8011310 con 0x560956d2a170 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.474+0000 7f83b57fa640 1 -- 172.21.15.77:0/3914165537 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c8003e50 con 0x560956d2a170 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.474+0000 7f83b57fa640 1 -- 172.21.15.77:0/3914165537 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c80045a0 con 0x560956d2a170 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.474+0000 7f83b7fff640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f83b80a2800 0x7f83b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.475+0000 7f83ced9d640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f83b80a2800 0x7f83b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.475+0000 7f83b7fff640 1 -- 172.21.15.77:0/3994608382 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f83c00f0290 con 0x7f83d01006f0 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.475+0000 7f83d7e0c740 1 -- 172.21.15.77:0/3914165537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2a170 msgr2=0x560956d2a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.475+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/3914165537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2a170 0x560956d2a590 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f83c80117e0 tx=0x7f83c8011ca0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.475+0000 7f83d7e0c740 1 -- 172.21.15.77:0/3914165537 shutdown_connections 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.475+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/3914165537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2fcd0 0x560956d32160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.475+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/3914165537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2ab60 0x560956d2f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.475+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/3914165537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2a170 0x560956d2a590 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.475+0000 7f83d7e0c740 1 -- 172.21.15.77:0/3914165537 >> 172.21.15.77:0/3914165537 conn(0x560956d256e0 msgr2=0x560956d25ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.475+0000 7f83ced9d640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f83b80a2800 0x7f83b80a4cc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x560956d2af60 tx=0x7f83c002bc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.475+0000 7f83d7e0c740 1 -- 172.21.15.77:0/3914165537 shutdown_connections 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.475+0000 7f83d7e0c740 1 -- 172.21.15.77:0/3914165537 wait complete. 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83d7e0c740 1 -- start start 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83d7e0c740 1 Processor -- start 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83d7e0c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2a170 0x560956d3cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83d7e0c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2ab60 0x560956d3d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83d7e0c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 0x560956d44490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83d7e0c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560956d32c80 con 0x560956d2fcd0 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83d7e0c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x560956c4b160 con 0x560956d2ab60 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83d7e0c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560956d32e00 con 0x560956d2a170 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83d7e0c740 10 client.0 _ll_register_callbacks cb 0x560956c2f150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83cf59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 0x560956d44490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2ab60 0x560956d3d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83cf59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 0x560956d44490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60010/0 (socket says 172.21.15.77:60010) 2024-03-16T10:21:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83cf59e640 1 -- 172.21.15.77:0/2953864660 learned_addr learned my addr 172.21.15.77:0/2953864660 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.476+0000 7f83cf59e640 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2a170 0x560956d3cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83ce59c640 1 -- 172.21.15.77:0/2953864660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2a170 msgr2=0x560956d3cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83ce59c640 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2a170 0x560956d3cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83ce59c640 1 -- 172.21.15.77:0/2953864660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 msgr2=0x560956d44490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83ce59c640 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 0x560956d44490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83ce59c640 1 -- 172.21.15.77:0/2953864660 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560956a6d7d0 con 0x560956d2ab60 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83cf59e640 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 0x560956d44490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83ce59c640 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2ab60 0x560956d3d450 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f83c4009640 tx=0x7f83c400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7149cecf8456c7e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83b2ff5640 1 -- 172.21.15.77:0/2953864660 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c40120d0 con 0x560956d2ab60 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83b2ff5640 1 -- 172.21.15.77:0/2953864660 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f83c4005020 con 0x560956d2ab60 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83d7e0c740 1 -- 172.21.15.77:0/2953864660 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83c8003e50 con 0x560956d2ab60 2024-03-16T10:21:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83b2ff5640 1 -- 172.21.15.77:0/2953864660 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f83c4012270 con 0x560956d2ab60 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83d7e0c740 10 client.26528 fetch_fsmap learned FSMap version 34 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.477+0000 7f83d7e0c740 1 -- 172.21.15.77:0/2953864660 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5609569a33f0 con 0x560956d2ab60 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.478+0000 7f83b2ff5640 1 -- 172.21.15.77:0/2953864660 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c8003e50 con 0x560956d2ab60 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.478+0000 7f83b2ff5640 1 -- 172.21.15.77:0/2953864660 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f83c401c470 con 0x560956d2ab60 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.478+0000 7f83d7e0c740 10 client.26528 fetch_fsmap finished waiting for FSMap version 34 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.478+0000 7f83d7e0c740 10 client.26528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.478+0000 7f83d7e0c740 20 client.26528 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.478+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560956d48ab0 0x560956d4aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.478+0000 7f83d7e0c740 4 client.26528 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.478+0000 7f83d7e0c740 1 -- 172.21.15.77:0/2953864660 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560956c9ca40 con 0x560956d48ab0 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.478+0000 7f83cf59e640 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560956d48ab0 0x560956d4aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.479+0000 7f83b2ff5640 10 client.26528 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.479+0000 7f83cf59e640 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560956d48ab0 0x560956d4aef0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f155ec7cbb01c20 server_cookie=62da425a430eb54 in_seq=0 out_seq=0 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.479+0000 7f83b2ff5640 1 -- 172.21.15.77:0/2953864660 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560956c9ca40 con 0x560956d48ab0 2024-03-16T10:21:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.479+0000 7f83b2ff5640 10 client.26528 handle_command_reply: tid=0 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.479+0000 7f83d7e0c740 1 client.26528 shutdown 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.480+0000 7f83d7e0c740 1 -- 172.21.15.77:0/2953864660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2ab60 msgr2=0x560956d3d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.480+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2ab60 0x560956d3d450 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f83c4009640 tx=0x7f83c400f790 comp rx=0 tx=0).stop 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.480+0000 7f83d7e0c740 1 -- 172.21.15.77:0/2953864660 shutdown_connections 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.480+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560956d48ab0 0x560956d4aef0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.480+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 0x560956d44490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.480+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2ab60 0x560956d3d450 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.480+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/2953864660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2a170 0x560956d3cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.480+0000 7f83b2ff5640 0 client.26528 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.480+0000 7f83d7e0c740 1 -- 172.21.15.77:0/2953864660 >> 172.21.15.77:0/2953864660 conn(0x560956d256e0 msgr2=0x560956d24900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.480+0000 7f83d7e0c740 1 -- 172.21.15.77:0/2953864660 shutdown_connections 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.480+0000 7f83d7e0c740 1 -- 172.21.15.77:0/2953864660 wait complete. 2024-03-16T10:21:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.480+0000 7f83d7e0c740 20 client.26528 trim_cache size 0 max 16384 2024-03-16T10:21:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.497+0000 7f83d7e0c740 1 Processor -- start 2024-03-16T10:21:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.497+0000 7f83d7e0c740 1 -- start start 2024-03-16T10:21:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.497+0000 7f83d7e0c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2a170 0x560956d3c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.497+0000 7f83d7e0c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2ab60 0x560956d44450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.497+0000 7f83d7e0c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 0x560956d3c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.497+0000 7f83d7e0c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc00e330 con 0x560956d2fcd0 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.497+0000 7f83d7e0c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc00ff30 con 0x560956d2a170 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.497+0000 7f83d7e0c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc00ea00 con 0x560956d2ab60 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2ab60 0x560956d44450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83cf59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 0x560956d3c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2ab60 0x560956d44450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:36732/0 (socket says 172.21.15.77:36732) 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83ce59c640 1 -- 172.21.15.77:0/1826617434 learned_addr learned my addr 172.21.15.77:0/1826617434 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83cf59e640 1 --2- 172.21.15.77:0/1826617434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2a170 0x560956d3c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83cf59e640 1 -- 172.21.15.77:0/1826617434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2ab60 msgr2=0x560956d44450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83cf59e640 1 --2- 172.21.15.77:0/1826617434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2ab60 0x560956d44450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83cf59e640 1 -- 172.21.15.77:0/1826617434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2a170 msgr2=0x560956d3c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83cf59e640 1 --2- 172.21.15.77:0/1826617434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2a170 0x560956d3c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83cf59e640 1 -- 172.21.15.77:0/1826617434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609569630f0 con 0x560956d2fcd0 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83ce59c640 1 --2- 172.21.15.77:0/1826617434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2ab60 0x560956d44450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83cf59e640 1 --2- 172.21.15.77:0/1826617434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 0x560956d3c760 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x560956d4b770 tx=0x7f83c800ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185bdeda0d9853a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83b4ff9640 1 -- 172.21.15.77:0/1826617434 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8025040 con 0x560956d2fcd0 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83b4ff9640 1 -- 172.21.15.77:0/1826617434 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c802f520 con 0x560956d2fcd0 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.498+0000 7f83b4ff9640 1 -- 172.21.15.77:0/1826617434 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c80057f0 con 0x560956d2fcd0 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.499+0000 7f83d7e0c740 1 -- 172.21.15.77:0/1826617434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 msgr2=0x560956d3c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.499+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/1826617434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 0x560956d3c760 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x560956d4b770 tx=0x7f83c800ff50 comp rx=0 tx=0).stop 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.499+0000 7f83d7e0c740 1 -- 172.21.15.77:0/1826617434 shutdown_connections 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.499+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/1826617434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2fcd0 0x560956d3c760 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.499+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/1826617434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2ab60 0x560956d44450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.499+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/1826617434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2a170 0x560956d3c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.499+0000 7f83d7e0c740 1 -- 172.21.15.77:0/1826617434 >> 172.21.15.77:0/1826617434 conn(0x560956d256e0 msgr2=0x560956d25ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.499+0000 7f83d7e0c740 1 -- 172.21.15.77:0/1826617434 shutdown_connections 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.499+0000 7f83d7e0c740 1 -- 172.21.15.77:0/1826617434 wait complete. 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83d7e0c740 1 -- start start 2024-03-16T10:21:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83d7e0c740 1 Processor -- start 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83d7e0c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2a170 0x560956d41870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83d7e0c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2ab60 0x560956d41db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83d7e0c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2fcd0 0x560956d42dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83d7e0c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc00f860 con 0x560956d2ab60 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83d7e0c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc00f190 con 0x560956d2fcd0 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83d7e0c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc00c310 con 0x560956d2a170 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83d7e0c740 10 client.0 _ll_register_callbacks cb 0x7f83c4009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2ab60 0x560956d41db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2ab60 0x560956d41db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60020/0 (socket says 172.21.15.77:60020) 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83ce59c640 1 -- 172.21.15.77:0/4175109673 learned_addr learned my addr 172.21.15.77:0/4175109673 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83cf59e640 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2a170 0x560956d41870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83cf59e640 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2fcd0 0x560956d42dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83ce59c640 1 -- 172.21.15.77:0/4175109673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2a170 msgr2=0x560956d41870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83ce59c640 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2a170 0x560956d41870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83ce59c640 1 -- 172.21.15.77:0/4175109673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2fcd0 msgr2=0x560956d42dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83ce59c640 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2fcd0 0x560956d42dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83ce59c640 1 -- 172.21.15.77:0/4175109673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560956c569d0 con 0x560956d2ab60 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83cf59e640 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2fcd0 0x560956d42dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83ce59c640 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2ab60 0x560956d41db0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f83c401cad0 tx=0x7f83c401c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95587cf4f387e28e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.500+0000 7f83b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83b2ff5640 1 -- 172.21.15.77:0/4175109673 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c40128a0 con 0x560956d2ab60 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83b2ff5640 1 -- 172.21.15.77:0/4175109673 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f83c4005250 con 0x560956d2ab60 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83d7e0c740 1 -- 172.21.15.77:0/4175109673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83c802f520 con 0x560956d2ab60 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83b2ff5640 1 -- 172.21.15.77:0/4175109673 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f83c4005450 con 0x560956d2ab60 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83d7e0c740 10 client.16770 fetch_fsmap learned FSMap version 34 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83d7e0c740 1 -- 172.21.15.77:0/4175109673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560956d44590 con 0x560956d2ab60 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83b2ff5640 1 -- 172.21.15.77:0/4175109673 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c802f520 con 0x560956d2ab60 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83b2ff5640 1 -- 172.21.15.77:0/4175109673 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7f83c40304e0 con 0x560956d2ab60 2024-03-16T10:21:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83d7e0c740 10 client.16770 fetch_fsmap finished waiting for FSMap version 34 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83d7e0c740 10 client.16770 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83d7e0c740 20 client.16770 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560956daf480 0x560956db18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83d7e0c740 4 client.16770 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83d7e0c740 1 -- 172.21.15.77:0/4175109673 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560956c96d90 con 0x560956daf480 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.501+0000 7f83cf59e640 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560956daf480 0x560956db18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83b2ff5640 10 client.16770 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83cf59e640 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560956daf480 0x560956db18a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=323937eecc2a2ee7 server_cookie=2cec4ea9e892fd1 in_seq=0 out_seq=0 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83b2ff5640 1 -- 172.21.15.77:0/4175109673 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560956c96d90 con 0x560956daf480 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83b2ff5640 10 client.16770 handle_command_reply: tid=0 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83d7e0c740 1 client.16770 shutdown 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83d7e0c740 1 -- 172.21.15.77:0/4175109673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2ab60 msgr2=0x560956d41db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2ab60 0x560956d41db0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f83c401cad0 tx=0x7f83c401c680 comp rx=0 tx=0).stop 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83d7e0c740 1 -- 172.21.15.77:0/4175109673 shutdown_connections 2024-03-16T10:21:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560956d2fcd0 0x560956d42dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560956daf480 0x560956db18a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560956d2ab60 0x560956d41db0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83d7e0c740 1 --2- 172.21.15.77:0/4175109673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560956d2a170 0x560956d41870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83b2ff5640 0 client.16770 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83d7e0c740 1 -- 172.21.15.77:0/4175109673 >> 172.21.15.77:0/4175109673 conn(0x560956d256e0 msgr2=0x560956d291d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83d7e0c740 1 -- 172.21.15.77:0/4175109673 shutdown_connections 2024-03-16T10:21:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83d7e0c740 1 -- 172.21.15.77:0/4175109673 wait complete. 2024-03-16T10:21:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.502+0000 7f83d7e0c740 20 client.16770 trim_cache size 0 max 16384 2024-03-16T10:21:31.510 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:21:31.510 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:21:31.510 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:21:31.511 INFO:teuthology.orchestra.run.smithi077.stdout: "127457ea-db1f-4f0c-887d-a265bb630cd6": { 2024-03-16T10:21:31.511 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:21:31.511 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "127457ea-db1f-4f0c-887d-a265bb630cd6", 2024-03-16T10:21:31.511 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:21:31.511 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:21:31.511 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:21:31.511 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:21:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.503+0000 7f83d5148640 1 -- 172.21.15.77:0/3994608382 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f83b80a2800 msgr2=0x7f83b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.503+0000 7f83d5148640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f83b80a2800 0x7f83b80a4cc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x560956d2af60 tx=0x7f83c002bc30 comp rx=0 tx=0).stop 2024-03-16T10:21:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.503+0000 7f83d5148640 1 -- 172.21.15.77:0/3994608382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83d01006f0 msgr2=0x7f83d0196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.503+0000 7f83d5148640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83d01006f0 0x7f83d0196a30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f83c002c170 tx=0x7f83c002c650 comp rx=0 tx=0).stop 2024-03-16T10:21:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.503+0000 7f83d5148640 1 -- 172.21.15.77:0/3994608382 shutdown_connections 2024-03-16T10:21:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.503+0000 7f83d5148640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83d01022a0 0x7f83d019dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.503+0000 7f83d5148640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83d01018e0 0x7f83d0196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.503+0000 7f83d5148640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f83b80a2800 0x7f83b80a4cc0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.503+0000 7f83d5148640 1 --2- 172.21.15.77:0/3994608382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83d01006f0 0x7f83d0196a30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.503+0000 7f83d5148640 1 -- 172.21.15.77:0/3994608382 >> 172.21.15.77:0/3994608382 conn(0x7f83d00fbc80 msgr2=0x7f83d0104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:31.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.503+0000 7f83d5148640 1 -- 172.21.15.77:0/3994608382 shutdown_connections 2024-03-16T10:21:31.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:21:31.503+0000 7f83d5148640 1 -- 172.21.15.77:0/3994608382 wait complete. 2024-03-16T10:21:31.514 INFO:tasks.cephfs.filesystem:scrub status for tag:127457ea-db1f-4f0c-887d-a265bb630cd6 - {'path': '/', 'tag': '127457ea-db1f-4f0c-887d-a265bb630cd6', 'options': 'recursive,force'} 2024-03-16T10:21:32.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:31 smithi192 ceph-mon[36354]: pgmap v483: 97 pgs: 97 active+clean; 522 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.0 MiB/s wr, 437 op/s 2024-03-16T10:21:32.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:31 smithi077 ceph-mon[32400]: pgmap v483: 97 pgs: 97 active+clean; 522 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.0 MiB/s wr, 437 op/s 2024-03-16T10:21:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:31 smithi193 ceph-mon[36311]: pgmap v483: 97 pgs: 97 active+clean; 522 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.0 MiB/s wr, 437 op/s 2024-03-16T10:21:34.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:33 smithi192 ceph-mon[36354]: pgmap v484: 97 pgs: 97 active+clean; 520 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.0 MiB/s wr, 452 op/s 2024-03-16T10:21:34.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:33 smithi077 ceph-mon[32400]: pgmap v484: 97 pgs: 97 active+clean; 520 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.0 MiB/s wr, 452 op/s 2024-03-16T10:21:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:33 smithi193 ceph-mon[36311]: pgmap v484: 97 pgs: 97 active+clean; 520 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.0 MiB/s wr, 452 op/s 2024-03-16T10:21:35.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:34 smithi077 ceph-mon[32400]: pgmap v485: 97 pgs: 97 active+clean; 528 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 337 op/s 2024-03-16T10:21:35.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:34 smithi193 ceph-mon[36311]: pgmap v485: 97 pgs: 97 active+clean; 528 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 337 op/s 2024-03-16T10:21:35.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:34 smithi192 ceph-mon[36354]: pgmap v485: 97 pgs: 97 active+clean; 528 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 337 op/s 2024-03-16T10:21:37.104 INFO:tasks.workunit.client.0.smithi077.stdout: 105 1858279 356 1234655 365 1062484 1231 2024-03-16T10:21:38.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:37 smithi192 ceph-mon[36354]: pgmap v486: 97 pgs: 97 active+clean; 527 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 262 op/s 2024-03-16T10:21:38.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:37 smithi077 ceph-mon[32400]: pgmap v486: 97 pgs: 97 active+clean; 527 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 262 op/s 2024-03-16T10:21:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:37 smithi193 ceph-mon[36311]: pgmap v486: 97 pgs: 97 active+clean; 527 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 262 op/s 2024-03-16T10:21:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.099+0000 7fe069110640 1 -- 172.21.15.77:0/3784810209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe064100700 msgr2=0x7fe064100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.099+0000 7fe069110640 1 --2- 172.21.15.77:0/3784810209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe064100700 0x7fe064100b20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe05c009900 tx=0x7fe05c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:21:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.099+0000 7fe069110640 1 -- 172.21.15.77:0/3784810209 shutdown_connections 2024-03-16T10:21:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.099+0000 7fe069110640 1 --2- 172.21.15.77:0/3784810209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0641022b0 0x7fe064108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.099+0000 7fe069110640 1 --2- 172.21.15.77:0/3784810209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0641018f0 0x7fe064101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.099+0000 7fe069110640 1 --2- 172.21.15.77:0/3784810209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe064100700 0x7fe064100b20 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.099+0000 7fe069110640 1 -- 172.21.15.77:0/3784810209 >> 172.21.15.77:0/3784810209 conn(0x7fe0640fbc70 msgr2=0x7fe0640fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.099+0000 7fe069110640 1 -- 172.21.15.77:0/3784810209 shutdown_connections 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.099+0000 7fe069110640 1 -- 172.21.15.77:0/3784810209 wait complete. 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.099+0000 7fe069110640 1 Processor -- start 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe069110640 1 -- start start 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe069110640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe064100700 0x7fe0640699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe069110640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0641018f0 0x7fe06419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe069110640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0641022b0 0x7fe0641a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe069110640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe06410c380 con 0x7fe064100700 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe069110640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe06410c200 con 0x7fe0641018f0 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe069110640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe06410c500 con 0x7fe0641022b0 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe062d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe064100700 0x7fe0640699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe062d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe064100700 0x7fe0640699d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43138/0 (socket says 172.21.15.77:43138) 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe062d9d640 1 -- 172.21.15.77:0/2328766097 learned_addr learned my addr 172.21.15.77:0/2328766097 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe06259c640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0641018f0 0x7fe06419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe06359e640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0641022b0 0x7fe0641a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe062d9d640 1 -- 172.21.15.77:0/2328766097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0641022b0 msgr2=0x7fe0641a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe062d9d640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0641022b0 0x7fe0641a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe062d9d640 1 -- 172.21.15.77:0/2328766097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0641018f0 msgr2=0x7fe06419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe062d9d640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0641018f0 0x7fe06419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe062d9d640 1 -- 172.21.15.77:0/2328766097 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0641a1920 con 0x7fe064100700 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe06259c640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0641018f0 0x7fe06419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.100+0000 7fe062d9d640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe064100700 0x7fe0640699d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe05c000c00 tx=0x7fe05c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.101+0000 7fe04bfff640 1 -- 172.21.15.77:0/2328766097 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c041070 con 0x7fe064100700 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.101+0000 7fe069110640 1 -- 172.21.15.77:0/2328766097 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0641a1b50 con 0x7fe064100700 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.101+0000 7fe04bfff640 1 -- 172.21.15.77:0/2328766097 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe05c004450 con 0x7fe064100700 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.101+0000 7fe04bfff640 1 -- 172.21.15.77:0/2328766097 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c034440 con 0x7fe064100700 2024-03-16T10:21:40.104 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.101+0000 7fe069110640 1 -- 172.21.15.77:0/2328766097 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0641a2040 con 0x7fe064100700 2024-03-16T10:21:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.103+0000 7fe04bfff640 1 -- 172.21.15.77:0/2328766097 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fe05c044070 con 0x7fe064100700 2024-03-16T10:21:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06bdde740 1 Processor -- start 2024-03-16T10:21:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06bdde740 1 -- start start 2024-03-16T10:21:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06bdde740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c61760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06bdde740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c61d30 0x564723c66830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06bdde740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c66ea0 0x564723c69330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06bdde740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564723bc6a40 con 0x564723c61340 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06bdde740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564723b6f920 con 0x564723c61d30 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06bdde740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564723bc2ae0 con 0x564723c66ea0 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c61760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c61760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43148/0 (socket says 172.21.15.77:43148) 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06359e640 1 -- 172.21.15.77:0/2164358643 learned_addr learned my addr 172.21.15.77:0/2164358643 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06259c640 1 --2- 172.21.15.77:0/2164358643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c66ea0 0x564723c69330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe062d9d640 1 --2- 172.21.15.77:0/2164358643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c61d30 0x564723c66830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe04bfff640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe04c0a28d0 0x7fe04c0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06359e640 1 -- 172.21.15.77:0/2164358643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c66ea0 msgr2=0x564723c69330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06359e640 1 --2- 172.21.15.77:0/2164358643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c66ea0 0x564723c69330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06359e640 1 -- 172.21.15.77:0/2164358643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c61d30 msgr2=0x564723c66830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06359e640 1 --2- 172.21.15.77:0/2164358643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c61d30 0x564723c66830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06359e640 1 -- 172.21.15.77:0/2164358643 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564723b87460 con 0x564723c61340 2024-03-16T10:21:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.104+0000 7fe06259c640 1 --2- 172.21.15.77:0/2164358643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c66ea0 0x564723c69330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.105+0000 7fe04bfff640 1 -- 172.21.15.77:0/2328766097 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe05c0f0430 con 0x7fe064100700 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.105+0000 7fe06359e640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe04c0a28d0 0x7fe04c0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.105+0000 7fe06359e640 1 --2- 172.21.15.77:0/2164358643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c61760 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe054009ca0 tx=0x7fe054011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad54de54084fff44 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.105+0000 7fe0497fa640 1 -- 172.21.15.77:0/2164358643 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05401b870 con 0x564723c61340 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.105+0000 7fe0497fa640 1 -- 172.21.15.77:0/2164358643 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe054019db0 con 0x564723c61340 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.105+0000 7fe0497fa640 1 -- 172.21.15.77:0/2164358643 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05400ebc0 con 0x564723c61340 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.106+0000 7fe06bdde740 1 -- 172.21.15.77:0/2164358643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 msgr2=0x564723c61760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.106+0000 7fe06359e640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe04c0a28d0 0x7fe04c0a4d90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe054020000 tx=0x7fe054021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.106+0000 7fe06bdde740 1 --2- 172.21.15.77:0/2164358643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c61760 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe054009ca0 tx=0x7fe054011810 comp rx=0 tx=0).stop 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.106+0000 7fe06bdde740 1 -- 172.21.15.77:0/2164358643 shutdown_connections 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.106+0000 7fe06bdde740 1 --2- 172.21.15.77:0/2164358643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c66ea0 0x564723c69330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.106+0000 7fe06bdde740 1 --2- 172.21.15.77:0/2164358643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c61d30 0x564723c66830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.106+0000 7fe06bdde740 1 --2- 172.21.15.77:0/2164358643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c61760 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.106+0000 7fe06bdde740 1 -- 172.21.15.77:0/2164358643 >> 172.21.15.77:0/2164358643 conn(0x564723c5c8b0 msgr2=0x564723c5ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.106+0000 7fe06bdde740 1 -- 172.21.15.77:0/2164358643 shutdown_connections 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.106+0000 7fe06bdde740 1 -- 172.21.15.77:0/2164358643 wait complete. 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.106+0000 7fe06bdde740 1 -- start start 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.106+0000 7fe06bdde740 1 Processor -- start 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe06bdde740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c6b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe06bdde740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c61d30 0x564723c6bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe06bdde740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c66ea0 0x564723c72bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe06bdde740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564723c69e50 con 0x564723c61340 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe06bdde740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564723b82d70 con 0x564723c66ea0 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe06bdde740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564723c69fd0 con 0x564723c61d30 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe062d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c6b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe06bdde740 10 client.0 _ll_register_callbacks cb 0x5647238daa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe062d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c6b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43160/0 (socket says 172.21.15.77:43160) 2024-03-16T10:21:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe062d9d640 1 -- 172.21.15.77:0/597879327 learned_addr learned my addr 172.21.15.77:0/597879327 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe06259c640 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c61d30 0x564723c6bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe06359e640 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c66ea0 0x564723c72bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe062d9d640 1 -- 172.21.15.77:0/597879327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c61d30 msgr2=0x564723c6bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe062d9d640 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c61d30 0x564723c6bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe062d9d640 1 -- 172.21.15.77:0/597879327 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c66ea0 msgr2=0x564723c72bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe062d9d640 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c66ea0 0x564723c72bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe062d9d640 1 -- 172.21.15.77:0/597879327 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647238db210 con 0x564723c61340 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe062d9d640 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c6b640 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe05c005ab0 tx=0x7fe05c003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ec3b134e8726f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe046ff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe046ff5640 1 -- 172.21.15.77:0/597879327 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c04b070 con 0x564723c61340 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe046ff5640 1 -- 172.21.15.77:0/597879327 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe05c0c30f0 con 0x564723c61340 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe06bdde740 1 -- 172.21.15.77:0/597879327 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe054019db0 con 0x564723c61340 2024-03-16T10:21:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe046ff5640 1 -- 172.21.15.77:0/597879327 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fe05c0bcd70 con 0x564723c61340 2024-03-16T10:21:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.107+0000 7fe06bdde740 10 client.16782 fetch_fsmap learned FSMap version 34 2024-03-16T10:21:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.108+0000 7fe06bdde740 1 -- 172.21.15.77:0/597879327 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5647239a7620 con 0x564723c61340 2024-03-16T10:21:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.108+0000 7fe046ff5640 1 -- 172.21.15.77:0/597879327 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe054019db0 con 0x564723c61340 2024-03-16T10:21:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.108+0000 7fe046ff5640 1 -- 172.21.15.77:0/597879327 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fe05c0c5170 con 0x564723c61340 2024-03-16T10:21:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.108+0000 7fe06bdde740 10 client.16782 fetch_fsmap finished waiting for FSMap version 34 2024-03-16T10:21:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.108+0000 7fe06bdde740 10 client.16782 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.108+0000 7fe06bdde740 20 client.16782 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.108+0000 7fe06bdde740 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564723c77470 0x564723c798c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.108+0000 7fe06bdde740 4 client.16782 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:21:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.108+0000 7fe06bdde740 1 -- 172.21.15.77:0/597879327 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564723bd3f80 con 0x564723c77470 2024-03-16T10:21:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.109+0000 7fe06259c640 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564723c77470 0x564723c798c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.109+0000 7fe046ff5640 10 client.16782 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:40.116 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.113+0000 7fe06259c640 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564723c77470 0x564723c798c0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ec1ec353fef61e server_cookie=33aeea49f2744fb4 in_seq=0 out_seq=0 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe046ff5640 1 -- 172.21.15.77:0/597879327 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564723bd3f80 con 0x564723c77470 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe046ff5640 10 client.16782 handle_command_reply: tid=0 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe06bdde740 1 client.16782 shutdown 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe06bdde740 1 -- 172.21.15.77:0/597879327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 msgr2=0x564723c6b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe06bdde740 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c6b640 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe05c005ab0 tx=0x7fe05c003f80 comp rx=0 tx=0).stop 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe06bdde740 1 -- 172.21.15.77:0/597879327 shutdown_connections 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe06bdde740 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c66ea0 0x564723c72bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe06bdde740 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c61d30 0x564723c6bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe06bdde740 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564723c77470 0x564723c798c0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe06bdde740 1 --2- 172.21.15.77:0/597879327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c6b640 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe06bdde740 1 -- 172.21.15.77:0/597879327 >> 172.21.15.77:0/597879327 conn(0x564723c5c8b0 msgr2=0x564723c5bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe046ff5640 0 client.16782 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe06bdde740 1 -- 172.21.15.77:0/597879327 shutdown_connections 2024-03-16T10:21:40.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe06bdde740 1 -- 172.21.15.77:0/597879327 wait complete. 2024-03-16T10:21:40.118 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.114+0000 7fe06bdde740 20 client.16782 trim_cache size 0 max 16384 2024-03-16T10:21:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.137+0000 7fe06bdde740 1 Processor -- start 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.137+0000 7fe06bdde740 1 -- start start 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.137+0000 7fe06bdde740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c6bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.137+0000 7fe06bdde740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c61d30 0x564723c72b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.137+0000 7fe06bdde740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c66ea0 0x564723c60220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.137+0000 7fe06bdde740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe0641aadb0 con 0x564723c61340 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.137+0000 7fe06bdde740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe0641ac9b0 con 0x564723c61d30 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.137+0000 7fe06bdde740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe0641ab480 con 0x564723c66ea0 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.137+0000 7fe062d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c6bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.137+0000 7fe062d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c6bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43174/0 (socket says 172.21.15.77:43174) 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.137+0000 7fe06359e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c66ea0 0x564723c60220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.138+0000 7fe062d9d640 1 -- 172.21.15.77:0/4199471732 learned_addr learned my addr 172.21.15.77:0/4199471732 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.138+0000 7fe06259c640 1 --2- 172.21.15.77:0/4199471732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c61d30 0x564723c72b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.138+0000 7fe06359e640 1 -- 172.21.15.77:0/4199471732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c61d30 msgr2=0x564723c72b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.138+0000 7fe06359e640 1 --2- 172.21.15.77:0/4199471732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c61d30 0x564723c72b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.138+0000 7fe06359e640 1 -- 172.21.15.77:0/4199471732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 msgr2=0x564723c6bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.138+0000 7fe06359e640 1 --2- 172.21.15.77:0/4199471732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c6bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.138+0000 7fe06359e640 1 -- 172.21.15.77:0/4199471732 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564723c5d620 con 0x564723c66ea0 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.138+0000 7fe062d9d640 1 --2- 172.21.15.77:0/4199471732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c6bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.138+0000 7fe06359e640 1 --2- 172.21.15.77:0/4199471732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c66ea0 0x564723c60220 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe054011d10 tx=0x7fe054027000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4715a90ca1f7942 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.138+0000 7fe048ff9640 1 -- 172.21.15.77:0/4199471732 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054009920 con 0x564723c66ea0 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.138+0000 7fe048ff9640 1 -- 172.21.15.77:0/4199471732 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe054019850 con 0x564723c66ea0 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.138+0000 7fe048ff9640 1 -- 172.21.15.77:0/4199471732 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054004510 con 0x564723c66ea0 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.139+0000 7fe06bdde740 1 -- 172.21.15.77:0/4199471732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c66ea0 msgr2=0x564723c60220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.139+0000 7fe06bdde740 1 --2- 172.21.15.77:0/4199471732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c66ea0 0x564723c60220 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe054011d10 tx=0x7fe054027000 comp rx=0 tx=0).stop 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.139+0000 7fe06bdde740 1 -- 172.21.15.77:0/4199471732 shutdown_connections 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.139+0000 7fe06bdde740 1 --2- 172.21.15.77:0/4199471732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c66ea0 0x564723c60220 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.139+0000 7fe06bdde740 1 --2- 172.21.15.77:0/4199471732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c61d30 0x564723c72b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.139+0000 7fe06bdde740 1 --2- 172.21.15.77:0/4199471732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c6bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.139+0000 7fe06bdde740 1 -- 172.21.15.77:0/4199471732 >> 172.21.15.77:0/4199471732 conn(0x564723c5c8b0 msgr2=0x564723c5fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.139+0000 7fe06bdde740 1 -- 172.21.15.77:0/4199471732 shutdown_connections 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.139+0000 7fe06bdde740 1 -- 172.21.15.77:0/4199471732 wait complete. 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.139+0000 7fe06bdde740 1 -- start start 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06bdde740 1 Processor -- start 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06bdde740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c709f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06bdde740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c61d30 0x564723c70f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06bdde740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c66ea0 0x564723c71ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06bdde740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe0641ac2e0 con 0x564723c61340 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06bdde740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe0641abc10 con 0x564723c66ea0 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06bdde740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe0641a8d90 con 0x564723c61d30 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06bdde740 10 client.0 _ll_register_callbacks cb 0x7fe05c005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe062d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c709f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06259c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c61d30 0x564723c70f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06259c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c61d30 0x564723c70f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:35588/0 (socket says 172.21.15.77:35588) 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06259c640 1 -- 172.21.15.77:0/3806875186 learned_addr learned my addr 172.21.15.77:0/3806875186 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06359e640 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c66ea0 0x564723c71ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06259c640 1 -- 172.21.15.77:0/3806875186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c66ea0 msgr2=0x564723c71ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06259c640 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c66ea0 0x564723c71ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06259c640 1 -- 172.21.15.77:0/3806875186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 msgr2=0x564723c709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06259c640 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c709f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06259c640 1 -- 172.21.15.77:0/3806875186 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564723c724c0 con 0x564723c61d30 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06359e640 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c66ea0 0x564723c71ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe06259c640 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c61d30 0x564723c70f30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe058009fd0 tx=0x7fe058010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c22a63bbd16dafc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.140+0000 7fe046ff5640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe046ff5640 1 -- 172.21.15.77:0/3806875186 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe058024070 con 0x564723c61d30 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe046ff5640 1 -- 172.21.15.77:0/3806875186 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe058003b70 con 0x564723c61d30 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe06bdde740 1 -- 172.21.15.77:0/3806875186 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe054019850 con 0x564723c61d30 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe046ff5640 1 -- 172.21.15.77:0/3806875186 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fe058030050 con 0x564723c61d30 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe06bdde740 10 client.26545 fetch_fsmap learned FSMap version 34 2024-03-16T10:21:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe06bdde740 1 -- 172.21.15.77:0/3806875186 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x564723cde020 con 0x564723c61d30 2024-03-16T10:21:40.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe046ff5640 1 -- 172.21.15.77:0/3806875186 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe054019850 con 0x564723c61d30 2024-03-16T10:21:40.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe046ff5640 1 -- 172.21.15.77:0/3806875186 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fe0580040e0 con 0x564723c61d30 2024-03-16T10:21:40.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe06bdde740 10 client.26545 fetch_fsmap finished waiting for FSMap version 34 2024-03-16T10:21:40.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe06bdde740 10 client.26545 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:40.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe06bdde740 20 client.26545 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:40.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe06bdde740 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564723cdea00 0x564723ce0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:40.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe06bdde740 4 client.26545 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:21:40.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe06bdde740 1 -- 172.21.15.77:0/3806875186 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564723c79c50 con 0x564723cdea00 2024-03-16T10:21:40.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.141+0000 7fe062d9d640 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564723cdea00 0x564723ce0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:40.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.142+0000 7fe046ff5640 10 client.26545 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:40.147 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.142+0000 7fe062d9d640 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564723cdea00 0x564723ce0e20 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6ee883550aa3d55 server_cookie=97595724776d44d7 in_seq=0 out_seq=0 2024-03-16T10:21:40.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:39 smithi192 ceph-mon[36354]: pgmap v487: 97 pgs: 97 active+clean; 529 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 165 op/s 2024-03-16T10:21:40.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:39 smithi077 ceph-mon[32400]: pgmap v487: 97 pgs: 97 active+clean; 529 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 165 op/s 2024-03-16T10:21:40.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:39 smithi193 ceph-mon[36311]: pgmap v487: 97 pgs: 97 active+clean; 529 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 165 op/s 2024-03-16T10:21:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.292+0000 7fe046ff5640 1 -- 172.21.15.77:0/3806875186 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564723c79c50 con 0x564723cdea00 2024-03-16T10:21:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.292+0000 7fe046ff5640 10 client.26545 handle_command_reply: tid=0 2024-03-16T10:21:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.292+0000 7fe06bdde740 1 client.26545 shutdown 2024-03-16T10:21:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.292+0000 7fe06bdde740 1 -- 172.21.15.77:0/3806875186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c61d30 msgr2=0x564723c70f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.292+0000 7fe06bdde740 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c61d30 0x564723c70f30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe058009fd0 tx=0x7fe058010800 comp rx=0 tx=0).stop 2024-03-16T10:21:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.292+0000 7fe06bdde740 1 -- 172.21.15.77:0/3806875186 shutdown_connections 2024-03-16T10:21:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.292+0000 7fe06bdde740 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564723c66ea0 0x564723c71ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.292+0000 7fe06bdde740 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564723c61d30 0x564723c70f30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.292+0000 7fe06bdde740 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564723cdea00 0x564723ce0e20 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.292+0000 7fe06bdde740 1 --2- 172.21.15.77:0/3806875186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564723c61340 0x564723c709f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.292+0000 7fe06bdde740 1 -- 172.21.15.77:0/3806875186 >> 172.21.15.77:0/3806875186 conn(0x564723c5c8b0 msgr2=0x564723c603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.292+0000 7fe046ff5640 0 client.26545 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.293+0000 7fe06bdde740 1 -- 172.21.15.77:0/3806875186 shutdown_connections 2024-03-16T10:21:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.293+0000 7fe06bdde740 1 -- 172.21.15.77:0/3806875186 wait complete. 2024-03-16T10:21:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.293+0000 7fe06bdde740 20 client.26545 trim_cache size 0 max 16384 2024-03-16T10:21:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:21:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:21:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:21:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:21:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.294+0000 7fe069110640 1 -- 172.21.15.77:0/2328766097 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe04c0a28d0 msgr2=0x7fe04c0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.294+0000 7fe069110640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe04c0a28d0 0x7fe04c0a4d90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe054020000 tx=0x7fe054021000 comp rx=0 tx=0).stop 2024-03-16T10:21:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.295+0000 7fe069110640 1 -- 172.21.15.77:0/2328766097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe064100700 msgr2=0x7fe0640699d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.295+0000 7fe069110640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe064100700 0x7fe0640699d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe05c000c00 tx=0x7fe05c02c640 comp rx=0 tx=0).stop 2024-03-16T10:21:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.295+0000 7fe069110640 1 -- 172.21.15.77:0/2328766097 shutdown_connections 2024-03-16T10:21:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.295+0000 7fe069110640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe04c0a28d0 0x7fe04c0a4d90 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.295+0000 7fe069110640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0641022b0 0x7fe0641a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.295+0000 7fe069110640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0641018f0 0x7fe06419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.295+0000 7fe069110640 1 --2- 172.21.15.77:0/2328766097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe064100700 0x7fe0640699d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:40.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.296+0000 7fe069110640 1 -- 172.21.15.77:0/2328766097 >> 172.21.15.77:0/2328766097 conn(0x7fe0640fbc70 msgr2=0x7fe064104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:40.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.296+0000 7fe069110640 1 -- 172.21.15.77:0/2328766097 shutdown_connections 2024-03-16T10:21:40.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:40.297+0000 7fe069110640 1 -- 172.21.15.77:0/2328766097 wait complete. 2024-03-16T10:21:42.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:41 smithi192 ceph-mon[36354]: pgmap v488: 97 pgs: 97 active+clean; 530 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-03-16T10:21:42.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:41 smithi077 ceph-mon[32400]: pgmap v488: 97 pgs: 97 active+clean; 530 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-03-16T10:21:42.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:41 smithi193 ceph-mon[36311]: pgmap v488: 97 pgs: 97 active+clean; 530 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-03-16T10:21:44.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:43 smithi192 ceph-mon[36354]: pgmap v489: 97 pgs: 97 active+clean; 530 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-03-16T10:21:44.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:43 smithi077 ceph-mon[32400]: pgmap v489: 97 pgs: 97 active+clean; 530 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-03-16T10:21:44.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:43 smithi193 ceph-mon[36311]: pgmap v489: 97 pgs: 97 active+clean; 530 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-03-16T10:21:46.170 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:45 smithi192 ceph-mon[36354]: pgmap v490: 97 pgs: 97 active+clean; 531 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-03-16T10:21:46.170 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:45 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:21:46.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:45 smithi077 ceph-mon[32400]: pgmap v490: 97 pgs: 97 active+clean; 531 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-03-16T10:21:46.251 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:45 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:21:46.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:45 smithi193 ceph-mon[36311]: pgmap v490: 97 pgs: 97 active+clean; 531 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-03-16T10:21:46.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:45 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:21:47.105 INFO:tasks.workunit.client.0.smithi077.stdout: 112 1907988 488 1294594 348 1080368 1122 2024-03-16T10:21:48.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:47 smithi192 ceph-mon[36354]: pgmap v491: 97 pgs: 97 active+clean; 532 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 1.1 MiB/s wr, 172 op/s 2024-03-16T10:21:48.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:47 smithi077 ceph-mon[32400]: pgmap v491: 97 pgs: 97 active+clean; 532 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 1.1 MiB/s wr, 172 op/s 2024-03-16T10:21:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:47 smithi193 ceph-mon[36311]: pgmap v491: 97 pgs: 97 active+clean; 532 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 1.1 MiB/s wr, 172 op/s 2024-03-16T10:21:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:49 smithi192 ceph-mon[36354]: pgmap v492: 97 pgs: 97 active+clean; 541 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 267 op/s 2024-03-16T10:21:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:49 smithi077 ceph-mon[32400]: pgmap v492: 97 pgs: 97 active+clean; 541 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 267 op/s 2024-03-16T10:21:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:49 smithi193 ceph-mon[36311]: pgmap v492: 97 pgs: 97 active+clean; 541 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 267 op/s 2024-03-16T10:21:52.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:51 smithi192 ceph-mon[36354]: pgmap v493: 97 pgs: 97 active+clean; 563 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-03-16T10:21:52.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:51 smithi077 ceph-mon[32400]: pgmap v493: 97 pgs: 97 active+clean; 563 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-03-16T10:21:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:51 smithi193 ceph-mon[36311]: pgmap v493: 97 pgs: 97 active+clean; 563 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-03-16T10:21:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.412+0000 7efcbefb2640 1 -- 172.21.15.77:0/2260690049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcb8101830 msgr2=0x7efcb8101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.412+0000 7efcbefb2640 1 --2- 172.21.15.77:0/2260690049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcb8101830 0x7efcb8101c50 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7efcac009900 tx=0x7efcac02bc30 comp rx=0 tx=0).stop 2024-03-16T10:21:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.412+0000 7efcbefb2640 1 -- 172.21.15.77:0/2260690049 shutdown_connections 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.413+0000 7efcbefb2640 1 --2- 172.21.15.77:0/2260690049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcb81033e0 0x7efcb8109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.413+0000 7efcbefb2640 1 --2- 172.21.15.77:0/2260690049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb8102a20 0x7efcb8102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.413+0000 7efcbefb2640 1 --2- 172.21.15.77:0/2260690049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcb8101830 0x7efcb8101c50 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.413+0000 7efcbefb2640 1 -- 172.21.15.77:0/2260690049 >> 172.21.15.77:0/2260690049 conn(0x7efcb80fcda0 msgr2=0x7efcb80ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.413+0000 7efcbefb2640 1 -- 172.21.15.77:0/2260690049 shutdown_connections 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.413+0000 7efcbefb2640 1 -- 172.21.15.77:0/2260690049 wait complete. 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.413+0000 7efcbefb2640 1 Processor -- start 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.413+0000 7efcbefb2640 1 -- start start 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.413+0000 7efcbefb2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcb8101830 0x7efcb8197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcbefb2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcb8102a20 0x7efcb8197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcbefb2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb81033e0 0x7efcb819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcbefb2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efcb810d4b0 con 0x7efcb8102a20 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcbefb2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7efcb810d330 con 0x7efcb81033e0 2024-03-16T10:21:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcbefb2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efcb810d630 con 0x7efcb8101830 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcb7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcb8102a20 0x7efcb8197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcbcd4e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcb8101830 0x7efcb8197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcb7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcb8102a20 0x7efcb8197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43488/0 (socket says 172.21.15.77:43488) 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcb7fff640 1 -- 172.21.15.77:0/2795373530 learned_addr learned my addr 172.21.15.77:0/2795373530 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcbd54f640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb81033e0 0x7efcb819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcb7fff640 1 -- 172.21.15.77:0/2795373530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcb8101830 msgr2=0x7efcb8197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcb7fff640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcb8101830 0x7efcb8197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcb7fff640 1 -- 172.21.15.77:0/2795373530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb81033e0 msgr2=0x7efcb819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcb7fff640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb81033e0 0x7efcb819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcb7fff640 1 -- 172.21.15.77:0/2795373530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb819f690 con 0x7efcb8102a20 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcbcd4e640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcb8101830 0x7efcb8197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcb7fff640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcb8102a20 0x7efcb8197fb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7efca400e9f0 tx=0x7efca400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcb5ffb640 1 -- 172.21.15.77:0/2795373530 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca400cd80 con 0x7efcb8102a20 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcbefb2640 1 -- 172.21.15.77:0/2795373530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb819f980 con 0x7efcb8102a20 2024-03-16T10:21:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcb5ffb640 1 -- 172.21.15.77:0/2795373530 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efca4010910 con 0x7efcb8102a20 2024-03-16T10:21:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.414+0000 7efcb5ffb640 1 -- 172.21.15.77:0/2795373530 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca40188f0 con 0x7efcb8102a20 2024-03-16T10:21:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.415+0000 7efcbefb2640 1 -- 172.21.15.77:0/2795373530 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb819fed0 con 0x7efcb8102a20 2024-03-16T10:21:53.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcb5ffb640 1 -- 172.21.15.77:0/2795373530 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7efca4010ac0 con 0x7efcb8102a20 2024-03-16T10:21:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcc1ba0740 1 Processor -- start 2024-03-16T10:21:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcc1ba0740 1 -- start start 2024-03-16T10:21:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcc1ba0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 0x563c7015db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcc1ba0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c70162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcc1ba0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c701632b0 0x563c70165740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcc1ba0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563c700bea40 con 0x563c7015d750 2024-03-16T10:21:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcc1ba0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563c70067920 con 0x563c701632b0 2024-03-16T10:21:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcc1ba0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c700baae0 con 0x563c7015e140 2024-03-16T10:21:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcb5ffb640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efca80a28a0 0x7efca80a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcbd54f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 0x563c7015db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcbd54f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 0x563c7015db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43492/0 (socket says 172.21.15.77:43492) 2024-03-16T10:21:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcbd54f640 1 -- 172.21.15.77:0/1163318975 learned_addr learned my addr 172.21.15.77:0/1163318975 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcb7fff640 1 --2- 172.21.15.77:0/1163318975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c701632b0 0x563c70165740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcbcd4e640 1 --2- 172.21.15.77:0/1163318975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c70162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcbd54f640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efca80a28a0 0x7efca80a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.417+0000 7efcb5ffb640 1 -- 172.21.15.77:0/2795373530 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7efca40ced80 con 0x7efcb8102a20 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.418+0000 7efcbd54f640 1 -- 172.21.15.77:0/1163318975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 msgr2=0x563c70162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.418+0000 7efcbd54f640 1 --2- 172.21.15.77:0/1163318975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c70162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.418+0000 7efcbd54f640 1 -- 172.21.15.77:0/1163318975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c701632b0 msgr2=0x563c70165740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.418+0000 7efcbd54f640 1 --2- 172.21.15.77:0/1163318975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c701632b0 0x563c70165740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.418+0000 7efcbd54f640 1 -- 172.21.15.77:0/1163318975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c7007f460 con 0x563c7015d750 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.418+0000 7efcbcd4e640 1 --2- 172.21.15.77:0/1163318975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c70162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.418+0000 7efcbd54f640 1 --2- 172.21.15.77:0/1163318975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 0x563c7015db70 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7efcb0011e20 tx=0x7efcb000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf11934c79b3ef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.418+0000 7efcbd54f640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efca80a28a0 0x7efca80a4d60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efcb000fc90 tx=0x7efcb001e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.418+0000 7efca2ffd640 1 -- 172.21.15.77:0/1163318975 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb001dcb0 con 0x563c7015d750 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.418+0000 7efca2ffd640 1 -- 172.21.15.77:0/1163318975 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcb001de50 con 0x563c7015d750 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.418+0000 7efca2ffd640 1 -- 172.21.15.77:0/1163318975 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0004280 con 0x563c7015d750 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.419+0000 7efcc1ba0740 1 -- 172.21.15.77:0/1163318975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 msgr2=0x563c7015db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.419+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/1163318975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 0x563c7015db70 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7efcb0011e20 tx=0x7efcb000f790 comp rx=0 tx=0).stop 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.419+0000 7efcc1ba0740 1 -- 172.21.15.77:0/1163318975 shutdown_connections 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.419+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/1163318975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c701632b0 0x563c70165740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.419+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/1163318975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c70162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.419+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/1163318975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 0x563c7015db70 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.419+0000 7efcc1ba0740 1 -- 172.21.15.77:0/1163318975 >> 172.21.15.77:0/1163318975 conn(0x563c701548b0 msgr2=0x563c70154cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.419+0000 7efcc1ba0740 1 -- 172.21.15.77:0/1163318975 shutdown_connections 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.419+0000 7efcc1ba0740 1 -- 172.21.15.77:0/1163318975 wait complete. 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.419+0000 7efcc1ba0740 1 -- start start 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.419+0000 7efcc1ba0740 1 Processor -- start 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcc1ba0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 0x563c701679f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcc1ba0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c70167f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcbcd4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 0x563c701679f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcc1ba0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c701632b0 0x563c7016ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcc1ba0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563c701661a0 con 0x563c7015d750 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcbcd4e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 0x563c701679f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43494/0 (socket says 172.21.15.77:43494) 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcbcd4e640 1 -- 172.21.15.77:0/3747361573 learned_addr learned my addr 172.21.15.77:0/3747361573 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcc1ba0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563c7007ad70 con 0x563c701632b0 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcc1ba0740 1 -- 172.21.15.77:0/3747361573 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c70166320 con 0x563c7015e140 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcc1ba0740 10 client.0 _ll_register_callbacks cb 0x563c6fdd2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcbd54f640 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c701632b0 0x563c7016ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcb7fff640 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c70167f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcbcd4e640 1 -- 172.21.15.77:0/3747361573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 msgr2=0x563c70167f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcbcd4e640 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c70167f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcbcd4e640 1 -- 172.21.15.77:0/3747361573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c701632b0 msgr2=0x563c7016ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcbcd4e640 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c701632b0 0x563c7016ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcbcd4e640 1 -- 172.21.15.77:0/3747361573 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c6fdd3210 con 0x563c7015d750 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcbd54f640 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c701632b0 0x563c7016ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcb7fff640 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c70167f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcbcd4e640 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 0x563c701679f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efcac02c7b0 tx=0x7efcac005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d94f8172804d1d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efc97fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efc97fff640 1 -- 172.21.15.77:0/3747361573 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac005d50 con 0x563c7015d750 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efc97fff640 1 -- 172.21.15.77:0/3747361573 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7efcac0048c0 con 0x563c7015d750 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.420+0000 7efcc1ba0740 1 -- 172.21.15.77:0/3747361573 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcb001de50 con 0x563c7015d750 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.421+0000 7efc97fff640 1 -- 172.21.15.77:0/3747361573 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efcac005650 con 0x563c7015d750 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.421+0000 7efcc1ba0740 10 client.16800 fetch_fsmap learned FSMap version 35 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.421+0000 7efcc1ba0740 1 -- 172.21.15.77:0/3747361573 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563c6fe9f620 con 0x563c7015d750 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.421+0000 7efc97fff640 1 -- 172.21.15.77:0/3747361573 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcb001de50 con 0x563c7015d750 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.421+0000 7efc97fff640 1 -- 172.21.15.77:0/3747361573 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7efcac003b40 con 0x563c7015d750 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.421+0000 7efcc1ba0740 10 client.16800 fetch_fsmap finished waiting for FSMap version 35 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.421+0000 7efcc1ba0740 10 client.16800 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.421+0000 7efcc1ba0740 20 client.16800 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.421+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563c70173c80 0x563c70176110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.421+0000 7efcc1ba0740 4 client.16800 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.421+0000 7efcc1ba0740 1 -- 172.21.15.77:0/3747361573 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c700cbf80 con 0x563c70173c80 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.421+0000 7efcb7fff640 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563c70173c80 0x563c70176110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.422+0000 7efc97fff640 10 client.16800 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.422+0000 7efcb7fff640 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563c70173c80 0x563c70176110 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27d7553fb4decf05 server_cookie=d13df05bade9b411 in_seq=0 out_seq=0 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efc97fff640 1 -- 172.21.15.77:0/3747361573 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563c700cbf80 con 0x563c70173c80 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efc97fff640 10 client.16800 handle_command_reply: tid=0 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efcc1ba0740 1 client.16800 shutdown 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efcc1ba0740 1 -- 172.21.15.77:0/3747361573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 msgr2=0x563c701679f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 0x563c701679f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efcac02c7b0 tx=0x7efcac005250 comp rx=0 tx=0).stop 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efcc1ba0740 1 -- 172.21.15.77:0/3747361573 shutdown_connections 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c701632b0 0x563c7016ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c70167f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563c70173c80 0x563c70176110 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/3747361573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c7015d750 0x563c701679f0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efc97fff640 0 client.16800 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efcc1ba0740 1 -- 172.21.15.77:0/3747361573 >> 172.21.15.77:0/3747361573 conn(0x563c701548b0 msgr2=0x563c70153bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efcc1ba0740 1 -- 172.21.15.77:0/3747361573 shutdown_connections 2024-03-16T10:21:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efcc1ba0740 1 -- 172.21.15.77:0/3747361573 wait complete. 2024-03-16T10:21:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.423+0000 7efcc1ba0740 20 client.16800 trim_cache size 0 max 16384 2024-03-16T10:21:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcc1ba0740 1 Processor -- start 2024-03-16T10:21:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcc1ba0740 1 -- start start 2024-03-16T10:21:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcc1ba0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015d750 0x563c70154c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcc1ba0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c7015e140 0x563c7017aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcc1ba0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 0x563c7017af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcc1ba0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81a68d0 con 0x563c701632b0 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcc1ba0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81a84a0 con 0x563c7015e140 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcc1ba0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81a6fa0 con 0x563c7015d750 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcbd54f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 0x563c7017af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcbd54f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 0x563c7017af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43496/0 (socket says 172.21.15.77:43496) 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcbd54f640 1 -- 172.21.15.77:0/2648769025 learned_addr learned my addr 172.21.15.77:0/2648769025 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcbcd4e640 1 --2- 172.21.15.77:0/2648769025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015d750 0x563c70154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcb7fff640 1 --2- 172.21.15.77:0/2648769025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c7015e140 0x563c7017aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcbd54f640 1 -- 172.21.15.77:0/2648769025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015d750 msgr2=0x563c70154c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcbd54f640 1 --2- 172.21.15.77:0/2648769025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015d750 0x563c70154c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcbd54f640 1 -- 172.21.15.77:0/2648769025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c7015e140 msgr2=0x563c7017aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcbd54f640 1 --2- 172.21.15.77:0/2648769025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c7015e140 0x563c7017aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.438+0000 7efcbd54f640 1 -- 172.21.15.77:0/2648769025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c70146140 con 0x563c701632b0 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efcbd54f640 1 --2- 172.21.15.77:0/2648769025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 0x563c7017af50 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7efcb00114e0 tx=0x7efcb0011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8d0dd8aadc1aa7b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efca27fc640 1 -- 172.21.15.77:0/2648769025 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb002a040 con 0x563c701632b0 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efca27fc640 1 -- 172.21.15.77:0/2648769025 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcb0003e50 con 0x563c701632b0 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efca27fc640 1 -- 172.21.15.77:0/2648769025 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb00056d0 con 0x563c701632b0 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2648769025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 msgr2=0x563c7017af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/2648769025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 0x563c7017af50 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7efcb00114e0 tx=0x7efcb0011b20 comp rx=0 tx=0).stop 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2648769025 shutdown_connections 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/2648769025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 0x563c7017af50 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/2648769025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c7015e140 0x563c7017aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/2648769025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015d750 0x563c70154c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2648769025 >> 172.21.15.77:0/2648769025 conn(0x563c701548b0 msgr2=0x563c70153bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2648769025 shutdown_connections 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.439+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2648769025 wait complete. 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcc1ba0740 1 -- start start 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcc1ba0740 1 Processor -- start 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcc1ba0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c7015d750 0x563c7016bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcc1ba0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c7016c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcc1ba0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 0x563c7016d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcc1ba0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81a7dd0 con 0x563c701632b0 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcc1ba0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81a7700 con 0x563c7015d750 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcc1ba0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81a48b0 con 0x563c7015e140 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcc1ba0740 10 client.0 _ll_register_callbacks cb 0x7efcac009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcbd54f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 0x563c7016d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcbd54f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 0x563c7016d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43504/0 (socket says 172.21.15.77:43504) 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcbd54f640 1 -- 172.21.15.77:0/2148914793 learned_addr learned my addr 172.21.15.77:0/2148914793 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcbcd4e640 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c7015d750 0x563c7016bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.440+0000 7efcb7fff640 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c7016c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efcbd54f640 1 -- 172.21.15.77:0/2148914793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 msgr2=0x563c7016c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efcbd54f640 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c7016c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efcbd54f640 1 -- 172.21.15.77:0/2148914793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c7015d750 msgr2=0x563c7016bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efcbd54f640 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c7015d750 0x563c7016bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efcbd54f640 1 -- 172.21.15.77:0/2148914793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c7016db30 con 0x563c701632b0 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efcbd54f640 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 0x563c7016d540 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7efcb000a010 tx=0x7efcb0011760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e433e0aa82f9ed00 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efc97fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efc97fff640 1 -- 172.21.15.77:0/2148914793 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb003b070 con 0x563c701632b0 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efc97fff640 1 -- 172.21.15.77:0/2148914793 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7efcb0011040 con 0x563c701632b0 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2148914793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcb0003e50 con 0x563c701632b0 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efc97fff640 1 -- 172.21.15.77:0/2148914793 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efcb0046050 con 0x563c701632b0 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efcc1ba0740 10 client.16806 fetch_fsmap learned FSMap version 35 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2148914793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563c7016ee40 con 0x563c701632b0 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efc97fff640 1 -- 172.21.15.77:0/2148914793 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcb0003e50 con 0x563c701632b0 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efc97fff640 1 -- 172.21.15.77:0/2148914793 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7efcb002d030 con 0x563c701632b0 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efcc1ba0740 10 client.16806 fetch_fsmap finished waiting for FSMap version 35 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efcc1ba0740 10 client.16806 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:21:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.441+0000 7efcc1ba0740 20 client.16806 populate_metadata read hostname 'smithi077' 2024-03-16T10:21:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.442+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563c701da5b0 0x563c701dc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:21:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.442+0000 7efcc1ba0740 4 client.16806 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:21:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.442+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2148914793 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563c700c62d0 con 0x563c701da5b0 2024-03-16T10:21:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.442+0000 7efcbcd4e640 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563c701da5b0 0x563c701dc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:21:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.442+0000 7efc97fff640 10 client.16806 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:53.442+0000 7efcbcd4e640 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563c701da5b0 0x563c701dc9d0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ca8dfe7f3653295 server_cookie=5e17cd2ee0e1c3b3 in_seq=0 out_seq=0 2024-03-16T10:21:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.142+0000 7efc97fff640 1 -- 172.21.15.77:0/2148914793 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563c700c62d0 con 0x563c701da5b0 2024-03-16T10:21:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.142+0000 7efc97fff640 10 client.16806 handle_command_reply: tid=0 2024-03-16T10:21:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.142+0000 7efcc1ba0740 1 client.16806 shutdown 2024-03-16T10:21:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.142+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2148914793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 msgr2=0x563c7016d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.142+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 0x563c7016d540 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7efcb000a010 tx=0x7efcb0011760 comp rx=0 tx=0).stop 2024-03-16T10:21:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.142+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2148914793 shutdown_connections 2024-03-16T10:21:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.142+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563c701da5b0 0x563c701dc9d0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.143+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c701632b0 0x563c7016d540 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.143+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c7015e140 0x563c7016c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.143+0000 7efcc1ba0740 1 --2- 172.21.15.77:0/2148914793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c7015d750 0x563c7016bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.143+0000 7efc97fff640 0 client.16806 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:21:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.143+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2148914793 >> 172.21.15.77:0/2148914793 conn(0x563c701548b0 msgr2=0x563c701583a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.143+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2148914793 shutdown_connections 2024-03-16T10:21:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.143+0000 7efcc1ba0740 1 -- 172.21.15.77:0/2148914793 wait complete. 2024-03-16T10:21:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.144+0000 7efcc1ba0740 20 client.16806 trim_cache size 0 max 16384 2024-03-16T10:21:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:21:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:21:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:21:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:21:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.165+0000 7efcbefb2640 1 -- 172.21.15.77:0/2795373530 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efca80a28a0 msgr2=0x7efca80a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.165+0000 7efcbefb2640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efca80a28a0 0x7efca80a4d60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efcb000fc90 tx=0x7efcb001e000 comp rx=0 tx=0).stop 2024-03-16T10:21:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.165+0000 7efcbefb2640 1 -- 172.21.15.77:0/2795373530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcb8102a20 msgr2=0x7efcb8197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:21:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.165+0000 7efcbefb2640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcb8102a20 0x7efcb8197fb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7efca400e9f0 tx=0x7efca400eeb0 comp rx=0 tx=0).stop 2024-03-16T10:21:54.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:53 smithi192 ceph-mon[36354]: pgmap v494: 97 pgs: 97 active+clean; 599 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.6 MiB/s wr, 697 op/s 2024-03-16T10:21:54.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:53 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.174+0000 7efcbefb2640 1 -- 172.21.15.77:0/2795373530 shutdown_connections 2024-03-16T10:21:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.174+0000 7efcbefb2640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb81033e0 0x7efcb819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.174+0000 7efcbefb2640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcb8102a20 0x7efcb8197fb0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.174+0000 7efcbefb2640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efca80a28a0 0x7efca80a4d60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.174+0000 7efcbefb2640 1 --2- 172.21.15.77:0/2795373530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcb8101830 0x7efcb8197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:21:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.174+0000 7efcbefb2640 1 -- 172.21.15.77:0/2795373530 >> 172.21.15.77:0/2795373530 conn(0x7efcb80fcda0 msgr2=0x7efcb8105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:21:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.179+0000 7efcbefb2640 1 -- 172.21.15.77:0/2795373530 shutdown_connections 2024-03-16T10:21:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:21:54.179+0000 7efcbefb2640 1 -- 172.21.15.77:0/2795373530 wait complete. 2024-03-16T10:21:54.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:53 smithi077 ceph-mon[32400]: pgmap v494: 97 pgs: 97 active+clean; 599 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.6 MiB/s wr, 697 op/s 2024-03-16T10:21:54.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:53 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:53 smithi193 ceph-mon[36311]: pgmap v494: 97 pgs: 97 active+clean; 599 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.6 MiB/s wr, 697 op/s 2024-03-16T10:21:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:53 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:21:56.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:55 smithi192 ceph-mon[36354]: pgmap v495: 97 pgs: 97 active+clean; 602 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.9 MiB/s wr, 730 op/s 2024-03-16T10:21:56.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:55 smithi077 ceph-mon[32400]: pgmap v495: 97 pgs: 97 active+clean; 602 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.9 MiB/s wr, 730 op/s 2024-03-16T10:21:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:55 smithi193 ceph-mon[36311]: pgmap v495: 97 pgs: 97 active+clean; 602 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.9 MiB/s wr, 730 op/s 2024-03-16T10:21:57.104 INFO:tasks.workunit.client.0.smithi077.stdout: 115 861131 256 570572 134 488805 552 2024-03-16T10:21:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:57 smithi192 ceph-mon[36354]: pgmap v496: 97 pgs: 97 active+clean; 599 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.8 MiB/s wr, 817 op/s 2024-03-16T10:21:58.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:57 smithi077 ceph-mon[32400]: pgmap v496: 97 pgs: 97 active+clean; 599 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.8 MiB/s wr, 817 op/s 2024-03-16T10:21:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:57 smithi193 ceph-mon[36311]: pgmap v496: 97 pgs: 97 active+clean; 599 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.8 MiB/s wr, 817 op/s 2024-03-16T10:22:00.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:21:59 smithi192 ceph-mon[36354]: pgmap v497: 97 pgs: 97 active+clean; 602 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.0 MiB/s wr, 876 op/s 2024-03-16T10:22:00.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:21:59 smithi077 ceph-mon[32400]: pgmap v497: 97 pgs: 97 active+clean; 602 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.0 MiB/s wr, 876 op/s 2024-03-16T10:22:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:21:59 smithi193 ceph-mon[36311]: pgmap v497: 97 pgs: 97 active+clean; 602 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.0 MiB/s wr, 876 op/s 2024-03-16T10:22:01.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:00 smithi077 ceph-mon[32400]: pgmap v498: 97 pgs: 97 active+clean; 597 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.2 MiB/s wr, 819 op/s 2024-03-16T10:22:01.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:00 smithi193 ceph-mon[36311]: pgmap v498: 97 pgs: 97 active+clean; 597 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.2 MiB/s wr, 819 op/s 2024-03-16T10:22:01.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:00 smithi192 ceph-mon[36354]: pgmap v498: 97 pgs: 97 active+clean; 597 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.2 MiB/s wr, 819 op/s 2024-03-16T10:22:01.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:22:01.516 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:22:01.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.780+0000 7f5648dff640 1 -- 172.21.15.77:0/1324111582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56440ffaa0 msgr2=0x7f56440ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:01.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.780+0000 7f5648dff640 1 --2- 172.21.15.77:0/1324111582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56440ffaa0 0x7f56440ffec0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f5634009930 tx=0x7f563402bc30 comp rx=0 tx=0).stop 2024-03-16T10:22:01.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.781+0000 7f5648dff640 1 -- 172.21.15.77:0/1324111582 shutdown_connections 2024-03-16T10:22:01.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.781+0000 7f5648dff640 1 --2- 172.21.15.77:0/1324111582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5644103430 0x7f5644109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.781+0000 7f5648dff640 1 --2- 172.21.15.77:0/1324111582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5644102a70 0x7f5644102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.781+0000 7f5648dff640 1 --2- 172.21.15.77:0/1324111582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56440ffaa0 0x7f56440ffec0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.781+0000 7f5648dff640 1 -- 172.21.15.77:0/1324111582 >> 172.21.15.77:0/1324111582 conn(0x7f56440fb3d0 msgr2=0x7f56440fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:01.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.781+0000 7f5648dff640 1 -- 172.21.15.77:0/1324111582 shutdown_connections 2024-03-16T10:22:01.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.781+0000 7f5648dff640 1 -- 172.21.15.77:0/1324111582 wait complete. 2024-03-16T10:22:01.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5648dff640 1 Processor -- start 2024-03-16T10:22:01.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5648dff640 1 -- start start 2024-03-16T10:22:01.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5648dff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56440ffaa0 0x7f5644197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5648dff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5644102a70 0x7f56441980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5648dff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5644103430 0x7f564419f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5648dff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f564410d4a0 con 0x7f5644102a70 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5648dff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f564410d320 con 0x7f56440ffaa0 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5648dff640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f564410d620 con 0x7f5644103430 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5641d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5644102a70 0x7f56441980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5642d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5644103430 0x7f564419f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5641d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5644102a70 0x7f56441980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59936/0 (socket says 172.21.15.77:59936) 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5642d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5644103430 0x7f564419f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58758/0 (socket says 172.21.15.77:58758) 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f5641d9b640 1 -- 172.21.15.77:0/488824435 learned_addr learned my addr 172.21.15.77:0/488824435 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.782+0000 7f564259c640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56440ffaa0 0x7f5644197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.783+0000 7f5642d9d640 1 -- 172.21.15.77:0/488824435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56440ffaa0 msgr2=0x7f5644197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.783+0000 7f5642d9d640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56440ffaa0 0x7f5644197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.783+0000 7f5642d9d640 1 -- 172.21.15.77:0/488824435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5644102a70 msgr2=0x7f56441980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:01.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.783+0000 7f5642d9d640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5644102a70 0x7f56441980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.783+0000 7f5642d9d640 1 -- 172.21.15.77:0/488824435 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f564419f7b0 con 0x7f5644103430 2024-03-16T10:22:01.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.783+0000 7f5641d9b640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5644102a70 0x7f56441980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:01.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.783+0000 7f5642d9d640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5644103430 0x7f564419f140 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f563c009fd0 tx=0x7f563c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:01.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.783+0000 7f56337fe640 1 -- 172.21.15.77:0/488824435 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c00ce20 con 0x7f5644103430 2024-03-16T10:22:01.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.783+0000 7f56337fe640 1 -- 172.21.15.77:0/488824435 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f563c0137d0 con 0x7f5644103430 2024-03-16T10:22:01.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.783+0000 7f56337fe640 1 -- 172.21.15.77:0/488824435 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c0055e0 con 0x7f5644103430 2024-03-16T10:22:01.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.783+0000 7f5648dff640 1 -- 172.21.15.77:0/488824435 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56441008c0 con 0x7f5644103430 2024-03-16T10:22:01.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.783+0000 7f5648dff640 1 -- 172.21.15.77:0/488824435 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5644100d60 con 0x7f5644103430 2024-03-16T10:22:01.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.786+0000 7f56337fe640 1 -- 172.21.15.77:0/488824435 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f563c013980 con 0x7f5644103430 2024-03-16T10:22:01.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.787+0000 7f564b9ef740 1 Processor -- start 2024-03-16T10:22:01.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.787+0000 7f564b9ef740 1 -- start start 2024-03-16T10:22:01.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.787+0000 7f56337fe640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f562c0a2880 0x7f562c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.787+0000 7f564b9ef740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 0x5568dbc55590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.787+0000 7f56337fe640 1 -- 172.21.15.77:0/488824435 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f563c0d30b0 con 0x7f5644103430 2024-03-16T10:22:01.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.787+0000 7f564b9ef740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 0x5568dbc5a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.787+0000 7f564b9ef740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc5acd0 0x5568dbc5d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.787+0000 7f564b9ef740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5568dbbba500 con 0x5568dbc5acd0 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.787+0000 7f564b9ef740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5568dbb62be0 con 0x5568dbc55b60 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.787+0000 7f564b9ef740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5568dbbb65a0 con 0x5568dbc55170 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.787+0000 7f5641d9b640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f562c0a2880 0x7f562c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f564259c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 0x5568dbc55590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f564259c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 0x5568dbc55590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58764/0 (socket says 172.21.15.77:58764) 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f564259c640 1 -- 172.21.15.77:0/741481801 learned_addr learned my addr 172.21.15.77:0/741481801 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f5642d9d640 1 --2- 172.21.15.77:0/741481801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc5acd0 0x5568dbc5d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f564259c640 1 -- 172.21.15.77:0/741481801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 msgr2=0x5568dbc5a660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f564259c640 1 --2- 172.21.15.77:0/741481801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 0x5568dbc5a660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f564259c640 1 -- 172.21.15.77:0/741481801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc5acd0 msgr2=0x5568dbc5d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:01.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f564259c640 1 --2- 172.21.15.77:0/741481801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc5acd0 0x5568dbc5d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f564259c640 1 -- 172.21.15.77:0/741481801 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568dbb764b0 con 0x5568dbc55170 2024-03-16T10:22:01.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f5642d9d640 1 --2- 172.21.15.77:0/741481801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc5acd0 0x5568dbc5d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:22:01.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f5641d9b640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f562c0a2880 0x7f562c0a4d40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f56380095f0 tx=0x7f5638005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:01.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f564259c640 1 --2- 172.21.15.77:0/741481801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 0x5568dbc55590 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5634036040 tx=0x7f563402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea05a8648ad4f09 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:01.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f5630ff9640 1 -- 172.21.15.77:0/741481801 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563402c980 con 0x5568dbc55170 2024-03-16T10:22:01.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f5630ff9640 1 -- 172.21.15.77:0/741481801 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f563402cb20 con 0x5568dbc55170 2024-03-16T10:22:01.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.788+0000 7f5630ff9640 1 -- 172.21.15.77:0/741481801 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56340046c0 con 0x5568dbc55170 2024-03-16T10:22:01.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.789+0000 7f564b9ef740 1 -- 172.21.15.77:0/741481801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 msgr2=0x5568dbc55590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.789+0000 7f564b9ef740 1 --2- 172.21.15.77:0/741481801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 0x5568dbc55590 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5634036040 tx=0x7f563402c650 comp rx=0 tx=0).stop 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.790+0000 7f564b9ef740 1 -- 172.21.15.77:0/741481801 shutdown_connections 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.790+0000 7f564b9ef740 1 --2- 172.21.15.77:0/741481801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc5acd0 0x5568dbc5d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.790+0000 7f564b9ef740 1 --2- 172.21.15.77:0/741481801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 0x5568dbc5a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.790+0000 7f564b9ef740 1 --2- 172.21.15.77:0/741481801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 0x5568dbc55590 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.790+0000 7f564b9ef740 1 -- 172.21.15.77:0/741481801 >> 172.21.15.77:0/741481801 conn(0x5568dbc506e0 msgr2=0x5568dbc50ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.790+0000 7f564b9ef740 1 -- 172.21.15.77:0/741481801 shutdown_connections 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.790+0000 7f564b9ef740 1 -- 172.21.15.77:0/741481801 wait complete. 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.790+0000 7f564b9ef740 1 -- start start 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.790+0000 7f564b9ef740 1 Processor -- start 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f564b9ef740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55170 0x5568dbc5f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f564b9ef740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc55b60 0x5568dbc5faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f564b9ef740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc5acd0 0x5568dbc66a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f564b9ef740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5568dbc5dc20 con 0x5568dbc55b60 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f564b9ef740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5568dbb76160 con 0x5568dbc55170 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f564b9ef740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5568dbc5dda0 con 0x5568dbc5acd0 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f564b9ef740 10 client.0 _ll_register_callbacks cb 0x5568dbb5a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f564259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55170 0x5568dbc5f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f564259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55170 0x5568dbc5f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54062/0 (socket says 172.21.15.77:54062) 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f564259c640 1 -- 172.21.15.77:0/742837381 learned_addr learned my addr 172.21.15.77:0/742837381 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f5641d9b640 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc55b60 0x5568dbc5faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f564259c640 1 -- 172.21.15.77:0/742837381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc5acd0 msgr2=0x5568dbc66a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.791+0000 7f5642d9d640 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc5acd0 0x5568dbc66a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.792+0000 7f564259c640 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc5acd0 0x5568dbc66a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.792+0000 7f564259c640 1 -- 172.21.15.77:0/742837381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc55b60 msgr2=0x5568dbc5faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.792+0000 7f564259c640 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc55b60 0x5568dbc5faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.792+0000 7f564259c640 1 -- 172.21.15.77:0/742837381 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568db9987d0 con 0x5568dbc55170 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.792+0000 7f5641d9b640 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc55b60 0x5568dbc5faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.792+0000 7f564259c640 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55170 0x5568dbc5f560 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5634000c00 tx=0x7f563403f930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74cd28b2bf231ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.792+0000 7f5629ffb640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.792+0000 7f5629ffb640 1 -- 172.21.15.77:0/742837381 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563402c980 con 0x5568dbc55170 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.792+0000 7f564b9ef740 1 -- 172.21.15.77:0/742837381 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f563402cb20 con 0x5568dbc55170 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.792+0000 7f5629ffb640 1 -- 172.21.15.77:0/742837381 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f5634034440 con 0x5568dbc55170 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.793+0000 7f5629ffb640 1 -- 172.21.15.77:0/742837381 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5634034680 con 0x5568dbc55170 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.793+0000 7f564b9ef740 10 client.26579 fetch_fsmap learned FSMap version 35 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.793+0000 7f564b9ef740 1 -- 172.21.15.77:0/742837381 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5568db8ce3f0 con 0x5568dbc55170 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.793+0000 7f5629ffb640 1 -- 172.21.15.77:0/742837381 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f563402cb20 con 0x5568dbc55170 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.793+0000 7f5629ffb640 1 -- 172.21.15.77:0/742837381 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f5634054970 con 0x5568dbc55170 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.793+0000 7f564b9ef740 10 client.26579 fetch_fsmap finished waiting for FSMap version 35 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.793+0000 7f564b9ef740 10 client.26579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.793+0000 7f564b9ef740 20 client.26579 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:01.797 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.793+0000 7f564b9ef740 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5568dbc6abd0 0x5568dbc6cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.793+0000 7f564b9ef740 4 client.26579 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:01.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.793+0000 7f564b9ef740 1 -- 172.21.15.77:0/742837381 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568dbbc7a40 con 0x5568dbc6abd0 2024-03-16T10:22:01.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.793+0000 7f5641d9b640 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5568dbc6abd0 0x5568dbc6cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.794+0000 7f5629ffb640 10 client.26579 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:01.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.794+0000 7f5641d9b640 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5568dbc6abd0 0x5568dbc6cfc0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54ce41b38f3f35f9 server_cookie=9529b98077fec531 in_seq=0 out_seq=0 2024-03-16T10:22:01.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f5629ffb640 1 -- 172.21.15.77:0/742837381 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5568dbbc7a40 con 0x5568dbc6abd0 2024-03-16T10:22:01.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f5629ffb640 10 client.26579 handle_command_reply: tid=0 2024-03-16T10:22:01.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f564b9ef740 1 client.26579 shutdown 2024-03-16T10:22:01.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f564b9ef740 1 -- 172.21.15.77:0/742837381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55170 msgr2=0x5568dbc5f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:01.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f564b9ef740 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55170 0x5568dbc5f560 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5634000c00 tx=0x7f563403f930 comp rx=0 tx=0).stop 2024-03-16T10:22:01.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f564b9ef740 1 -- 172.21.15.77:0/742837381 shutdown_connections 2024-03-16T10:22:01.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f564b9ef740 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc5acd0 0x5568dbc66a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f564b9ef740 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5568dbc6abd0 0x5568dbc6cfc0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f564b9ef740 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc55b60 0x5568dbc5faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f564b9ef740 1 --2- 172.21.15.77:0/742837381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55170 0x5568dbc5f560 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f5629ffb640 0 client.26579 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:01.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f564b9ef740 1 -- 172.21.15.77:0/742837381 >> 172.21.15.77:0/742837381 conn(0x5568dbc506e0 msgr2=0x5568dbc4f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:01.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f564b9ef740 1 -- 172.21.15.77:0/742837381 shutdown_connections 2024-03-16T10:22:01.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f564b9ef740 1 -- 172.21.15.77:0/742837381 wait complete. 2024-03-16T10:22:01.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.795+0000 7f564b9ef740 20 client.26579 trim_cache size 0 max 16384 2024-03-16T10:22:01.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.827+0000 7f564b9ef740 1 Processor -- start 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.827+0000 7f564b9ef740 1 -- start start 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f564b9ef740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc55170 0x5568dbc541d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f564b9ef740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 0x5568dbc66990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f564b9ef740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc5acd0 0x5568dbc4f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f564b9ef740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f56441a6980 con 0x5568dbc55170 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f564b9ef740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f56441a8550 con 0x5568dbc55b60 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f564b9ef740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f56441a7050 con 0x5568dbc5acd0 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f564259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc55170 0x5568dbc541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f564259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc55170 0x5568dbc541d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59958/0 (socket says 172.21.15.77:59958) 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f564259c640 1 -- 172.21.15.77:0/1832693324 learned_addr learned my addr 172.21.15.77:0/1832693324 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f5642d9d640 1 --2- 172.21.15.77:0/1832693324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc5acd0 0x5568dbc4f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f5641d9b640 1 --2- 172.21.15.77:0/1832693324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 0x5568dbc66990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f5642d9d640 1 -- 172.21.15.77:0/1832693324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 msgr2=0x5568dbc66990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:01.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f5642d9d640 1 --2- 172.21.15.77:0/1832693324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 0x5568dbc66990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f5642d9d640 1 -- 172.21.15.77:0/1832693324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc55170 msgr2=0x5568dbc541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:01.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f5642d9d640 1 --2- 172.21.15.77:0/1832693324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc55170 0x5568dbc541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f5642d9d640 1 -- 172.21.15.77:0/1832693324 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568db88e0f0 con 0x5568dbc5acd0 2024-03-16T10:22:01.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.828+0000 7f5642d9d640 1 --2- 172.21.15.77:0/1832693324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc5acd0 0x5568dbc4f780 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f563c0d0c90 tx=0x7f563c013cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b5506d90915b579 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:01.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.829+0000 7f562bfff640 1 -- 172.21.15.77:0/1832693324 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c09fae0 con 0x5568dbc5acd0 2024-03-16T10:22:01.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.829+0000 7f562bfff640 1 -- 172.21.15.77:0/1832693324 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f563c0a3c30 con 0x5568dbc5acd0 2024-03-16T10:22:01.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.829+0000 7f562bfff640 1 -- 172.21.15.77:0/1832693324 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c09c2f0 con 0x5568dbc5acd0 2024-03-16T10:22:01.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.830+0000 7f564b9ef740 1 -- 172.21.15.77:0/1832693324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc5acd0 msgr2=0x5568dbc4f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:01.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.830+0000 7f564b9ef740 1 --2- 172.21.15.77:0/1832693324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc5acd0 0x5568dbc4f780 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f563c0d0c90 tx=0x7f563c013cf0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.830+0000 7f564b9ef740 1 -- 172.21.15.77:0/1832693324 shutdown_connections 2024-03-16T10:22:01.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.830+0000 7f564b9ef740 1 --2- 172.21.15.77:0/1832693324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc5acd0 0x5568dbc4f780 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.830+0000 7f564b9ef740 1 --2- 172.21.15.77:0/1832693324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 0x5568dbc66990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.830+0000 7f564b9ef740 1 --2- 172.21.15.77:0/1832693324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc55170 0x5568dbc541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.830+0000 7f564b9ef740 1 -- 172.21.15.77:0/1832693324 >> 172.21.15.77:0/1832693324 conn(0x5568dbc506e0 msgr2=0x5568dbc4f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:01.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.830+0000 7f564b9ef740 1 -- 172.21.15.77:0/1832693324 shutdown_connections 2024-03-16T10:22:01.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.830+0000 7f564b9ef740 1 -- 172.21.15.77:0/1832693324 wait complete. 2024-03-16T10:22:01.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.830+0000 7f564b9ef740 1 -- start start 2024-03-16T10:22:01.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.830+0000 7f564b9ef740 1 Processor -- start 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564b9ef740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 0x5568dbc64490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564b9ef740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 0x5568dbc649d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564b9ef740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc5acd0 0x5568dbc659e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564b9ef740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f56441a7e80 con 0x5568dbc5acd0 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564b9ef740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f56441a77b0 con 0x5568dbc55b60 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564b9ef740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f56441a49b0 con 0x5568dbc55170 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564b9ef740 10 client.0 _ll_register_callbacks cb 0x7f5634009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f5642d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc5acd0 0x5568dbc659e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564259c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 0x5568dbc64490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564259c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 0x5568dbc64490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58798/0 (socket says 172.21.15.77:58798) 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564259c640 1 -- 172.21.15.77:0/3549584001 learned_addr learned my addr 172.21.15.77:0/3549584001 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564259c640 1 -- 172.21.15.77:0/3549584001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 msgr2=0x5568dbc649d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f5641d9b640 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 0x5568dbc649d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564259c640 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 0x5568dbc649d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564259c640 1 -- 172.21.15.77:0/3549584001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc5acd0 msgr2=0x5568dbc659e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564259c640 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc5acd0 0x5568dbc659e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f564259c640 1 -- 172.21.15.77:0/3549584001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568dbb819d0 con 0x5568dbc55170 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.831+0000 7f5642d9d640 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc5acd0 0x5568dbc659e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.832+0000 7f5641d9b640 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 0x5568dbc649d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.832+0000 7f564259c640 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 0x5568dbc64490 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5634000c00 tx=0x7f563402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f0d59f04ee47fb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.832+0000 7f5629ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.832+0000 7f5629ffb640 1 -- 172.21.15.77:0/3549584001 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5634005d50 con 0x5568dbc55170 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.832+0000 7f564b9ef740 1 -- 172.21.15.77:0/3549584001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f563c0a3c30 con 0x5568dbc55170 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.832+0000 7f5629ffb640 1 -- 172.21.15.77:0/3549584001 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f5634034c30 con 0x5568dbc55170 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.832+0000 7f5629ffb640 1 -- 172.21.15.77:0/3549584001 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5634034e30 con 0x5568dbc55170 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.832+0000 7f564b9ef740 10 client.26575 fetch_fsmap learned FSMap version 35 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.832+0000 7f564b9ef740 1 -- 172.21.15.77:0/3549584001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5568dbcd0e20 con 0x5568dbc55170 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.832+0000 7f5629ffb640 1 -- 172.21.15.77:0/3549584001 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f563c0a3c30 con 0x5568dbc55170 2024-03-16T10:22:01.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.833+0000 7f5629ffb640 1 -- 172.21.15.77:0/3549584001 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f56340568d0 con 0x5568dbc55170 2024-03-16T10:22:01.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.833+0000 7f564b9ef740 10 client.26575 fetch_fsmap finished waiting for FSMap version 35 2024-03-16T10:22:01.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.833+0000 7f564b9ef740 10 client.26575 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:01.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.833+0000 7f564b9ef740 20 client.26575 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:01.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.833+0000 7f564b9ef740 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5568dbcd1790 0x5568dbcd3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:01.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.833+0000 7f564b9ef740 4 client.26575 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:22:01.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.833+0000 7f564b9ef740 1 -- 172.21.15.77:0/3549584001 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5568dbbc1d90 con 0x5568dbcd1790 2024-03-16T10:22:01.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.833+0000 7f5641d9b640 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5568dbcd1790 0x5568dbcd3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:01.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f5641d9b640 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5568dbcd1790 0x5568dbcd3bb0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ee2675ac855d2a3 server_cookie=714471b6c3fe95d in_seq=0 out_seq=0 2024-03-16T10:22:01.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f5629ffb640 10 client.26575 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:01.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f5629ffb640 1 -- 172.21.15.77:0/3549584001 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5568dbbc1d90 con 0x5568dbcd1790 2024-03-16T10:22:01.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f5629ffb640 10 client.26575 handle_command_reply: tid=0 2024-03-16T10:22:01.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f564b9ef740 1 client.26575 shutdown 2024-03-16T10:22:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f564b9ef740 1 -- 172.21.15.77:0/3549584001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 msgr2=0x5568dbc64490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f564b9ef740 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 0x5568dbc64490 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5634000c00 tx=0x7f563402c650 comp rx=0 tx=0).stop 2024-03-16T10:22:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f564b9ef740 1 -- 172.21.15.77:0/3549584001 shutdown_connections 2024-03-16T10:22:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f564b9ef740 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5568dbcd1790 0x5568dbcd3bb0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f564b9ef740 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568dbc5acd0 0x5568dbc659e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f564b9ef740 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5568dbc55b60 0x5568dbc649d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f564b9ef740 1 --2- 172.21.15.77:0/3549584001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568dbc55170 0x5568dbc64490 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f564b9ef740 1 -- 172.21.15.77:0/3549584001 >> 172.21.15.77:0/3549584001 conn(0x5568dbc506e0 msgr2=0x5568dbc4f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.834+0000 7f5629ffb640 0 client.26575 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f564b9ef740 1 -- 172.21.15.77:0/3549584001 shutdown_connections 2024-03-16T10:22:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f564b9ef740 1 -- 172.21.15.77:0/3549584001 wait complete. 2024-03-16T10:22:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f564b9ef740 20 client.26575 trim_cache size 0 max 16384 2024-03-16T10:22:01.840 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:22:01.840 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T10:22:01.840 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T10:22:01.840 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:22:01.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f5648dff640 1 -- 172.21.15.77:0/488824435 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f562c0a2880 msgr2=0x7f562c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:01.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f5648dff640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f562c0a2880 0x7f562c0a4d40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f56380095f0 tx=0x7f5638005f90 comp rx=0 tx=0).stop 2024-03-16T10:22:01.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f5648dff640 1 -- 172.21.15.77:0/488824435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5644103430 msgr2=0x7f564419f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:01.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f5648dff640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5644103430 0x7f564419f140 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f563c009fd0 tx=0x7f563c00ef10 comp rx=0 tx=0).stop 2024-03-16T10:22:01.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f5648dff640 1 -- 172.21.15.77:0/488824435 shutdown_connections 2024-03-16T10:22:01.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f5648dff640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f562c0a2880 0x7f562c0a4d40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f5648dff640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5644103430 0x7f564419f140 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f5648dff640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5644102a70 0x7f56441980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f5648dff640 1 --2- 172.21.15.77:0/488824435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56440ffaa0 0x7f5644197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:01.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.835+0000 7f5648dff640 1 -- 172.21.15.77:0/488824435 >> 172.21.15.77:0/488824435 conn(0x7f56440fb3d0 msgr2=0x7f5644105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:01.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.836+0000 7f5648dff640 1 -- 172.21.15.77:0/488824435 shutdown_connections 2024-03-16T10:22:01.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:01.836+0000 7f5648dff640 1 -- 172.21.15.77:0/488824435 wait complete. 2024-03-16T10:22:01.849 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T10:22:01.849 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:22:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.187+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/3620801687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3b40763b0 msgr2=0x7fd3b4078820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.187+0000 7fd3b8ea1640 1 --2- 172.21.15.77:0/3620801687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3b40763b0 0x7fd3b4078820 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009ca0 tx=0x7fd3ac02c120 comp rx=0 tx=0).stop 2024-03-16T10:22:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.187+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/3620801687 shutdown_connections 2024-03-16T10:22:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.187+0000 7fd3b8ea1640 1 --2- 172.21.15.77:0/3620801687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b4078d60 0x7fd3b407b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.187+0000 7fd3b8ea1640 1 --2- 172.21.15.77:0/3620801687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3b40763b0 0x7fd3b4078820 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009ca0 tx=0x7fd3ac02c120 comp rx=0 tx=0).stop 2024-03-16T10:22:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.187+0000 7fd3b8ea1640 1 --2- 172.21.15.77:0/3620801687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3b40734d0 0x7fd3b40738f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.187+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/3620801687 >> 172.21.15.77:0/3620801687 conn(0x7fd3b406e530 msgr2=0x7fd3b4070970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:02.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.190+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/3620801687 shutdown_connections 2024-03-16T10:22:02.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.190+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/3620801687 wait complete. 2024-03-16T10:22:02.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.191+0000 7fd3b8ea1640 1 Processor -- start 2024-03-16T10:22:02.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.191+0000 7fd3b8ea1640 1 -- start start 2024-03-16T10:22:02.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.191+0000 7fd3b8ea1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3b40734d0 0x7fd3b40809e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:02.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.191+0000 7fd3b8ea1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3b4078d60 0x7fd3b41b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:02.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.191+0000 7fd3b8ea1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b41b4470 0x7fd3b41b4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:02.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.191+0000 7fd3b8ea1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b407ebf0 con 0x7fd3b41b4470 2024-03-16T10:22:02.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.191+0000 7fd3b8ea1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b407ea70 con 0x7fd3b40734d0 2024-03-16T10:22:02.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.191+0000 7fd3b8ea1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b407ed70 con 0x7fd3b4078d60 2024-03-16T10:22:02.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.196+0000 7fd3b2d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b41b4470 0x7fd3b41b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:02.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.196+0000 7fd3b2d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b41b4470 0x7fd3b41b4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59982/0 (socket says 172.21.15.77:59982) 2024-03-16T10:22:02.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.196+0000 7fd3b2d9d640 1 -- 172.21.15.77:0/1715290133 learned_addr learned my addr 172.21.15.77:0/1715290133 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:02.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.197+0000 7fd3b2d9d640 1 -- 172.21.15.77:0/1715290133 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3b4078d60 msgr2=0x7fd3b41b3ee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:22:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.197+0000 7fd3b2d9d640 1 --2- 172.21.15.77:0/1715290133 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3b4078d60 0x7fd3b41b3ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.197+0000 7fd3b2d9d640 1 -- 172.21.15.77:0/1715290133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3b40734d0 msgr2=0x7fd3b40809e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:22:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.197+0000 7fd3b2d9d640 1 --2- 172.21.15.77:0/1715290133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3b40734d0 0x7fd3b40809e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.197+0000 7fd3b2d9d640 1 -- 172.21.15.77:0/1715290133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b41b51e0 con 0x7fd3b41b4470 2024-03-16T10:22:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.197+0000 7fd3b2d9d640 1 --2- 172.21.15.77:0/1715290133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b41b4470 0x7fd3b41b4920 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd3a800da10 tx=0x7fd3a800ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:02.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.220+0000 7fd3a37fe640 1 -- 172.21.15.77:0/1715290133 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8017030 con 0x7fd3b41b4470 2024-03-16T10:22:02.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.221+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/1715290133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b41b54d0 con 0x7fd3b41b4470 2024-03-16T10:22:02.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.221+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/1715290133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b41b5a20 con 0x7fd3b41b4470 2024-03-16T10:22:02.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.227+0000 7fd3a37fe640 1 -- 172.21.15.77:0/1715290133 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3a8009940 con 0x7fd3b41b4470 2024-03-16T10:22:02.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.227+0000 7fd3a37fe640 1 -- 172.21.15.77:0/1715290133 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8003cc0 con 0x7fd3b41b4470 2024-03-16T10:22:02.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.227+0000 7fd3a37fe640 1 -- 172.21.15.77:0/1715290133 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fd3a801a070 con 0x7fd3b41b4470 2024-03-16T10:22:02.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.232+0000 7fd3a37fe640 1 --2- 172.21.15.77:0/1715290133 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd39c0a2a50 0x7fd39c0a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:02.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.232+0000 7fd3a37fe640 1 -- 172.21.15.77:0/1715290133 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fd3a80cf5c0 con 0x7fd3b41b4470 2024-03-16T10:22:02.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.239+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/1715290133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd394005290 con 0x7fd3b41b4470 2024-03-16T10:22:02.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.241+0000 7fd3b259c640 1 --2- 172.21.15.77:0/1715290133 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd39c0a2a50 0x7fd39c0a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:02.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:01 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:22:02.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:01 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:22:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:01 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:22:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:01 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:22:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:01 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:22:02.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.250+0000 7fd3a37fe640 1 -- 172.21.15.77:0/1715290133 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd3a809c6d0 con 0x7fd3b41b4470 2024-03-16T10:22:02.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:02.251+0000 7fd3b259c640 1 --2- 172.21.15.77:0/1715290133 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd39c0a2a50 0x7fd39c0a4f10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400b990 tx=0x7fd3a400b370 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:01 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:22:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:01 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:22:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:01 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:22:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:01 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:22:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:01 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:22:02.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:01 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:22:02.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:01 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:22:02.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:01 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:22:02.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:01 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:22:02.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:01 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:22:03.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:02 smithi077 ceph-mon[32400]: pgmap v499: 97 pgs: 97 active+clean; 598 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 674 op/s 2024-03-16T10:22:03.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:02 smithi193 ceph-mon[36311]: pgmap v499: 97 pgs: 97 active+clean; 598 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 674 op/s 2024-03-16T10:22:03.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:02 smithi192 ceph-mon[36354]: pgmap v499: 97 pgs: 97 active+clean; 598 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 674 op/s 2024-03-16T10:22:03.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.495+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/1715290133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd394005580 con 0x7fd3b41b4470 2024-03-16T10:22:03.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.497+0000 7fd3a37fe640 1 -- 172.21.15.77:0/1715290133 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11943 (secure 0 0 0) 0x7fd3a809bfd0 con 0x7fd3b41b4470 2024-03-16T10:22:03.500 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:22:03.500 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:21:44.704487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3768363028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3768363028},{"type":"v1","addr":"172.21.15.193:6835","nonce":3768363028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24475,24454,24482,14658,24455]},"id":1}]} 2024-03-16T10:22:03.501 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 35 2024-03-16T10:22:03.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.535+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/1715290133 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd39c0a2a50 msgr2=0x7fd39c0a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:03.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.535+0000 7fd3b8ea1640 1 --2- 172.21.15.77:0/1715290133 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd39c0a2a50 0x7fd39c0a4f10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400b990 tx=0x7fd3a400b370 comp rx=0 tx=0).stop 2024-03-16T10:22:03.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.536+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/1715290133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b41b4470 msgr2=0x7fd3b41b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:03.537 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.536+0000 7fd3b8ea1640 1 --2- 172.21.15.77:0/1715290133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b41b4470 0x7fd3b41b4920 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd3a800da10 tx=0x7fd3a800ded0 comp rx=0 tx=0).stop 2024-03-16T10:22:03.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.538+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/1715290133 shutdown_connections 2024-03-16T10:22:03.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.538+0000 7fd3b8ea1640 1 --2- 172.21.15.77:0/1715290133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b41b4470 0x7fd3b41b4920 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:03.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.538+0000 7fd3b8ea1640 1 --2- 172.21.15.77:0/1715290133 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fd39c0a2a50 0x7fd39c0a4f10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:03.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.538+0000 7fd3b8ea1640 1 --2- 172.21.15.77:0/1715290133 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3b4078d60 0x7fd3b41b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:03.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.538+0000 7fd3b8ea1640 1 --2- 172.21.15.77:0/1715290133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3b40734d0 0x7fd3b40809e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:03.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.538+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/1715290133 >> 172.21.15.77:0/1715290133 conn(0x7fd3b406e530 msgr2=0x7fd3b4079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:03.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.539+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/1715290133 shutdown_connections 2024-03-16T10:22:03.540 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:03.539+0000 7fd3b8ea1640 1 -- 172.21.15.77:0/1715290133 wait complete. 2024-03-16T10:22:03.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-16T10:22:03.977 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:03 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1715290133' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:22:04.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.001+0000 7fc747863640 1 -- 172.21.15.77:0/1127493685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc74010b150 msgr2=0x7fc74010d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:04.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.001+0000 7fc747863640 1 --2- 172.21.15.77:0/1127493685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc74010b150 0x7fc74010d5c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc738009ca0 tx=0x7fc73802c120 comp rx=0 tx=0).stop 2024-03-16T10:22:04.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.001+0000 7fc747863640 1 -- 172.21.15.77:0/1127493685 shutdown_connections 2024-03-16T10:22:04.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.001+0000 7fc747863640 1 --2- 172.21.15.77:0/1127493685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc74010db00 0x7fc74010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.001+0000 7fc747863640 1 --2- 172.21.15.77:0/1127493685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc74010b150 0x7fc74010d5c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.002+0000 7fc747863640 1 --2- 172.21.15.77:0/1127493685 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7401087e0 0x7fc74010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.002+0000 7fc747863640 1 -- 172.21.15.77:0/1127493685 >> 172.21.15.77:0/1127493685 conn(0x7fc74006e640 msgr2=0x7fc740070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:04.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.005+0000 7fc747863640 1 -- 172.21.15.77:0/1127493685 shutdown_connections 2024-03-16T10:22:04.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.005+0000 7fc747863640 1 -- 172.21.15.77:0/1127493685 wait complete. 2024-03-16T10:22:04.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.005+0000 7fc747863640 1 Processor -- start 2024-03-16T10:22:04.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc747863640 1 -- start start 2024-03-16T10:22:04.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc747863640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7401087e0 0x7fc7401a6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc747863640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc74010b150 0x7fc7401a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc747863640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc74010db00 0x7fc7401ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc747863640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc74011c3a0 con 0x7fc74010b150 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc747863640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc74011c220 con 0x7fc74010db00 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc747863640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc74011c520 con 0x7fc7401087e0 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc7455ff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7401087e0 0x7fc7401a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc7455ff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7401087e0 0x7fc7401a6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58810/0 (socket says 172.21.15.77:58810) 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc745e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc74010db00 0x7fc7401ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc7455ff640 1 -- 172.21.15.77:0/2849372376 learned_addr learned my addr 172.21.15.77:0/2849372376 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc744dfe640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc74010b150 0x7fc7401a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc7455ff640 1 -- 172.21.15.77:0/2849372376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc74010db00 msgr2=0x7fc7401ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc7455ff640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc74010db00 0x7fc7401ae090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.006+0000 7fc7455ff640 1 -- 172.21.15.77:0/2849372376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc74010b150 msgr2=0x7fc7401a7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.007+0000 7fc7455ff640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc74010b150 0x7fc7401a7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.007+0000 7fc7455ff640 1 -- 172.21.15.77:0/2849372376 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7401ae700 con 0x7fc7401087e0 2024-03-16T10:22:04.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.007+0000 7fc745e00640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc74010db00 0x7fc7401ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:04.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.007+0000 7fc744dfe640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc74010b150 0x7fc7401a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:22:04.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.007+0000 7fc7455ff640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7401087e0 0x7fc7401a6ae0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc73000b450 tx=0x7fc73000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:04.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.012+0000 7fc7367fc640 1 -- 172.21.15.77:0/2849372376 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730015020 con 0x7fc7401087e0 2024-03-16T10:22:04.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.012+0000 7fc7367fc640 1 -- 172.21.15.77:0/2849372376 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc730011d70 con 0x7fc7401087e0 2024-03-16T10:22:04.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.012+0000 7fc7367fc640 1 -- 172.21.15.77:0/2849372376 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73000f340 con 0x7fc7401087e0 2024-03-16T10:22:04.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.012+0000 7fc747863640 1 -- 172.21.15.77:0/2849372376 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7401ae9f0 con 0x7fc7401087e0 2024-03-16T10:22:04.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.012+0000 7fc747863640 1 -- 172.21.15.77:0/2849372376 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7401aef40 con 0x7fc7401087e0 2024-03-16T10:22:04.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.016+0000 7fc74a44a740 1 Processor -- start 2024-03-16T10:22:04.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.016+0000 7fc74a44a740 1 -- start start 2024-03-16T10:22:04.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.017+0000 7fc74a44a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b7850 0x56262a0b7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.017+0000 7fc74a44a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b8240 0x56262a0bcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc74a44a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0bd3b0 0x56262a0bf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc74a44a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56262a01c490 con 0x56262a0b8240 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc74a44a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x562629fc4c90 con 0x56262a0bd3b0 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc74a44a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56262a018530 con 0x56262a0b7850 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc744dfe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0bd3b0 0x56262a0bf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc745e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b7850 0x56262a0b7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc744dfe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0bd3b0 0x56262a0bf840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54108/0 (socket says 172.21.15.77:54108) 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc744dfe640 1 -- 172.21.15.77:0/3506947283 learned_addr learned my addr 172.21.15.77:0/3506947283 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc745e00640 1 --2- 172.21.15.77:0/3506947283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b7850 0x56262a0b7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58812/0 (socket says 172.21.15.77:58812) 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc7455ff640 1 --2- 172.21.15.77:0/3506947283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b8240 0x56262a0bcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc745e00640 1 -- 172.21.15.77:0/3506947283 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0bd3b0 msgr2=0x56262a0bf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc745e00640 1 --2- 172.21.15.77:0/3506947283 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0bd3b0 0x56262a0bf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc745e00640 1 -- 172.21.15.77:0/3506947283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b8240 msgr2=0x56262a0bcd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:04.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc745e00640 1 --2- 172.21.15.77:0/3506947283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b8240 0x56262a0bcd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc745e00640 1 -- 172.21.15.77:0/3506947283 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56262a07f500 con 0x56262a0b7850 2024-03-16T10:22:04.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc744dfe640 1 --2- 172.21.15.77:0/3506947283 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0bd3b0 0x56262a0bf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:04.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.018+0000 7fc7455ff640 1 --2- 172.21.15.77:0/3506947283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b8240 0x56262a0bcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:22:04.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.019+0000 7fc7367fc640 1 -- 172.21.15.77:0/2849372376 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fc730018020 con 0x7fc7401087e0 2024-03-16T10:22:04.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.019+0000 7fc745e00640 1 --2- 172.21.15.77:0/3506947283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b7850 0x56262a0b7c70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc73c009cb0 tx=0x7fc73c00e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d46710fe2382de5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:04.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.020+0000 7fc7367fc640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc72c0a2a00 0x7fc72c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.020+0000 7fc72b7fe640 1 -- 172.21.15.77:0/3506947283 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c017070 con 0x56262a0b7850 2024-03-16T10:22:04.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.020+0000 7fc72b7fe640 1 -- 172.21.15.77:0/3506947283 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc73c013950 con 0x56262a0b7850 2024-03-16T10:22:04.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.020+0000 7fc7367fc640 1 -- 172.21.15.77:0/2849372376 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc7300cf4b0 con 0x7fc7401087e0 2024-03-16T10:22:04.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.020+0000 7fc7455ff640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc72c0a2a00 0x7fc72c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.021+0000 7fc74a44a740 1 -- 172.21.15.77:0/3506947283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b7850 msgr2=0x56262a0b7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:04.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.021+0000 7fc7455ff640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc72c0a2a00 0x7fc72c0a4ec0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc730013000 tx=0x7fc730006df0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:04.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.021+0000 7fc74a44a740 1 --2- 172.21.15.77:0/3506947283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b7850 0x56262a0b7c70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc73c009cb0 tx=0x7fc73c00e5a0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.031+0000 7fc74a44a740 1 -- 172.21.15.77:0/3506947283 shutdown_connections 2024-03-16T10:22:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.031+0000 7fc74a44a740 1 --2- 172.21.15.77:0/3506947283 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0bd3b0 0x56262a0bf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.031+0000 7fc74a44a740 1 --2- 172.21.15.77:0/3506947283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b8240 0x56262a0bcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.031+0000 7fc74a44a740 1 --2- 172.21.15.77:0/3506947283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b7850 0x56262a0b7c70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.031+0000 7fc74a44a740 1 -- 172.21.15.77:0/3506947283 >> 172.21.15.77:0/3506947283 conn(0x56262a0b2dc0 msgr2=0x56262a0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:04.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.031+0000 7fc74a44a740 1 -- 172.21.15.77:0/3506947283 shutdown_connections 2024-03-16T10:22:04.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.031+0000 7fc74a44a740 1 -- 172.21.15.77:0/3506947283 wait complete. 2024-03-16T10:22:04.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.032+0000 7fc74a44a740 1 -- start start 2024-03-16T10:22:04.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.032+0000 7fc74a44a740 1 Processor -- start 2024-03-16T10:22:04.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.032+0000 7fc74a44a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 0x56262a0c1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.032+0000 7fc74a44a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0b8240 0x56262a0c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.032+0000 7fc74a44a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0bd3b0 0x56262a0c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.032+0000 7fc74a44a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56262a0c03b0 con 0x56262a0b7850 2024-03-16T10:22:04.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.032+0000 7fc74a44a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x562629fd8230 con 0x56262a0b8240 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.032+0000 7fc74a44a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56262a0c0530 con 0x56262a0bd3b0 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.032+0000 7fc74a44a740 10 client.0 _ll_register_callbacks cb 0x56262a014010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.033+0000 7fc745e00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 0x56262a0c1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.033+0000 7fc745e00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 0x56262a0c1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60028/0 (socket says 172.21.15.77:60028) 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.033+0000 7fc745e00640 1 -- 172.21.15.77:0/2488030410 learned_addr learned my addr 172.21.15.77:0/2488030410 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.033+0000 7fc744dfe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0b8240 0x56262a0c22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.033+0000 7fc745e00640 1 --2- 172.21.15.77:0/2488030410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0bd3b0 0x56262a0c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.033+0000 7fc745e00640 1 -- 172.21.15.77:0/2488030410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0b8240 msgr2=0x56262a0c22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.033+0000 7fc745e00640 1 --2- 172.21.15.77:0/2488030410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0b8240 0x56262a0c22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.033+0000 7fc745e00640 1 -- 172.21.15.77:0/2488030410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 msgr2=0x56262a0c1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.033+0000 7fc745e00640 1 --2- 172.21.15.77:0/2488030410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 0x56262a0c1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.033+0000 7fc745e00640 1 -- 172.21.15.77:0/2488030410 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562629fe7100 con 0x56262a0bd3b0 2024-03-16T10:22:04.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.034+0000 7fc745e00640 1 --2- 172.21.15.77:0/2488030410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0bd3b0 0x56262a0c92f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc73c006aa0 tx=0x7fc73c0047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8edfab65c15ed9ef server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:04.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.076+0000 7fc728ff9640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:22:04.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.077+0000 7fc728ff9640 1 -- 172.21.15.77:0/2488030410 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c017040 con 0x56262a0bd3b0 2024-03-16T10:22:04.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.077+0000 7fc74a44a740 1 -- 172.21.15.77:0/2488030410 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc73c013950 con 0x56262a0bd3b0 2024-03-16T10:22:04.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.077+0000 7fc728ff9640 1 -- 172.21.15.77:0/2488030410 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc73c012a30 con 0x56262a0bd3b0 2024-03-16T10:22:04.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.077+0000 7fc728ff9640 1 -- 172.21.15.77:0/2488030410 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc73c012c30 con 0x56262a0bd3b0 2024-03-16T10:22:04.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.077+0000 7fc74a44a740 10 client.26590 fetch_fsmap learned FSMap version 35 2024-03-16T10:22:04.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.077+0000 7fc74a44a740 1 -- 172.21.15.77:0/2488030410 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562629d303f0 con 0x56262a0bd3b0 2024-03-16T10:22:04.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.078+0000 7fc728ff9640 1 -- 172.21.15.77:0/2488030410 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc73c013950 con 0x56262a0bd3b0 2024-03-16T10:22:04.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.093+0000 7fc728ff9640 1 -- 172.21.15.77:0/2488030410 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fc73c033050 con 0x56262a0bd3b0 2024-03-16T10:22:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.093+0000 7fc74a44a740 10 client.26590 fetch_fsmap finished waiting for FSMap version 35 2024-03-16T10:22:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.093+0000 7fc74a44a740 10 client.26590 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:22:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.093+0000 7fc74a44a740 20 client.26590 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.093+0000 7fc74a44a740 1 --2- 172.21.15.77:0/2488030410 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56262a0cde30 0x56262a0d0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.093+0000 7fc74a44a740 4 client.26590 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.093+0000 7fc74a44a740 1 -- 172.21.15.77:0/2488030410 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56262a0299d0 con 0x56262a0cde30 2024-03-16T10:22:04.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.093+0000 7fc744dfe640 1 --2- 172.21.15.77:0/2488030410 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56262a0cde30 0x56262a0d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.094+0000 7fc744dfe640 1 --2- 172.21.15.77:0/2488030410 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56262a0cde30 0x56262a0d0270 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c24a1d5b9675a6bb server_cookie=10cedae135ccbaee in_seq=0 out_seq=0 2024-03-16T10:22:04.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.095+0000 7fc728ff9640 10 client.26590 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:22:04.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.095+0000 7fc728ff9640 1 -- 172.21.15.77:0/2488030410 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56262a0299d0 con 0x56262a0cde30 2024-03-16T10:22:04.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.095+0000 7fc728ff9640 10 client.26590 handle_command_reply: tid=0 2024-03-16T10:22:04.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.099+0000 7fc74a44a740 1 client.26590 shutdown 2024-03-16T10:22:04.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.101+0000 7fc74a44a740 1 -- 172.21.15.77:0/2488030410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0bd3b0 msgr2=0x56262a0c92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:04.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.101+0000 7fc74a44a740 1 --2- 172.21.15.77:0/2488030410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0bd3b0 0x56262a0c92f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc73c006aa0 tx=0x7fc73c0047d0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.101+0000 7fc74a44a740 1 -- 172.21.15.77:0/2488030410 shutdown_connections 2024-03-16T10:22:04.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.101+0000 7fc74a44a740 1 --2- 172.21.15.77:0/2488030410 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56262a0cde30 0x56262a0d0270 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.101+0000 7fc74a44a740 1 --2- 172.21.15.77:0/2488030410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0bd3b0 0x56262a0c92f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.101+0000 7fc74a44a740 1 --2- 172.21.15.77:0/2488030410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0b8240 0x56262a0c22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.101+0000 7fc74a44a740 1 --2- 172.21.15.77:0/2488030410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 0x56262a0c1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.101+0000 7fc74a44a740 1 -- 172.21.15.77:0/2488030410 >> 172.21.15.77:0/2488030410 conn(0x56262a0b2dc0 msgr2=0x56262a0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:04.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.101+0000 7fc728ff9640 0 client.26590 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:22:04.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.102+0000 7fc74a44a740 1 -- 172.21.15.77:0/2488030410 shutdown_connections 2024-03-16T10:22:04.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.102+0000 7fc74a44a740 1 -- 172.21.15.77:0/2488030410 wait complete. 2024-03-16T10:22:04.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.102+0000 7fc74a44a740 20 client.26590 trim_cache size 0 max 16384 2024-03-16T10:22:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:03 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1715290133' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:22:04.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.255+0000 7fc74a44a740 1 Processor -- start 2024-03-16T10:22:04.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.255+0000 7fc74a44a740 1 -- start start 2024-03-16T10:22:04.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.269+0000 7fc74a44a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0b7850 0x56262a0b96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.269+0000 7fc74a44a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b8240 0x56262a0b9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.269+0000 7fc74a44a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0bd3b0 0x56262a0c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.269+0000 7fc745e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0b7850 0x56262a0b96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.269+0000 7fc74a44a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc72400e2b0 con 0x56262a0bd3b0 2024-03-16T10:22:04.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.269+0000 7fc74a44a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc72400fe80 con 0x56262a0b7850 2024-03-16T10:22:04.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.269+0000 7fc74a44a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc72400e980 con 0x56262a0b8240 2024-03-16T10:22:04.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.270+0000 7fc744dfe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b8240 0x56262a0b9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.270+0000 7fc744dfe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b8240 0x56262a0b9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58820/0 (socket says 172.21.15.77:58820) 2024-03-16T10:22:04.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.270+0000 7fc744dfe640 1 -- 172.21.15.77:0/574306025 learned_addr learned my addr 172.21.15.77:0/574306025 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:04.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.269+0000 7fc745e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0b7850 0x56262a0b96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54116/0 (socket says 172.21.15.77:54116) 2024-03-16T10:22:04.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.270+0000 7fc745e00640 1 --2- 172.21.15.77:0/574306025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0bd3b0 0x56262a0c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.270+0000 7fc744dfe640 1 -- 172.21.15.77:0/574306025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0b7850 msgr2=0x56262a0b96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:04.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.270+0000 7fc744dfe640 1 --2- 172.21.15.77:0/574306025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0b7850 0x56262a0b96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.270+0000 7fc744dfe640 1 -- 172.21.15.77:0/574306025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0bd3b0 msgr2=0x56262a0c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:04.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.270+0000 7fc744dfe640 1 --2- 172.21.15.77:0/574306025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0bd3b0 0x56262a0c92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.270+0000 7fc744dfe640 1 -- 172.21.15.77:0/574306025 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562629dfa7d0 con 0x56262a0b8240 2024-03-16T10:22:04.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.270+0000 7fc745e00640 1 --2- 172.21.15.77:0/574306025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0bd3b0 0x56262a0c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:22:04.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.271+0000 7fc744dfe640 1 --2- 172.21.15.77:0/574306025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b8240 0x56262a0b9be0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc738032530 tx=0x7fc73802c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fb8be5e14d2798b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:04.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.281+0000 7fc72affd640 1 -- 172.21.15.77:0/574306025 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73804f070 con 0x56262a0b8240 2024-03-16T10:22:04.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.281+0000 7fc72affd640 1 -- 172.21.15.77:0/574306025 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc738003970 con 0x56262a0b8240 2024-03-16T10:22:04.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.282+0000 7fc72affd640 1 -- 172.21.15.77:0/574306025 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738039040 con 0x56262a0b8240 2024-03-16T10:22:04.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.284+0000 7fc74a44a740 1 -- 172.21.15.77:0/574306025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b8240 msgr2=0x56262a0b9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:04.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.284+0000 7fc74a44a740 1 --2- 172.21.15.77:0/574306025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b8240 0x56262a0b9be0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc738032530 tx=0x7fc73802c640 comp rx=0 tx=0).stop 2024-03-16T10:22:04.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.299+0000 7fc74a44a740 1 -- 172.21.15.77:0/574306025 shutdown_connections 2024-03-16T10:22:04.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.299+0000 7fc74a44a740 1 --2- 172.21.15.77:0/574306025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0bd3b0 0x56262a0c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.299+0000 7fc74a44a740 1 --2- 172.21.15.77:0/574306025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b8240 0x56262a0b9be0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.299+0000 7fc74a44a740 1 --2- 172.21.15.77:0/574306025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0b7850 0x56262a0b96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.299+0000 7fc74a44a740 1 -- 172.21.15.77:0/574306025 >> 172.21.15.77:0/574306025 conn(0x56262a0b2dc0 msgr2=0x56262a0b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:04.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.302+0000 7fc74a44a740 1 -- 172.21.15.77:0/574306025 shutdown_connections 2024-03-16T10:22:04.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.302+0000 7fc74a44a740 1 -- 172.21.15.77:0/574306025 wait complete. 2024-03-16T10:22:04.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.303+0000 7fc74a44a740 1 -- start start 2024-03-16T10:22:04.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.303+0000 7fc74a44a740 1 Processor -- start 2024-03-16T10:22:04.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.303+0000 7fc74a44a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 0x56262a0c6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.304+0000 7fc74a44a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b8240 0x56262a0c6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.304+0000 7fc745e00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 0x56262a0c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.304+0000 7fc74a44a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0bd3b0 0x56262a1321a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.304+0000 7fc74a44a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc72400f7b0 con 0x56262a0b7850 2024-03-16T10:22:04.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.304+0000 7fc745e00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 0x56262a0c6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60052/0 (socket says 172.21.15.77:60052) 2024-03-16T10:22:04.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.304+0000 7fc745e00640 1 -- 172.21.15.77:0/3930158863 learned_addr learned my addr 172.21.15.77:0/3930158863 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:04.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.304+0000 7fc744dfe640 1 --2- 172.21.15.77:0/3930158863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b8240 0x56262a0c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.304+0000 7fc74a44a740 1 -- 172.21.15.77:0/3930158863 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc72400f0e0 con 0x56262a0bd3b0 2024-03-16T10:22:04.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.304+0000 7fc74a44a740 1 -- 172.21.15.77:0/3930158863 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc72400c340 con 0x56262a0b8240 2024-03-16T10:22:04.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.305+0000 7fc745e00640 1 -- 172.21.15.77:0/3930158863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b8240 msgr2=0x56262a0c6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:04.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.305+0000 7fc745e00640 1 --2- 172.21.15.77:0/3930158863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b8240 0x56262a0c6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.305+0000 7fc745e00640 1 -- 172.21.15.77:0/3930158863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0bd3b0 msgr2=0x56262a1321a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:22:04.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.305+0000 7fc745e00640 1 --2- 172.21.15.77:0/3930158863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0bd3b0 0x56262a1321a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.305+0000 7fc745e00640 1 -- 172.21.15.77:0/3930158863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56262a0c6b70 con 0x56262a0b7850 2024-03-16T10:22:04.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.305+0000 7fc745e00640 1 --2- 172.21.15.77:0/3930158863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 0x56262a0c6040 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc73c034600 tx=0x7fc73c034f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90badc01f49afe94 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:04.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.306+0000 7fc74a44a740 10 client.0 _ll_register_callbacks cb 0x7fc73c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:04.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.306+0000 7fc74a44a740 1 -- 172.21.15.77:0/3930158863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc738003970 con 0x56262a0b7850 2024-03-16T10:22:04.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.330+0000 7fc728ff9640 10 client.16842 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:22:04.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.331+0000 7fc728ff9640 1 -- 172.21.15.77:0/3930158863 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c017070 con 0x56262a0b7850 2024-03-16T10:22:04.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.331+0000 7fc728ff9640 1 -- 172.21.15.77:0/3930158863 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc73c025040 con 0x56262a0b7850 2024-03-16T10:22:04.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.331+0000 7fc728ff9640 1 -- 172.21.15.77:0/3930158863 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc73c0123d0 con 0x56262a0b7850 2024-03-16T10:22:04.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.331+0000 7fc74a44a740 10 client.16842 fetch_fsmap learned FSMap version 35 2024-03-16T10:22:04.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.331+0000 7fc74a44a740 1 -- 172.21.15.77:0/3930158863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=35}) -- 0x56262a133420 con 0x56262a0b7850 2024-03-16T10:22:04.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.337+0000 7fc728ff9640 1 -- 172.21.15.77:0/3930158863 <== mon.0 v2:172.21.15.77:3300/0 4 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fc73c033040 con 0x56262a0b7850 2024-03-16T10:22:04.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.340+0000 7fc74a44a740 10 client.16842 fetch_fsmap finished waiting for FSMap version 35 2024-03-16T10:22:04.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.340+0000 7fc74a44a740 10 client.16842 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:22:04.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.341+0000 7fc74a44a740 20 client.16842 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:04.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.342+0000 7fc74a44a740 1 --2- 172.21.15.77:0/3930158863 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56262a133db0 0x56262a136270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:04.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.342+0000 7fc744dfe640 1 --2- 172.21.15.77:0/3930158863 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56262a133db0 0x56262a136270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:04.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.343+0000 7fc744dfe640 1 --2- 172.21.15.77:0/3930158863 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56262a133db0 0x56262a136270 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e227298a69975a00 server_cookie=e4ccdbd6f7bc199 in_seq=0 out_seq=0 2024-03-16T10:22:04.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.343+0000 7fc74a44a740 4 client.16842 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:22:04.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.343+0000 7fc74a44a740 1 -- 172.21.15.77:0/3930158863 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fc738003970 con 0x56262a133db0 2024-03-16T10:22:04.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.344+0000 7fc728ff9640 10 client.16842 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:22:04.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.344+0000 7fc728ff9640 1 -- 172.21.15.77:0/3930158863 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fc738003970 con 0x56262a133db0 2024-03-16T10:22:04.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.344+0000 7fc728ff9640 10 client.16842 handle_command_reply: tid=0 2024-03-16T10:22:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.345+0000 7fc74a44a740 1 client.16842 shutdown 2024-03-16T10:22:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.346+0000 7fc74a44a740 1 -- 172.21.15.77:0/3930158863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 msgr2=0x56262a0c6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:04.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.346+0000 7fc74a44a740 1 --2- 172.21.15.77:0/3930158863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 0x56262a0c6040 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc73c034600 tx=0x7fc73c034f60 comp rx=0 tx=0).stop 2024-03-16T10:22:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.346+0000 7fc74a44a740 1 -- 172.21.15.77:0/3930158863 shutdown_connections 2024-03-16T10:22:04.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.346+0000 7fc74a44a740 1 --2- 172.21.15.77:0/3930158863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56262a0bd3b0 0x56262a1321a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.346+0000 7fc74a44a740 1 --2- 172.21.15.77:0/3930158863 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x56262a133db0 0x56262a136270 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.346+0000 7fc74a44a740 1 --2- 172.21.15.77:0/3930158863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56262a0b8240 0x56262a0c6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.346+0000 7fc74a44a740 1 --2- 172.21.15.77:0/3930158863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56262a0b7850 0x56262a0c6040 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.347+0000 7fc74a44a740 1 -- 172.21.15.77:0/3930158863 >> 172.21.15.77:0/3930158863 conn(0x56262a0b2dc0 msgr2=0x56262a0c1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:04.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.347+0000 7fc728ff9640 0 client.16842 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:22:04.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.349+0000 7fc74a44a740 1 -- 172.21.15.77:0/3930158863 shutdown_connections 2024-03-16T10:22:04.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.349+0000 7fc74a44a740 1 -- 172.21.15.77:0/3930158863 wait complete. 2024-03-16T10:22:04.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.349+0000 7fc74a44a740 20 client.16842 trim_cache size 0 max 16384 2024-03-16T10:22:04.350 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:22:04.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.368+0000 7fc747863640 1 -- 172.21.15.77:0/2849372376 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc72c0a2a00 msgr2=0x7fc72c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:04.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.368+0000 7fc747863640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc72c0a2a00 0x7fc72c0a4ec0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc730013000 tx=0x7fc730006df0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.368+0000 7fc747863640 1 -- 172.21.15.77:0/2849372376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7401087e0 msgr2=0x7fc7401a6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:04.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.369+0000 7fc747863640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7401087e0 0x7fc7401a6ae0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc73000b450 tx=0x7fc73000b910 comp rx=0 tx=0).stop 2024-03-16T10:22:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.370+0000 7fc747863640 1 -- 172.21.15.77:0/2849372376 shutdown_connections 2024-03-16T10:22:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.370+0000 7fc747863640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc74010db00 0x7fc7401ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.370+0000 7fc747863640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc74010b150 0x7fc7401a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.370+0000 7fc747863640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc72c0a2a00 0x7fc72c0a4ec0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.370+0000 7fc747863640 1 --2- 172.21.15.77:0/2849372376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7401087e0 0x7fc7401a6ae0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.370+0000 7fc747863640 1 -- 172.21.15.77:0/2849372376 >> 172.21.15.77:0/2849372376 conn(0x7fc74006e640 msgr2=0x7fc740109b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.370+0000 7fc747863640 1 -- 172.21.15.77:0/2849372376 shutdown_connections 2024-03-16T10:22:04.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:04.370+0000 7fc747863640 1 -- 172.21.15.77:0/2849372376 wait complete. 2024-03-16T10:22:04.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:03 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1715290133' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:22:04.595 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-16T10:22:04.986 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:04 smithi077 ceph-mon[32400]: pgmap v500: 97 pgs: 97 active+clean; 596 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 435 op/s 2024-03-16T10:22:05.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:04 smithi193 ceph-mon[36311]: pgmap v500: 97 pgs: 97 active+clean; 596 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 435 op/s 2024-03-16T10:22:05.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:04 smithi192 ceph-mon[36354]: pgmap v500: 97 pgs: 97 active+clean; 596 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 435 op/s 2024-03-16T10:22:06.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.786+0000 7fc5ddd59640 1 -- 172.21.15.77:0/505334726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5d8108ef0 msgr2=0x7fc5d8109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:06.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.786+0000 7fc5ddd59640 1 --2- 172.21.15.77:0/505334726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5d8108ef0 0x7fc5d8109350 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009ca0 tx=0x7fc5d002c1a0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.786+0000 7fc5ddd59640 1 -- 172.21.15.77:0/505334726 shutdown_connections 2024-03-16T10:22:06.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.786+0000 7fc5ddd59640 1 --2- 172.21.15.77:0/505334726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d8109890 0x7fc5d810e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.786+0000 7fc5ddd59640 1 --2- 172.21.15.77:0/505334726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5d8108ef0 0x7fc5d8109350 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.786+0000 7fc5ddd59640 1 --2- 172.21.15.77:0/505334726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5d8072bc0 0x7fc5d8072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.786+0000 7fc5ddd59640 1 -- 172.21.15.77:0/505334726 >> 172.21.15.77:0/505334726 conn(0x7fc5d806e6c0 msgr2=0x7fc5d8070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:06.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.788+0000 7fc5ddd59640 1 -- 172.21.15.77:0/505334726 shutdown_connections 2024-03-16T10:22:06.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.789+0000 7fc5ddd59640 1 -- 172.21.15.77:0/505334726 wait complete. 2024-03-16T10:22:06.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.790+0000 7fc5ddd59640 1 Processor -- start 2024-03-16T10:22:06.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.790+0000 7fc5ddd59640 1 -- start start 2024-03-16T10:22:06.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.791+0000 7fc5ddd59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5d8072bc0 0x7fc5d81a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:06.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.791+0000 7fc5ddd59640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d8109890 0x7fc5d81a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:06.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.791+0000 7fc5ddd59640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5d81abca0 0x7fc5d81ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:06.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.791+0000 7fc5ddd59640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d811c260 con 0x7fc5d8072bc0 2024-03-16T10:22:06.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.791+0000 7fc5ddd59640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d811c0e0 con 0x7fc5d81abca0 2024-03-16T10:22:06.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.791+0000 7fc5ddd59640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d811c3e0 con 0x7fc5d8109890 2024-03-16T10:22:06.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.791+0000 7fc5d6ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d8109890 0x7fc5d81a6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:06.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.791+0000 7fc5d7fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5d81abca0 0x7fc5d81ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:06.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.791+0000 7fc5d7fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5d81abca0 0x7fc5d81ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54138/0 (socket says 172.21.15.77:54138) 2024-03-16T10:22:06.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.791+0000 7fc5d7fff640 1 -- 172.21.15.77:0/3506195806 learned_addr learned my addr 172.21.15.77:0/3506195806 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:06.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.792+0000 7fc5d7fff640 1 -- 172.21.15.77:0/3506195806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d8109890 msgr2=0x7fc5d81a6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:06.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.792+0000 7fc5d7fff640 1 --2- 172.21.15.77:0/3506195806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d8109890 0x7fc5d81a6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.792+0000 7fc5d7fff640 1 -- 172.21.15.77:0/3506195806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5d8072bc0 msgr2=0x7fc5d81a6700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:22:06.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.792+0000 7fc5d7fff640 1 --2- 172.21.15.77:0/3506195806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5d8072bc0 0x7fc5d81a6700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.792+0000 7fc5d7fff640 1 -- 172.21.15.77:0/3506195806 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d8110040 con 0x7fc5d81abca0 2024-03-16T10:22:06.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.792+0000 7fc5d7fff640 1 --2- 172.21.15.77:0/3506195806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5d81abca0 0x7fc5d81ae090 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800e970 tx=0x7fc5c800ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:06.806 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.806+0000 7fc5d4ff9640 1 -- 172.21.15.77:0/3506195806 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c800ccc0 con 0x7fc5d81abca0 2024-03-16T10:22:06.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.806+0000 7fc5ddd59640 1 -- 172.21.15.77:0/3506195806 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d8110330 con 0x7fc5d81abca0 2024-03-16T10:22:06.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.806+0000 7fc5ddd59640 1 -- 172.21.15.77:0/3506195806 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d8110850 con 0x7fc5d81abca0 2024-03-16T10:22:06.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.812+0000 7fc5e0a15740 1 Processor -- start 2024-03-16T10:22:06.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.813+0000 7fc5e0a15740 1 -- start start 2024-03-16T10:22:06.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.813+0000 7fc5e0a15740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 0x5648331ace60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:06.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.813+0000 7fc5e0a15740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331ad430 0x5648331b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:06.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.813+0000 7fc5e0a15740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331b25a0 0x5648331b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:06.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.813+0000 7fc5e0a15740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564833111490 con 0x5648331ad430 2024-03-16T10:22:06.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.813+0000 7fc5e0a15740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5648330b9c90 con 0x5648331b25a0 2024-03-16T10:22:06.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.813+0000 7fc5e0a15740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56483310d530 con 0x5648331aca40 2024-03-16T10:22:06.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.813+0000 7fc5d6ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 0x5648331ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.813+0000 7fc5d6ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 0x5648331ace60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58854/0 (socket says 172.21.15.77:58854) 2024-03-16T10:22:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.813+0000 7fc5d6ffd640 1 -- 172.21.15.77:0/496521173 learned_addr learned my addr 172.21.15.77:0/496521173 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.813+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/496521173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331b25a0 0x5648331b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.813+0000 7fc5d77fe640 1 --2- 172.21.15.77:0/496521173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331ad430 0x5648331b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:06.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.814+0000 7fc5d6ffd640 1 -- 172.21.15.77:0/496521173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331b25a0 msgr2=0x5648331b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:06.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.814+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/496521173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331b25a0 0x5648331b4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.814+0000 7fc5d6ffd640 1 -- 172.21.15.77:0/496521173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331ad430 msgr2=0x5648331b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:06.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.814+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/496521173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331ad430 0x5648331b1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.814+0000 7fc5d6ffd640 1 -- 172.21.15.77:0/496521173 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564833174500 con 0x5648331aca40 2024-03-16T10:22:06.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.814+0000 7fc5d77fe640 1 --2- 172.21.15.77:0/496521173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331ad430 0x5648331b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:06.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.814+0000 7fc5d4ff9640 1 -- 172.21.15.77:0/3506195806 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5c800ce60 con 0x7fc5d81abca0 2024-03-16T10:22:06.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.814+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/496521173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 0x5648331ace60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009ca0 tx=0x7fc5d002e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c58cf5e7bc97e68b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:06.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.814+0000 7fc5d4ff9640 1 -- 172.21.15.77:0/3506195806 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8010430 con 0x7fc5d81abca0 2024-03-16T10:22:06.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.815+0000 7fc5d4ff9640 1 -- 172.21.15.77:0/3506195806 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fc5c8010690 con 0x7fc5d81abca0 2024-03-16T10:22:06.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.816+0000 7fc5d4ff9640 1 --2- 172.21.15.77:0/3506195806 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc5c00a2a40 0x7fc5c00a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:06.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.816+0000 7fc5d4ff9640 1 -- 172.21.15.77:0/3506195806 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc5c80cfd50 con 0x7fc5d81abca0 2024-03-16T10:22:06.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.816+0000 7fc5d77fe640 1 --2- 172.21.15.77:0/3506195806 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc5c00a2a40 0x7fc5c00a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:06.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.817+0000 7fc5d77fe640 1 --2- 172.21.15.77:0/3506195806 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc5c00a2a40 0x7fc5c00a4f00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x5648331ad830 tx=0x7fc5cc009060 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:06.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.843+0000 7fc5bdffb640 1 -- 172.21.15.77:0/496521173 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d004a070 con 0x5648331aca40 2024-03-16T10:22:06.844 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.843+0000 7fc5bdffb640 1 -- 172.21.15.77:0/496521173 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5d0034440 con 0x5648331aca40 2024-03-16T10:22:06.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.844+0000 7fc5e0a15740 1 -- 172.21.15.77:0/496521173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 msgr2=0x5648331ace60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:06.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.844+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/496521173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 0x5648331ace60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009ca0 tx=0x7fc5d002e040 comp rx=0 tx=0).stop 2024-03-16T10:22:06.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.851+0000 7fc5e0a15740 1 -- 172.21.15.77:0/496521173 shutdown_connections 2024-03-16T10:22:06.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.851+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/496521173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331b25a0 0x5648331b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.851+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/496521173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331ad430 0x5648331b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.851+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/496521173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 0x5648331ace60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.851+0000 7fc5e0a15740 1 -- 172.21.15.77:0/496521173 >> 172.21.15.77:0/496521173 conn(0x5648331a7fb0 msgr2=0x5648331a83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.855+0000 7fc5e0a15740 1 -- 172.21.15.77:0/496521173 shutdown_connections 2024-03-16T10:22:06.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.856+0000 7fc5e0a15740 1 -- 172.21.15.77:0/496521173 wait complete. 2024-03-16T10:22:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.856+0000 7fc5e0a15740 1 -- start start 2024-03-16T10:22:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.856+0000 7fc5e0a15740 1 Processor -- start 2024-03-16T10:22:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5e0a15740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 0x5648331b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5e0a15740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331ad430 0x5648331b7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:06.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5e0a15740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331b25a0 0x5648331be370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5d77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 0x5648331b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5e0a15740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5648331b5540 con 0x5648331ad430 2024-03-16T10:22:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5d77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 0x5648331b6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58870/0 (socket says 172.21.15.77:58870) 2024-03-16T10:22:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5d77fe640 1 -- 172.21.15.77:0/2570792245 learned_addr learned my addr 172.21.15.77:0/2570792245 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331ad430 0x5648331b7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2570792245 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5648330cd230 con 0x5648331b25a0 2024-03-16T10:22:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5d7fff640 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331b25a0 0x5648331be370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:06.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2570792245 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5648331b56c0 con 0x5648331aca40 2024-03-16T10:22:06.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5e0a15740 10 client.0 _ll_register_callbacks cb 0x564833109010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:06.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5d77fe640 1 -- 172.21.15.77:0/2570792245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331b25a0 msgr2=0x5648331be370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:06.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5d77fe640 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331b25a0 0x5648331be370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.857+0000 7fc5d77fe640 1 -- 172.21.15.77:0/2570792245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331ad430 msgr2=0x5648331b7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:06.859 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.858+0000 7fc5d77fe640 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331ad430 0x5648331b7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.858+0000 7fc5d77fe640 1 -- 172.21.15.77:0/2570792245 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648330dc100 con 0x5648331aca40 2024-03-16T10:22:06.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.858+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331ad430 0x5648331b7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:06.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.858+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2570792245 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5d0034440 con 0x5648331aca40 2024-03-16T10:22:06.860 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.858+0000 7fc5d77fe640 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 0x5648331b6df0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc0156b0 tx=0x7fc5cc004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10308db5868e2fe server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.875+0000 7fc5bb7f6640 10 client.26608 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:22:06.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.875+0000 7fc5bb7f6640 1 -- 172.21.15.77:0/2570792245 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc023020 con 0x5648331aca40 2024-03-16T10:22:06.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.875+0000 7fc5bb7f6640 1 -- 172.21.15.77:0/2570792245 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc5cc013510 con 0x5648331aca40 2024-03-16T10:22:06.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.885+0000 7fc5bb7f6640 1 -- 172.21.15.77:0/2570792245 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc5cc01b040 con 0x5648331aca40 2024-03-16T10:22:06.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.886+0000 7fc5e0a15740 10 client.26608 fetch_fsmap learned FSMap version 35 2024-03-16T10:22:06.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.886+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2570792245 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=35}) -- 0x564832e253f0 con 0x5648331aca40 2024-03-16T10:22:06.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.887+0000 7fc5bb7f6640 1 -- 172.21.15.77:0/2570792245 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fc5cc01e030 con 0x5648331aca40 2024-03-16T10:22:06.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.887+0000 7fc5e0a15740 10 client.26608 fetch_fsmap finished waiting for FSMap version 35 2024-03-16T10:22:06.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.887+0000 7fc5e0a15740 10 client.26608 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:22:06.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.887+0000 7fc5e0a15740 20 client.26608 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:06.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.887+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5648331c2ef0 0x5648331c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:06.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.888+0000 7fc5e0a15740 4 client.26608 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:06.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.888+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2570792245 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56483311e9d0 con 0x5648331c2ef0 2024-03-16T10:22:06.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.888+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5648331c2ef0 0x5648331c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:06.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.888+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5648331c2ef0 0x5648331c53b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ac57efced124fccf server_cookie=c97e742974022fab in_seq=0 out_seq=0 2024-03-16T10:22:06.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.891+0000 7fc5bb7f6640 10 client.26608 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:22:06.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.891+0000 7fc5bb7f6640 1 -- 172.21.15.77:0/2570792245 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56483311e9d0 con 0x5648331c2ef0 2024-03-16T10:22:06.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.891+0000 7fc5bb7f6640 10 client.26608 handle_command_reply: tid=0 2024-03-16T10:22:06.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.891+0000 7fc5e0a15740 1 client.26608 shutdown 2024-03-16T10:22:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.908+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2570792245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 msgr2=0x5648331b6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.908+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 0x5648331b6df0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc0156b0 tx=0x7fc5cc004770 comp rx=0 tx=0).stop 2024-03-16T10:22:06.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.910+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2570792245 shutdown_connections 2024-03-16T10:22:06.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.910+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5648331c2ef0 0x5648331c53b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.910+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331b25a0 0x5648331be370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.910+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331ad430 0x5648331b7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.910+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2570792245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331aca40 0x5648331b6df0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:06.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.910+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2570792245 >> 172.21.15.77:0/2570792245 conn(0x5648331a7fb0 msgr2=0x5648331a7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:06.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.910+0000 7fc5bb7f6640 0 client.26608 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:22:06.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.910+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2570792245 shutdown_connections 2024-03-16T10:22:06.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.910+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2570792245 wait complete. 2024-03-16T10:22:06.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:06.912+0000 7fc5e0a15740 20 client.26608 trim_cache size 0 max 16384 2024-03-16T10:22:07.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.000+0000 7fc5e0a15740 1 Processor -- start 2024-03-16T10:22:07.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.000+0000 7fc5e0a15740 1 -- start start 2024-03-16T10:22:07.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.004+0000 7fc5e0a15740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331a7fb0 0x5648331cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:07.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.004+0000 7fc5e0a15740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331aca40 0x5648331be300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:07.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.004+0000 7fc5d77fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331a7fb0 0x5648331cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:07.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.004+0000 7fc5d77fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331a7fb0 0x5648331cacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54166/0 (socket says 172.21.15.77:54166) 2024-03-16T10:22:07.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.004+0000 7fc5d6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331aca40 0x5648331be300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:07.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.005+0000 7fc5d6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331aca40 0x5648331be300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60080/0 (socket says 172.21.15.77:60080) 2024-03-16T10:22:07.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.005+0000 7fc5d77fe640 1 -- 172.21.15.77:0/2198302943 learned_addr learned my addr 172.21.15.77:0/2198302943 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:07.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.005+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2198302943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331ad430 0x5648331ab210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:07.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.005+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2198302943 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c400e430 con 0x5648331aca40 2024-03-16T10:22:07.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.005+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2198302943 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c4010000 con 0x5648331a7fb0 2024-03-16T10:22:07.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.005+0000 7fc5d7fff640 1 --2- 172.21.15.77:0/2198302943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331ad430 0x5648331ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:07.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.005+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2198302943 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c400eb00 con 0x5648331ad430 2024-03-16T10:22:07.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.006+0000 7fc5d6ffd640 1 -- 172.21.15.77:0/2198302943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331ad430 msgr2=0x5648331ab210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:07.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.006+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/2198302943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331ad430 0x5648331ab210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.006+0000 7fc5d6ffd640 1 -- 172.21.15.77:0/2198302943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331a7fb0 msgr2=0x5648331cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:07.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.006+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/2198302943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331a7fb0 0x5648331cacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.006+0000 7fc5d6ffd640 1 -- 172.21.15.77:0/2198302943 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564832eef7d0 con 0x5648331aca40 2024-03-16T10:22:07.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.007+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/2198302943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331aca40 0x5648331be300 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0048040 tx=0x7fc5d002c1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9b81f9b679f61f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.012+0000 7fc5bd7fa640 1 -- 172.21.15.77:0/2198302943 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d004a040 con 0x5648331aca40 2024-03-16T10:22:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.012+0000 7fc5bd7fa640 1 -- 172.21.15.77:0/2198302943 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5d0055650 con 0x5648331aca40 2024-03-16T10:22:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.012+0000 7fc5bd7fa640 1 -- 172.21.15.77:0/2198302943 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0034e60 con 0x5648331aca40 2024-03-16T10:22:07.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.014+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2198302943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331aca40 msgr2=0x5648331be300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:07.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.014+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2198302943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331aca40 0x5648331be300 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0048040 tx=0x7fc5d002c1a0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.014+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2198302943 shutdown_connections 2024-03-16T10:22:07.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.014+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2198302943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331ad430 0x5648331ab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.014+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2198302943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331aca40 0x5648331be300 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.014+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2198302943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331a7fb0 0x5648331cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.014+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2198302943 >> 172.21.15.77:0/2198302943 conn(0x5648331a35a0 msgr2=0x5648331ae6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:07.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.014+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2198302943 shutdown_connections 2024-03-16T10:22:07.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.015+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2198302943 wait complete. 2024-03-16T10:22:07.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.015+0000 7fc5e0a15740 1 -- start start 2024-03-16T10:22:07.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.016+0000 7fc5e0a15740 1 Processor -- start 2024-03-16T10:22:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.016+0000 7fc5e0a15740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331a7fb0 0x5648331bae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.016+0000 7fc5e0a15740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331ad430 0x5648331bb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.017+0000 7fc5d77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331a7fb0 0x5648331bae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.017+0000 7fc5d77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331a7fb0 0x5648331bae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58886/0 (socket says 172.21.15.77:58886) 2024-03-16T10:22:07.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.017+0000 7fc5d77fe640 1 -- 172.21.15.77:0/2134969902 learned_addr learned my addr 172.21.15.77:0/2134969902 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:07.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.017+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331ad430 0x5648331bb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:07.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.017+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331b25a0 0x5648332272e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:07.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.017+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2134969902 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c400f930 con 0x5648331b25a0 2024-03-16T10:22:07.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.017+0000 7fc5d7fff640 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331b25a0 0x5648332272e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:07.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.017+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2134969902 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c400f260 con 0x5648331ad430 2024-03-16T10:22:07.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.017+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2134969902 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c400c410 con 0x5648331a7fb0 2024-03-16T10:22:07.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.018+0000 7fc5d77fe640 1 -- 172.21.15.77:0/2134969902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331ad430 msgr2=0x5648331bb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:07.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.018+0000 7fc5d77fe640 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331ad430 0x5648331bb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.018+0000 7fc5d77fe640 1 -- 172.21.15.77:0/2134969902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331b25a0 msgr2=0x5648332272e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:07.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.018+0000 7fc5e0a15740 10 client.0 _ll_register_callbacks cb 0x7fc5cc0097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:07.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.018+0000 7fc5d77fe640 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331b25a0 0x5648332272e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.018+0000 7fc5d77fe640 1 -- 172.21.15.77:0/2134969902 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648331be410 con 0x5648331a7fb0 2024-03-16T10:22:07.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.018+0000 7fc5d77fe640 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331a7fb0 0x5648331bae90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00ed80 tx=0x7fc5cc013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38761406be621c3d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:07.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.021+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2134969902 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5d0055650 con 0x5648331a7fb0 2024-03-16T10:22:07.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.021+0000 7fc5d7fff640 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331b25a0 0x5648332272e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:07.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.031+0000 7fc5bb7f6640 10 client.26614 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:22:07.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.031+0000 7fc5bb7f6640 1 -- 172.21.15.77:0/2134969902 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc037070 con 0x5648331a7fb0 2024-03-16T10:22:07.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.031+0000 7fc5bb7f6640 1 -- 172.21.15.77:0/2134969902 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fc5cc005b90 con 0x5648331a7fb0 2024-03-16T10:22:07.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.031+0000 7fc5bb7f6640 1 -- 172.21.15.77:0/2134969902 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc5cc023040 con 0x5648331a7fb0 2024-03-16T10:22:07.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.031+0000 7fc5e0a15740 10 client.26614 fetch_fsmap learned FSMap version 35 2024-03-16T10:22:07.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.031+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2134969902 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=35}) -- 0x564833228580 con 0x5648331a7fb0 2024-03-16T10:22:07.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.036+0000 7fc5bb7f6640 1 -- 172.21.15.77:0/2134969902 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fc5cc01e040 con 0x5648331a7fb0 2024-03-16T10:22:07.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.036+0000 7fc5e0a15740 10 client.26614 fetch_fsmap finished waiting for FSMap version 35 2024-03-16T10:22:07.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.036+0000 7fc5e0a15740 10 client.26614 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:22:07.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.036+0000 7fc5e0a15740 20 client.26614 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:07.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.036+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x564833228ef0 0x56483322b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:07.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.036+0000 7fc5e0a15740 4 client.26614 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:22:07.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.036+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2134969902 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564833118d20 con 0x564833228ef0 2024-03-16T10:22:07.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.036+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x564833228ef0 0x56483322b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:07.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.037+0000 7fc5d6ffd640 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x564833228ef0 0x56483322b310 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dac324b798363613 server_cookie=39842e1cc3cee9ae in_seq=0 out_seq=0 2024-03-16T10:22:07.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.039+0000 7fc5bb7f6640 10 client.26614 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:22:07.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.039+0000 7fc5bb7f6640 1 -- 172.21.15.77:0/2134969902 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564833118d20 con 0x564833228ef0 2024-03-16T10:22:07.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.039+0000 7fc5bb7f6640 10 client.26614 handle_command_reply: tid=0 2024-03-16T10:22:07.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.039+0000 7fc5e0a15740 1 client.26614 shutdown 2024-03-16T10:22:07.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.043+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2134969902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331a7fb0 msgr2=0x5648331bae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:07.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.043+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331a7fb0 0x5648331bae90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00ed80 tx=0x7fc5cc013000 comp rx=0 tx=0).stop 2024-03-16T10:22:07.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.044+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2134969902 shutdown_connections 2024-03-16T10:22:07.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.044+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x564833228ef0 0x56483322b310 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.044+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648331b25a0 0x5648332272e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.044+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648331ad430 0x5648331bb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.044+0000 7fc5e0a15740 1 --2- 172.21.15.77:0/2134969902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648331a7fb0 0x5648331bae90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.044+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2134969902 >> 172.21.15.77:0/2134969902 conn(0x5648331a35a0 msgr2=0x5648331ae6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:07.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.044+0000 7fc5bb7f6640 0 client.26614 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:22:07.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.044+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2134969902 shutdown_connections 2024-03-16T10:22:07.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.044+0000 7fc5e0a15740 1 -- 172.21.15.77:0/2134969902 wait complete. 2024-03-16T10:22:07.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.045+0000 7fc5e0a15740 20 client.26614 trim_cache size 0 max 16384 2024-03-16T10:22:07.047 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:22:07.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.065+0000 7fc5ddd59640 1 -- 172.21.15.77:0/3506195806 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc5c00a2a40 msgr2=0x7fc5c00a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:07.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.065+0000 7fc5ddd59640 1 --2- 172.21.15.77:0/3506195806 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc5c00a2a40 0x7fc5c00a4f00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x5648331ad830 tx=0x7fc5cc009060 comp rx=0 tx=0).stop 2024-03-16T10:22:07.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.065+0000 7fc5ddd59640 1 -- 172.21.15.77:0/3506195806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5d81abca0 msgr2=0x7fc5d81ae090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:07.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.065+0000 7fc5ddd59640 1 --2- 172.21.15.77:0/3506195806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5d81abca0 0x7fc5d81ae090 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800e970 tx=0x7fc5c800ee30 comp rx=0 tx=0).stop 2024-03-16T10:22:07.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.071+0000 7fc5ddd59640 1 -- 172.21.15.77:0/3506195806 shutdown_connections 2024-03-16T10:22:07.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.071+0000 7fc5ddd59640 1 --2- 172.21.15.77:0/3506195806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5d81abca0 0x7fc5d81ae090 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.071+0000 7fc5ddd59640 1 --2- 172.21.15.77:0/3506195806 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc5c00a2a40 0x7fc5c00a4f00 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.071+0000 7fc5ddd59640 1 --2- 172.21.15.77:0/3506195806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d8109890 0x7fc5d81a6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.071+0000 7fc5ddd59640 1 --2- 172.21.15.77:0/3506195806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc5d8072bc0 0x7fc5d81a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:07.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.071+0000 7fc5ddd59640 1 -- 172.21.15.77:0/3506195806 >> 172.21.15.77:0/3506195806 conn(0x7fc5d806e6c0 msgr2=0x7fc5d810a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:07.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.074+0000 7fc5ddd59640 1 -- 172.21.15.77:0/3506195806 shutdown_connections 2024-03-16T10:22:07.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:07.074+0000 7fc5ddd59640 1 -- 172.21.15.77:0/3506195806 wait complete. 2024-03-16T10:22:07.106 INFO:tasks.workunit.client.0.smithi077.stdout: 123 2128395 404 1407960 452 1197073 1134 2024-03-16T10:22:07.138 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T10:22:08.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:07 smithi192 ceph-mon[36354]: pgmap v501: 97 pgs: 97 active+clean; 593 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 413 op/s 2024-03-16T10:22:08.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:07 smithi077 ceph-mon[32400]: pgmap v501: 97 pgs: 97 active+clean; 593 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 413 op/s 2024-03-16T10:22:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:07 smithi193 ceph-mon[36311]: pgmap v501: 97 pgs: 97 active+clean; 593 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 413 op/s 2024-03-16T10:22:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.648+0000 7fade6ffd640 1 -- 172.21.15.77:0/2468775 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade0032e50 con 0x7fade8073410 2024-03-16T10:22:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.648+0000 7fadee95b640 1 -- 172.21.15.77:0/2468775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fade8073410 msgr2=0x7fade8108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.648+0000 7fadee95b640 1 --2- 172.21.15.77:0/2468775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fade8073410 0x7fade8108e70 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fade0009f00 tx=0x7fade002c3a0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.658+0000 7fadee95b640 1 -- 172.21.15.77:0/2468775 shutdown_connections 2024-03-16T10:22:08.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.658+0000 7fadee95b640 1 --2- 172.21.15.77:0/2468775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fade8109510 0x7fade810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.658+0000 7fadee95b640 1 --2- 172.21.15.77:0/2468775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fade8073410 0x7fade8108e70 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.658+0000 7fadee95b640 1 --2- 172.21.15.77:0/2468775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fade8072ab0 0x7fade8072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.658+0000 7fadee95b640 1 -- 172.21.15.77:0/2468775 >> 172.21.15.77:0/2468775 conn(0x7fade806e650 msgr2=0x7fade8070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:08.660 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.660+0000 7fadee95b640 1 -- 172.21.15.77:0/2468775 shutdown_connections 2024-03-16T10:22:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.660+0000 7fadee95b640 1 -- 172.21.15.77:0/2468775 wait complete. 2024-03-16T10:22:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.661+0000 7fadee95b640 1 Processor -- start 2024-03-16T10:22:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.661+0000 7fadee95b640 1 -- start start 2024-03-16T10:22:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.661+0000 7fadee95b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fade8072ab0 0x7fade8117eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.661+0000 7fadee95b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fade8073410 0x7fade81183f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.661+0000 7fadee95b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fade8109510 0x7fade8119420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.661+0000 7fadee95b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fade8110520 con 0x7fade8109510 2024-03-16T10:22:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.661+0000 7fadee95b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fade81103a0 con 0x7fade8072ab0 2024-03-16T10:22:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.661+0000 7fadee95b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fade81106a0 con 0x7fade8073410 2024-03-16T10:22:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.661+0000 7fade7fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fade8072ab0 0x7fade8117eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.661+0000 7fade7fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fade8072ab0 0x7fade8117eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54184/0 (socket says 172.21.15.77:54184) 2024-03-16T10:22:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.661+0000 7fade7fff640 1 -- 172.21.15.77:0/2138185289 learned_addr learned my addr 172.21.15.77:0/2138185289 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.662+0000 7fade7fff640 1 -- 172.21.15.77:0/2138185289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fade8073410 msgr2=0x7fade81183f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.662+0000 7fade7fff640 1 --2- 172.21.15.77:0/2138185289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fade8073410 0x7fade81183f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.662+0000 7fade7fff640 1 -- 172.21.15.77:0/2138185289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fade8109510 msgr2=0x7fade8119420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.662+0000 7fade7fff640 1 --2- 172.21.15.77:0/2138185289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fade8109510 0x7fade8119420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.662+0000 7fade7fff640 1 -- 172.21.15.77:0/2138185289 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fade8119ac0 con 0x7fade8072ab0 2024-03-16T10:22:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.662+0000 7fade7fff640 1 --2- 172.21.15.77:0/2138185289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fade8072ab0 0x7fade8117eb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faddc00b450 tx=0x7faddc00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.687+0000 7fade57fa640 1 -- 172.21.15.77:0/2138185289 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faddc015020 con 0x7fade8072ab0 2024-03-16T10:22:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.687+0000 7fadee95b640 1 -- 172.21.15.77:0/2138185289 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fade8119db0 con 0x7fade8072ab0 2024-03-16T10:22:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.687+0000 7fadee95b640 1 -- 172.21.15.77:0/2138185289 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fade811a380 con 0x7fade8072ab0 2024-03-16T10:22:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.691+0000 7fadf1629740 1 Processor -- start 2024-03-16T10:22:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.691+0000 7fadf1629740 1 -- start start 2024-03-16T10:22:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fadf1629740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 0x557253895950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fadf1629740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557253895f20 0x55725389aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fadf1629740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55725389b090 0x55725389d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fadf1629740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5572537faa40 con 0x557253895530 2024-03-16T10:22:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fadf1629740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5572537a3920 con 0x55725389b090 2024-03-16T10:22:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fadf1629740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5572537f6ae0 con 0x557253895f20 2024-03-16T10:22:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fade57fa640 1 -- 172.21.15.77:0/2138185289 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faddc011d70 con 0x7fade8072ab0 2024-03-16T10:22:08.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fadecef8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 0x557253895950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fadecef8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 0x557253895950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60096/0 (socket says 172.21.15.77:60096) 2024-03-16T10:22:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fadecef8640 1 -- 172.21.15.77:0/3667458749 learned_addr learned my addr 172.21.15.77:0/3667458749 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fade77fe640 1 --2- 172.21.15.77:0/3667458749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55725389b090 0x55725389d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fade7fff640 1 --2- 172.21.15.77:0/3667458749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557253895f20 0x55725389aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fadecef8640 1 -- 172.21.15.77:0/3667458749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557253895f20 msgr2=0x55725389aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.692+0000 7fadecef8640 1 --2- 172.21.15.77:0/3667458749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557253895f20 0x55725389aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.693+0000 7fadecef8640 1 -- 172.21.15.77:0/3667458749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55725389b090 msgr2=0x55725389d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.693+0000 7fadecef8640 1 --2- 172.21.15.77:0/3667458749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55725389b090 0x55725389d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.693+0000 7fadecef8640 1 -- 172.21.15.77:0/3667458749 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572537bb460 con 0x557253895530 2024-03-16T10:22:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.694+0000 7fadecef8640 1 --2- 172.21.15.77:0/3667458749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 0x557253895950 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fadd8009760 tx=0x7fadd8009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73117d18361370dc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.694+0000 7fade77fe640 1 --2- 172.21.15.77:0/3667458749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55725389b090 0x55725389d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.694+0000 7fade57fa640 1 -- 172.21.15.77:0/2138185289 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faddc00f340 con 0x7fade8072ab0 2024-03-16T10:22:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.700+0000 7fade57fa640 1 -- 172.21.15.77:0/2138185289 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7faddc018020 con 0x7fade8072ab0 2024-03-16T10:22:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.701+0000 7fade57fa640 1 --2- 172.21.15.77:0/2138185289 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fadd00a2a00 0x7fadd00a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.707+0000 7fade57fa640 1 -- 172.21.15.77:0/2138185289 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7faddc0cfaf0 con 0x7fade8072ab0 2024-03-16T10:22:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.707+0000 7fade7fff640 1 --2- 172.21.15.77:0/2138185289 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fadd00a2a00 0x7fadd00a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.708+0000 7fade7fff640 1 --2- 172.21.15.77:0/2138185289 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fadd00a2a00 0x7fadd00a4ec0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7faddc010bb0 tx=0x7faddc010fc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.712+0000 7fadd67fc640 1 -- 172.21.15.77:0/3667458749 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8015020 con 0x557253895530 2024-03-16T10:22:08.713 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.712+0000 7fadd67fc640 1 -- 172.21.15.77:0/3667458749 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadd8011420 con 0x557253895530 2024-03-16T10:22:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.712+0000 7fadd67fc640 1 -- 172.21.15.77:0/3667458749 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd80105c0 con 0x557253895530 2024-03-16T10:22:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.713+0000 7fadf1629740 1 -- 172.21.15.77:0/3667458749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 msgr2=0x557253895950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.713+0000 7fadf1629740 1 --2- 172.21.15.77:0/3667458749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 0x557253895950 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fadd8009760 tx=0x7fadd8009c20 comp rx=0 tx=0).stop 2024-03-16T10:22:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.713+0000 7fadf1629740 1 -- 172.21.15.77:0/3667458749 shutdown_connections 2024-03-16T10:22:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.713+0000 7fadf1629740 1 --2- 172.21.15.77:0/3667458749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55725389b090 0x55725389d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.713+0000 7fadf1629740 1 --2- 172.21.15.77:0/3667458749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557253895f20 0x55725389aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.713+0000 7fadf1629740 1 --2- 172.21.15.77:0/3667458749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 0x557253895950 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.713+0000 7fadf1629740 1 -- 172.21.15.77:0/3667458749 >> 172.21.15.77:0/3667458749 conn(0x557253890aa0 msgr2=0x557253890ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.718+0000 7fadf1629740 1 -- 172.21.15.77:0/3667458749 shutdown_connections 2024-03-16T10:22:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.718+0000 7fadf1629740 1 -- 172.21.15.77:0/3667458749 wait complete. 2024-03-16T10:22:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.718+0000 7fadf1629740 1 -- start start 2024-03-16T10:22:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.719+0000 7fadf1629740 1 Processor -- start 2024-03-16T10:22:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.719+0000 7fadf1629740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895530 0x55725389f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.719+0000 7fadf1629740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895f20 0x55725389fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.719+0000 7fadf1629740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55725389b090 0x5572538a6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.719+0000 7fadf1629740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55725389e040 con 0x557253895f20 2024-03-16T10:22:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.719+0000 7fadf1629740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5572537b6d70 con 0x557253895530 2024-03-16T10:22:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.719+0000 7fadf1629740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55725389e1c0 con 0x55725389b090 2024-03-16T10:22:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.719+0000 7fadf1629740 10 client.0 _ll_register_callbacks cb 0x55725350ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.720+0000 7fadecef8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895530 0x55725389f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.720+0000 7fadecef8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895530 0x55725389f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54210/0 (socket says 172.21.15.77:54210) 2024-03-16T10:22:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.720+0000 7fadecef8640 1 -- 172.21.15.77:0/1353550181 learned_addr learned my addr 172.21.15.77:0/1353550181 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.720+0000 7fadecef8640 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55725389b090 0x5572538a6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.720+0000 7fade77fe640 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895f20 0x55725389fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.720+0000 7fadecef8640 1 -- 172.21.15.77:0/1353550181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55725389b090 msgr2=0x5572538a6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.720+0000 7fadecef8640 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55725389b090 0x5572538a6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.720+0000 7fadecef8640 1 -- 172.21.15.77:0/1353550181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895f20 msgr2=0x55725389fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.720+0000 7fadecef8640 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895f20 0x55725389fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.720+0000 7fadecef8640 1 -- 172.21.15.77:0/1353550181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55725350f210 con 0x557253895530 2024-03-16T10:22:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.720+0000 7fade77fe640 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895f20 0x55725389fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:22:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.720+0000 7fadecef8640 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895530 0x55725389f890 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fadd8011da0 tx=0x7fadd80090e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=745710cc2ec54032 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.743+0000 7fadcb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:22:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.743+0000 7fadcb7fe640 1 -- 172.21.15.77:0/1353550181 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8015040 con 0x557253895530 2024-03-16T10:22:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.743+0000 7fadcb7fe640 1 -- 172.21.15.77:0/1353550181 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fadd8011a10 con 0x557253895530 2024-03-16T10:22:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.743+0000 7fadf1629740 1 -- 172.21.15.77:0/1353550181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadd8011420 con 0x557253895530 2024-03-16T10:22:08.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.748+0000 7fadcb7fe640 1 -- 172.21.15.77:0/1353550181 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fadd800b8d0 con 0x557253895530 2024-03-16T10:22:08.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.748+0000 7fadf1629740 10 client.26624 fetch_fsmap learned FSMap version 35 2024-03-16T10:22:08.749 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.749+0000 7fadf1629740 1 -- 172.21.15.77:0/1353550181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5572535db620 con 0x557253895530 2024-03-16T10:22:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.749+0000 7fadcb7fe640 1 -- 172.21.15.77:0/1353550181 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadd8011420 con 0x557253895530 2024-03-16T10:22:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.751+0000 7fadcb7fe640 1 -- 172.21.15.77:0/1353550181 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fadd801a420 con 0x557253895530 2024-03-16T10:22:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.751+0000 7fadf1629740 10 client.26624 fetch_fsmap finished waiting for FSMap version 35 2024-03-16T10:22:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.751+0000 7fadf1629740 10 client.26624 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.751+0000 7fadf1629740 20 client.26624 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.751+0000 7fadf1629740 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5572538ab4a0 0x5572538ad8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.751+0000 7fadf1629740 4 client.26624 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.751+0000 7fadf1629740 1 -- 172.21.15.77:0/1353550181 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557253807f80 con 0x5572538ab4a0 2024-03-16T10:22:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.751+0000 7fade77fe640 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5572538ab4a0 0x5572538ad8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.752+0000 7fade77fe640 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5572538ab4a0 0x5572538ad8f0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8345f96240d98e86 server_cookie=46ec9a5f936822f in_seq=0 out_seq=0 2024-03-16T10:22:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.767+0000 7fadcb7fe640 10 client.26624 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.767+0000 7fadcb7fe640 1 -- 172.21.15.77:0/1353550181 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557253807f80 con 0x5572538ab4a0 2024-03-16T10:22:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.767+0000 7fadcb7fe640 10 client.26624 handle_command_reply: tid=0 2024-03-16T10:22:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.768+0000 7fadf1629740 1 client.26624 shutdown 2024-03-16T10:22:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.768+0000 7fadf1629740 1 -- 172.21.15.77:0/1353550181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895530 msgr2=0x55725389f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.768+0000 7fadf1629740 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895530 0x55725389f890 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fadd8011da0 tx=0x7fadd80090e0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.768+0000 7fadf1629740 1 -- 172.21.15.77:0/1353550181 shutdown_connections 2024-03-16T10:22:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.768+0000 7fadf1629740 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55725389b090 0x5572538a6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.768+0000 7fadf1629740 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5572538ab4a0 0x5572538ad8f0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.768+0000 7fadf1629740 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895f20 0x55725389fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.768+0000 7fadf1629740 1 --2- 172.21.15.77:0/1353550181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895530 0x55725389f890 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.768+0000 7fadf1629740 1 -- 172.21.15.77:0/1353550181 >> 172.21.15.77:0/1353550181 conn(0x557253890aa0 msgr2=0x55725388fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.768+0000 7fadcb7fe640 0 client.26624 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.768+0000 7fadf1629740 1 -- 172.21.15.77:0/1353550181 shutdown_connections 2024-03-16T10:22:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.768+0000 7fadf1629740 1 -- 172.21.15.77:0/1353550181 wait complete. 2024-03-16T10:22:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.769+0000 7fadf1629740 20 client.26624 trim_cache size 0 max 16384 2024-03-16T10:22:08.868 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.867+0000 7fadf1629740 1 Processor -- start 2024-03-16T10:22:08.868 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.867+0000 7fadf1629740 1 -- start start 2024-03-16T10:22:08.868 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.867+0000 7fadf1629740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 0x55725389eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.868+0000 7fadf1629740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895f20 0x5572538a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.868+0000 7fadecef8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 0x55725389eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.868+0000 7fadf1629740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55725389b090 0x55725389fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.868+0000 7fadf1629740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc00e3c0 con 0x557253895530 2024-03-16T10:22:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.868+0000 7fadecef8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 0x55725389eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60114/0 (socket says 172.21.15.77:60114) 2024-03-16T10:22:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.868+0000 7fadecef8640 1 -- 172.21.15.77:0/3941075915 learned_addr learned my addr 172.21.15.77:0/3941075915 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.868+0000 7fade77fe640 1 --2- 172.21.15.77:0/3941075915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895f20 0x5572538a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.871+0000 7fadf1629740 1 -- 172.21.15.77:0/3941075915 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc00ff90 con 0x557253895f20 2024-03-16T10:22:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.871+0000 7fadf1629740 1 -- 172.21.15.77:0/3941075915 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc00ea90 con 0x55725389b090 2024-03-16T10:22:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.871+0000 7fade77fe640 1 -- 172.21.15.77:0/3941075915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55725389b090 msgr2=0x55725389fbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T10:22:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.871+0000 7fade77fe640 1 --2- 172.21.15.77:0/3941075915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55725389b090 0x55725389fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.871+0000 7fade77fe640 1 -- 172.21.15.77:0/3941075915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 msgr2=0x55725389eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.871+0000 7fade77fe640 1 --2- 172.21.15.77:0/3941075915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 0x55725389eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.871+0000 7fade77fe640 1 -- 172.21.15.77:0/3941075915 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557253895910 con 0x557253895f20 2024-03-16T10:22:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.871+0000 7fadecef8640 1 --2- 172.21.15.77:0/3941075915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 0x55725389eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.871+0000 7fade77fe640 1 --2- 172.21.15.77:0/3941075915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895f20 0x5572538a6dd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fade0009e50 tx=0x7fade0032a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f42a9c7ae267f925 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.882+0000 7fadd5ffb640 1 -- 172.21.15.77:0/3941075915 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade0048ca0 con 0x557253895f20 2024-03-16T10:22:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.882+0000 7fadd5ffb640 1 -- 172.21.15.77:0/3941075915 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fade0048e40 con 0x557253895f20 2024-03-16T10:22:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.882+0000 7fadd5ffb640 1 -- 172.21.15.77:0/3941075915 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade0003bd0 con 0x557253895f20 2024-03-16T10:22:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.883+0000 7fadf1629740 1 -- 172.21.15.77:0/3941075915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895f20 msgr2=0x5572538a6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.883+0000 7fadf1629740 1 --2- 172.21.15.77:0/3941075915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895f20 0x5572538a6dd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fade0009e50 tx=0x7fade0032a70 comp rx=0 tx=0).stop 2024-03-16T10:22:08.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.883+0000 7fadf1629740 1 -- 172.21.15.77:0/3941075915 shutdown_connections 2024-03-16T10:22:08.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.883+0000 7fadf1629740 1 --2- 172.21.15.77:0/3941075915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55725389b090 0x55725389fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.883+0000 7fadf1629740 1 --2- 172.21.15.77:0/3941075915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895f20 0x5572538a6dd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.883+0000 7fadf1629740 1 --2- 172.21.15.77:0/3941075915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557253895530 0x55725389eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.884 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.883+0000 7fadf1629740 1 -- 172.21.15.77:0/3941075915 >> 172.21.15.77:0/3941075915 conn(0x557253890aa0 msgr2=0x55725389f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.887+0000 7fadf1629740 1 -- 172.21.15.77:0/3941075915 shutdown_connections 2024-03-16T10:22:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.887+0000 7fadf1629740 1 -- 172.21.15.77:0/3941075915 wait complete. 2024-03-16T10:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.888+0000 7fadf1629740 1 -- start start 2024-03-16T10:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.888+0000 7fadf1629740 1 Processor -- start 2024-03-16T10:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.889+0000 7fadf1629740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557253895530 0x5572538a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.889+0000 7fadf1629740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895f20 0x5572538a4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.889+0000 7fadecef8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557253895530 0x5572538a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.889+0000 7fadecef8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557253895530 0x5572538a3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58928/0 (socket says 172.21.15.77:58928) 2024-03-16T10:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.889+0000 7fadecef8640 1 -- 172.21.15.77:0/2293266292 learned_addr learned my addr 172.21.15.77:0/2293266292 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.889+0000 7fadf1629740 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55725389b090 0x5572538a5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.889+0000 7fade77fe640 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895f20 0x5572538a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.889+0000 7fadf1629740 1 -- 172.21.15.77:0/2293266292 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc00f8c0 con 0x55725389b090 2024-03-16T10:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.889+0000 7fadf1629740 1 -- 172.21.15.77:0/2293266292 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc00f1f0 con 0x557253895f20 2024-03-16T10:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.890+0000 7fadf1629740 1 -- 172.21.15.77:0/2293266292 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc00c3a0 con 0x557253895530 2024-03-16T10:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.890+0000 7fadecef8640 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55725389b090 0x5572538a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.890+0000 7fade77fe640 1 -- 172.21.15.77:0/2293266292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557253895530 msgr2=0x5572538a3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.890+0000 7fade77fe640 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557253895530 0x5572538a3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.890+0000 7fade77fe640 1 -- 172.21.15.77:0/2293266292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55725389b090 msgr2=0x5572538a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.890+0000 7fade77fe640 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55725389b090 0x5572538a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.890+0000 7fade77fe640 1 -- 172.21.15.77:0/2293266292 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572538a5a70 con 0x557253895f20 2024-03-16T10:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.890+0000 7fadecef8640 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55725389b090 0x5572538a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.891+0000 7fade77fe640 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895f20 0x5572538a4470 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fade004b040 tx=0x7fade00485b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=117bd4716b81d0e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.890+0000 7fadf1629740 10 client.0 _ll_register_callbacks cb 0x7fade0009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.893+0000 7fadf1629740 1 -- 172.21.15.77:0/2293266292 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5572538022d0 con 0x557253895f20 2024-03-16T10:22:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.903+0000 7fadcb7fe640 10 client.26630 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:22:08.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.903+0000 7fadcb7fe640 1 -- 172.21.15.77:0/2293266292 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade0059060 con 0x557253895f20 2024-03-16T10:22:08.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.904+0000 7fadcb7fe640 1 -- 172.21.15.77:0/2293266292 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fade0048870 con 0x557253895f20 2024-03-16T10:22:08.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.904+0000 7fadcb7fe640 1 -- 172.21.15.77:0/2293266292 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fade0048a70 con 0x557253895f20 2024-03-16T10:22:08.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.904+0000 7fadf1629740 10 client.26630 fetch_fsmap learned FSMap version 35 2024-03-16T10:22:08.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.904+0000 7fadf1629740 1 -- 172.21.15.77:0/2293266292 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({fsmap=35}) -- 0x5572538a6ca0 con 0x557253895f20 2024-03-16T10:22:08.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.905+0000 7fadcb7fe640 1 -- 172.21.15.77:0/2293266292 <== mon.2 v2:172.21.15.192:3300/0 4 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fade0003970 con 0x557253895f20 2024-03-16T10:22:08.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.905+0000 7fadf1629740 10 client.26630 fetch_fsmap finished waiting for FSMap version 35 2024-03-16T10:22:08.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.905+0000 7fadf1629740 10 client.26630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:08.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.905+0000 7fadf1629740 20 client.26630 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:08.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.905+0000 7fadf1629740 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5572539119f0 0x557253913e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.905+0000 7fadf1629740 4 client.26630 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:22:08.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.905+0000 7fadf1629740 1 -- 172.21.15.77:0/2293266292 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5572538022d0 con 0x5572539119f0 2024-03-16T10:22:08.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.905+0000 7fadecef8640 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5572539119f0 0x557253913e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.907 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.906+0000 7fadecef8640 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5572539119f0 0x557253913e10 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc63eeb1980ae3e3 server_cookie=9b12b38065450f41 in_seq=0 out_seq=0 2024-03-16T10:22:08.907 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:08.906+0000 7fadcb7fe640 10 client.26630 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:08.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.988+0000 7f8492fd2640 1 -- 172.21.15.77:0/1177945086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f848c078de0 msgr2=0x7f848c07b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:08.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.988+0000 7f8492fd2640 1 --2- 172.21.15.77:0/1177945086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f848c078de0 0x7f848c07b270 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f847c009860 tx=0x7f847c02bbb0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.988+0000 7f8492fd2640 1 -- 172.21.15.77:0/1177945086 shutdown_connections 2024-03-16T10:22:08.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.988+0000 7f8492fd2640 1 --2- 172.21.15.77:0/1177945086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f848c078de0 0x7f848c07b270 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.988+0000 7f8492fd2640 1 --2- 172.21.15.77:0/1177945086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f848c076430 0x7f848c0788a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.988+0000 7f8492fd2640 1 --2- 172.21.15.77:0/1177945086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f848c073550 0x7f848c073970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.988+0000 7f8492fd2640 1 -- 172.21.15.77:0/1177945086 >> 172.21.15.77:0/1177945086 conn(0x7f848c06e6c0 msgr2=0x7f848c070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:08.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.989+0000 7f8492fd2640 1 -- 172.21.15.77:0/1177945086 shutdown_connections 2024-03-16T10:22:08.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.989+0000 7f8492fd2640 1 -- 172.21.15.77:0/1177945086 wait complete. 2024-03-16T10:22:08.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.990+0000 7f8492fd2640 1 Processor -- start 2024-03-16T10:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.990+0000 7f8492fd2640 1 -- start start 2024-03-16T10:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.990+0000 7f8492fd2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f848c073550 0x7f848c1b1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.991+0000 7f8492fd2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f848c076430 0x7f848c1b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.991+0000 7f8492fd2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f848c078de0 0x7f848c1b6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.991+0000 7f8492fd2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f848c07ec70 con 0x7f848c078de0 2024-03-16T10:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.991+0000 7f8492fd2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f848c07eaf0 con 0x7f848c076430 2024-03-16T10:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.991+0000 7f8492fd2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f848c07edf0 con 0x7f848c073550 2024-03-16T10:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.991+0000 7f8491fd0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f848c073550 0x7f848c1b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.991+0000 7f8491fd0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f848c073550 0x7f848c1b1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58940/0 (socket says 172.21.15.77:58940) 2024-03-16T10:22:08.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.991+0000 7f8491fd0640 1 -- 172.21.15.77:0/485171570 learned_addr learned my addr 172.21.15.77:0/485171570 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:08.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.992+0000 7f8491fd0640 1 -- 172.21.15.77:0/485171570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f848c076430 msgr2=0x7f848c1b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:08.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.992+0000 7f8491fd0640 1 --2- 172.21.15.77:0/485171570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f848c076430 0x7f848c1b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.992+0000 7f8491fd0640 1 -- 172.21.15.77:0/485171570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f848c078de0 msgr2=0x7f848c1b6270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:22:08.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.992+0000 7f8491fd0640 1 --2- 172.21.15.77:0/485171570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f848c078de0 0x7f848c1b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:08.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.992+0000 7f8491fd0640 1 -- 172.21.15.77:0/485171570 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848c1b6b30 con 0x7f848c073550 2024-03-16T10:22:08.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:08.993+0000 7f8491fd0640 1 --2- 172.21.15.77:0/485171570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f848c073550 0x7f848c1b1730 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8488007860 tx=0x7f8488007d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:09.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.024+0000 7f8482ffd640 1 -- 172.21.15.77:0/485171570 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488019020 con 0x7f848c073550 2024-03-16T10:22:09.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.024+0000 7f8482ffd640 1 -- 172.21.15.77:0/485171570 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8488015470 con 0x7f848c073550 2024-03-16T10:22:09.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.024+0000 7f8482ffd640 1 -- 172.21.15.77:0/485171570 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488014630 con 0x7f848c073550 2024-03-16T10:22:09.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.026+0000 7f8492fd2640 1 -- 172.21.15.77:0/485171570 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f848c1b6e20 con 0x7f848c073550 2024-03-16T10:22:09.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.027+0000 7f8492fd2640 1 -- 172.21.15.77:0/485171570 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f848c1b72f0 con 0x7f848c073550 2024-03-16T10:22:09.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.031+0000 7f8496e19740 1 Processor -- start 2024-03-16T10:22:09.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.031+0000 7f8496e19740 1 -- start start 2024-03-16T10:22:09.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.031+0000 7f8496e19740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7ca40 0x559ecce7ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.031+0000 7f8496e19740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7d430 0x559ecce81f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.031+0000 7f8496e19740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce825a0 0x559ecce84a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.032+0000 7f84917cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce825a0 0x559ecce84a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:09.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.032+0000 7f8496e19740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559eccde1490 con 0x559ecce825a0 2024-03-16T10:22:09.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.032+0000 7f8496e19740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559eccd89c90 con 0x559ecce7ca40 2024-03-16T10:22:09.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.032+0000 7f8496e19740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559eccddd530 con 0x559ecce7d430 2024-03-16T10:22:09.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.032+0000 7f84927d1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7ca40 0x559ecce7ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:09.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.033+0000 7f84927d1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7ca40 0x559ecce7ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54250/0 (socket says 172.21.15.77:54250) 2024-03-16T10:22:09.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.033+0000 7f84927d1640 1 -- 172.21.15.77:0/308754150 learned_addr learned my addr 172.21.15.77:0/308754150 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:09.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.035+0000 7f8482ffd640 1 -- 172.21.15.77:0/485171570 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f848801c020 con 0x7f848c073550 2024-03-16T10:22:09.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.035+0000 7f8491fd0640 1 --2- 172.21.15.77:0/308754150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7d430 0x559ecce81f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:09.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.036+0000 7f8482ffd640 1 --2- 172.21.15.77:0/485171570 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84780a2a10 0x7f84780a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.036+0000 7f8482ffd640 1 -- 172.21.15.77:0/485171570 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f84880ce6a0 con 0x7f848c073550 2024-03-16T10:22:09.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.044+0000 7f84927d1640 1 -- 172.21.15.77:0/308754150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7d430 msgr2=0x559ecce81f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:09.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.044+0000 7f84927d1640 1 --2- 172.21.15.77:0/308754150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7d430 0x559ecce81f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.044+0000 7f84927d1640 1 -- 172.21.15.77:0/308754150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce825a0 msgr2=0x559ecce84a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:09.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.044+0000 7f84927d1640 1 --2- 172.21.15.77:0/308754150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce825a0 0x559ecce84a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.044+0000 7f84927d1640 1 -- 172.21.15.77:0/308754150 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ecce44500 con 0x559ecce7ca40 2024-03-16T10:22:09.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.044+0000 7f8491fd0640 1 --2- 172.21.15.77:0/308754150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7d430 0x559ecce81f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:22:09.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.044+0000 7f84917cf640 1 --2- 172.21.15.77:0/308754150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce825a0 0x559ecce84a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:22:09.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.044+0000 7f84927d1640 1 --2- 172.21.15.77:0/308754150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7ca40 0x559ecce7ce60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f847c02c140 tx=0x7f847c02c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b39957b4e237bebc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:09.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.045+0000 7f84927d1640 1 --2- 172.21.15.77:0/485171570 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84780a2a10 0x7f84780a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:09.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.051+0000 7f84927d1640 1 --2- 172.21.15.77:0/485171570 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84780a2a10 0x7f84780a4ed0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f847c03e000 tx=0x7f847c02cd70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:09.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.055+0000 7f8477fff640 1 -- 172.21.15.77:0/308754150 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f847c039040 con 0x559ecce7ca40 2024-03-16T10:22:09.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.056+0000 7f8477fff640 1 -- 172.21.15.77:0/308754150 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f847c042070 con 0x559ecce7ca40 2024-03-16T10:22:09.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.057+0000 7f8496e19740 1 -- 172.21.15.77:0/308754150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7ca40 msgr2=0x559ecce7ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:09.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.057+0000 7f8496e19740 1 --2- 172.21.15.77:0/308754150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7ca40 0x559ecce7ce60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f847c02c140 tx=0x7f847c02c6a0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.083+0000 7f8496e19740 1 -- 172.21.15.77:0/308754150 shutdown_connections 2024-03-16T10:22:09.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.083+0000 7f8496e19740 1 --2- 172.21.15.77:0/308754150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce825a0 0x559ecce84a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.083+0000 7f8496e19740 1 --2- 172.21.15.77:0/308754150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7d430 0x559ecce81f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.083+0000 7f8496e19740 1 --2- 172.21.15.77:0/308754150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7ca40 0x559ecce7ce60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.083+0000 7f8496e19740 1 -- 172.21.15.77:0/308754150 >> 172.21.15.77:0/308754150 conn(0x559ecce77fb0 msgr2=0x559ecce783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:09.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.083+0000 7f8496e19740 1 -- 172.21.15.77:0/308754150 shutdown_connections 2024-03-16T10:22:09.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.083+0000 7f8496e19740 1 -- 172.21.15.77:0/308754150 wait complete. 2024-03-16T10:22:09.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.084+0000 7f8496e19740 1 -- start start 2024-03-16T10:22:09.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.084+0000 7f8496e19740 1 Processor -- start 2024-03-16T10:22:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.085+0000 7f8496e19740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7ca40 0x559ecce86e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.085+0000 7f8496e19740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce7d430 0x559ecce873c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.085+0000 7f8496e19740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce825a0 0x559ecce8e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.085+0000 7f8496e19740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559ecce85540 con 0x559ecce7d430 2024-03-16T10:22:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.085+0000 7f8496e19740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559eccd9d230 con 0x559ecce7ca40 2024-03-16T10:22:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.085+0000 7f8496e19740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559ecce856c0 con 0x559ecce825a0 2024-03-16T10:22:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.085+0000 7f8496e19740 10 client.0 _ll_register_callbacks cb 0x559eccdd9010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.085+0000 7f84917cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce7d430 0x559ecce873c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.085+0000 7f84917cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce7d430 0x559ecce873c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60164/0 (socket says 172.21.15.77:60164) 2024-03-16T10:22:09.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.085+0000 7f84917cf640 1 -- 172.21.15.77:0/358488959 learned_addr learned my addr 172.21.15.77:0/358488959 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:09.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.085+0000 7f84927d1640 1 --2- 172.21.15.77:0/358488959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce825a0 0x559ecce8e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:09.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.086+0000 7f84927d1640 1 -- 172.21.15.77:0/358488959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7ca40 msgr2=0x559ecce86e80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T10:22:09.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.086+0000 7f84927d1640 1 --2- 172.21.15.77:0/358488959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7ca40 0x559ecce86e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.086+0000 7f84927d1640 1 -- 172.21.15.77:0/358488959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce7d430 msgr2=0x559ecce873c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:09.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.086+0000 7f84927d1640 1 --2- 172.21.15.77:0/358488959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce7d430 0x559ecce873c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.086+0000 7f84927d1640 1 -- 172.21.15.77:0/358488959 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559eccdac100 con 0x559ecce825a0 2024-03-16T10:22:09.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.086+0000 7f84927d1640 1 --2- 172.21.15.77:0/358488959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce825a0 0x559ecce8e400 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f847c03f040 tx=0x7f847c0348a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96293bde01904dbe server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.094+0000 7f84757fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:22:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.094+0000 7f84757fa640 1 -- 172.21.15.77:0/358488959 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f847c039050 con 0x559ecce825a0 2024-03-16T10:22:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.094+0000 7f8496e19740 1 -- 172.21.15.77:0/358488959 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f847c042070 con 0x559ecce825a0 2024-03-16T10:22:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.094+0000 7f84757fa640 1 -- 172.21.15.77:0/358488959 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f847c03d4f0 con 0x559ecce825a0 2024-03-16T10:22:09.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.094+0000 7f84757fa640 1 -- 172.21.15.77:0/358488959 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f847c061050 con 0x559ecce825a0 2024-03-16T10:22:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.096+0000 7f8496e19740 10 client.26632 fetch_fsmap learned FSMap version 35 2024-03-16T10:22:09.096 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.096+0000 7f8496e19740 1 -- 172.21.15.77:0/358488959 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559eccaf53f0 con 0x559ecce825a0 2024-03-16T10:22:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.096+0000 7f84757fa640 1 -- 172.21.15.77:0/358488959 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f847c042070 con 0x559ecce825a0 2024-03-16T10:22:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.096+0000 7f84757fa640 1 -- 172.21.15.77:0/358488959 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f847c03c070 con 0x559ecce825a0 2024-03-16T10:22:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.096+0000 7f8496e19740 10 client.26632 fetch_fsmap finished waiting for FSMap version 35 2024-03-16T10:22:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.096+0000 7f8496e19740 10 client.26632 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:22:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.096+0000 7f8496e19740 20 client.26632 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:09.097 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.096+0000 7f8496e19740 1 --2- 172.21.15.77:0/358488959 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x559ecce92e40 0x559ecce952c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.097+0000 7f8496e19740 4 client.26632 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:09.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.097+0000 7f8496e19740 1 -- 172.21.15.77:0/358488959 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559eccdee9d0 con 0x559ecce92e40 2024-03-16T10:22:09.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.110+0000 7f8491fd0640 1 --2- 172.21.15.77:0/358488959 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x559ecce92e40 0x559ecce952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:09.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.119+0000 7f84757fa640 10 client.26632 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:22:09.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.119+0000 7f8491fd0640 1 --2- 172.21.15.77:0/358488959 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x559ecce92e40 0x559ecce952c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e28bcdff06d6c33a server_cookie=d19fa5a349ebf346 in_seq=0 out_seq=0 2024-03-16T10:22:09.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.127+0000 7f84757fa640 1 -- 172.21.15.77:0/358488959 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559eccdee9d0 con 0x559ecce92e40 2024-03-16T10:22:09.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.127+0000 7f84757fa640 10 client.26632 handle_command_reply: tid=0 2024-03-16T10:22:09.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.127+0000 7f8496e19740 1 client.26632 shutdown 2024-03-16T10:22:09.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.127+0000 7f8496e19740 1 -- 172.21.15.77:0/358488959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce825a0 msgr2=0x559ecce8e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:09.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.127+0000 7f8496e19740 1 --2- 172.21.15.77:0/358488959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce825a0 0x559ecce8e400 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f847c03f040 tx=0x7f847c0348a0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.133+0000 7f8496e19740 1 -- 172.21.15.77:0/358488959 shutdown_connections 2024-03-16T10:22:09.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.133+0000 7f8496e19740 1 --2- 172.21.15.77:0/358488959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce825a0 0x559ecce8e400 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.133+0000 7f8496e19740 1 --2- 172.21.15.77:0/358488959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce7d430 0x559ecce873c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.133+0000 7f8496e19740 1 --2- 172.21.15.77:0/358488959 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x559ecce92e40 0x559ecce952c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.133+0000 7f8496e19740 1 --2- 172.21.15.77:0/358488959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7ca40 0x559ecce86e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.133+0000 7f8496e19740 1 -- 172.21.15.77:0/358488959 >> 172.21.15.77:0/358488959 conn(0x559ecce77fb0 msgr2=0x559ecce77140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:09.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.133+0000 7f84757fa640 0 client.26632 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:22:09.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.134+0000 7f8496e19740 1 -- 172.21.15.77:0/358488959 shutdown_connections 2024-03-16T10:22:09.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.134+0000 7f8496e19740 1 -- 172.21.15.77:0/358488959 wait complete. 2024-03-16T10:22:09.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.134+0000 7f8496e19740 20 client.26632 trim_cache size 0 max 16384 2024-03-16T10:22:09.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.260+0000 7f8496e19740 1 Processor -- start 2024-03-16T10:22:09.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.260+0000 7f8496e19740 1 -- start start 2024-03-16T10:22:09.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.260+0000 7f8496e19740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7ca40 0x559ecce72fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.260+0000 7f8496e19740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7d430 0x559ecce8e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.260+0000 7f8496e19740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce825a0 0x559ecce86e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.261+0000 7f8496e19740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f848c1d1030 con 0x559ecce825a0 2024-03-16T10:22:09.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.261+0000 7f84917cf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7d430 0x559ecce8e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:09.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.261+0000 7f84917cf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7d430 0x559ecce8e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54264/0 (socket says 172.21.15.77:54264) 2024-03-16T10:22:09.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.261+0000 7f84917cf640 1 -- 172.21.15.77:0/1316303841 learned_addr learned my addr 172.21.15.77:0/1316303841 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:09.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.261+0000 7f8496e19740 1 -- 172.21.15.77:0/1316303841 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f848c1d2c00 con 0x559ecce7d430 2024-03-16T10:22:09.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.261+0000 7f8496e19740 1 -- 172.21.15.77:0/1316303841 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f848c1d1700 con 0x559ecce7ca40 2024-03-16T10:22:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.261+0000 7f84917cf640 1 -- 172.21.15.77:0/1316303841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7ca40 msgr2=0x559ecce72fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.261+0000 7f84917cf640 1 --2- 172.21.15.77:0/1316303841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7ca40 0x559ecce72fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.261+0000 7f84917cf640 1 -- 172.21.15.77:0/1316303841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce825a0 msgr2=0x559ecce86e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.261+0000 7f84917cf640 1 --2- 172.21.15.77:0/1316303841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce825a0 0x559ecce86e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.261+0000 7f84917cf640 1 -- 172.21.15.77:0/1316303841 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559eccbbf7d0 con 0x559ecce7d430 2024-03-16T10:22:09.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.266+0000 7f84917cf640 1 --2- 172.21.15.77:0/1316303841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7d430 0x559ecce8e3c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8484008a40 tx=0x7f8484008b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47525f1744799167 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:09.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.284+0000 7f84777fe640 1 -- 172.21.15.77:0/1316303841 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484004570 con 0x559ecce7d430 2024-03-16T10:22:09.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.284+0000 7f84777fe640 1 -- 172.21.15.77:0/1316303841 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8484004710 con 0x559ecce7d430 2024-03-16T10:22:09.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.284+0000 7f84777fe640 1 -- 172.21.15.77:0/1316303841 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484007440 con 0x559ecce7d430 2024-03-16T10:22:09.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.286+0000 7f8496e19740 1 -- 172.21.15.77:0/1316303841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7d430 msgr2=0x559ecce8e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:09.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.286+0000 7f8496e19740 1 --2- 172.21.15.77:0/1316303841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7d430 0x559ecce8e3c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8484008a40 tx=0x7f8484008b20 comp rx=0 tx=0).stop 2024-03-16T10:22:09.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.286+0000 7f8496e19740 1 -- 172.21.15.77:0/1316303841 shutdown_connections 2024-03-16T10:22:09.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.286+0000 7f8496e19740 1 --2- 172.21.15.77:0/1316303841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce825a0 0x559ecce86e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.286+0000 7f8496e19740 1 --2- 172.21.15.77:0/1316303841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce7d430 0x559ecce8e3c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.286+0000 7f8496e19740 1 --2- 172.21.15.77:0/1316303841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7ca40 0x559ecce72fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.286+0000 7f8496e19740 1 -- 172.21.15.77:0/1316303841 >> 172.21.15.77:0/1316303841 conn(0x559ecce77fb0 msgr2=0x559ecce85ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:09.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.286+0000 7f8496e19740 1 -- 172.21.15.77:0/1316303841 shutdown_connections 2024-03-16T10:22:09.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.286+0000 7f8496e19740 1 -- 172.21.15.77:0/1316303841 wait complete. 2024-03-16T10:22:09.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.288+0000 7f8496e19740 1 -- start start 2024-03-16T10:22:09.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.288+0000 7f8496e19740 1 Processor -- start 2024-03-16T10:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.289+0000 7f8496e19740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7ca40 0x559ecce8ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.289+0000 7f8496e19740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce7d430 0x559ecce8e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.289+0000 7f8496e19740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce825a0 0x559ecce8a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.289+0000 7f8491fd0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7ca40 0x559ecce8ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.289+0000 7f8491fd0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7ca40 0x559ecce8ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58974/0 (socket says 172.21.15.77:58974) 2024-03-16T10:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.289+0000 7f8496e19740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f848c1d2530 con 0x559ecce7d430 2024-03-16T10:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.289+0000 7f8496e19740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f848c1d1e60 con 0x559ecce825a0 2024-03-16T10:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.289+0000 7f8496e19740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f848c1cf010 con 0x559ecce7ca40 2024-03-16T10:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.289+0000 7f8496e19740 10 client.0 _ll_register_callbacks cb 0x7f8488005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:09.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.289+0000 7f8491fd0640 1 -- 172.21.15.77:0/1183875439 learned_addr learned my addr 172.21.15.77:0/1183875439 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.289+0000 7f84927d1640 1 --2- 172.21.15.77:0/1183875439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce825a0 0x559ecce8a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.290+0000 7f8491fd0640 1 -- 172.21.15.77:0/1183875439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce825a0 msgr2=0x559ecce8a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.290+0000 7f8491fd0640 1 --2- 172.21.15.77:0/1183875439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce825a0 0x559ecce8a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.290+0000 7f8491fd0640 1 -- 172.21.15.77:0/1183875439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce7d430 msgr2=0x559ecce8e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.290+0000 7f8491fd0640 1 --2- 172.21.15.77:0/1183875439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce7d430 0x559ecce8e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.290+0000 7f8491fd0640 1 -- 172.21.15.77:0/1183875439 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ecce8b110 con 0x559ecce7ca40 2024-03-16T10:22:09.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.290+0000 7f8491fd0640 1 --2- 172.21.15.77:0/1183875439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7ca40 0x559ecce8ddd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f848809adc0 tx=0x7f848809b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e5e7f58827c7011 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:09.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.308+0000 7f84757fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:22:09.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.308+0000 7f84757fa640 1 -- 172.21.15.77:0/1183875439 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84880a37d0 con 0x559ecce7ca40 2024-03-16T10:22:09.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.309+0000 7f84757fa640 1 -- 172.21.15.77:0/1183875439 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8488015990 con 0x559ecce7ca40 2024-03-16T10:22:09.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.309+0000 7f8496e19740 1 -- 172.21.15.77:0/1183875439 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8484004710 con 0x559ecce7ca40 2024-03-16T10:22:09.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.313+0000 7f84757fa640 1 -- 172.21.15.77:0/1183875439 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8488097730 con 0x559ecce7ca40 2024-03-16T10:22:09.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.313+0000 7f8496e19740 10 client.26635 fetch_fsmap learned FSMap version 35 2024-03-16T10:22:09.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.313+0000 7f8496e19740 1 -- 172.21.15.77:0/1183875439 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559eccef7f90 con 0x559ecce7ca40 2024-03-16T10:22:09.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.315+0000 7f84757fa640 1 -- 172.21.15.77:0/1183875439 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8488015620 con 0x559ecce7ca40 2024-03-16T10:22:09.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.315+0000 7f84757fa640 1 -- 172.21.15.77:0/1183875439 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f84880d1ba0 con 0x559ecce7ca40 2024-03-16T10:22:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.316+0000 7f8496e19740 10 client.26635 fetch_fsmap finished waiting for FSMap version 35 2024-03-16T10:22:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.316+0000 7f8496e19740 10 client.26635 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:22:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.316+0000 7f8496e19740 20 client.26635 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.316+0000 7f8496e19740 1 --2- 172.21.15.77:0/1183875439 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x559eccef89b0 0x559eccefadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.316+0000 7f8496e19740 4 client.26635 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:22:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.316+0000 7f8496e19740 1 -- 172.21.15.77:0/1183875439 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f8484004710 con 0x559eccef89b0 2024-03-16T10:22:09.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.318+0000 7f84917cf640 1 --2- 172.21.15.77:0/1183875439 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x559eccef89b0 0x559eccefadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:09.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.360+0000 7f84757fa640 10 client.26635 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:22:09.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.360+0000 7f84917cf640 1 --2- 172.21.15.77:0/1183875439 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x559eccef89b0 0x559eccefadd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d7a8792b8cdd692 server_cookie=ad78ab5f6496273c in_seq=0 out_seq=0 2024-03-16T10:22:09.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.382+0000 7f84757fa640 1 -- 172.21.15.77:0/1183875439 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f8484004710 con 0x559eccef89b0 2024-03-16T10:22:09.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.382+0000 7f84757fa640 10 client.26635 handle_command_reply: tid=0 2024-03-16T10:22:09.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.382+0000 7f8496e19740 1 client.26635 shutdown 2024-03-16T10:22:09.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.413+0000 7f8496e19740 1 -- 172.21.15.77:0/1183875439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7ca40 msgr2=0x559ecce8ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:09.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.413+0000 7f8496e19740 1 --2- 172.21.15.77:0/1183875439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7ca40 0x559ecce8ddd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f848809adc0 tx=0x7f848809b880 comp rx=0 tx=0).stop 2024-03-16T10:22:09.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.413+0000 7f8496e19740 1 -- 172.21.15.77:0/1183875439 shutdown_connections 2024-03-16T10:22:09.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.413+0000 7f8496e19740 1 --2- 172.21.15.77:0/1183875439 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x559eccef89b0 0x559eccefadd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.413+0000 7f8496e19740 1 --2- 172.21.15.77:0/1183875439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559ecce825a0 0x559ecce8a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.413+0000 7f8496e19740 1 --2- 172.21.15.77:0/1183875439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ecce7d430 0x559ecce8e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.413+0000 7f8496e19740 1 --2- 172.21.15.77:0/1183875439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ecce7ca40 0x559ecce8ddd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.414 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.413+0000 7f8496e19740 1 -- 172.21.15.77:0/1183875439 >> 172.21.15.77:0/1183875439 conn(0x559ecce77fb0 msgr2=0x559ecce72fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:09.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.413+0000 7f84757fa640 0 client.26635 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:22:09.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.413+0000 7f8496e19740 1 -- 172.21.15.77:0/1183875439 shutdown_connections 2024-03-16T10:22:09.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.413+0000 7f8496e19740 1 -- 172.21.15.77:0/1183875439 wait complete. 2024-03-16T10:22:09.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.414+0000 7f8496e19740 20 client.26635 trim_cache size 0 max 16384 2024-03-16T10:22:09.416 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:22:09.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.428+0000 7f8492fd2640 1 -- 172.21.15.77:0/485171570 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84780a2a10 msgr2=0x7f84780a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.428+0000 7f8492fd2640 1 --2- 172.21.15.77:0/485171570 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84780a2a10 0x7f84780a4ed0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f847c03e000 tx=0x7f847c02cd70 comp rx=0 tx=0).stop 2024-03-16T10:22:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.428+0000 7f8492fd2640 1 -- 172.21.15.77:0/485171570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f848c073550 msgr2=0x7f848c1b1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.428+0000 7f8492fd2640 1 --2- 172.21.15.77:0/485171570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f848c073550 0x7f848c1b1730 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8488007860 tx=0x7f8488007d20 comp rx=0 tx=0).stop 2024-03-16T10:22:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.428+0000 7f8492fd2640 1 -- 172.21.15.77:0/485171570 shutdown_connections 2024-03-16T10:22:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.428+0000 7f8492fd2640 1 --2- 172.21.15.77:0/485171570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f848c078de0 0x7f848c1b6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.429+0000 7f8492fd2640 1 --2- 172.21.15.77:0/485171570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f848c076430 0x7f848c1b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.429+0000 7f8492fd2640 1 --2- 172.21.15.77:0/485171570 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84780a2a10 0x7f84780a4ed0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.429+0000 7f8492fd2640 1 --2- 172.21.15.77:0/485171570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f848c073550 0x7f848c1b1730 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.429+0000 7f8492fd2640 1 -- 172.21.15.77:0/485171570 >> 172.21.15.77:0/485171570 conn(0x7f848c06e6c0 msgr2=0x7f848c077200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.429+0000 7f8492fd2640 1 -- 172.21.15.77:0/485171570 shutdown_connections 2024-03-16T10:22:09.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:09.429+0000 7f8492fd2640 1 -- 172.21.15.77:0/485171570 wait complete. 2024-03-16T10:22:09.484 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T10:22:10.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.099+0000 7fafb67b0640 1 -- 172.21.15.77:0/1670952428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb00ff560 msgr2=0x7fafb00ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.099+0000 7fafb67b0640 1 --2- 172.21.15.77:0/1670952428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb00ff560 0x7fafb00ff980 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fafac009900 tx=0x7fafac02bc60 comp rx=0 tx=0).stop 2024-03-16T10:22:10.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.104+0000 7fafb67b0640 1 -- 172.21.15.77:0/1670952428 shutdown_connections 2024-03-16T10:22:10.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.104+0000 7fafb67b0640 1 --2- 172.21.15.77:0/1670952428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb0101030 0x7fafb01058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.104+0000 7fafb67b0640 1 --2- 172.21.15.77:0/1670952428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb0100690 0x7fafb0100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.104+0000 7fafb67b0640 1 --2- 172.21.15.77:0/1670952428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb00ff560 0x7fafb00ff980 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.104+0000 7fafb67b0640 1 -- 172.21.15.77:0/1670952428 >> 172.21.15.77:0/1670952428 conn(0x7fafb00fae60 msgr2=0x7fafb00fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.109 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.107+0000 7fafb67b0640 1 -- 172.21.15.77:0/1670952428 shutdown_connections 2024-03-16T10:22:10.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.109+0000 7fafb67b0640 1 -- 172.21.15.77:0/1670952428 wait complete. 2024-03-16T10:22:10.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.109+0000 7fafb67b0640 1 Processor -- start 2024-03-16T10:22:10.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.110+0000 7fafb67b0640 1 -- start start 2024-03-16T10:22:10.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.110+0000 7fafb67b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb00ff560 0x7fafb0110d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.110+0000 7fafb67b0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb0100690 0x7fafb0111240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.110+0000 7fafb67b0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb0101030 0x7fafb010f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.110+0000 7fafb67b0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0107bd0 con 0x7fafb00ff560 2024-03-16T10:22:10.111 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.110+0000 7fafb5faf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb0101030 0x7fafb010f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.111+0000 7fafb5faf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb0101030 0x7fafb010f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54286/0 (socket says 172.21.15.77:54286) 2024-03-16T10:22:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.111+0000 7fafb5faf640 1 -- 172.21.15.77:0/4195080208 learned_addr learned my addr 172.21.15.77:0/4195080208 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.111+0000 7fafb67b0640 1 -- 172.21.15.77:0/4195080208 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0107a50 con 0x7fafb0101030 2024-03-16T10:22:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.111+0000 7fafb67b0640 1 -- 172.21.15.77:0/4195080208 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0107d50 con 0x7fafb0100690 2024-03-16T10:22:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.111+0000 7fafb57ae640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb00ff560 0x7fafb0110d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.112 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.111+0000 7fafb4fad640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb0100690 0x7fafb0111240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.111+0000 7fafb5faf640 1 -- 172.21.15.77:0/4195080208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb0100690 msgr2=0x7fafb0111240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.111+0000 7fafb5faf640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb0100690 0x7fafb0111240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.111+0000 7fafb5faf640 1 -- 172.21.15.77:0/4195080208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb00ff560 msgr2=0x7fafb0110d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.111+0000 7fafb5faf640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb00ff560 0x7fafb0110d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.111+0000 7fafb5faf640 1 -- 172.21.15.77:0/4195080208 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb010fb10 con 0x7fafb0101030 2024-03-16T10:22:10.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.112+0000 7fafb5faf640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb0101030 0x7fafb010f390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009fd0 tx=0x7fafa000b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.112+0000 7fafa67fc640 1 -- 172.21.15.77:0/4195080208 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa0011830 con 0x7fafb0101030 2024-03-16T10:22:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.113+0000 7fafb67b0640 1 -- 172.21.15.77:0/4195080208 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb010fe00 con 0x7fafb0101030 2024-03-16T10:22:10.114 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.113+0000 7fafb67b0640 1 -- 172.21.15.77:0/4195080208 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb01aed40 con 0x7fafb0101030 2024-03-16T10:22:10.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.117+0000 7fafba602740 1 Processor -- start 2024-03-16T10:22:10.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafba602740 1 -- start start 2024-03-16T10:22:10.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafba602740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46972a40 0x55dc46972e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafba602740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc46973430 0x55dc46977f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafba602740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc469785a0 0x55dc4697aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafba602740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55dc468d7490 con 0x55dc46973430 2024-03-16T10:22:10.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafba602740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55dc4687fc90 con 0x55dc469785a0 2024-03-16T10:22:10.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafba602740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dc468d3530 con 0x55dc46972a40 2024-03-16T10:22:10.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafa67fc640 1 -- 172.21.15.77:0/4195080208 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafa0016070 con 0x7fafb0101030 2024-03-16T10:22:10.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafa67fc640 1 -- 172.21.15.77:0/4195080208 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa000f2e0 con 0x7fafb0101030 2024-03-16T10:22:10.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafb4fad640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46972a40 0x55dc46972e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafb4fad640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46972a40 0x55dc46972e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58998/0 (socket says 172.21.15.77:58998) 2024-03-16T10:22:10.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafb4fad640 1 -- 172.21.15.77:0/1193844203 learned_addr learned my addr 172.21.15.77:0/1193844203 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:10.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.118+0000 7fafb4fad640 1 --2- 172.21.15.77:0/1193844203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc469785a0 0x55dc4697aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.119+0000 7fafb4fad640 1 -- 172.21.15.77:0/1193844203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc469785a0 msgr2=0x55dc4697aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.119+0000 7fafb4fad640 1 --2- 172.21.15.77:0/1193844203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc469785a0 0x55dc4697aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.119+0000 7fafb4fad640 1 -- 172.21.15.77:0/1193844203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc46973430 msgr2=0x55dc46977f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.119+0000 7fafb4fad640 1 --2- 172.21.15.77:0/1193844203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc46973430 0x55dc46977f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.119+0000 7fafb4fad640 1 -- 172.21.15.77:0/1193844203 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc4693a500 con 0x55dc46972a40 2024-03-16T10:22:10.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.119+0000 7fafb4fad640 1 --2- 172.21.15.77:0/1193844203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46972a40 0x55dc46972e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fafa800ac90 tx=0x7fafa8009810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50dc917bbe35db63 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.120+0000 7fafa67fc640 1 -- 172.21.15.77:0/4195080208 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fafa000f540 con 0x7fafb0101030 2024-03-16T10:22:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.121+0000 7fafa67fc640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7faf9c0a2950 0x7faf9c0a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.121+0000 7fafa67fc640 1 -- 172.21.15.77:0/4195080208 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fafa00cf1b0 con 0x7fafb0101030 2024-03-16T10:22:10.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.123+0000 7fafb57ae640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7faf9c0a2950 0x7faf9c0a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.127+0000 7fafb57ae640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7faf9c0a2950 0x7faf9c0a4e10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fafac02c270 tx=0x7fafac02c8d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:10.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.129+0000 7faf9b7fe640 1 -- 172.21.15.77:0/1193844203 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8013040 con 0x55dc46972a40 2024-03-16T10:22:10.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.129+0000 7faf9b7fe640 1 -- 172.21.15.77:0/1193844203 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafa801a430 con 0x55dc46972a40 2024-03-16T10:22:10.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.129+0000 7faf9b7fe640 1 -- 172.21.15.77:0/1193844203 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8019640 con 0x55dc46972a40 2024-03-16T10:22:10.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.131+0000 7fafba602740 1 -- 172.21.15.77:0/1193844203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46972a40 msgr2=0x55dc46972e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.131+0000 7fafba602740 1 --2- 172.21.15.77:0/1193844203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46972a40 0x55dc46972e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fafa800ac90 tx=0x7fafa8009810 comp rx=0 tx=0).stop 2024-03-16T10:22:10.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.136+0000 7fafba602740 1 -- 172.21.15.77:0/1193844203 shutdown_connections 2024-03-16T10:22:10.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.136+0000 7fafba602740 1 --2- 172.21.15.77:0/1193844203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc469785a0 0x55dc4697aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.136+0000 7fafba602740 1 --2- 172.21.15.77:0/1193844203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc46973430 0x55dc46977f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.136+0000 7fafba602740 1 --2- 172.21.15.77:0/1193844203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46972a40 0x55dc46972e60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.136+0000 7fafba602740 1 -- 172.21.15.77:0/1193844203 >> 172.21.15.77:0/1193844203 conn(0x55dc4696dfb0 msgr2=0x55dc4696e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.138+0000 7fafba602740 1 -- 172.21.15.77:0/1193844203 shutdown_connections 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.140+0000 7fafba602740 1 -- 172.21.15.77:0/1193844203 wait complete. 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.140+0000 7fafba602740 1 -- start start 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.140+0000 7fafba602740 1 Processor -- start 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.141+0000 7fafba602740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc46972a40 0x55dc4697cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.141+0000 7fafba602740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 0x55dc4697d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.141+0000 7fafba602740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc469785a0 0x55dc46984540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.141+0000 7fafba602740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55dc4697b4f0 con 0x55dc46972a40 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.141+0000 7fafba602740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55dc46893230 con 0x55dc469785a0 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.141+0000 7fafba602740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dc4697b670 con 0x55dc46973430 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.141+0000 7fafba602740 10 client.0 _ll_register_callbacks cb 0x55dc468cf010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.142+0000 7fafb4fad640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 0x55dc4697d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.142+0000 7fafb4fad640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 0x55dc4697d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:59014/0 (socket says 172.21.15.77:59014) 2024-03-16T10:22:10.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.142+0000 7fafb4fad640 1 -- 172.21.15.77:0/744809843 learned_addr learned my addr 172.21.15.77:0/744809843 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:10.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.142+0000 7fafb5faf640 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc469785a0 0x55dc46984540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.142+0000 7fafb57ae640 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc46972a40 0x55dc4697cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.142+0000 7fafb4fad640 1 -- 172.21.15.77:0/744809843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc469785a0 msgr2=0x55dc46984540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.142+0000 7fafb4fad640 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc469785a0 0x55dc46984540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.142+0000 7fafb4fad640 1 -- 172.21.15.77:0/744809843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc46972a40 msgr2=0x55dc4697cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.142+0000 7fafb4fad640 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc46972a40 0x55dc4697cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.142+0000 7fafb4fad640 1 -- 172.21.15.77:0/744809843 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc468a2100 con 0x55dc46973430 2024-03-16T10:22:10.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.144+0000 7fafb4fad640 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 0x55dc4697d500 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fafa800a010 tx=0x7fafa8009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac90ababe9ad61eb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:10.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.146+0000 7faf98ff9640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:22:10.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.147+0000 7faf98ff9640 1 -- 172.21.15.77:0/744809843 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa800a950 con 0x55dc46973430 2024-03-16T10:22:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.147+0000 7fafba602740 1 -- 172.21.15.77:0/744809843 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafa801a430 con 0x55dc46973430 2024-03-16T10:22:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.147+0000 7faf98ff9640 1 -- 172.21.15.77:0/744809843 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fafa801a7f0 con 0x55dc46973430 2024-03-16T10:22:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.147+0000 7faf98ff9640 1 -- 172.21.15.77:0/744809843 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fafa8014a50 con 0x55dc46973430 2024-03-16T10:22:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.148+0000 7fafba602740 10 client.26647 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.148+0000 7fafba602740 1 -- 172.21.15.77:0/744809843 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55dc465eb3f0 con 0x55dc46973430 2024-03-16T10:22:10.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.148+0000 7faf98ff9640 1 -- 172.21.15.77:0/744809843 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafa801a430 con 0x55dc46973430 2024-03-16T10:22:10.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.149+0000 7faf98ff9640 1 -- 172.21.15.77:0/744809843 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fafa8023cc0 con 0x55dc46973430 2024-03-16T10:22:10.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.149+0000 7fafba602740 10 client.26647 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:10.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.149+0000 7fafba602740 10 client.26647 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:22:10.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.149+0000 7fafba602740 20 client.26647 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.149+0000 7fafba602740 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55dc469ab190 0x55dc469ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.149+0000 7fafba602740 4 client.26647 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:10.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.149+0000 7fafba602740 1 -- 172.21.15.77:0/744809843 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc468e49d0 con 0x55dc469ab190 2024-03-16T10:22:10.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.152+0000 7fafb57ae640 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55dc469ab190 0x55dc469ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.155+0000 7fafb57ae640 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55dc469ab190 0x55dc469ad5b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=de7434edc4b08b78 server_cookie=13ed5b1eb83d8a0e in_seq=0 out_seq=0 2024-03-16T10:22:10.155 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.155+0000 7faf98ff9640 10 client.26647 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:22:10.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.156+0000 7faf98ff9640 1 -- 172.21.15.77:0/744809843 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dc468e49d0 con 0x55dc469ab190 2024-03-16T10:22:10.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.156+0000 7faf98ff9640 10 client.26647 handle_command_reply: tid=0 2024-03-16T10:22:10.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.156+0000 7fafba602740 1 client.26647 shutdown 2024-03-16T10:22:10.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.157+0000 7fafba602740 1 -- 172.21.15.77:0/744809843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 msgr2=0x55dc4697d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.158 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.157+0000 7fafba602740 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 0x55dc4697d500 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fafa800a010 tx=0x7fafa8009700 comp rx=0 tx=0).stop 2024-03-16T10:22:10.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.163+0000 7fafba602740 1 -- 172.21.15.77:0/744809843 shutdown_connections 2024-03-16T10:22:10.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.163+0000 7fafba602740 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55dc469ab190 0x55dc469ad5b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.163+0000 7fafba602740 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc469785a0 0x55dc46984540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.163+0000 7fafba602740 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 0x55dc4697d500 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.163+0000 7fafba602740 1 --2- 172.21.15.77:0/744809843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc46972a40 0x55dc4697cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.163+0000 7fafba602740 1 -- 172.21.15.77:0/744809843 >> 172.21.15.77:0/744809843 conn(0x55dc4696dfb0 msgr2=0x55dc4696d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.164+0000 7faf98ff9640 0 client.26647 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:22:10.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:09 smithi192 ceph-mon[36354]: pgmap v502: 97 pgs: 97 active+clean; 573 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 352 op/s 2024-03-16T10:22:10.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.167+0000 7fafba602740 1 -- 172.21.15.77:0/744809843 shutdown_connections 2024-03-16T10:22:10.169 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.169+0000 7fafba602740 1 -- 172.21.15.77:0/744809843 wait complete. 2024-03-16T10:22:10.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.171+0000 7fafba602740 20 client.26647 trim_cache size 0 max 16384 2024-03-16T10:22:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.195+0000 7fadcb7fe640 1 -- 172.21.15.77:0/2293266292 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5572538022d0 con 0x5572539119f0 2024-03-16T10:22:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.195+0000 7fadcb7fe640 10 client.26630 handle_command_reply: tid=0 2024-03-16T10:22:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.195+0000 7fadf1629740 1 client.26630 shutdown 2024-03-16T10:22:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.195+0000 7fadf1629740 1 -- 172.21.15.77:0/2293266292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895f20 msgr2=0x5572538a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.195+0000 7fadf1629740 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895f20 0x5572538a4470 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fade004b040 tx=0x7fade00485b0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.195+0000 7fadf1629740 1 -- 172.21.15.77:0/2293266292 shutdown_connections 2024-03-16T10:22:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.195+0000 7fadf1629740 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5572539119f0 0x557253913e10 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.195+0000 7fadf1629740 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55725389b090 0x5572538a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.195+0000 7fadf1629740 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557253895f20 0x5572538a4470 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.195+0000 7fadf1629740 1 --2- 172.21.15.77:0/2293266292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557253895530 0x5572538a3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.195+0000 7fadf1629740 1 -- 172.21.15.77:0/2293266292 >> 172.21.15.77:0/2293266292 conn(0x557253890aa0 msgr2=0x557253893d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.196+0000 7fadcb7fe640 0 client.26630 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.196+0000 7fadf1629740 1 -- 172.21.15.77:0/2293266292 shutdown_connections 2024-03-16T10:22:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.196+0000 7fadf1629740 1 -- 172.21.15.77:0/2293266292 wait complete. 2024-03-16T10:22:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.196+0000 7fadf1629740 20 client.26630 trim_cache size 0 max 16384 2024-03-16T10:22:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:22:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:22:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:22:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:22:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.199+0000 7fadee95b640 1 -- 172.21.15.77:0/2138185289 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fadd00a2a00 msgr2=0x7fadd00a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.200+0000 7fadee95b640 1 --2- 172.21.15.77:0/2138185289 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fadd00a2a00 0x7fadd00a4ec0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7faddc010bb0 tx=0x7faddc010fc0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.200+0000 7fadee95b640 1 -- 172.21.15.77:0/2138185289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fade8072ab0 msgr2=0x7fade8117eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.200+0000 7fadee95b640 1 --2- 172.21.15.77:0/2138185289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fade8072ab0 0x7fade8117eb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faddc00b450 tx=0x7faddc00b910 comp rx=0 tx=0).stop 2024-03-16T10:22:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.200+0000 7fadee95b640 1 -- 172.21.15.77:0/2138185289 shutdown_connections 2024-03-16T10:22:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.200+0000 7fadee95b640 1 --2- 172.21.15.77:0/2138185289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fade8109510 0x7fade8119420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.200+0000 7fadee95b640 1 --2- 172.21.15.77:0/2138185289 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fadd00a2a00 0x7fadd00a4ec0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.200+0000 7fadee95b640 1 --2- 172.21.15.77:0/2138185289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fade8073410 0x7fade81183f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.200+0000 7fadee95b640 1 --2- 172.21.15.77:0/2138185289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fade8072ab0 0x7fade8117eb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.200+0000 7fadee95b640 1 -- 172.21.15.77:0/2138185289 >> 172.21.15.77:0/2138185289 conn(0x7fade806e650 msgr2=0x7fade810c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.200+0000 7fadee95b640 1 -- 172.21.15.77:0/2138185289 shutdown_connections 2024-03-16T10:22:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:10.200+0000 7fadee95b640 1 -- 172.21.15.77:0/2138185289 wait complete. 2024-03-16T10:22:10.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.243+0000 7fafba602740 1 Processor -- start 2024-03-16T10:22:10.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafba602740 1 -- start start 2024-03-16T10:22:10.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafba602740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46972a40 0x55dc46968fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafba602740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 0x55dc46984500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafba602740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc469785a0 0x55dc4697cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafb57ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46972a40 0x55dc46968fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafba602740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fafb011b9d0 con 0x55dc469785a0 2024-03-16T10:22:10.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafba602740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fafb011d5a0 con 0x55dc46972a40 2024-03-16T10:22:10.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafba602740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fafb011c0a0 con 0x55dc46973430 2024-03-16T10:22:10.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafb57ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46972a40 0x55dc46968fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54314/0 (socket says 172.21.15.77:54314) 2024-03-16T10:22:10.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafb57ae640 1 -- 172.21.15.77:0/831005332 learned_addr learned my addr 172.21.15.77:0/831005332 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:10.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafb4fad640 1 --2- 172.21.15.77:0/831005332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 0x55dc46984500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:09 smithi077 ceph-mon[32400]: pgmap v502: 97 pgs: 97 active+clean; 573 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 352 op/s 2024-03-16T10:22:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafb5faf640 1 --2- 172.21.15.77:0/831005332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc469785a0 0x55dc4697cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafb57ae640 1 -- 172.21.15.77:0/831005332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 msgr2=0x55dc46984500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafb57ae640 1 --2- 172.21.15.77:0/831005332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 0x55dc46984500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafb57ae640 1 -- 172.21.15.77:0/831005332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc469785a0 msgr2=0x55dc4697cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafb57ae640 1 --2- 172.21.15.77:0/831005332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc469785a0 0x55dc4697cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.244+0000 7fafb57ae640 1 -- 172.21.15.77:0/831005332 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc466b57d0 con 0x55dc46972a40 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.245+0000 7fafb5faf640 1 --2- 172.21.15.77:0/831005332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc469785a0 0x55dc4697cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.245+0000 7fafb4fad640 1 --2- 172.21.15.77:0/831005332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 0x55dc46984500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.245+0000 7fafb57ae640 1 --2- 172.21.15.77:0/831005332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46972a40 0x55dc46968fd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fafac02c7e0 tx=0x7fafac041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c247a1135024a043 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.245+0000 7faf9affd640 1 -- 172.21.15.77:0/831005332 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafac054c30 con 0x55dc46972a40 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.245+0000 7faf9affd640 1 -- 172.21.15.77:0/831005332 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafac059070 con 0x55dc46972a40 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.245+0000 7faf9affd640 1 -- 172.21.15.77:0/831005332 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafac0046c0 con 0x55dc46972a40 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.246+0000 7fafba602740 1 -- 172.21.15.77:0/831005332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46972a40 msgr2=0x55dc46968fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.246+0000 7fafba602740 1 --2- 172.21.15.77:0/831005332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46972a40 0x55dc46968fd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fafac02c7e0 tx=0x7fafac041000 comp rx=0 tx=0).stop 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.246+0000 7fafba602740 1 -- 172.21.15.77:0/831005332 shutdown_connections 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.246+0000 7fafba602740 1 --2- 172.21.15.77:0/831005332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc469785a0 0x55dc4697cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.246+0000 7fafba602740 1 --2- 172.21.15.77:0/831005332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46973430 0x55dc46984500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.246+0000 7fafba602740 1 --2- 172.21.15.77:0/831005332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46972a40 0x55dc46968fd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.246+0000 7fafba602740 1 -- 172.21.15.77:0/831005332 >> 172.21.15.77:0/831005332 conn(0x55dc4696dfb0 msgr2=0x55dc46973e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.248+0000 7fafba602740 1 -- 172.21.15.77:0/831005332 shutdown_connections 2024-03-16T10:22:10.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.249+0000 7fafba602740 1 -- 172.21.15.77:0/831005332 wait complete. 2024-03-16T10:22:10.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.250+0000 7fafba602740 1 -- start start 2024-03-16T10:22:10.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.250+0000 7fafba602740 1 Processor -- start 2024-03-16T10:22:10.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:09 smithi193 ceph-mon[36311]: pgmap v502: 97 pgs: 97 active+clean; 573 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 352 op/s 2024-03-16T10:22:10.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.252+0000 7fafba602740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46972a40 0x55dc46980640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.252+0000 7fafba602740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46973430 0x55dc46980b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.252+0000 7fafba602740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc469785a0 0x55dc46983ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.252+0000 7fafba602740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fafb011ced0 con 0x55dc469785a0 2024-03-16T10:22:10.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.252+0000 7fafba602740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fafb011c800 con 0x55dc46973430 2024-03-16T10:22:10.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.252+0000 7fafba602740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fafb01199b0 con 0x55dc46972a40 2024-03-16T10:22:10.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.252+0000 7fafba602740 10 client.0 _ll_register_callbacks cb 0x7fafa800a5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:10.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.252+0000 7fafb4fad640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46973430 0x55dc46980b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.253+0000 7fafb4fad640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46973430 0x55dc46980b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54318/0 (socket says 172.21.15.77:54318) 2024-03-16T10:22:10.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.253+0000 7fafb4fad640 1 -- 172.21.15.77:0/1784994125 learned_addr learned my addr 172.21.15.77:0/1784994125 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:10.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.253+0000 7fafb4fad640 1 -- 172.21.15.77:0/1784994125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46972a40 msgr2=0x55dc46980640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T10:22:10.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.253+0000 7fafb4fad640 1 --2- 172.21.15.77:0/1784994125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46972a40 0x55dc46980640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.253+0000 7fafb4fad640 1 -- 172.21.15.77:0/1784994125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc469785a0 msgr2=0x55dc46983ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:22:10.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.253+0000 7fafb4fad640 1 --2- 172.21.15.77:0/1784994125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc469785a0 0x55dc46983ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.253+0000 7fafb4fad640 1 -- 172.21.15.77:0/1784994125 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc46984190 con 0x55dc46973430 2024-03-16T10:22:10.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.253+0000 7fafb4fad640 1 --2- 172.21.15.77:0/1784994125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46973430 0x55dc46980b80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fafa8002230 tx=0x7fafa800ac00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f2448cd71bf8bae server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:10.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.259+0000 7faf98ff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:22:10.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.259+0000 7faf98ff9640 1 -- 172.21.15.77:0/1784994125 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa801a430 con 0x55dc46973430 2024-03-16T10:22:10.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.259+0000 7fafba602740 1 -- 172.21.15.77:0/1784994125 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafac059070 con 0x55dc46973430 2024-03-16T10:22:10.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.259+0000 7faf98ff9640 1 -- 172.21.15.77:0/1784994125 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fafa8014bf0 con 0x55dc46973430 2024-03-16T10:22:10.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.259+0000 7faf98ff9640 1 -- 172.21.15.77:0/1784994125 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fafa801a9f0 con 0x55dc46973430 2024-03-16T10:22:10.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.259+0000 7fafba602740 10 client.26657 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:10.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.259+0000 7fafba602740 1 -- 172.21.15.77:0/1784994125 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55dc46a20eb0 con 0x55dc46973430 2024-03-16T10:22:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.261+0000 7faf98ff9640 1 -- 172.21.15.77:0/1784994125 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafac059070 con 0x55dc46973430 2024-03-16T10:22:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.261+0000 7faf98ff9640 1 -- 172.21.15.77:0/1784994125 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fafa8023720 con 0x55dc46973430 2024-03-16T10:22:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.261+0000 7fafba602740 10 client.26657 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.261+0000 7fafba602740 10 client.26657 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:22:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.261+0000 7fafba602740 20 client.26657 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.261+0000 7fafba602740 1 --2- 172.21.15.77:0/1784994125 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55dc46a21770 0x55dc46a23b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.261+0000 7fafba602740 4 client.26657 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:22:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.261+0000 7fafba602740 1 -- 172.21.15.77:0/1784994125 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dc468ded20 con 0x55dc46a21770 2024-03-16T10:22:10.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.262+0000 7fafb57ae640 1 --2- 172.21.15.77:0/1784994125 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55dc46a21770 0x55dc46a23b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.263+0000 7fafb57ae640 1 --2- 172.21.15.77:0/1784994125 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55dc46a21770 0x55dc46a23b90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=89e82ee8ad20e1b5 server_cookie=dffe84457d4542d7 in_seq=0 out_seq=0 2024-03-16T10:22:10.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.263+0000 7faf98ff9640 10 client.26657 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:22:10.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.263+0000 7faf98ff9640 1 -- 172.21.15.77:0/1784994125 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dc468ded20 con 0x55dc46a21770 2024-03-16T10:22:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.264+0000 7faf98ff9640 10 client.26657 handle_command_reply: tid=0 2024-03-16T10:22:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.264+0000 7fafba602740 1 client.26657 shutdown 2024-03-16T10:22:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.264+0000 7fafba602740 1 -- 172.21.15.77:0/1784994125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46973430 msgr2=0x55dc46980b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.264+0000 7fafba602740 1 --2- 172.21.15.77:0/1784994125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46973430 0x55dc46980b80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fafa8002230 tx=0x7fafa800ac00 comp rx=0 tx=0).stop 2024-03-16T10:22:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.269+0000 7fafba602740 1 -- 172.21.15.77:0/1784994125 shutdown_connections 2024-03-16T10:22:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.269+0000 7fafba602740 1 --2- 172.21.15.77:0/1784994125 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55dc46a21770 0x55dc46a23b90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.269+0000 7fafba602740 1 --2- 172.21.15.77:0/1784994125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55dc469785a0 0x55dc46983ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.269+0000 7fafba602740 1 --2- 172.21.15.77:0/1784994125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc46973430 0x55dc46980b80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.269+0000 7fafba602740 1 --2- 172.21.15.77:0/1784994125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc46972a40 0x55dc46980640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.269+0000 7faf98ff9640 0 client.26657 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:22:10.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.269+0000 7fafba602740 1 -- 172.21.15.77:0/1784994125 >> 172.21.15.77:0/1784994125 conn(0x55dc4696dfb0 msgr2=0x55dc46974a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.271+0000 7fafba602740 1 -- 172.21.15.77:0/1784994125 shutdown_connections 2024-03-16T10:22:10.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.272+0000 7fafba602740 1 -- 172.21.15.77:0/1784994125 wait complete. 2024-03-16T10:22:10.279 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:22:10.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.278+0000 7fafba602740 20 client.26657 trim_cache size 0 max 16384 2024-03-16T10:22:10.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.282+0000 7fafb67b0640 1 -- 172.21.15.77:0/4195080208 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7faf9c0a2950 msgr2=0x7faf9c0a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.282+0000 7fafb67b0640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7faf9c0a2950 0x7faf9c0a4e10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fafac02c270 tx=0x7fafac02c8d0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.282+0000 7fafb67b0640 1 -- 172.21.15.77:0/4195080208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb0101030 msgr2=0x7fafb010f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.282+0000 7fafb67b0640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb0101030 0x7fafb010f390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009fd0 tx=0x7fafa000b8e0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.284+0000 7fafb67b0640 1 -- 172.21.15.77:0/4195080208 shutdown_connections 2024-03-16T10:22:10.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.284+0000 7fafb67b0640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fafb0101030 0x7fafb010f390 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.284+0000 7fafb67b0640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7faf9c0a2950 0x7faf9c0a4e10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.284+0000 7fafb67b0640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafb0100690 0x7fafb0111240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.284+0000 7fafb67b0640 1 --2- 172.21.15.77:0/4195080208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb00ff560 0x7fafb0110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.284+0000 7fafb67b0640 1 -- 172.21.15.77:0/4195080208 >> 172.21.15.77:0/4195080208 conn(0x7fafb00fae60 msgr2=0x7fafb01039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.285+0000 7fafb67b0640 1 -- 172.21.15.77:0/4195080208 shutdown_connections 2024-03-16T10:22:10.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.285+0000 7fafb67b0640 1 -- 172.21.15.77:0/4195080208 wait complete. 2024-03-16T10:22:10.311 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T10:22:10.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.454+0000 7f2f347cd640 1 -- 172.21.15.77:0/3290603817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f2c1022a0 msgr2=0x7f2f2c108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.454+0000 7f2f347cd640 1 --2- 172.21.15.77:0/3290603817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f2c1022a0 0x7f2f2c108b30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009ca0 tx=0x7f2f2402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 -- 172.21.15.77:0/3290603817 shutdown_connections 2024-03-16T10:22:10.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 --2- 172.21.15.77:0/3290603817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f2c1022a0 0x7f2f2c108b30 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 --2- 172.21.15.77:0/3290603817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f2c1018e0 0x7f2f2c101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 --2- 172.21.15.77:0/3290603817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c1006f0 0x7f2f2c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 -- 172.21.15.77:0/3290603817 >> 172.21.15.77:0/3290603817 conn(0x7f2f2c0fbc80 msgr2=0x7f2f2c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 -- 172.21.15.77:0/3290603817 shutdown_connections 2024-03-16T10:22:10.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 -- 172.21.15.77:0/3290603817 wait complete. 2024-03-16T10:22:10.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 Processor -- start 2024-03-16T10:22:10.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 -- start start 2024-03-16T10:22:10.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f2c1006f0 0x7f2f2c1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f2c1018e0 0x7f2f2c196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c1022a0 0x7f2f2c19df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c10c370 con 0x7f2f2c1018e0 2024-03-16T10:22:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c10c1f0 con 0x7f2f2c1022a0 2024-03-16T10:22:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f347cd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c10c4f0 con 0x7f2f2c1006f0 2024-03-16T10:22:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f31d68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f2c1018e0 0x7f2f2c196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f31d68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f2c1018e0 0x7f2f2c196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60244/0 (socket says 172.21.15.77:60244) 2024-03-16T10:22:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f32569640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f2c1006f0 0x7f2f2c1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f32d6a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c1022a0 0x7f2f2c19df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.455+0000 7f2f31d68640 1 -- 172.21.15.77:0/56717897 learned_addr learned my addr 172.21.15.77:0/56717897 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.456+0000 7f2f32569640 1 -- 172.21.15.77:0/56717897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c1022a0 msgr2=0x7f2f2c19df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.456+0000 7f2f32569640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c1022a0 0x7f2f2c19df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.456+0000 7f2f32569640 1 -- 172.21.15.77:0/56717897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f2c1018e0 msgr2=0x7f2f2c196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.456+0000 7f2f32569640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f2c1018e0 0x7f2f2c196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.456+0000 7f2f32569640 1 -- 172.21.15.77:0/56717897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2c19e5e0 con 0x7f2f2c1006f0 2024-03-16T10:22:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.456+0000 7f2f31d68640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f2c1018e0 0x7f2f2c196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.456+0000 7f2f32569640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f2c1006f0 0x7f2f2c1969c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800e9c0 tx=0x7f2f2800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.456+0000 7f2f237fe640 1 -- 172.21.15.77:0/56717897 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2800ccc0 con 0x7f2f2c1006f0 2024-03-16T10:22:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.456+0000 7f2f347cd640 1 -- 172.21.15.77:0/56717897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2c19e8d0 con 0x7f2f2c1006f0 2024-03-16T10:22:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.456+0000 7f2f237fe640 1 -- 172.21.15.77:0/56717897 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f280035f0 con 0x7f2f2c1006f0 2024-03-16T10:22:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.456+0000 7f2f237fe640 1 -- 172.21.15.77:0/56717897 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2800ce60 con 0x7f2f2c1006f0 2024-03-16T10:22:10.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.456+0000 7f2f347cd640 1 -- 172.21.15.77:0/56717897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f2c19ee70 con 0x7f2f2c1006f0 2024-03-16T10:22:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.458+0000 7f2f237fe640 1 -- 172.21.15.77:0/56717897 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f2f280037a0 con 0x7f2f2c1006f0 2024-03-16T10:22:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.459+0000 7f2f237fe640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2f180a2800 0x7f2f180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.459+0000 7f2f373bb740 1 Processor -- start 2024-03-16T10:22:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.459+0000 7f2f31d68640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2f180a2800 0x7f2f180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f237fe640 1 -- 172.21.15.77:0/56717897 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2f280cee10 con 0x7f2f2c1006f0 2024-03-16T10:22:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f373bb740 1 -- start start 2024-03-16T10:22:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f373bb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc575d850 0x562dc575dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f373bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575e240 0x562dc5762d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f373bb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc57633b0 0x562dc5765840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f373bb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562dc56c2490 con 0x562dc57633b0 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f373bb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x562dc566ac90 con 0x562dc575d850 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f373bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562dc56be530 con 0x562dc575e240 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f32d6a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc57633b0 0x562dc5765840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f32569640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc575d850 0x562dc575dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f32d6a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc57633b0 0x562dc5765840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60246/0 (socket says 172.21.15.77:60246) 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f32569640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc575d850 0x562dc575dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54342/0 (socket says 172.21.15.77:54342) 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f32d6a640 1 -- 172.21.15.77:0/3336828815 learned_addr learned my addr 172.21.15.77:0/3336828815 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f31d68640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2f180a2800 0x7f2f180a4cc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c197ed0 tx=0x7f2f1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f31d68640 1 --2- 172.21.15.77:0/3336828815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575e240 0x562dc5762d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f32569640 1 -- 172.21.15.77:0/3336828815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575e240 msgr2=0x562dc5762d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f32569640 1 --2- 172.21.15.77:0/3336828815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575e240 0x562dc5762d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f32569640 1 -- 172.21.15.77:0/3336828815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc57633b0 msgr2=0x562dc5765840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f32569640 1 --2- 172.21.15.77:0/3336828815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc57633b0 0x562dc5765840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.460+0000 7f2f32569640 1 -- 172.21.15.77:0/3336828815 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dc5725500 con 0x562dc575d850 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.461+0000 7f2f32569640 1 --2- 172.21.15.77:0/3336828815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc575d850 0x562dc575dc70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2f280cc9d0 tx=0x7f2f2809b280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93eaaec6d948bdcb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.461+0000 7f2f20ff9640 1 -- 172.21.15.77:0/3336828815 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f280b3070 con 0x562dc575d850 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.461+0000 7f2f20ff9640 1 -- 172.21.15.77:0/3336828815 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f280a3fc0 con 0x562dc575d850 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.461+0000 7f2f20ff9640 1 -- 172.21.15.77:0/3336828815 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f28098020 con 0x562dc575d850 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.462+0000 7f2f373bb740 1 -- 172.21.15.77:0/3336828815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc575d850 msgr2=0x562dc575dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.462+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3336828815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc575d850 0x562dc575dc70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2f280cc9d0 tx=0x7f2f2809b280 comp rx=0 tx=0).stop 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.462+0000 7f2f373bb740 1 -- 172.21.15.77:0/3336828815 shutdown_connections 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.462+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3336828815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc57633b0 0x562dc5765840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.462+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3336828815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575e240 0x562dc5762d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.462+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3336828815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc575d850 0x562dc575dc70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.462+0000 7f2f373bb740 1 -- 172.21.15.77:0/3336828815 >> 172.21.15.77:0/3336828815 conn(0x562dc5758dc0 msgr2=0x562dc57591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.462+0000 7f2f373bb740 1 -- 172.21.15.77:0/3336828815 shutdown_connections 2024-03-16T10:22:10.469 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.462+0000 7f2f373bb740 1 -- 172.21.15.77:0/3336828815 wait complete. 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.462+0000 7f2f373bb740 1 -- start start 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.462+0000 7f2f373bb740 1 Processor -- start 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f373bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc5767d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f373bb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc57682b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f373bb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f373bb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562dc5766360 con 0x562dc575e240 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f373bb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x562dc567e230 con 0x562dc57633b0 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f373bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562dc57664e0 con 0x562dc575d850 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f373bb740 10 client.0 _ll_register_callbacks cb 0x562dc56ba010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f31d68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc57682b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f32d6a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f32569640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc5767d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f32569640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc5767d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:59066/0 (socket says 172.21.15.77:59066) 2024-03-16T10:22:10.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f32d6a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:54350/0 (socket says 172.21.15.77:54350) 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f32569640 1 -- 172.21.15.77:0/3341672206 learned_addr learned my addr 172.21.15.77:0/3341672206 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f31d68640 1 -- 172.21.15.77:0/3341672206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 msgr2=0x562dc5767d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f31d68640 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc5767d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f31d68640 1 -- 172.21.15.77:0/3341672206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 msgr2=0x562dc576f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f31d68640 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f31d68640 1 -- 172.21.15.77:0/3341672206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dc568d100 con 0x562dc575e240 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f32569640 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc5767d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f32d6a640 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f31d68640 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc57682b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x562dc575e640 tx=0x7f2f1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e17d0cbc2bbd406 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f15ffb640 1 -- 172.21.15.77:0/3341672206 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c02a060 con 0x562dc575e240 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.463+0000 7f2f15ffb640 1 -- 172.21.15.77:0/3341672206 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2f1c01cbb0 con 0x562dc575e240 2024-03-16T10:22:10.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f373bb740 1 -- 172.21.15.77:0/3341672206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f280a3fc0 con 0x562dc575e240 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f15ffb640 1 -- 172.21.15.77:0/3341672206 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2f1c01cdb0 con 0x562dc575e240 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f373bb740 10 client.16893 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f373bb740 1 -- 172.21.15.77:0/3341672206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x562dc53d63f0 con 0x562dc575e240 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f15ffb640 1 -- 172.21.15.77:0/3341672206 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f280a3fc0 con 0x562dc575e240 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f15ffb640 1 -- 172.21.15.77:0/3341672206 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f2f1c024070 con 0x562dc575e240 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f373bb740 10 client.16893 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f373bb740 10 client.16893 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f373bb740 20 client.16893 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x562dc5773c50 0x562dc57760b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f373bb740 4 client.16893 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f373bb740 1 -- 172.21.15.77:0/3341672206 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562dc56cf9d0 con 0x562dc5773c50 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.464+0000 7f2f32569640 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x562dc5773c50 0x562dc57760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.465+0000 7f2f32569640 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x562dc5773c50 0x562dc57760b0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c753413d38766f94 server_cookie=8bff67a829e149b5 in_seq=0 out_seq=0 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.465+0000 7f2f15ffb640 10 client.16893 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:10.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.465+0000 7f2f15ffb640 1 -- 172.21.15.77:0/3341672206 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562dc56cf9d0 con 0x562dc5773c50 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.465+0000 7f2f15ffb640 10 client.16893 handle_command_reply: tid=0 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f373bb740 1 client.16893 shutdown 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f373bb740 1 -- 172.21.15.77:0/3341672206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 msgr2=0x562dc57682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc57682b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x562dc575e640 tx=0x7f2f1c004730 comp rx=0 tx=0).stop 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f373bb740 1 -- 172.21.15.77:0/3341672206 shutdown_connections 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x562dc5773c50 0x562dc57760b0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc57682b0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3341672206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc5767d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f15ffb640 0 client.16893 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f373bb740 1 -- 172.21.15.77:0/3341672206 >> 172.21.15.77:0/3341672206 conn(0x562dc5758dc0 msgr2=0x562dc5757fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f373bb740 1 -- 172.21.15.77:0/3341672206 shutdown_connections 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f373bb740 1 -- 172.21.15.77:0/3341672206 wait complete. 2024-03-16T10:22:10.473 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.466+0000 7f2f373bb740 20 client.16893 trim_cache size 0 max 16384 2024-03-16T10:22:10.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f373bb740 1 Processor -- start 2024-03-16T10:22:10.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f373bb740 1 -- start start 2024-03-16T10:22:10.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f373bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc5767d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f373bb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc57682b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f373bb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f373bb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c1a6a00 con 0x562dc575e240 2024-03-16T10:22:10.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f373bb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c1a85d0 con 0x562dc57633b0 2024-03-16T10:22:10.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f373bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c1a70d0 con 0x562dc575d850 2024-03-16T10:22:10.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f31d68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc57682b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f31d68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc57682b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60254/0 (socket says 172.21.15.77:60254) 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f31d68640 1 -- 172.21.15.77:0/2717664947 learned_addr learned my addr 172.21.15.77:0/2717664947 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f32569640 1 --2- 172.21.15.77:0/2717664947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc5767d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f32d6a640 1 --2- 172.21.15.77:0/2717664947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f31d68640 1 -- 172.21.15.77:0/2717664947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 msgr2=0x562dc5767d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f31d68640 1 --2- 172.21.15.77:0/2717664947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc5767d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f31d68640 1 -- 172.21.15.77:0/2717664947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 msgr2=0x562dc576f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.480+0000 7f2f31d68640 1 --2- 172.21.15.77:0/2717664947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.481+0000 7f2f31d68640 1 -- 172.21.15.77:0/2717664947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dc54a07d0 con 0x562dc575e240 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.481+0000 7f2f32569640 1 --2- 172.21.15.77:0/2717664947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc5767d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.481+0000 7f2f31d68640 1 --2- 172.21.15.77:0/2717664947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc57682b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c009f20 tx=0x7f2f1c004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5483cb9be2494a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.481+0000 7f2f17fff640 1 -- 172.21.15.77:0/2717664947 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c00f8e0 con 0x562dc575e240 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.481+0000 7f2f17fff640 1 -- 172.21.15.77:0/2717664947 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f1c01cbb0 con 0x562dc575e240 2024-03-16T10:22:10.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.481+0000 7f2f17fff640 1 -- 172.21.15.77:0/2717664947 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c025e60 con 0x562dc575e240 2024-03-16T10:22:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 -- 172.21.15.77:0/2717664947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 msgr2=0x562dc57682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 --2- 172.21.15.77:0/2717664947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc57682b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c009f20 tx=0x7f2f1c004680 comp rx=0 tx=0).stop 2024-03-16T10:22:10.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 -- 172.21.15.77:0/2717664947 shutdown_connections 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 --2- 172.21.15.77:0/2717664947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 --2- 172.21.15.77:0/2717664947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc57682b0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 --2- 172.21.15.77:0/2717664947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc5767d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 -- 172.21.15.77:0/2717664947 >> 172.21.15.77:0/2717664947 conn(0x562dc5758dc0 msgr2=0x562dc575f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 -- 172.21.15.77:0/2717664947 shutdown_connections 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 -- 172.21.15.77:0/2717664947 wait complete. 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 -- start start 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 Processor -- start 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc576f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc576bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c1a7f00 con 0x562dc575e240 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c1a7830 con 0x562dc57633b0 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c1a49e0 con 0x562dc575d850 2024-03-16T10:22:10.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f373bb740 10 client.0 _ll_register_callbacks cb 0x7f2f1c009370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f31d68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc576bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f31d68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc576bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60270/0 (socket says 172.21.15.77:60270) 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.482+0000 7f2f31d68640 1 -- 172.21.15.77:0/3354376056 learned_addr learned my addr 172.21.15.77:0/3354376056 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f32d6a640 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f32569640 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc576f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f31d68640 1 -- 172.21.15.77:0/3354376056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 msgr2=0x562dc576f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f31d68640 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc576f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f31d68640 1 -- 172.21.15.77:0/3354376056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 msgr2=0x562dc576c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f31d68640 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f31d68640 1 -- 172.21.15.77:0/3354376056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dc576c6e0 con 0x562dc575e240 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f31d68640 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc576bb30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c013010 tx=0x7f2f1c005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f291c080b702c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f15ffb640 1 -- 172.21.15.77:0/3354376056 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c045060 con 0x562dc575e240 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f15ffb640 1 -- 172.21.15.77:0/3354376056 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f2f1c0055d0 con 0x562dc575e240 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f373bb740 1 -- 172.21.15.77:0/3354376056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f1c01cbb0 con 0x562dc575e240 2024-03-16T10:22:10.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f15ffb640 1 -- 172.21.15.77:0/3354376056 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2f1c004730 con 0x562dc575e240 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f373bb740 10 client.16899 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.483+0000 7f2f373bb740 1 -- 172.21.15.77:0/3354376056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x562dc57d9080 con 0x562dc575e240 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.484+0000 7f2f15ffb640 1 -- 172.21.15.77:0/3354376056 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f1c01cbb0 con 0x562dc575e240 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.484+0000 7f2f15ffb640 1 -- 172.21.15.77:0/3354376056 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f2f1c024040 con 0x562dc575e240 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.484+0000 7f2f373bb740 10 client.16899 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.484+0000 7f2f373bb740 10 client.16899 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.484+0000 7f2f373bb740 20 client.16899 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.484+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x562dc57d99b0 0x562dc57dbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.484+0000 7f2f373bb740 4 client.16899 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.484+0000 7f2f373bb740 1 -- 172.21.15.77:0/3354376056 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562dc56c9d20 con 0x562dc57d99b0 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.484+0000 7f2f32569640 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x562dc57d99b0 0x562dc57dbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f32569640 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x562dc57d99b0 0x562dc57dbdd0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18d1d3c58c05a2b3 server_cookie=9a870c4ec65865e3 in_seq=0 out_seq=0 2024-03-16T10:22:10.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f15ffb640 10 client.16899 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f15ffb640 1 -- 172.21.15.77:0/3354376056 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562dc56c9d20 con 0x562dc57d99b0 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f15ffb640 10 client.16899 handle_command_reply: tid=0 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f373bb740 1 client.16899 shutdown 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f373bb740 1 -- 172.21.15.77:0/3354376056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 msgr2=0x562dc576bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc576bb30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c013010 tx=0x7f2f1c005250 comp rx=0 tx=0).stop 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f373bb740 1 -- 172.21.15.77:0/3354376056 shutdown_connections 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562dc57633b0 0x562dc576c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x562dc57d99b0 0x562dc57dbdd0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dc575e240 0x562dc576bb30 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f373bb740 1 --2- 172.21.15.77:0/3354376056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc575d850 0x562dc576f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f373bb740 1 -- 172.21.15.77:0/3354376056 >> 172.21.15.77:0/3354376056 conn(0x562dc5758dc0 msgr2=0x562dc5767ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f15ffb640 0 client.16899 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f373bb740 1 -- 172.21.15.77:0/3354376056 shutdown_connections 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f373bb740 1 -- 172.21.15.77:0/3354376056 wait complete. 2024-03-16T10:22:10.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.485+0000 7f2f373bb740 20 client.16899 trim_cache size 0 max 16384 2024-03-16T10:22:10.492 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:22:10.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.486+0000 7f2f347cd640 1 -- 172.21.15.77:0/56717897 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2f180a2800 msgr2=0x7f2f180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.486+0000 7f2f347cd640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2f180a2800 0x7f2f180a4cc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c197ed0 tx=0x7f2f1c005f90 comp rx=0 tx=0).stop 2024-03-16T10:22:10.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.486+0000 7f2f347cd640 1 -- 172.21.15.77:0/56717897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f2c1006f0 msgr2=0x7f2f2c1969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:10.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.486+0000 7f2f347cd640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f2c1006f0 0x7f2f2c1969c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800e9c0 tx=0x7f2f2800ee80 comp rx=0 tx=0).stop 2024-03-16T10:22:10.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.486+0000 7f2f347cd640 1 -- 172.21.15.77:0/56717897 shutdown_connections 2024-03-16T10:22:10.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.486+0000 7f2f347cd640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f2c1022a0 0x7f2f2c19df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.486+0000 7f2f347cd640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f2c1018e0 0x7f2f2c196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.486+0000 7f2f347cd640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f2f180a2800 0x7f2f180a4cc0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.486+0000 7f2f347cd640 1 --2- 172.21.15.77:0/56717897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f2c1006f0 0x7f2f2c1969c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:10.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.486+0000 7f2f347cd640 1 -- 172.21.15.77:0/56717897 >> 172.21.15.77:0/56717897 conn(0x7f2f2c0fbc80 msgr2=0x7f2f2c104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:10.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.486+0000 7f2f347cd640 1 -- 172.21.15.77:0/56717897 shutdown_connections 2024-03-16T10:22:10.493 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:10.486+0000 7f2f347cd640 1 -- 172.21.15.77:0/56717897 wait complete. 2024-03-16T10:22:11.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:10 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:22:11.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:10 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:22:11.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:10 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:22:11.499 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T10:22:11.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:22:11.500 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T10:22:11.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.611+0000 7fb50b11d640 1 -- 172.21.15.77:0/332458237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb504100700 msgr2=0x7fb504100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.611+0000 7fb50b11d640 1 --2- 172.21.15.77:0/332458237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb504100700 0x7fb504100b20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009900 tx=0x7fb4f802bc30 comp rx=0 tx=0).stop 2024-03-16T10:22:11.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.611+0000 7fb50b11d640 1 -- 172.21.15.77:0/332458237 shutdown_connections 2024-03-16T10:22:11.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.611+0000 7fb50b11d640 1 --2- 172.21.15.77:0/332458237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5041022b0 0x7fb504108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.611+0000 7fb50b11d640 1 --2- 172.21.15.77:0/332458237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5041018f0 0x7fb504101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.611+0000 7fb50b11d640 1 --2- 172.21.15.77:0/332458237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb504100700 0x7fb504100b20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.611+0000 7fb50b11d640 1 -- 172.21.15.77:0/332458237 >> 172.21.15.77:0/332458237 conn(0x7fb5040fbc90 msgr2=0x7fb5040fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:11.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.611+0000 7fb50b11d640 1 -- 172.21.15.77:0/332458237 shutdown_connections 2024-03-16T10:22:11.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.611+0000 7fb50b11d640 1 -- 172.21.15.77:0/332458237 wait complete. 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb50b11d640 1 Processor -- start 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb50b11d640 1 -- start start 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb50b11d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb504100700 0x7fb50410b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb50b11d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5041018f0 0x7fb5041096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb50b11d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5041022b0 0x7fb504109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb50b11d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb50410d460 con 0x7fb5041018f0 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb50b11d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb50410d2e0 con 0x7fb504100700 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb50b11d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb50410d5e0 con 0x7fb5041022b0 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb508eb9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb504100700 0x7fb50410b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb503fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5041018f0 0x7fb5041096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb5096ba640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5041022b0 0x7fb504109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb508eb9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb504100700 0x7fb50410b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:52216/0 (socket says 172.21.15.77:52216) 2024-03-16T10:22:11.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.612+0000 7fb508eb9640 1 -- 172.21.15.77:0/3712407587 learned_addr learned my addr 172.21.15.77:0/3712407587 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:11.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.613+0000 7fb5096ba640 1 -- 172.21.15.77:0/3712407587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb504100700 msgr2=0x7fb50410b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.613+0000 7fb5096ba640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb504100700 0x7fb50410b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.613+0000 7fb5096ba640 1 -- 172.21.15.77:0/3712407587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5041018f0 msgr2=0x7fb5041096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.613+0000 7fb5096ba640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5041018f0 0x7fb5041096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.613+0000 7fb5096ba640 1 -- 172.21.15.77:0/3712407587 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50410a490 con 0x7fb5041022b0 2024-03-16T10:22:11.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.613+0000 7fb508eb9640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb504100700 0x7fb50410b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:11.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.613+0000 7fb5096ba640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5041022b0 0x7fb504109c00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc00ec50 tx=0x7fb4fc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:11.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.613+0000 7fb501ffb640 1 -- 172.21.15.77:0/3712407587 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc003b70 con 0x7fb5041022b0 2024-03-16T10:22:11.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.613+0000 7fb50b11d640 1 -- 172.21.15.77:0/3712407587 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5041a1cb0 con 0x7fb5041022b0 2024-03-16T10:22:11.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.613+0000 7fb501ffb640 1 -- 172.21.15.77:0/3712407587 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4fc003d10 con 0x7fb5041022b0 2024-03-16T10:22:11.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.613+0000 7fb501ffb640 1 -- 172.21.15.77:0/3712407587 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc010890 con 0x7fb5041022b0 2024-03-16T10:22:11.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.613+0000 7fb50b11d640 1 -- 172.21.15.77:0/3712407587 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5041a21b0 con 0x7fb5041022b0 2024-03-16T10:22:11.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.615+0000 7fb501ffb640 1 -- 172.21.15.77:0/3712407587 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fb4fc003fc0 con 0x7fb5041022b0 2024-03-16T10:22:11.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb50dde0740 1 Processor -- start 2024-03-16T10:22:11.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb50dde0740 1 -- start start 2024-03-16T10:22:11.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb50dde0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 0x555586666db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb50dde0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555586667380 0x55558666be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb50dde0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55558666c4f0 0x55558666e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb50dde0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5555865cb0d0 con 0x555586666990 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb50dde0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x555586573c70 con 0x555586667380 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb50dde0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5555865c7170 con 0x55558666c4f0 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb501ffb640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4f40a2880 0x7fb4f40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb503fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 0x555586666db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb503fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 0x555586666db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57978/0 (socket says 172.21.15.77:57978) 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb503fff640 1 -- 172.21.15.77:0/2522530962 learned_addr learned my addr 172.21.15.77:0/2522530962 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb508eb9640 1 --2- 172.21.15.77:0/2522530962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555586667380 0x55558666be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb503fff640 1 --2- 172.21.15.77:0/2522530962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55558666c4f0 0x55558666e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb5096ba640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4f40a2880 0x7fb4f40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb501ffb640 1 -- 172.21.15.77:0/3712407587 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb4fc0cf060 con 0x7fb5041022b0 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb503fff640 1 -- 172.21.15.77:0/2522530962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55558666c4f0 msgr2=0x55558666e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb503fff640 1 --2- 172.21.15.77:0/2522530962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55558666c4f0 0x55558666e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb503fff640 1 -- 172.21.15.77:0/2522530962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555586667380 msgr2=0x55558666be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb503fff640 1 --2- 172.21.15.77:0/2522530962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555586667380 0x55558666be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb503fff640 1 -- 172.21.15.77:0/2522530962 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555863a77e0 con 0x555586666990 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.616+0000 7fb508eb9640 1 --2- 172.21.15.77:0/2522530962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555586667380 0x55558666be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.617+0000 7fb503fff640 1 --2- 172.21.15.77:0/2522530962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 0x555586666db0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009fd0 tx=0x7fb4f0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4bfdc4a4d69e75 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.617+0000 7fb5096ba640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4f40a2880 0x7fb4f40a4d40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc09b980 tx=0x7fb4fc01d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.617+0000 7fb4eeffd640 1 -- 172.21.15.77:0/2522530962 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0013d50 con 0x555586666990 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.617+0000 7fb4eeffd640 1 -- 172.21.15.77:0/2522530962 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f0013130 con 0x555586666990 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.617+0000 7fb4eeffd640 1 -- 172.21.15.77:0/2522530962 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f00047a0 con 0x555586666990 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.617+0000 7fb50dde0740 1 -- 172.21.15.77:0/2522530962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 msgr2=0x555586666db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.617+0000 7fb50dde0740 1 --2- 172.21.15.77:0/2522530962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 0x555586666db0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009fd0 tx=0x7fb4f0005850 comp rx=0 tx=0).stop 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.617+0000 7fb50dde0740 1 -- 172.21.15.77:0/2522530962 shutdown_connections 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.617+0000 7fb50dde0740 1 --2- 172.21.15.77:0/2522530962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55558666c4f0 0x55558666e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.617+0000 7fb50dde0740 1 --2- 172.21.15.77:0/2522530962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555586667380 0x55558666be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.617+0000 7fb50dde0740 1 --2- 172.21.15.77:0/2522530962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 0x555586666db0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.617+0000 7fb50dde0740 1 -- 172.21.15.77:0/2522530962 >> 172.21.15.77:0/2522530962 conn(0x55558662ec50 msgr2=0x555586662300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.618+0000 7fb50dde0740 1 -- 172.21.15.77:0/2522530962 shutdown_connections 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.618+0000 7fb50dde0740 1 -- 172.21.15.77:0/2522530962 wait complete. 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.618+0000 7fb50dde0740 1 -- start start 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.618+0000 7fb50dde0740 1 Processor -- start 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.618+0000 7fb50dde0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 0x555586670df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.618+0000 7fb50dde0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555586667380 0x555586671330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.618+0000 7fb50dde0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55558666c4f0 0x555586678370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.618+0000 7fb50dde0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55558666f440 con 0x555586666990 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb50dde0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5555865870d0 con 0x555586667380 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb50dde0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55558666f5c0 con 0x55558666c4f0 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb50dde0740 10 client.0 _ll_register_callbacks cb 0x555586550760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb503fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 0x555586670df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb503fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 0x555586670df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57992/0 (socket says 172.21.15.77:57992) 2024-03-16T10:22:11.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb503fff640 1 -- 172.21.15.77:0/3769532141 learned_addr learned my addr 172.21.15.77:0/3769532141 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb508eb9640 1 --2- 172.21.15.77:0/3769532141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555586667380 0x555586671330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb503fff640 1 -- 172.21.15.77:0/3769532141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55558666c4f0 msgr2=0x555586678370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb503fff640 1 --2- 172.21.15.77:0/3769532141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55558666c4f0 0x555586678370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb503fff640 1 -- 172.21.15.77:0/3769532141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555586667380 msgr2=0x555586671330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb503fff640 1 --2- 172.21.15.77:0/3769532141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555586667380 0x555586671330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb503fff640 1 -- 172.21.15.77:0/3769532141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555586407f70 con 0x555586666990 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb503fff640 1 --2- 172.21.15.77:0/3769532141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 0x555586670df0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0006010 tx=0x7fb4f00050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce4d0d27103876b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb4e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb4e3fff640 1 -- 172.21.15.77:0/3769532141 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0023070 con 0x555586666990 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb4e3fff640 1 -- 172.21.15.77:0/3769532141 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb4f0004320 con 0x555586666990 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb50dde0740 1 -- 172.21.15.77:0/3769532141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4f0013130 con 0x555586666990 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb4e3fff640 1 -- 172.21.15.77:0/3769532141 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb4f002f050 con 0x555586666990 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb50dde0740 10 client.16911 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.619+0000 7fb50dde0740 1 -- 172.21.15.77:0/3769532141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55558629f170 con 0x555586666990 2024-03-16T10:22:11.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.620+0000 7fb4e3fff640 1 -- 172.21.15.77:0/3769532141 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f0013130 con 0x555586666990 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.620+0000 7fb4e3fff640 1 -- 172.21.15.77:0/3769532141 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fb4f0004a50 con 0x555586666990 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.620+0000 7fb50dde0740 10 client.16911 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.620+0000 7fb50dde0740 10 client.16911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.620+0000 7fb50dde0740 20 client.16911 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.620+0000 7fb50dde0740 1 --2- 172.21.15.77:0/3769532141 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55558667c5a0 0x55558667e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.620+0000 7fb50dde0740 4 client.16911 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.620+0000 7fb50dde0740 1 -- 172.21.15.77:0/3769532141 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555865d8610 con 0x55558667c5a0 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.620+0000 7fb508eb9640 1 --2- 172.21.15.77:0/3769532141 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55558667c5a0 0x55558667e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.620+0000 7fb4e3fff640 10 client.16911 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.620+0000 7fb508eb9640 1 --2- 172.21.15.77:0/3769532141 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55558667c5a0 0x55558667e9e0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bca2be47ce6ca77f server_cookie=ac7d66e707eab1e5 in_seq=0 out_seq=0 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb4e3fff640 1 -- 172.21.15.77:0/3769532141 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5555865d8610 con 0x55558667c5a0 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb4e3fff640 10 client.16911 handle_command_reply: tid=0 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb50dde0740 1 client.16911 shutdown 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb50dde0740 1 -- 172.21.15.77:0/3769532141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 msgr2=0x555586670df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb50dde0740 1 --2- 172.21.15.77:0/3769532141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 0x555586670df0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0006010 tx=0x7fb4f00050d0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb50dde0740 1 -- 172.21.15.77:0/3769532141 shutdown_connections 2024-03-16T10:22:11.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb50dde0740 1 --2- 172.21.15.77:0/3769532141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55558666c4f0 0x555586678370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb50dde0740 1 --2- 172.21.15.77:0/3769532141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555586667380 0x555586671330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb50dde0740 1 --2- 172.21.15.77:0/3769532141 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55558667c5a0 0x55558667e9e0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb50dde0740 1 --2- 172.21.15.77:0/3769532141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586666990 0x555586670df0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb4e3fff640 0 client.16911 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:11.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb50dde0740 1 -- 172.21.15.77:0/3769532141 >> 172.21.15.77:0/3769532141 conn(0x55558662ec50 msgr2=0x5555866614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:11.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb50dde0740 1 -- 172.21.15.77:0/3769532141 shutdown_connections 2024-03-16T10:22:11.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.621+0000 7fb50dde0740 1 -- 172.21.15.77:0/3769532141 wait complete. 2024-03-16T10:22:11.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.622+0000 7fb50dde0740 20 client.16911 trim_cache size 0 max 16384 2024-03-16T10:22:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb50dde0740 1 Processor -- start 2024-03-16T10:22:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb50dde0740 1 -- start start 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb50dde0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555586667380 0x555586665c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb50dde0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55558666c4f0 0x555586678330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb50dde0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5555866842b0 0x555586670a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb50dde0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb5041aae20 con 0x5555866842b0 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb50dde0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb5041ac9f0 con 0x55558666c4f0 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb50dde0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5041ab4f0 con 0x555586667380 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb503fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5555866842b0 0x555586670a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb503fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5555866842b0 0x555586670a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58004/0 (socket says 172.21.15.77:58004) 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb503fff640 1 -- 172.21.15.77:0/3464277415 learned_addr learned my addr 172.21.15.77:0/3464277415 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb508eb9640 1 --2- 172.21.15.77:0/3464277415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55558666c4f0 0x555586678330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb503fff640 1 --2- 172.21.15.77:0/3464277415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555586667380 0x555586665c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb503fff640 1 -- 172.21.15.77:0/3464277415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555586667380 msgr2=0x555586665c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb503fff640 1 --2- 172.21.15.77:0/3464277415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555586667380 0x555586665c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb503fff640 1 -- 172.21.15.77:0/3464277415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55558666c4f0 msgr2=0x555586678330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb503fff640 1 --2- 172.21.15.77:0/3464277415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55558666c4f0 0x555586678330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.640+0000 7fb503fff640 1 -- 172.21.15.77:0/3464277415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555586666190 con 0x5555866842b0 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.641+0000 7fb503fff640 1 --2- 172.21.15.77:0/3464277415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5555866842b0 0x555586670a00 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00059f0 tx=0x7fb4f0005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a8f809afc79765 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.641+0000 7fb4ee7fc640 1 -- 172.21.15.77:0/3464277415 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0023040 con 0x5555866842b0 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.641+0000 7fb4ee7fc640 1 -- 172.21.15.77:0/3464277415 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f0009aa0 con 0x5555866842b0 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.641+0000 7fb4ee7fc640 1 -- 172.21.15.77:0/3464277415 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0003a30 con 0x5555866842b0 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.641+0000 7fb50dde0740 1 -- 172.21.15.77:0/3464277415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5555866842b0 msgr2=0x555586670a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.641+0000 7fb50dde0740 1 --2- 172.21.15.77:0/3464277415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5555866842b0 0x555586670a00 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00059f0 tx=0x7fb4f0005730 comp rx=0 tx=0).stop 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.641+0000 7fb50dde0740 1 -- 172.21.15.77:0/3464277415 shutdown_connections 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.641+0000 7fb50dde0740 1 --2- 172.21.15.77:0/3464277415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5555866842b0 0x555586670a00 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.641+0000 7fb50dde0740 1 --2- 172.21.15.77:0/3464277415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55558666c4f0 0x555586678330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.641+0000 7fb50dde0740 1 --2- 172.21.15.77:0/3464277415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555586667380 0x555586665c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.641+0000 7fb50dde0740 1 -- 172.21.15.77:0/3464277415 >> 172.21.15.77:0/3464277415 conn(0x55558662ec50 msgr2=0x555586665540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb50dde0740 1 -- 172.21.15.77:0/3464277415 shutdown_connections 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb50dde0740 1 -- 172.21.15.77:0/3464277415 wait complete. 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb50dde0740 1 -- start start 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb50dde0740 1 Processor -- start 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb50dde0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586667380 0x555586676090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb50dde0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55558666c4f0 0x5555866765d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb50dde0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555866842b0 0x5555866775e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb50dde0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb5041ac320 con 0x555586667380 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb50dde0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb5041abc50 con 0x55558666c4f0 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb50dde0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5041a8e20 con 0x5555866842b0 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb50dde0740 10 client.0 _ll_register_callbacks cb 0x7fb4f0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb503fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586667380 0x555586676090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb503fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586667380 0x555586676090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58018/0 (socket says 172.21.15.77:58018) 2024-03-16T10:22:11.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb503fff640 1 -- 172.21.15.77:0/48089225 learned_addr learned my addr 172.21.15.77:0/48089225 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb508eb9640 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55558666c4f0 0x5555866765d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.642+0000 7fb503fff640 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555866842b0 0x5555866775e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb503fff640 1 -- 172.21.15.77:0/48089225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555866842b0 msgr2=0x5555866775e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb503fff640 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555866842b0 0x5555866775e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb503fff640 1 -- 172.21.15.77:0/48089225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55558666c4f0 msgr2=0x5555866765d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb503fff640 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55558666c4f0 0x5555866765d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb503fff640 1 -- 172.21.15.77:0/48089225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555586677bd0 con 0x555586667380 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb503fff640 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586667380 0x555586676090 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009980 tx=0x7fb4f0013130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872059e49d5c3fb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb4e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb4e3fff640 1 -- 172.21.15.77:0/48089225 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0023040 con 0x555586667380 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb4e3fff640 1 -- 172.21.15.77:0/48089225 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fb4f0004fe0 con 0x555586667380 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb50dde0740 1 -- 172.21.15.77:0/48089225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4f0009aa0 con 0x555586667380 2024-03-16T10:22:11.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb4e3fff640 1 -- 172.21.15.77:0/48089225 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb4f00051e0 con 0x555586667380 2024-03-16T10:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb50dde0740 10 client.16917 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb50dde0740 1 -- 172.21.15.77:0/48089225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5555866e2ee0 con 0x555586667380 2024-03-16T10:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb4e3fff640 1 -- 172.21.15.77:0/48089225 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f0009aa0 con 0x555586667380 2024-03-16T10:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb4e3fff640 1 -- 172.21.15.77:0/48089225 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fb4f0004440 con 0x555586667380 2024-03-16T10:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb50dde0740 10 client.16917 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb50dde0740 10 client.16917 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb50dde0740 20 client.16917 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb50dde0740 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5555866e37d0 0x5555866e5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb50dde0740 4 client.16917 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T10:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.643+0000 7fb50dde0740 1 -- 172.21.15.77:0/48089225 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5555865d2960 con 0x5555866e37d0 2024-03-16T10:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.644+0000 7fb508eb9640 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5555866e37d0 0x5555866e5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.644+0000 7fb4e3fff640 10 client.16917 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:11.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.644+0000 7fb508eb9640 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5555866e37d0 0x5555866e5bf0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9cf7c56c29cefe0 server_cookie=abc4b2954e409b73 in_seq=0 out_seq=0 2024-03-16T10:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.645+0000 7fb4e3fff640 1 -- 172.21.15.77:0/48089225 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5555865d2960 con 0x5555866e37d0 2024-03-16T10:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.645+0000 7fb4e3fff640 10 client.16917 handle_command_reply: tid=0 2024-03-16T10:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.645+0000 7fb50dde0740 1 client.16917 shutdown 2024-03-16T10:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.645+0000 7fb50dde0740 1 -- 172.21.15.77:0/48089225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586667380 msgr2=0x555586676090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.645+0000 7fb50dde0740 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586667380 0x555586676090 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009980 tx=0x7fb4f0013130 comp rx=0 tx=0).stop 2024-03-16T10:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.645+0000 7fb50dde0740 1 -- 172.21.15.77:0/48089225 shutdown_connections 2024-03-16T10:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.645+0000 7fb50dde0740 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555866842b0 0x5555866775e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.645+0000 7fb50dde0740 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55558666c4f0 0x5555866765d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.645+0000 7fb50dde0740 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5555866e37d0 0x5555866e5bf0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.645+0000 7fb4e3fff640 0 client.16917 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:11.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.645+0000 7fb50dde0740 1 --2- 172.21.15.77:0/48089225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555586667380 0x555586676090 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.645+0000 7fb50dde0740 1 -- 172.21.15.77:0/48089225 >> 172.21.15.77:0/48089225 conn(0x55558662ec50 msgr2=0x5555866832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:11.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50dde0740 1 -- 172.21.15.77:0/48089225 shutdown_connections 2024-03-16T10:22:11.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50dde0740 1 -- 172.21.15.77:0/48089225 wait complete. 2024-03-16T10:22:11.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50dde0740 20 client.16917 trim_cache size 0 max 16384 2024-03-16T10:22:11.653 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:22:11.653 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T10:22:11.653 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "1f1adeef-2126-4ee3-aca7-819aadf3baf6", 2024-03-16T10:22:11.653 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T10:22:11.653 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:22:11.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50b11d640 1 -- 172.21.15.77:0/3712407587 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4f40a2880 msgr2=0x7fb4f40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50b11d640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4f40a2880 0x7fb4f40a4d40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc09b980 tx=0x7fb4fc01d000 comp rx=0 tx=0).stop 2024-03-16T10:22:11.654 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50b11d640 1 -- 172.21.15.77:0/3712407587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5041022b0 msgr2=0x7fb504109c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50b11d640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5041022b0 0x7fb504109c00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc00ec50 tx=0x7fb4fc00c6a0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50b11d640 1 -- 172.21.15.77:0/3712407587 shutdown_connections 2024-03-16T10:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50b11d640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb4f40a2880 0x7fb4f40a4d40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50b11d640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5041022b0 0x7fb504109c00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50b11d640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5041018f0 0x7fb5041096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50b11d640 1 --2- 172.21.15.77:0/3712407587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb504100700 0x7fb50410b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50b11d640 1 -- 172.21.15.77:0/3712407587 >> 172.21.15.77:0/3712407587 conn(0x7fb5040fbc90 msgr2=0x7fb504104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50b11d640 1 -- 172.21.15.77:0/3712407587 shutdown_connections 2024-03-16T10:22:11.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.646+0000 7fb50b11d640 1 -- 172.21.15.77:0/3712407587 wait complete. 2024-03-16T10:22:11.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:22:11.659 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:22:11.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.774+0000 7f95d8737640 1 -- 172.21.15.77:0/3616941767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95d01018d0 msgr2=0x7f95d0101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.774+0000 7f95d8737640 1 --2- 172.21.15.77:0/3616941767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95d01018d0 0x7f95d0101d50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009900 tx=0x7f95c002bc30 comp rx=0 tx=0).stop 2024-03-16T10:22:11.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.774+0000 7f95d8737640 1 -- 172.21.15.77:0/3616941767 shutdown_connections 2024-03-16T10:22:11.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.774+0000 7f95d8737640 1 --2- 172.21.15.77:0/3616941767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d0102290 0x7f95d0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.774+0000 7f95d8737640 1 --2- 172.21.15.77:0/3616941767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95d01018d0 0x7f95d0101d50 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.774+0000 7f95d8737640 1 --2- 172.21.15.77:0/3616941767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95d01006e0 0x7f95d0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.774+0000 7f95d8737640 1 -- 172.21.15.77:0/3616941767 >> 172.21.15.77:0/3616941767 conn(0x7f95d00fbc70 msgr2=0x7f95d00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:11.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.774+0000 7f95d8737640 1 -- 172.21.15.77:0/3616941767 shutdown_connections 2024-03-16T10:22:11.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.774+0000 7f95d8737640 1 -- 172.21.15.77:0/3616941767 wait complete. 2024-03-16T10:22:11.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.775+0000 7f95d8737640 1 Processor -- start 2024-03-16T10:22:11.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.775+0000 7f95d8737640 1 -- start start 2024-03-16T10:22:11.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.775+0000 7f95d8737640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d01006e0 0x7f95d0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.775+0000 7f95d8737640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95d01018d0 0x7f95d0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.775+0000 7f95d8737640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95d0102290 0x7f95d019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.775+0000 7f95d8737640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f95d010c360 con 0x7f95d0102290 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.775+0000 7f95d8737640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f95d010c1e0 con 0x7f95d01006e0 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.775+0000 7f95d8737640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f95d010c4e0 con 0x7f95d01018d0 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.775+0000 7f95d6cd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95d0102290 0x7f95d019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.775+0000 7f95d64d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d01006e0 0x7f95d0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d5cd2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95d01018d0 0x7f95d0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d5cd2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95d01018d0 0x7f95d0198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:45488/0 (socket says 172.21.15.77:45488) 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d64d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d01006e0 0x7f95d0197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:52276/0 (socket says 172.21.15.77:52276) 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d5cd2640 1 -- 172.21.15.77:0/412932143 learned_addr learned my addr 172.21.15.77:0/412932143 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d6cd4640 1 -- 172.21.15.77:0/412932143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95d01018d0 msgr2=0x7f95d0198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d6cd4640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95d01018d0 0x7f95d0198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d6cd4640 1 -- 172.21.15.77:0/412932143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d01006e0 msgr2=0x7f95d0197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d6cd4640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d01006e0 0x7f95d0197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d6cd4640 1 -- 172.21.15.77:0/412932143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d019f750 con 0x7f95d0102290 2024-03-16T10:22:11.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d5cd2640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95d01018d0 0x7f95d0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:11.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d6cd4640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95d0102290 0x7f95d019f0e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009fd0 tx=0x7f95c800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:11.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95c77fe640 1 -- 172.21.15.77:0/412932143 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c800ce20 con 0x7f95d0102290 2024-03-16T10:22:11.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d8737640 1 -- 172.21.15.77:0/412932143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95d019fa40 con 0x7f95d0102290 2024-03-16T10:22:11.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95c77fe640 1 -- 172.21.15.77:0/412932143 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95c8005020 con 0x7f95d0102290 2024-03-16T10:22:11.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95d8737640 1 -- 172.21.15.77:0/412932143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95d019ff60 con 0x7f95d0102290 2024-03-16T10:22:11.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.776+0000 7f95c77fe640 1 -- 172.21.15.77:0/412932143 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c8010720 con 0x7f95d0102290 2024-03-16T10:22:11.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.779+0000 7f95c77fe640 1 -- 172.21.15.77:0/412932143 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f95c8005540 con 0x7f95d0102290 2024-03-16T10:22:11.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.780+0000 7f95db3fb740 1 Processor -- start 2024-03-16T10:22:11.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.780+0000 7f95db3fb740 1 -- start start 2024-03-16T10:22:11.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.780+0000 7f95c77fe640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f95bc0a2800 0x7f95bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95db3fb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6d6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95db3fb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6b60 0x564d5f6db660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95c77fe640 1 -- 172.21.15.77:0/412932143 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f95c80cf080 con 0x7f95d0102290 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95db3fb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6de160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95db3fb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564d5f63b500 con 0x564d5f6d6170 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95d64d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6d6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95d64d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6d6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58060/0 (socket says 172.21.15.77:58060) 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95d64d3640 1 -- 172.21.15.77:0/843779824 learned_addr learned my addr 172.21.15.77:0/843779824 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95d6cd4640 1 --2- 172.21.15.77:0/843779824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6de160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95db3fb740 1 -- 172.21.15.77:0/843779824 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564d5f5e3be0 con 0x564d5f6d6b60 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95db3fb740 1 -- 172.21.15.77:0/843779824 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564d5f6375a0 con 0x564d5f6dbcd0 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95d5cd2640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f95bc0a2800 0x7f95bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95d6cd4640 1 -- 172.21.15.77:0/843779824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6b60 msgr2=0x564d5f6db660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95d6cd4640 1 --2- 172.21.15.77:0/843779824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6b60 0x564d5f6db660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95d6cd4640 1 -- 172.21.15.77:0/843779824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 msgr2=0x564d5f6d6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95d6cd4640 1 --2- 172.21.15.77:0/843779824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6d6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95d6cd4640 1 -- 172.21.15.77:0/843779824 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d5f5f74b0 con 0x564d5f6dbcd0 2024-03-16T10:22:11.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.781+0000 7f95d64d3640 1 --2- 172.21.15.77:0/843779824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6d6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:11.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.782+0000 7f95d6cd4640 1 --2- 172.21.15.77:0/843779824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6de160 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f95c80ccc60 tx=0x7f95c80054b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eb221dbe6d3bb82 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:11.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.782+0000 7f95c4ff9640 1 -- 172.21.15.77:0/843779824 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c8099b50 con 0x564d5f6dbcd0 2024-03-16T10:22:11.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.782+0000 7f95d5cd2640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f95bc0a2800 0x7f95bc0a4cc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f95c0000c00 tx=0x7f95c002c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:11.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.782+0000 7f95c4ff9640 1 -- 172.21.15.77:0/843779824 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95c8099db0 con 0x564d5f6dbcd0 2024-03-16T10:22:11.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.782+0000 7f95c4ff9640 1 -- 172.21.15.77:0/843779824 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c80a3dc0 con 0x564d5f6dbcd0 2024-03-16T10:22:11.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.782+0000 7f95db3fb740 1 -- 172.21.15.77:0/843779824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 msgr2=0x564d5f6de160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.782+0000 7f95db3fb740 1 --2- 172.21.15.77:0/843779824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6de160 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f95c80ccc60 tx=0x7f95c80054b0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 -- 172.21.15.77:0/843779824 shutdown_connections 2024-03-16T10:22:11.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 --2- 172.21.15.77:0/843779824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6de160 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 --2- 172.21.15.77:0/843779824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6b60 0x564d5f6db660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 --2- 172.21.15.77:0/843779824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6d6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 -- 172.21.15.77:0/843779824 >> 172.21.15.77:0/843779824 conn(0x564d5f6d16e0 msgr2=0x564d5f6d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 -- 172.21.15.77:0/843779824 shutdown_connections 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 -- 172.21.15.77:0/843779824 wait complete. 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 -- start start 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 Processor -- start 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6e0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6d6b60 0x564d5f6e0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6dbcd0 0x564d5f6e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564d5f6dec80 con 0x564d5f6d6170 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564d5f5f7160 con 0x564d5f6dbcd0 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564d5f6dee00 con 0x564d5f6d6b60 2024-03-16T10:22:11.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95db3fb740 10 client.0 _ll_register_callbacks cb 0x564d5f5db150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:11.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95d64d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6e0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95d5cd2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6d6b60 0x564d5f6e0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95d64d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6e0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58072/0 (socket says 172.21.15.77:58072) 2024-03-16T10:22:11.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95d64d3640 1 -- 172.21.15.77:0/3197461394 learned_addr learned my addr 172.21.15.77:0/3197461394 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:11.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.783+0000 7f95d6cd4640 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6dbcd0 0x564d5f6e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95d5cd2640 1 -- 172.21.15.77:0/3197461394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6dbcd0 msgr2=0x564d5f6e7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95d5cd2640 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6dbcd0 0x564d5f6e7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95d5cd2640 1 -- 172.21.15.77:0/3197461394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 msgr2=0x564d5f6e0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95d5cd2640 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6e0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95d5cd2640 1 -- 172.21.15.77:0/3197461394 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d5f4197d0 con 0x564d5f6d6b60 2024-03-16T10:22:11.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95d64d3640 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6e0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:11.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95d5cd2640 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6d6b60 0x564d5f6e0c50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f95c002c7b0 tx=0x7f95c002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=223822111da93250 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:11.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:22:11.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95b9ffb640 1 -- 172.21.15.77:0/3197461394 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c004f070 con 0x564d5f6d6b60 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95db3fb740 1 -- 172.21.15.77:0/3197461394 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95c8099db0 con 0x564d5f6d6b60 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95b9ffb640 1 -- 172.21.15.77:0/3197461394 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f95c0005300 con 0x564d5f6d6b60 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95b9ffb640 1 -- 172.21.15.77:0/3197461394 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f95c005b050 con 0x564d5f6d6b60 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95db3fb740 10 client.26695 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.784+0000 7f95db3fb740 1 -- 172.21.15.77:0/3197461394 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564d5f34f3f0 con 0x564d5f6d6b60 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.785+0000 7f95b9ffb640 1 -- 172.21.15.77:0/3197461394 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95c8099db0 con 0x564d5f6d6b60 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.785+0000 7f95b9ffb640 1 -- 172.21.15.77:0/3197461394 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f95c004b450 con 0x564d5f6d6b60 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.785+0000 7f95db3fb740 10 client.26695 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.785+0000 7f95db3fb740 10 client.26695 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.785+0000 7f95db3fb740 20 client.26695 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.785+0000 7f95db3fb740 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d5f6ec130 0x564d5f6ee520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.785+0000 7f95db3fb740 4 client.26695 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:11.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.785+0000 7f95db3fb740 1 -- 172.21.15.77:0/3197461394 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d5f648a40 con 0x564d5f6ec130 2024-03-16T10:22:11.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.785+0000 7f95d64d3640 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d5f6ec130 0x564d5f6ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.785+0000 7f95d64d3640 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d5f6ec130 0x564d5f6ee520 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8a99314a74c1446 server_cookie=845a471d747efa55 in_seq=0 out_seq=0 2024-03-16T10:22:11.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.785+0000 7f95b9ffb640 10 client.26695 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:11.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.786+0000 7f95b9ffb640 1 -- 172.21.15.77:0/3197461394 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564d5f648a40 con 0x564d5f6ec130 2024-03-16T10:22:11.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.786+0000 7f95b9ffb640 10 client.26695 handle_command_reply: tid=0 2024-03-16T10:22:11.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.786+0000 7f95db3fb740 1 client.26695 shutdown 2024-03-16T10:22:11.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.787+0000 7f95db3fb740 1 -- 172.21.15.77:0/3197461394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6d6b60 msgr2=0x564d5f6e0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.787+0000 7f95db3fb740 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6d6b60 0x564d5f6e0c50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f95c002c7b0 tx=0x7f95c002bc30 comp rx=0 tx=0).stop 2024-03-16T10:22:11.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.787+0000 7f95db3fb740 1 -- 172.21.15.77:0/3197461394 shutdown_connections 2024-03-16T10:22:11.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.787+0000 7f95db3fb740 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6dbcd0 0x564d5f6e7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.787+0000 7f95db3fb740 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6d6b60 0x564d5f6e0c50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.787+0000 7f95db3fb740 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d5f6ec130 0x564d5f6ee520 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.787+0000 7f95b9ffb640 0 client.26695 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:11.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.787+0000 7f95db3fb740 1 --2- 172.21.15.77:0/3197461394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6e0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.787+0000 7f95db3fb740 1 -- 172.21.15.77:0/3197461394 >> 172.21.15.77:0/3197461394 conn(0x564d5f6d16e0 msgr2=0x564d5f6d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:11.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.787+0000 7f95db3fb740 1 -- 172.21.15.77:0/3197461394 shutdown_connections 2024-03-16T10:22:11.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.787+0000 7f95db3fb740 1 -- 172.21.15.77:0/3197461394 wait complete. 2024-03-16T10:22:11.794 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.787+0000 7f95db3fb740 20 client.26695 trim_cache size 0 max 16384 2024-03-16T10:22:11.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.804+0000 7f95db3fb740 1 Processor -- start 2024-03-16T10:22:11.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.804+0000 7f95db3fb740 1 -- start start 2024-03-16T10:22:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.804+0000 7f95db3fb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.804+0000 7f95db3fb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6b60 0x564d5f6e7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95db3fb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6d8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95db3fb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f95d01aac90 con 0x564d5f6d6170 2024-03-16T10:22:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95db3fb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f95d01ac860 con 0x564d5f6d6b60 2024-03-16T10:22:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95db3fb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f95d01ab360 con 0x564d5f6dbcd0 2024-03-16T10:22:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d64d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d64d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6d80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58086/0 (socket says 172.21.15.77:58086) 2024-03-16T10:22:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d5cd2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6b60 0x564d5f6e7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d64d3640 1 -- 172.21.15.77:0/3286321777 learned_addr learned my addr 172.21.15.77:0/3286321777 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d6cd4640 1 --2- 172.21.15.77:0/3286321777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6d8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d64d3640 1 -- 172.21.15.77:0/3286321777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 msgr2=0x564d5f6d8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d64d3640 1 --2- 172.21.15.77:0/3286321777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6d8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d64d3640 1 -- 172.21.15.77:0/3286321777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6b60 msgr2=0x564d5f6e7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d64d3640 1 --2- 172.21.15.77:0/3286321777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6b60 0x564d5f6e7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d64d3640 1 -- 172.21.15.77:0/3286321777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d5f6ee890 con 0x564d5f6d6170 2024-03-16T10:22:11.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d5cd2640 1 --2- 172.21.15.77:0/3286321777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6b60 0x564d5f6e7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:11.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d6cd4640 1 --2- 172.21.15.77:0/3286321777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6d8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:22:11.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95d64d3640 1 --2- 172.21.15.77:0/3286321777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6d80c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f95cc012ae0 tx=0x7f95cc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ef4465fd2d6264 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:11.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.805+0000 7f95bbfff640 1 -- 172.21.15.77:0/3286321777 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc004050 con 0x564d5f6d6170 2024-03-16T10:22:11.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.806+0000 7f95bbfff640 1 -- 172.21.15.77:0/3286321777 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95cc00fda0 con 0x564d5f6d6170 2024-03-16T10:22:11.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.806+0000 7f95bbfff640 1 -- 172.21.15.77:0/3286321777 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc005440 con 0x564d5f6d6170 2024-03-16T10:22:11.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.806+0000 7f95db3fb740 1 -- 172.21.15.77:0/3286321777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 msgr2=0x564d5f6d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.806+0000 7f95db3fb740 1 --2- 172.21.15.77:0/3286321777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6d80c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f95cc012ae0 tx=0x7f95cc005f90 comp rx=0 tx=0).stop 2024-03-16T10:22:11.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.806+0000 7f95db3fb740 1 -- 172.21.15.77:0/3286321777 shutdown_connections 2024-03-16T10:22:11.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.806+0000 7f95db3fb740 1 --2- 172.21.15.77:0/3286321777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6d8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.806+0000 7f95db3fb740 1 --2- 172.21.15.77:0/3286321777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6b60 0x564d5f6e7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.806+0000 7f95db3fb740 1 --2- 172.21.15.77:0/3286321777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6170 0x564d5f6d80c0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.806+0000 7f95db3fb740 1 -- 172.21.15.77:0/3286321777 >> 172.21.15.77:0/3286321777 conn(0x564d5f6d16e0 msgr2=0x564d5f6d1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:11.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.806+0000 7f95db3fb740 1 -- 172.21.15.77:0/3286321777 shutdown_connections 2024-03-16T10:22:11.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.806+0000 7f95db3fb740 1 -- 172.21.15.77:0/3286321777 wait complete. 2024-03-16T10:22:11.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95db3fb740 1 -- start start 2024-03-16T10:22:11.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95db3fb740 1 Processor -- start 2024-03-16T10:22:11.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95db3fb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6170 0x564d5f6e51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95db3fb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6b60 0x564d5f6e5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95db3fb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6e6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95db3fb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f95d01ac190 con 0x564d5f6d6b60 2024-03-16T10:22:11.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95db3fb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f95d01abac0 con 0x564d5f6d6170 2024-03-16T10:22:11.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95db3fb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f95d01a8c70 con 0x564d5f6dbcd0 2024-03-16T10:22:11.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95db3fb740 10 client.0 _ll_register_callbacks cb 0x7f95c0042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:11.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95d5cd2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6b60 0x564d5f6e5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95d64d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6170 0x564d5f6e51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95d5cd2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6b60 0x564d5f6e5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58098/0 (socket says 172.21.15.77:58098) 2024-03-16T10:22:11.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95d5cd2640 1 -- 172.21.15.77:0/1316422945 learned_addr learned my addr 172.21.15.77:0/1316422945 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:11.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95d6cd4640 1 --2- 172.21.15.77:0/1316422945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6e6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95d5cd2640 1 -- 172.21.15.77:0/1316422945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 msgr2=0x564d5f6e6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95d5cd2640 1 --2- 172.21.15.77:0/1316422945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6e6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95d5cd2640 1 -- 172.21.15.77:0/1316422945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6170 msgr2=0x564d5f6e51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95d5cd2640 1 --2- 172.21.15.77:0/1316422945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6170 0x564d5f6e51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.807+0000 7f95d5cd2640 1 -- 172.21.15.77:0/1316422945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d5f30f0f0 con 0x564d5f6d6b60 2024-03-16T10:22:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95d5cd2640 1 --2- 172.21.15.77:0/1316422945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6b60 0x564d5f6e5700 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f95c002ce70 tx=0x7f95c0005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d46f2cc04c1551f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:22:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95b9ffb640 1 -- 172.21.15.77:0/1316422945 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0064060 con 0x564d5f6d6b60 2024-03-16T10:22:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95b9ffb640 1 -- 172.21.15.77:0/1316422945 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f95c0005300 con 0x564d5f6d6b60 2024-03-16T10:22:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95db3fb740 1 -- 172.21.15.77:0/1316422945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95cc00fda0 con 0x564d5f6d6b60 2024-03-16T10:22:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95b9ffb640 1 -- 172.21.15.77:0/1316422945 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f95c004f040 con 0x564d5f6d6b60 2024-03-16T10:22:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95db3fb740 10 client.16935 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95db3fb740 1 -- 172.21.15.77:0/1316422945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564d5f6029d0 con 0x564d5f6d6b60 2024-03-16T10:22:11.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95b9ffb640 1 -- 172.21.15.77:0/1316422945 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95cc00fda0 con 0x564d5f6d6b60 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95b9ffb640 1 -- 172.21.15.77:0/1316422945 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f95c0003bd0 con 0x564d5f6d6b60 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95db3fb740 10 client.16935 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95db3fb740 10 client.16935 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95db3fb740 20 client.16935 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95db3fb740 1 --2- 172.21.15.77:0/1316422945 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d5f6ec130 0x564d5f754940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95db3fb740 4 client.16935 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.808+0000 7f95db3fb740 1 -- 172.21.15.77:0/1316422945 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564d5f642d90 con 0x564d5f6ec130 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95d64d3640 1 --2- 172.21.15.77:0/1316422945 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d5f6ec130 0x564d5f754940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95d64d3640 1 --2- 172.21.15.77:0/1316422945 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d5f6ec130 0x564d5f754940 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98524fb62697d60b server_cookie=b4e9181892b84757 in_seq=0 out_seq=0 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95b9ffb640 10 client.16935 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95b9ffb640 1 -- 172.21.15.77:0/1316422945 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564d5f642d90 con 0x564d5f6ec130 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95b9ffb640 10 client.16935 handle_command_reply: tid=0 2024-03-16T10:22:11.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95db3fb740 1 client.16935 shutdown 2024-03-16T10:22:11.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95db3fb740 1 -- 172.21.15.77:0/1316422945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6b60 msgr2=0x564d5f6e5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95db3fb740 1 --2- 172.21.15.77:0/1316422945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6b60 0x564d5f6e5700 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f95c002ce70 tx=0x7f95c0005da0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95db3fb740 1 -- 172.21.15.77:0/1316422945 shutdown_connections 2024-03-16T10:22:11.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95db3fb740 1 --2- 172.21.15.77:0/1316422945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d5f6dbcd0 0x564d5f6e6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95db3fb740 1 --2- 172.21.15.77:0/1316422945 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d5f6ec130 0x564d5f754940 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95db3fb740 1 --2- 172.21.15.77:0/1316422945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d5f6d6b60 0x564d5f6e5700 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95db3fb740 1 --2- 172.21.15.77:0/1316422945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d5f6d6170 0x564d5f6e51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95db3fb740 1 -- 172.21.15.77:0/1316422945 >> 172.21.15.77:0/1316422945 conn(0x564d5f6d16e0 msgr2=0x564d5f6e06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:11.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95b9ffb640 0 client.16935 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:11.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95db3fb740 1 -- 172.21.15.77:0/1316422945 shutdown_connections 2024-03-16T10:22:11.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95db3fb740 1 -- 172.21.15.77:0/1316422945 wait complete. 2024-03-16T10:22:11.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.809+0000 7f95db3fb740 20 client.16935 trim_cache size 0 max 16384 2024-03-16T10:22:11.819 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:22:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:22:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:22:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: "1f1adeef-2126-4ee3-aca7-819aadf3baf6": { 2024-03-16T10:22:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:22:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "1f1adeef-2126-4ee3-aca7-819aadf3baf6", 2024-03-16T10:22:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:22:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:22:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:22:11.819 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:22:11.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.810+0000 7f95d8737640 1 -- 172.21.15.77:0/412932143 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f95bc0a2800 msgr2=0x7f95bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.810+0000 7f95d8737640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f95bc0a2800 0x7f95bc0a4cc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f95c0000c00 tx=0x7f95c002c880 comp rx=0 tx=0).stop 2024-03-16T10:22:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.810+0000 7f95d8737640 1 -- 172.21.15.77:0/412932143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95d0102290 msgr2=0x7f95d019f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.810+0000 7f95d8737640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95d0102290 0x7f95d019f0e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009fd0 tx=0x7f95c800ef10 comp rx=0 tx=0).stop 2024-03-16T10:22:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.810+0000 7f95d8737640 1 -- 172.21.15.77:0/412932143 shutdown_connections 2024-03-16T10:22:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.810+0000 7f95d8737640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95d0102290 0x7f95d019f0e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.810+0000 7f95d8737640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f95bc0a2800 0x7f95bc0a4cc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.810+0000 7f95d8737640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95d01018d0 0x7f95d0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.810+0000 7f95d8737640 1 --2- 172.21.15.77:0/412932143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d01006e0 0x7f95d0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:11.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.810+0000 7f95d8737640 1 -- 172.21.15.77:0/412932143 >> 172.21.15.77:0/412932143 conn(0x7f95d00fbc70 msgr2=0x7f95d00fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:11.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.810+0000 7f95d8737640 1 -- 172.21.15.77:0/412932143 shutdown_connections 2024-03-16T10:22:11.822 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:11.810+0000 7f95d8737640 1 -- 172.21.15.77:0/412932143 wait complete. 2024-03-16T10:22:11.824 INFO:tasks.cephfs.filesystem:scrub status for tag:1f1adeef-2126-4ee3-aca7-819aadf3baf6 - {'path': '/', 'tag': '1f1adeef-2126-4ee3-aca7-819aadf3baf6', 'options': 'recursive,force'} 2024-03-16T10:22:12.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:11 smithi192 ceph-mon[36354]: pgmap v503: 97 pgs: 97 active+clean; 592 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 356 op/s 2024-03-16T10:22:12.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:11 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:22:12.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:11 smithi077 ceph-mon[32400]: pgmap v503: 97 pgs: 97 active+clean; 592 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 356 op/s 2024-03-16T10:22:12.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:11 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:22:12.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:11 smithi193 ceph-mon[36311]: pgmap v503: 97 pgs: 97 active+clean; 592 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 356 op/s 2024-03-16T10:22:12.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:11 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:22:13.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:12 smithi192 ceph-mon[36354]: scrub queued for path: / 2024-03-16T10:22:13.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:12 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:13.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:12 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:22:13.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:12 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:13.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:12 smithi077 ceph-mon[32400]: scrub queued for path: / 2024-03-16T10:22:13.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:12 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:13.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:12 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:22:13.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:12 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:13.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:12 smithi193 ceph-mon[36311]: scrub queued for path: / 2024-03-16T10:22:13.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:12 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:13.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:12 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:22:13.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:12 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:14.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:13 smithi192 ceph-mon[36354]: pgmap v504: 97 pgs: 97 active+clean; 620 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 475 op/s 2024-03-16T10:22:14.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:13 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:22:14.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:13 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:14.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:13 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:22:14.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:13 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:14.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:13 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:22:14.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:13 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:14.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:13 smithi077 ceph-mon[32400]: pgmap v504: 97 pgs: 97 active+clean; 620 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 475 op/s 2024-03-16T10:22:14.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:13 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:22:14.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:13 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:14.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:13 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:22:14.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:13 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:14.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:13 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:22:14.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:13 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:14.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:13 smithi193 ceph-mon[36311]: pgmap v504: 97 pgs: 97 active+clean; 620 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 475 op/s 2024-03-16T10:22:14.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:13 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:22:14.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:13 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:14.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:13 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:22:14.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:13 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:14.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:13 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:22:14.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:13 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:15.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:14 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:22:15.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:14 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:15.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:14 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:22:15.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:14 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:15.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:14 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:22:15.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:14 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:22:15.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:14 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:15.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:14 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:15.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:14 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:22:15.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:14 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:15.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:14 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:22:15.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:14 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:15.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:14 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:22:15.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:14 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:22:15.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:14 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:15.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:14 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:15.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:14 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:22:15.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:14 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:15.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:14 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:22:15.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:14 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:15.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:14 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:22:15.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:14 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:22:15.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:14 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:15.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:14 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:22:16.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:15 smithi077 ceph-mon[32400]: pgmap v505: 97 pgs: 97 active+clean; 627 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 489 op/s 2024-03-16T10:22:16.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:15 smithi193 ceph-mon[36311]: pgmap v505: 97 pgs: 97 active+clean; 627 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 489 op/s 2024-03-16T10:22:16.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:15 smithi192 ceph-mon[36354]: pgmap v505: 97 pgs: 97 active+clean; 627 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 489 op/s 2024-03-16T10:22:17.104 INFO:tasks.workunit.client.0.smithi077.stdout: 126 1297161 318 865857 146 743399 702 2024-03-16T10:22:17.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:16 smithi077 ceph-mon[32400]: pgmap v506: 97 pgs: 97 active+clean; 626 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.4 MiB/s wr, 514 op/s 2024-03-16T10:22:17.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:16 smithi193 ceph-mon[36311]: pgmap v506: 97 pgs: 97 active+clean; 626 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.4 MiB/s wr, 514 op/s 2024-03-16T10:22:17.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:16 smithi192 ceph-mon[36354]: pgmap v506: 97 pgs: 97 active+clean; 626 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.4 MiB/s wr, 514 op/s 2024-03-16T10:22:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:19 smithi192 ceph-mon[36354]: pgmap v507: 97 pgs: 97 active+clean; 631 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 522 op/s 2024-03-16T10:22:20.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:19 smithi077 ceph-mon[32400]: pgmap v507: 97 pgs: 97 active+clean; 631 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 522 op/s 2024-03-16T10:22:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:19 smithi193 ceph-mon[36311]: pgmap v507: 97 pgs: 97 active+clean; 631 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 522 op/s 2024-03-16T10:22:22.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:21 smithi192 ceph-mon[36354]: pgmap v508: 97 pgs: 97 active+clean; 633 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 487 op/s 2024-03-16T10:22:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:21 smithi077 ceph-mon[32400]: pgmap v508: 97 pgs: 97 active+clean; 633 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 487 op/s 2024-03-16T10:22:22.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:21 smithi193 ceph-mon[36311]: pgmap v508: 97 pgs: 97 active+clean; 633 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 487 op/s 2024-03-16T10:22:24.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:23 smithi192 ceph-mon[36354]: pgmap v509: 97 pgs: 97 active+clean; 636 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 426 op/s 2024-03-16T10:22:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.200+0000 7f8d8c153640 1 -- 172.21.15.77:0/3902399605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8410a230 msgr2=0x7f8d84110ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.200+0000 7f8d8c153640 1 --2- 172.21.15.77:0/3902399605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8410a230 0x7f8d84110ae0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009ca0 tx=0x7f8d7c02c120 comp rx=0 tx=0).stop 2024-03-16T10:22:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.201+0000 7f8d8c153640 1 -- 172.21.15.77:0/3902399605 shutdown_connections 2024-03-16T10:22:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.201+0000 7f8d8c153640 1 --2- 172.21.15.77:0/3902399605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8410a230 0x7f8d84110ae0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.201+0000 7f8d8c153640 1 --2- 172.21.15.77:0/3902399605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d84109890 0x7f8d84109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.201+0000 7f8d8c153640 1 --2- 172.21.15.77:0/3902399605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d84108b80 0x7f8d84108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.201+0000 7f8d8c153640 1 -- 172.21.15.77:0/3902399605 >> 172.21.15.77:0/3902399605 conn(0x7f8d8406e820 msgr2=0x7f8d84070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.204+0000 7f8d8c153640 1 -- 172.21.15.77:0/3902399605 shutdown_connections 2024-03-16T10:22:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.204+0000 7f8d8c153640 1 -- 172.21.15.77:0/3902399605 wait complete. 2024-03-16T10:22:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.205+0000 7f8d8c153640 1 Processor -- start 2024-03-16T10:22:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.205+0000 7f8d8c153640 1 -- start start 2024-03-16T10:22:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.205+0000 7f8d8c153640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d84108b80 0x7f8d840714a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.205+0000 7f8d8c153640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d84109890 0x7f8d841a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.205+0000 7f8d8c153640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d841a2fe0 0x7f8d841a3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.205+0000 7f8d8c153640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d84114320 con 0x7f8d84109890 2024-03-16T10:22:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.205+0000 7f8d8c153640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8d841141a0 con 0x7f8d841a2fe0 2024-03-16T10:22:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.205+0000 7f8d8c153640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8d841144a0 con 0x7f8d84108b80 2024-03-16T10:22:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.205+0000 7f8d89eef640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d84108b80 0x7f8d840714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.207 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.205+0000 7f8d896ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d84109890 0x7f8d841a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.207 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.206+0000 7f8d8a6f0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d841a2fe0 0x7f8d841a3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.207 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.206+0000 7f8d8a6f0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d841a2fe0 0x7f8d841a3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:36438/0 (socket says 172.21.15.77:36438) 2024-03-16T10:22:24.207 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.206+0000 7f8d8a6f0640 1 -- 172.21.15.77:0/721905550 learned_addr learned my addr 172.21.15.77:0/721905550 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:24.207 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.206+0000 7f8d8a6f0640 1 -- 172.21.15.77:0/721905550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d84108b80 msgr2=0x7f8d840714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:24.207 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.206+0000 7f8d8a6f0640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d84108b80 0x7f8d840714a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.206+0000 7f8d8a6f0640 1 -- 172.21.15.77:0/721905550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d84109890 msgr2=0x7f8d841a2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.206+0000 7f8d8a6f0640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d84109890 0x7f8d841a2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.206+0000 7f8d8a6f0640 1 -- 172.21.15.77:0/721905550 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d841a3d50 con 0x7f8d841a2fe0 2024-03-16T10:22:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.207+0000 7f8d896ee640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d84109890 0x7f8d841a2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.207+0000 7f8d8a6f0640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d841a2fe0 0x7f8d841a3490 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009330 tx=0x7f8d7c003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.207+0000 7f8d7affd640 1 -- 172.21.15.77:0/721905550 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c039040 con 0x7f8d841a2fe0 2024-03-16T10:22:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.208+0000 7f8d8c153640 1 -- 172.21.15.77:0/721905550 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d841a3fe0 con 0x7f8d841a2fe0 2024-03-16T10:22:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.208+0000 7f8d7affd640 1 -- 172.21.15.77:0/721905550 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d7c044070 con 0x7f8d841a2fe0 2024-03-16T10:22:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.214+0000 7f8d8c153640 1 -- 172.21.15.77:0/721905550 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d841a4520 con 0x7f8d841a2fe0 2024-03-16T10:22:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.214+0000 7f8d7affd640 1 -- 172.21.15.77:0/721905550 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c004370 con 0x7f8d841a2fe0 2024-03-16T10:22:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.214+0000 7f8d7affd640 1 -- 172.21.15.77:0/721905550 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f8d7c0045d0 con 0x7f8d841a2fe0 2024-03-16T10:22:24.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:23 smithi077 ceph-mon[32400]: pgmap v509: 97 pgs: 97 active+clean; 636 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 426 op/s 2024-03-16T10:22:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:23 smithi193 ceph-mon[36311]: pgmap v509: 97 pgs: 97 active+clean; 636 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 426 op/s 2024-03-16T10:22:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.269+0000 7f8d8ee18740 1 Processor -- start 2024-03-16T10:22:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.269+0000 7f8d8ee18740 1 -- start start 2024-03-16T10:22:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.269+0000 7f8d8ee18740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106340 0x563aaf106760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.269+0000 7f8d8ee18740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106d30 0x563aaf10b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.269+0000 7f8d8ee18740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf10e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.269+0000 7f8d8ee18740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563aaf06ba40 con 0x563aaf106d30 2024-03-16T10:22:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.269+0000 7f8d8ee18740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563aaf014920 con 0x563aaf106340 2024-03-16T10:22:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.269+0000 7f8d8ee18740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563aaf067ae0 con 0x563aaf10bea0 2024-03-16T10:22:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.270+0000 7f8d896ee640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf10e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.270+0000 7f8d896ee640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf10e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:43080/0 (socket says 172.21.15.77:43080) 2024-03-16T10:22:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.270+0000 7f8d896ee640 1 -- 172.21.15.77:0/2862117864 learned_addr learned my addr 172.21.15.77:0/2862117864 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.270+0000 7f8d896ee640 1 --2- 172.21.15.77:0/2862117864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106340 0x563aaf106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.270+0000 7f8d896ee640 1 -- 172.21.15.77:0/2862117864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106340 msgr2=0x563aaf106760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.270+0000 7f8d896ee640 1 --2- 172.21.15.77:0/2862117864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106340 0x563aaf106760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.270+0000 7f8d896ee640 1 -- 172.21.15.77:0/2862117864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106d30 msgr2=0x563aaf10b830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:22:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.270+0000 7f8d896ee640 1 --2- 172.21.15.77:0/2862117864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106d30 0x563aaf10b830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.270+0000 7f8d896ee640 1 -- 172.21.15.77:0/2862117864 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aaf02c460 con 0x563aaf10bea0 2024-03-16T10:22:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.271+0000 7f8d896ee640 1 --2- 172.21.15.77:0/2862117864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf10e330 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8d800097e0 tx=0x7f8d80009770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=513d77466fcee7cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.274+0000 7f8d7affd640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8d700a29a0 0x7f8d700a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.274+0000 7f8d7affd640 1 -- 172.21.15.77:0/721905550 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8d7c0f5170 con 0x7f8d841a2fe0 2024-03-16T10:22:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.282+0000 7f8d6ffff640 1 -- 172.21.15.77:0/2862117864 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8000fc20 con 0x563aaf10bea0 2024-03-16T10:22:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.282+0000 7f8d6ffff640 1 -- 172.21.15.77:0/2862117864 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d8000fdc0 con 0x563aaf10bea0 2024-03-16T10:22:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.283+0000 7f8d8ee18740 1 -- 172.21.15.77:0/2862117864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 msgr2=0x563aaf10e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.283+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/2862117864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf10e330 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8d800097e0 tx=0x7f8d80009770 comp rx=0 tx=0).stop 2024-03-16T10:22:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.284+0000 7f8d8ee18740 1 -- 172.21.15.77:0/2862117864 shutdown_connections 2024-03-16T10:22:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.284+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/2862117864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf10e330 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.284+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/2862117864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106d30 0x563aaf10b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.284+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/2862117864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106340 0x563aaf106760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.284+0000 7f8d8ee18740 1 -- 172.21.15.77:0/2862117864 >> 172.21.15.77:0/2862117864 conn(0x563aaf1018b0 msgr2=0x563aaf101cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.289+0000 7f8d89eef640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8d700a29a0 0x7f8d700a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.289+0000 7f8d8ee18740 1 -- 172.21.15.77:0/2862117864 shutdown_connections 2024-03-16T10:22:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.290+0000 7f8d8ee18740 1 -- 172.21.15.77:0/2862117864 wait complete. 2024-03-16T10:22:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.290+0000 7f8d8ee18740 1 -- start start 2024-03-16T10:22:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.291+0000 7f8d8ee18740 1 Processor -- start 2024-03-16T10:22:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.291+0000 7f8d8ee18740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf110980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.291+0000 7f8d8ee18740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106d30 0x563aaf110ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.291+0000 7f8d8ee18740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf117f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.291+0000 7f8d8ee18740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563aaf10eef0 con 0x563aaf106340 2024-03-16T10:22:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.291+0000 7f8d8ee18740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563aaf027d70 con 0x563aaf106d30 2024-03-16T10:22:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.291+0000 7f8d8ee18740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563aaf10f070 con 0x563aaf10bea0 2024-03-16T10:22:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.291+0000 7f8d8ee18740 10 client.0 _ll_register_callbacks cb 0x563aaed7fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.292+0000 7f8d8a6f0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf117f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.292+0000 7f8d8a6f0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf117f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:43094/0 (socket says 172.21.15.77:43094) 2024-03-16T10:22:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.292+0000 7f8d8a6f0640 1 -- 172.21.15.77:0/974529070 learned_addr learned my addr 172.21.15.77:0/974529070 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.292+0000 7f8d896ee640 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106d30 0x563aaf110ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.292+0000 7f8d8a6f0640 1 -- 172.21.15.77:0/974529070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106d30 msgr2=0x563aaf110ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.292+0000 7f8d8a6f0640 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106d30 0x563aaf110ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.292+0000 7f8d8a6f0640 1 -- 172.21.15.77:0/974529070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 msgr2=0x563aaf110980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.292+0000 7f8d8a6f0640 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf110980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.292+0000 7f8d8a6f0640 1 -- 172.21.15.77:0/974529070 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aaed80210 con 0x563aaf10bea0 2024-03-16T10:22:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.292+0000 7f8d896ee640 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106d30 0x563aaf110ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.292+0000 7f8d8a6f0640 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf117f00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0c6700 tx=0x7f8d7c0bc9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cee659e9aea7ec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.293+0000 7f8d6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:22:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.293+0000 7f8d6d7fa640 1 -- 172.21.15.77:0/974529070 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c0f6060 con 0x563aaf10bea0 2024-03-16T10:22:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.293+0000 7f8d8ee18740 1 -- 172.21.15.77:0/974529070 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563aaf078f80 con 0x563aaf10bea0 2024-03-16T10:22:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.294+0000 7f8d89eef640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8d700a29a0 0x7f8d700a4e60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x563aaf102820 tx=0x7f8d7400a8d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.293+0000 7f8d6d7fa640 1 -- 172.21.15.77:0/974529070 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8d7c0c2160 con 0x563aaf10bea0 2024-03-16T10:22:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.296+0000 7f8d6d7fa640 1 -- 172.21.15.77:0/974529070 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8d7c0bddf0 con 0x563aaf10bea0 2024-03-16T10:22:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.296+0000 7f8d8ee18740 10 client.26707 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.296+0000 7f8d8ee18740 1 -- 172.21.15.77:0/974529070 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563aaee4c620 con 0x563aaf10bea0 2024-03-16T10:22:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.296+0000 7f8d6d7fa640 1 -- 172.21.15.77:0/974529070 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563aaf078f80 con 0x563aaf10bea0 2024-03-16T10:22:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.296+0000 7f8d6d7fa640 1 -- 172.21.15.77:0/974529070 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f8d7c03c030 con 0x563aaf10bea0 2024-03-16T10:22:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.296+0000 7f8d8ee18740 10 client.26707 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.296+0000 7f8d8ee18740 10 client.26707 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.296+0000 7f8d8ee18740 20 client.26707 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.296+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563aaf12d960 0x563aaf12fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.296+0000 7f8d8ee18740 4 client.26707 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.296+0000 7f8d8ee18740 1 -- 172.21.15.77:0/974529070 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563aaf0732d0 con 0x563aaf12d960 2024-03-16T10:22:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.297+0000 7f8d89eef640 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563aaf12d960 0x563aaf12fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.297+0000 7f8d6d7fa640 10 client.26707 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.297+0000 7f8d89eef640 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563aaf12d960 0x563aaf12fd90 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=697b2db205feffe5 server_cookie=eada81c4339491b1 in_seq=0 out_seq=0 2024-03-16T10:22:24.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.298+0000 7f8d6d7fa640 1 -- 172.21.15.77:0/974529070 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563aaf0732d0 con 0x563aaf12d960 2024-03-16T10:22:24.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.299+0000 7f8d6d7fa640 10 client.26707 handle_command_reply: tid=0 2024-03-16T10:22:24.299 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.299+0000 7f8d8ee18740 1 client.26707 shutdown 2024-03-16T10:22:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.312+0000 7f8d8ee18740 1 -- 172.21.15.77:0/974529070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 msgr2=0x563aaf117f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.312+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf117f00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0c6700 tx=0x7f8d7c0bc9a0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.314 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.314+0000 7f8d8ee18740 1 -- 172.21.15.77:0/974529070 shutdown_connections 2024-03-16T10:22:24.315 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.314+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf117f00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.315 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.314+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106d30 0x563aaf110ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.315 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.314+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563aaf12d960 0x563aaf12fd90 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.315 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.314+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/974529070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.315 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.314+0000 7f8d8ee18740 1 -- 172.21.15.77:0/974529070 >> 172.21.15.77:0/974529070 conn(0x563aaf1018b0 msgr2=0x563aaf100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:24.315 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.314+0000 7f8d6d7fa640 0 client.26707 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.318+0000 7f8d8ee18740 1 -- 172.21.15.77:0/974529070 shutdown_connections 2024-03-16T10:22:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.318+0000 7f8d8ee18740 1 -- 172.21.15.77:0/974529070 wait complete. 2024-03-16T10:22:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.318+0000 7f8d8ee18740 20 client.26707 trim_cache size 0 max 16384 2024-03-16T10:22:24.380 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.379+0000 7f8d8ee18740 1 Processor -- start 2024-03-16T10:22:24.380 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.379+0000 7f8d8ee18740 1 -- start start 2024-03-16T10:22:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.381+0000 7f8d8ee18740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf110980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.383+0000 7f8d89eef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf110980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.383+0000 7f8d89eef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf110980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58826/0 (socket says 172.21.15.77:58826) 2024-03-16T10:22:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.383+0000 7f8d89eef640 1 -- 172.21.15.77:0/842123176 learned_addr learned my addr 172.21.15.77:0/842123176 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.383+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/842123176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106d30 0x563aaf117ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.383+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/842123176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf110ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.383+0000 7f8d8ee18740 1 -- 172.21.15.77:0/842123176 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d841ad350 con 0x563aaf106340 2024-03-16T10:22:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.383+0000 7f8d8ee18740 1 -- 172.21.15.77:0/842123176 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8d841b0f30 con 0x563aaf106d30 2024-03-16T10:22:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.383+0000 7f8d896ee640 1 --2- 172.21.15.77:0/842123176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106d30 0x563aaf117ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.383+0000 7f8d8ee18740 1 -- 172.21.15.77:0/842123176 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8d841afa30 con 0x563aaf10bea0 2024-03-16T10:22:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.383+0000 7f8d8a6f0640 1 --2- 172.21.15.77:0/842123176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf110ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.384+0000 7f8d8a6f0640 1 -- 172.21.15.77:0/842123176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106d30 msgr2=0x563aaf117ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.384+0000 7f8d8a6f0640 1 --2- 172.21.15.77:0/842123176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106d30 0x563aaf117ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.384+0000 7f8d8a6f0640 1 -- 172.21.15.77:0/842123176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 msgr2=0x563aaf110980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:24.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.384+0000 7f8d8a6f0640 1 --2- 172.21.15.77:0/842123176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf110980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.384+0000 7f8d8a6f0640 1 -- 172.21.15.77:0/842123176 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aaf0fd310 con 0x563aaf10bea0 2024-03-16T10:22:24.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.384+0000 7f8d89eef640 1 --2- 172.21.15.77:0/842123176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:24.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.384+0000 7f8d8a6f0640 1 --2- 172.21.15.77:0/842123176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf110ec0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0be190 tx=0x7f8d7c0be090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbe10e86998067ff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.388+0000 7f8d6f7fe640 1 -- 172.21.15.77:0/842123176 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c0be260 con 0x563aaf10bea0 2024-03-16T10:22:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.388+0000 7f8d6f7fe640 1 -- 172.21.15.77:0/842123176 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d7c035e50 con 0x563aaf10bea0 2024-03-16T10:22:24.389 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.388+0000 7f8d6f7fe640 1 -- 172.21.15.77:0/842123176 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c0be490 con 0x563aaf10bea0 2024-03-16T10:22:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.396+0000 7f8d8ee18740 1 -- 172.21.15.77:0/842123176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 msgr2=0x563aaf110ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.396+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/842123176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf110ec0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0be190 tx=0x7f8d7c0be090 comp rx=0 tx=0).stop 2024-03-16T10:22:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.397+0000 7f8d8ee18740 1 -- 172.21.15.77:0/842123176 shutdown_connections 2024-03-16T10:22:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.397+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/842123176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf10bea0 0x563aaf110ec0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.397+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/842123176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf106d30 0x563aaf117ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.397+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/842123176 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.397+0000 7f8d8ee18740 1 -- 172.21.15.77:0/842123176 >> 172.21.15.77:0/842123176 conn(0x563aaf1018b0 msgr2=0x563aaf0fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.397+0000 7f8d8ee18740 1 -- 172.21.15.77:0/842123176 shutdown_connections 2024-03-16T10:22:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.398+0000 7f8d8ee18740 1 -- 172.21.15.77:0/842123176 wait complete. 2024-03-16T10:22:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.398+0000 7f8d8ee18740 1 -- start start 2024-03-16T10:22:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.398+0000 7f8d8ee18740 1 Processor -- start 2024-03-16T10:22:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d8ee18740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf114b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d8ee18740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf106d30 0x563aaf115050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d8ee18740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf10bea0 0x563aaf191cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d8ee18740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d841b0860 con 0x563aaf106340 2024-03-16T10:22:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d8ee18740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8d841b0190 con 0x563aaf10bea0 2024-03-16T10:22:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d896ee640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf106d30 0x563aaf115050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d8ee18740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8d841ab330 con 0x563aaf106d30 2024-03-16T10:22:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d896ee640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf106d30 0x563aaf115050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:43110/0 (socket says 172.21.15.77:43110) 2024-03-16T10:22:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d896ee640 1 -- 172.21.15.77:0/1946727188 learned_addr learned my addr 172.21.15.77:0/1946727188 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d89eef640 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf114b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d8ee18740 10 client.0 _ll_register_callbacks cb 0x7f8d7c0f4070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d8a6f0640 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf10bea0 0x563aaf191cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d896ee640 1 -- 172.21.15.77:0/1946727188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf10bea0 msgr2=0x563aaf191cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d896ee640 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf10bea0 0x563aaf191cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.399+0000 7f8d896ee640 1 -- 172.21.15.77:0/1946727188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 msgr2=0x563aaf114b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.400+0000 7f8d896ee640 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf114b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.400+0000 7f8d896ee640 1 -- 172.21.15.77:0/1946727188 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aaf1180b0 con 0x563aaf106d30 2024-03-16T10:22:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.400+0000 7f8d89eef640 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf114b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.400+0000 7f8d896ee640 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf106d30 0x563aaf115050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8d800097e0 tx=0x7f8d80003a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84b356a2212e447d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.400+0000 7f8d6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:22:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.400+0000 7f8d6d7fa640 1 -- 172.21.15.77:0/1946727188 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80005270 con 0x563aaf106d30 2024-03-16T10:22:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.400+0000 7f8d8ee18740 1 -- 172.21.15.77:0/1946727188 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d7c035e50 con 0x563aaf106d30 2024-03-16T10:22:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.400+0000 7f8d6d7fa640 1 -- 172.21.15.77:0/1946727188 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f8d80005410 con 0x563aaf106d30 2024-03-16T10:22:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.401+0000 7f8d6d7fa640 1 -- 172.21.15.77:0/1946727188 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8d80005650 con 0x563aaf106d30 2024-03-16T10:22:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.401+0000 7f8d8ee18740 10 client.26713 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.401+0000 7f8d8ee18740 1 -- 172.21.15.77:0/1946727188 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563aaf193180 con 0x563aaf106d30 2024-03-16T10:22:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.401+0000 7f8d6d7fa640 1 -- 172.21.15.77:0/1946727188 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d7c035e50 con 0x563aaf106d30 2024-03-16T10:22:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.402+0000 7f8d6d7fa640 1 -- 172.21.15.77:0/1946727188 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f8d80036050 con 0x563aaf106d30 2024-03-16T10:22:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.402+0000 7f8d8ee18740 10 client.26713 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.402+0000 7f8d8ee18740 10 client.26713 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.402+0000 7f8d8ee18740 20 client.26713 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.402+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563aaf193890 0x563aaf195cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.402+0000 7f8d8ee18740 4 client.26713 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:22:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.402+0000 7f8d8ee18740 1 -- 172.21.15.77:0/1946727188 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563aaeff1770 con 0x563aaf193890 2024-03-16T10:22:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.402+0000 7f8d89eef640 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563aaf193890 0x563aaf195cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:24.404 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.403+0000 7f8d89eef640 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563aaf193890 0x563aaf195cb0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59dc33e5a837155 server_cookie=6aeac8ac0c8fbba2 in_seq=0 out_seq=0 2024-03-16T10:22:24.405 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.403+0000 7f8d6d7fa640 10 client.26713 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.420+0000 7f8d6d7fa640 1 -- 172.21.15.77:0/1946727188 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563aaeff1770 con 0x563aaf193890 2024-03-16T10:22:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.420+0000 7f8d6d7fa640 10 client.26713 handle_command_reply: tid=0 2024-03-16T10:22:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.420+0000 7f8d8ee18740 1 client.26713 shutdown 2024-03-16T10:22:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.420+0000 7f8d8ee18740 1 -- 172.21.15.77:0/1946727188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf106d30 msgr2=0x563aaf115050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.420+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf106d30 0x563aaf115050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8d800097e0 tx=0x7f8d80003a10 comp rx=0 tx=0).stop 2024-03-16T10:22:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.420+0000 7f8d8ee18740 1 -- 172.21.15.77:0/1946727188 shutdown_connections 2024-03-16T10:22:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.420+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563aaf10bea0 0x563aaf191cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.420+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563aaf106d30 0x563aaf115050 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.420+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563aaf193890 0x563aaf195cb0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.420+0000 7f8d8ee18740 1 --2- 172.21.15.77:0/1946727188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563aaf106340 0x563aaf114b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.420+0000 7f8d6d7fa640 0 client.26713 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.420+0000 7f8d8ee18740 1 -- 172.21.15.77:0/1946727188 >> 172.21.15.77:0/1946727188 conn(0x563aaf1018b0 msgr2=0x563aaf110980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.421+0000 7f8d8ee18740 1 -- 172.21.15.77:0/1946727188 shutdown_connections 2024-03-16T10:22:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.421+0000 7f8d8ee18740 1 -- 172.21.15.77:0/1946727188 wait complete. 2024-03-16T10:22:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.421+0000 7f8d8ee18740 20 client.26713 trim_cache size 0 max 16384 2024-03-16T10:22:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:22:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:22:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:22:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:22:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.422+0000 7f8d8c153640 1 -- 172.21.15.77:0/721905550 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8d700a29a0 msgr2=0x7f8d700a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.422+0000 7f8d8c153640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8d700a29a0 0x7f8d700a4e60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x563aaf102820 tx=0x7f8d7400a8d0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.422+0000 7f8d8c153640 1 -- 172.21.15.77:0/721905550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d841a2fe0 msgr2=0x7f8d841a3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.422+0000 7f8d8c153640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d841a2fe0 0x7f8d841a3490 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009330 tx=0x7f8d7c003ba0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.422+0000 7f8d8c153640 1 -- 172.21.15.77:0/721905550 shutdown_connections 2024-03-16T10:22:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.422+0000 7f8d8c153640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d841a2fe0 0x7f8d841a3490 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.422+0000 7f8d8c153640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d84109890 0x7f8d841a2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.422+0000 7f8d8c153640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f8d700a29a0 0x7f8d700a4e60 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.422+0000 7f8d8c153640 1 --2- 172.21.15.77:0/721905550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d84108b80 0x7f8d840714a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.422+0000 7f8d8c153640 1 -- 172.21.15.77:0/721905550 >> 172.21.15.77:0/721905550 conn(0x7f8d8406e820 msgr2=0x7f8d8410cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.422+0000 7f8d8c153640 1 -- 172.21.15.77:0/721905550 shutdown_connections 2024-03-16T10:22:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:24.423+0000 7f8d8c153640 1 -- 172.21.15.77:0/721905550 wait complete. 2024-03-16T10:22:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:25 smithi192 ceph-mon[36354]: pgmap v510: 97 pgs: 97 active+clean; 636 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 281 op/s 2024-03-16T10:22:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:25 smithi077 ceph-mon[32400]: pgmap v510: 97 pgs: 97 active+clean; 636 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 281 op/s 2024-03-16T10:22:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:25 smithi193 ceph-mon[36311]: pgmap v510: 97 pgs: 97 active+clean; 636 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 281 op/s 2024-03-16T10:22:27.104 INFO:tasks.workunit.client.0.smithi077.stdout: 134 2148680 381 1433664 620 1224801 1079 2024-03-16T10:22:28.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:27 smithi192 ceph-mon[36354]: pgmap v511: 97 pgs: 97 active+clean; 637 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 251 op/s 2024-03-16T10:22:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:27 smithi077 ceph-mon[32400]: pgmap v511: 97 pgs: 97 active+clean; 637 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 251 op/s 2024-03-16T10:22:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:27 smithi193 ceph-mon[36311]: pgmap v511: 97 pgs: 97 active+clean; 637 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 251 op/s 2024-03-16T10:22:29.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:28 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:22:29.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:28 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:22:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:28 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:22:30.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:29 smithi077 ceph-mon[32400]: pgmap v512: 97 pgs: 97 active+clean; 619 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 238 op/s 2024-03-16T10:22:30.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:29 smithi193 ceph-mon[36311]: pgmap v512: 97 pgs: 97 active+clean; 619 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 238 op/s 2024-03-16T10:22:30.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:29 smithi192 ceph-mon[36354]: pgmap v512: 97 pgs: 97 active+clean; 619 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 238 op/s 2024-03-16T10:22:31.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:30 smithi077 ceph-mon[32400]: pgmap v513: 97 pgs: 97 active+clean; 632 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 279 op/s 2024-03-16T10:22:31.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:30 smithi193 ceph-mon[36311]: pgmap v513: 97 pgs: 97 active+clean; 632 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 279 op/s 2024-03-16T10:22:31.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:30 smithi192 ceph-mon[36354]: pgmap v513: 97 pgs: 97 active+clean; 632 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 279 op/s 2024-03-16T10:22:32.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:22:32.264 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:22:32.265 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:22:32.265 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:22:32.265 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:22:32.265 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:22:32.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:22:32.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:22:32.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:22:33.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:32 smithi077 ceph-mon[32400]: pgmap v514: 97 pgs: 97 active+clean; 657 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 451 op/s 2024-03-16T10:22:33.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:32 smithi193 ceph-mon[36311]: pgmap v514: 97 pgs: 97 active+clean; 657 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 451 op/s 2024-03-16T10:22:33.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:32 smithi192 ceph-mon[36354]: pgmap v514: 97 pgs: 97 active+clean; 657 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 451 op/s 2024-03-16T10:22:36.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:35 smithi192 ceph-mon[36354]: pgmap v515: 97 pgs: 97 active+clean; 670 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.6 MiB/s wr, 497 op/s 2024-03-16T10:22:36.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:35 smithi077 ceph-mon[32400]: pgmap v515: 97 pgs: 97 active+clean; 670 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.6 MiB/s wr, 497 op/s 2024-03-16T10:22:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:35 smithi193 ceph-mon[36311]: pgmap v515: 97 pgs: 97 active+clean; 670 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.6 MiB/s wr, 497 op/s 2024-03-16T10:22:37.105 INFO:tasks.workunit.client.0.smithi077.stdout: 139 1305759 267 888702 234 764312 664 2024-03-16T10:22:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.892+0000 7fe77db37640 1 -- 172.21.15.77:0/1915878110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe778100ea0 msgr2=0x7fe778101320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.892+0000 7fe77db37640 1 --2- 172.21.15.77:0/1915878110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe778100ea0 0x7fe778101320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe76400d960 tx=0x7fe76402db10 comp rx=0 tx=0).stop 2024-03-16T10:22:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.892+0000 7fe77db37640 1 -- 172.21.15.77:0/1915878110 shutdown_connections 2024-03-16T10:22:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.892+0000 7fe77db37640 1 --2- 172.21.15.77:0/1915878110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe778101860 0x7fe7781080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.892+0000 7fe77db37640 1 --2- 172.21.15.77:0/1915878110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe778100ea0 0x7fe778101320 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.892+0000 7fe77db37640 1 --2- 172.21.15.77:0/1915878110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7780ffcb0 0x7fe7781000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.892+0000 7fe77db37640 1 -- 172.21.15.77:0/1915878110 >> 172.21.15.77:0/1915878110 conn(0x7fe7780fb220 msgr2=0x7fe7780fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.892+0000 7fe77db37640 1 -- 172.21.15.77:0/1915878110 shutdown_connections 2024-03-16T10:22:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.892+0000 7fe77db37640 1 -- 172.21.15.77:0/1915878110 wait complete. 2024-03-16T10:22:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.893+0000 7fe77db37640 1 Processor -- start 2024-03-16T10:22:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.893+0000 7fe77db37640 1 -- start start 2024-03-16T10:22:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.893+0000 7fe77db37640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7780ffcb0 0x7fe778199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.893+0000 7fe77db37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe778100ea0 0x7fe77819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.893+0000 7fe77db37640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe778101860 0x7fe7781a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.893+0000 7fe77db37640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe77810b110 con 0x7fe778100ea0 2024-03-16T10:22:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.893+0000 7fe77db37640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe77810af90 con 0x7fe7780ffcb0 2024-03-16T10:22:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.893+0000 7fe77db37640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe77810b290 con 0x7fe778101860 2024-03-16T10:22:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.894+0000 7fe7777fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7780ffcb0 0x7fe778199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.894+0000 7fe776ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe778100ea0 0x7fe77819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.894+0000 7fe7777fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7780ffcb0 0x7fe778199c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:43812/0 (socket says 172.21.15.77:43812) 2024-03-16T10:22:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.894+0000 7fe7777fe640 1 -- 172.21.15.77:0/652242456 learned_addr learned my addr 172.21.15.77:0/652242456 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.894+0000 7fe777fff640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe778101860 0x7fe7781a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.894+0000 7fe7777fe640 1 -- 172.21.15.77:0/652242456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe778101860 msgr2=0x7fe7781a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.894+0000 7fe7777fe640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe778101860 0x7fe7781a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.894+0000 7fe7777fe640 1 -- 172.21.15.77:0/652242456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe778100ea0 msgr2=0x7fe77819a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.894+0000 7fe7777fe640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe778100ea0 0x7fe77819a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.895+0000 7fe7777fe640 1 -- 172.21.15.77:0/652242456 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7781a1790 con 0x7fe7780ffcb0 2024-03-16T10:22:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.895+0000 7fe777fff640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe778101860 0x7fe7781a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:22:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.895+0000 7fe7777fe640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7780ffcb0 0x7fe778199c10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00fb90 tx=0x7fe76c00d7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.895+0000 7fe774ff9640 1 -- 172.21.15.77:0/652242456 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c004220 con 0x7fe7780ffcb0 2024-03-16T10:22:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.895+0000 7fe774ff9640 1 -- 172.21.15.77:0/652242456 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe76c0043c0 con 0x7fe7780ffcb0 2024-03-16T10:22:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.895+0000 7fe77db37640 1 -- 172.21.15.77:0/652242456 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7781a1a80 con 0x7fe7780ffcb0 2024-03-16T10:22:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.895+0000 7fe77db37640 1 -- 172.21.15.77:0/652242456 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7781a1ff0 con 0x7fe7780ffcb0 2024-03-16T10:22:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.897+0000 7fe774ff9640 1 -- 172.21.15.77:0/652242456 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c00bbe0 con 0x7fe7780ffcb0 2024-03-16T10:22:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.900+0000 7fe7807fa740 1 Processor -- start 2024-03-16T10:22:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.900+0000 7fe7807fa740 1 -- start start 2024-03-16T10:22:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe774ff9640 1 -- 172.21.15.77:0/652242456 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fe76c022430 con 0x7fe7780ffcb0 2024-03-16T10:22:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe7807fa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 0x5645568a5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe7807fa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 0x5645568aa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe7807fa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568ab060 0x5645568ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe7807fa740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56455680aa40 con 0x5645568a5500 2024-03-16T10:22:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe7807fa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5645567b3920 con 0x5645568ab060 2024-03-16T10:22:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe7807fa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564556806ae0 con 0x5645568a5ef0 2024-03-16T10:22:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe7777fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 0x5645568aa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe777fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 0x5645568a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe777fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 0x5645568a5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45208/0 (socket says 172.21.15.77:45208) 2024-03-16T10:22:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe7777fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 0x5645568aa9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:57530/0 (socket says 172.21.15.77:57530) 2024-03-16T10:22:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe777fff640 1 -- 172.21.15.77:0/1714170763 learned_addr learned my addr 172.21.15.77:0/1714170763 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.901+0000 7fe776ffd640 1 --2- 172.21.15.77:0/1714170763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568ab060 0x5645568ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe777fff640 1 -- 172.21.15.77:0/1714170763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 msgr2=0x5645568aa9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe777fff640 1 --2- 172.21.15.77:0/1714170763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 0x5645568aa9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe777fff640 1 -- 172.21.15.77:0/1714170763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568ab060 msgr2=0x5645568ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe777fff640 1 --2- 172.21.15.77:0/1714170763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568ab060 0x5645568ad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe777fff640 1 -- 172.21.15.77:0/1714170763 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645567cb460 con 0x5645568a5500 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe7777fe640 1 --2- 172.21.15.77:0/1714170763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 0x5645568aa9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe774ff9640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe7680a28b0 0x7fe7680a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe777fff640 1 --2- 172.21.15.77:0/1714170763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 0x5645568a5920 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe77000f410 tx=0x7fe77000f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4e424cb8901737 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe774ff9640 1 -- 172.21.15.77:0/652242456 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe76c0cf0d0 con 0x7fe7780ffcb0 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe761ffb640 1 -- 172.21.15.77:0/1714170763 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77000d660 con 0x5645568a5500 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe761ffb640 1 -- 172.21.15.77:0/1714170763 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe77000dce0 con 0x5645568a5500 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe761ffb640 1 -- 172.21.15.77:0/1714170763 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe770019e10 con 0x5645568a5500 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.902+0000 7fe7777fe640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe7680a28b0 0x7fe7680a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.903+0000 7fe7807fa740 1 -- 172.21.15.77:0/1714170763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 msgr2=0x5645568a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.903+0000 7fe7807fa740 1 --2- 172.21.15.77:0/1714170763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 0x5645568a5920 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe77000f410 tx=0x7fe77000f810 comp rx=0 tx=0).stop 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.903+0000 7fe7777fe640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe7680a28b0 0x7fe7680a4d70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0a3cd0 tx=0x7fe76c0d2b10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.904+0000 7fe7807fa740 1 -- 172.21.15.77:0/1714170763 shutdown_connections 2024-03-16T10:22:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.904+0000 7fe7807fa740 1 --2- 172.21.15.77:0/1714170763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568ab060 0x5645568ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.904+0000 7fe7807fa740 1 --2- 172.21.15.77:0/1714170763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 0x5645568aa9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.904+0000 7fe7807fa740 1 --2- 172.21.15.77:0/1714170763 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 0x5645568a5920 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.904+0000 7fe7807fa740 1 -- 172.21.15.77:0/1714170763 >> 172.21.15.77:0/1714170763 conn(0x5645568a0a70 msgr2=0x5645568a0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.904+0000 7fe7807fa740 1 -- 172.21.15.77:0/1714170763 shutdown_connections 2024-03-16T10:22:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.904+0000 7fe7807fa740 1 -- 172.21.15.77:0/1714170763 wait complete. 2024-03-16T10:22:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.905+0000 7fe7807fa740 1 -- start start 2024-03-16T10:22:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.905+0000 7fe7807fa740 1 Processor -- start 2024-03-16T10:22:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.906+0000 7fe7807fa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5500 0x5645568b8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.906+0000 7fe7807fa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 0x5645568b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.906+0000 7fe777fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5500 0x5645568b8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.906+0000 7fe776ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 0x5645568b85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.906+0000 7fe776ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 0x5645568b85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:57538/0 (socket says 172.21.15.77:57538) 2024-03-16T10:22:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.906+0000 7fe777fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5500 0x5645568b8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:43826/0 (socket says 172.21.15.77:43826) 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.906+0000 7fe776ffd640 1 -- 172.21.15.77:0/2925436225 learned_addr learned my addr 172.21.15.77:0/2925436225 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.906+0000 7fe7807fa740 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568ab060 0x5645568bf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.906+0000 7fe7807fa740 1 -- 172.21.15.77:0/2925436225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5645568b6820 con 0x5645568ab060 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.906+0000 7fe7807fa740 1 -- 172.21.15.77:0/2925436225 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5645567c6d70 con 0x5645568a5500 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.906+0000 7fe7807fa740 1 -- 172.21.15.77:0/2925436225 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5645568b69a0 con 0x5645568a5ef0 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.907+0000 7fe7807fa740 10 client.0 _ll_register_callbacks cb 0x56455651ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.907+0000 7fe777fff640 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568ab060 0x5645568bf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.907+0000 7fe776ffd640 1 -- 172.21.15.77:0/2925436225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5500 msgr2=0x5645568b8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.907+0000 7fe776ffd640 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5500 0x5645568b8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.907+0000 7fe776ffd640 1 -- 172.21.15.77:0/2925436225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568ab060 msgr2=0x5645568bf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.907+0000 7fe776ffd640 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568ab060 0x5645568bf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.907+0000 7fe776ffd640 1 -- 172.21.15.77:0/2925436225 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56455651f210 con 0x5645568a5ef0 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.907+0000 7fe7807fa740 1 -- 172.21.15.77:0/2925436225 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe77000dce0 con 0x5645568a5ef0 2024-03-16T10:22:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.907+0000 7fe776ffd640 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 0x5645568b85b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe764039040 tx=0x7fe76400d6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d4d9847ad5b3c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.907+0000 7fe756ffd640 10 client.26725 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.907+0000 7fe756ffd640 1 -- 172.21.15.77:0/2925436225 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe764005230 con 0x5645568a5ef0 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.907+0000 7fe756ffd640 1 -- 172.21.15.77:0/2925436225 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe7640053d0 con 0x5645568a5ef0 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.908+0000 7fe756ffd640 1 -- 172.21.15.77:0/2925436225 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fe7640055d0 con 0x5645568a5ef0 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.908+0000 7fe7807fa740 10 client.26725 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.908+0000 7fe7807fa740 1 -- 172.21.15.77:0/2925436225 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=36}) -- 0x5645565eb620 con 0x5645568a5ef0 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.908+0000 7fe756ffd640 1 -- 172.21.15.77:0/2925436225 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fe7640046c0 con 0x5645568a5ef0 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.908+0000 7fe7807fa740 10 client.26725 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.908+0000 7fe7807fa740 10 client.26725 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.908+0000 7fe7807fa740 20 client.26725 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.908+0000 7fe7807fa740 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645568c3ca0 0x5645568c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.908+0000 7fe7807fa740 4 client.26725 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.908+0000 7fe7807fa740 1 -- 172.21.15.77:0/2925436225 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564556817f80 con 0x5645568c3ca0 2024-03-16T10:22:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.908+0000 7fe777fff640 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645568c3ca0 0x5645568c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.909+0000 7fe777fff640 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645568c3ca0 0x5645568c60b0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71c97899fcf5677a server_cookie=55605c76fd2cfd92 in_seq=0 out_seq=0 2024-03-16T10:22:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.909+0000 7fe756ffd640 10 client.26725 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.910+0000 7fe756ffd640 1 -- 172.21.15.77:0/2925436225 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564556817f80 con 0x5645568c3ca0 2024-03-16T10:22:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.910+0000 7fe756ffd640 10 client.26725 handle_command_reply: tid=0 2024-03-16T10:22:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.910+0000 7fe7807fa740 1 client.26725 shutdown 2024-03-16T10:22:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.910+0000 7fe7807fa740 1 -- 172.21.15.77:0/2925436225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 msgr2=0x5645568b85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.910+0000 7fe7807fa740 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 0x5645568b85b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe764039040 tx=0x7fe76400d6b0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.910+0000 7fe7807fa740 1 -- 172.21.15.77:0/2925436225 shutdown_connections 2024-03-16T10:22:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.910+0000 7fe7807fa740 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645568c3ca0 0x5645568c60b0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.910+0000 7fe756ffd640 0 client.26725 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.910+0000 7fe7807fa740 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568ab060 0x5645568bf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.910+0000 7fe7807fa740 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5ef0 0x5645568b85b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.910+0000 7fe7807fa740 1 --2- 172.21.15.77:0/2925436225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5500 0x5645568b8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.910+0000 7fe7807fa740 1 -- 172.21.15.77:0/2925436225 >> 172.21.15.77:0/2925436225 conn(0x5645568a0a70 msgr2=0x56455689fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.911+0000 7fe7807fa740 1 -- 172.21.15.77:0/2925436225 shutdown_connections 2024-03-16T10:22:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.911+0000 7fe7807fa740 1 -- 172.21.15.77:0/2925436225 wait complete. 2024-03-16T10:22:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.911+0000 7fe7807fa740 20 client.26725 trim_cache size 0 max 16384 2024-03-16T10:22:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.989+0000 7fe7807fa740 1 Processor -- start 2024-03-16T10:22:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.989+0000 7fe7807fa740 1 -- start start 2024-03-16T10:22:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.990+0000 7fe7807fa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 0x5645568a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.991 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.990+0000 7fe777fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 0x5645568a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.990+0000 7fe777fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 0x5645568a3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45234/0 (socket says 172.21.15.77:45234) 2024-03-16T10:22:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.990+0000 7fe777fff640 1 -- 172.21.15.77:0/183246406 learned_addr learned my addr 172.21.15.77:0/183246406 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.990+0000 7fe7807fa740 1 --2- 172.21.15.77:0/183246406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5ef0 0x5645568bf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.990+0000 7fe7807fa740 1 --2- 172.21.15.77:0/183246406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568ab060 0x5645568a4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.991+0000 7fe7807fa740 1 -- 172.21.15.77:0/183246406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c00e2e0 con 0x5645568a5500 2024-03-16T10:22:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.991+0000 7fe776ffd640 1 --2- 172.21.15.77:0/183246406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5ef0 0x5645568bf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.991+0000 7fe7807fa740 1 -- 172.21.15.77:0/183246406 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c00feb0 con 0x5645568a5ef0 2024-03-16T10:22:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.991+0000 7fe7807fa740 1 -- 172.21.15.77:0/183246406 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c00e9b0 con 0x5645568ab060 2024-03-16T10:22:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.991+0000 7fe777fff640 1 -- 172.21.15.77:0/183246406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568ab060 msgr2=0x5645568a4210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:22:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.991+0000 7fe777fff640 1 --2- 172.21.15.77:0/183246406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568ab060 0x5645568a4210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.991+0000 7fe777fff640 1 -- 172.21.15.77:0/183246406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5ef0 msgr2=0x5645568bf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.991+0000 7fe777fff640 1 --2- 172.21.15.77:0/183246406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5ef0 0x5645568bf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.991+0000 7fe777fff640 1 -- 172.21.15.77:0/183246406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645568aa470 con 0x5645568a5500 2024-03-16T10:22:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:37.992+0000 7fe777fff640 1 --2- 172.21.15.77:0/183246406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 0x5645568a3cd0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe77000f410 tx=0x7fe77000f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b0a98e99d844d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.013+0000 7fe7617fa640 1 -- 172.21.15.77:0/183246406 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe770021040 con 0x5645568a5500 2024-03-16T10:22:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.013+0000 7fe7617fa640 1 -- 172.21.15.77:0/183246406 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe77002f020 con 0x5645568a5500 2024-03-16T10:22:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.013+0000 7fe7617fa640 1 -- 172.21.15.77:0/183246406 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77000d930 con 0x5645568a5500 2024-03-16T10:22:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.014+0000 7fe7807fa740 1 -- 172.21.15.77:0/183246406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 msgr2=0x5645568a3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.014+0000 7fe7807fa740 1 --2- 172.21.15.77:0/183246406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 0x5645568a3cd0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe77000f410 tx=0x7fe77000f810 comp rx=0 tx=0).stop 2024-03-16T10:22:38.019 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.018+0000 7fe7807fa740 1 -- 172.21.15.77:0/183246406 shutdown_connections 2024-03-16T10:22:38.019 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.018+0000 7fe7807fa740 1 --2- 172.21.15.77:0/183246406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568ab060 0x5645568a4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.019 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.018+0000 7fe7807fa740 1 --2- 172.21.15.77:0/183246406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5ef0 0x5645568bf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.019 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.018+0000 7fe7807fa740 1 --2- 172.21.15.77:0/183246406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568a5500 0x5645568a3cd0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.019 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.018+0000 7fe7807fa740 1 -- 172.21.15.77:0/183246406 >> 172.21.15.77:0/183246406 conn(0x5645568a0a70 msgr2=0x5645568a71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.019+0000 7fe7807fa740 1 -- 172.21.15.77:0/183246406 shutdown_connections 2024-03-16T10:22:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.019+0000 7fe7807fa740 1 -- 172.21.15.77:0/183246406 wait complete. 2024-03-16T10:22:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.019+0000 7fe7807fa740 1 -- start start 2024-03-16T10:22:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.020+0000 7fe7807fa740 1 Processor -- start 2024-03-16T10:22:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.020+0000 7fe7807fa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5500 0x5645568bcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.020+0000 7fe7807fa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5ef0 0x5645568bd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.020+0000 7fe7807fa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568ab060 0x5645568be210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.020+0000 7fe7807fa740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c00f7e0 con 0x5645568ab060 2024-03-16T10:22:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.020+0000 7fe7807fa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c00f110 con 0x5645568a5ef0 2024-03-16T10:22:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.020+0000 7fe7807fa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c00c2c0 con 0x5645568a5500 2024-03-16T10:22:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.020+0000 7fe7807fa740 10 client.0 _ll_register_callbacks cb 0x7fe76c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.021+0000 7fe776ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5ef0 0x5645568bd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.021+0000 7fe777fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568ab060 0x5645568be210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.021+0000 7fe777fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568ab060 0x5645568be210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45246/0 (socket says 172.21.15.77:45246) 2024-03-16T10:22:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.021+0000 7fe777fff640 1 -- 172.21.15.77:0/1211942427 learned_addr learned my addr 172.21.15.77:0/1211942427 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.021+0000 7fe777fff640 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5500 0x5645568bcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.021+0000 7fe777fff640 1 -- 172.21.15.77:0/1211942427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5500 msgr2=0x5645568bcd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.021+0000 7fe777fff640 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5500 0x5645568bcd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.021+0000 7fe777fff640 1 -- 172.21.15.77:0/1211942427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5ef0 msgr2=0x5645568bd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.021+0000 7fe777fff640 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5ef0 0x5645568bd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.021+0000 7fe777fff640 1 -- 172.21.15.77:0/1211942427 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645568be800 con 0x5645568ab060 2024-03-16T10:22:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.022+0000 7fe776ffd640 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5ef0 0x5645568bd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.022+0000 7fe777fff640 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568ab060 0x5645568be210 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe770000c00 tx=0x7fe77000d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a6e88dcb425882 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.034+0000 7fe756ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:22:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.034+0000 7fe756ffd640 1 -- 172.21.15.77:0/1211942427 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77000da10 con 0x5645568ab060 2024-03-16T10:22:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.034+0000 7fe7807fa740 1 -- 172.21.15.77:0/1211942427 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe77002f020 con 0x5645568ab060 2024-03-16T10:22:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.034+0000 7fe756ffd640 1 -- 172.21.15.77:0/1211942427 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fe770030040 con 0x5645568ab060 2024-03-16T10:22:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.037+0000 7fe756ffd640 1 -- 172.21.15.77:0/1211942427 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fe770019a80 con 0x5645568ab060 2024-03-16T10:22:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.037+0000 7fe7807fa740 10 client.16965 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.037+0000 7fe7807fa740 1 -- 172.21.15.77:0/1211942427 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56455693b020 con 0x5645568ab060 2024-03-16T10:22:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.038+0000 7fe756ffd640 1 -- 172.21.15.77:0/1211942427 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe77002f020 con 0x5645568ab060 2024-03-16T10:22:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.038+0000 7fe756ffd640 1 -- 172.21.15.77:0/1211942427 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fe770027040 con 0x5645568ab060 2024-03-16T10:22:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.039+0000 7fe7807fa740 10 client.16965 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.039+0000 7fe7807fa740 10 client.16965 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.039+0000 7fe7807fa740 20 client.16965 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.039+0000 7fe7807fa740 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645568c3ca0 0x56455693d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.039+0000 7fe7807fa740 4 client.16965 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:22:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.039+0000 7fe7807fa740 1 -- 172.21.15.77:0/1211942427 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5645568122d0 con 0x5645568c3ca0 2024-03-16T10:22:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.039+0000 7fe776ffd640 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645568c3ca0 0x56455693d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.040+0000 7fe776ffd640 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645568c3ca0 0x56455693d720 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba75579771c0a4be server_cookie=97d176f6126eac61 in_seq=0 out_seq=0 2024-03-16T10:22:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.040+0000 7fe756ffd640 10 client.16965 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.063+0000 7fe756ffd640 1 -- 172.21.15.77:0/1211942427 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5645568122d0 con 0x5645568c3ca0 2024-03-16T10:22:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.063+0000 7fe756ffd640 10 client.16965 handle_command_reply: tid=0 2024-03-16T10:22:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.063+0000 7fe7807fa740 1 client.16965 shutdown 2024-03-16T10:22:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.064+0000 7fe7807fa740 1 -- 172.21.15.77:0/1211942427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568ab060 msgr2=0x5645568be210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.064+0000 7fe7807fa740 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568ab060 0x5645568be210 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe770000c00 tx=0x7fe77000d660 comp rx=0 tx=0).stop 2024-03-16T10:22:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.064+0000 7fe7807fa740 1 -- 172.21.15.77:0/1211942427 shutdown_connections 2024-03-16T10:22:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.064+0000 7fe7807fa740 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5645568c3ca0 0x56455693d720 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.064+0000 7fe7807fa740 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645568ab060 0x5645568be210 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.064+0000 7fe7807fa740 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5645568a5ef0 0x5645568bd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.064+0000 7fe7807fa740 1 --2- 172.21.15.77:0/1211942427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645568a5500 0x5645568bcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.064+0000 7fe7807fa740 1 -- 172.21.15.77:0/1211942427 >> 172.21.15.77:0/1211942427 conn(0x5645568a0a70 msgr2=0x5645568a71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.064+0000 7fe756ffd640 0 client.16965 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.065+0000 7fe7807fa740 1 -- 172.21.15.77:0/1211942427 shutdown_connections 2024-03-16T10:22:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.065+0000 7fe7807fa740 1 -- 172.21.15.77:0/1211942427 wait complete. 2024-03-16T10:22:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.065+0000 7fe7807fa740 20 client.16965 trim_cache size 0 max 16384 2024-03-16T10:22:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:22:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:22:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:22:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:22:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.086+0000 7fe77db37640 1 -- 172.21.15.77:0/652242456 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe7680a28b0 msgr2=0x7fe7680a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.086+0000 7fe77db37640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe7680a28b0 0x7fe7680a4d70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0a3cd0 tx=0x7fe76c0d2b10 comp rx=0 tx=0).stop 2024-03-16T10:22:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.086+0000 7fe77db37640 1 -- 172.21.15.77:0/652242456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7780ffcb0 msgr2=0x7fe778199c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.086+0000 7fe77db37640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7780ffcb0 0x7fe778199c10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00fb90 tx=0x7fe76c00d7b0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.088+0000 7fe77db37640 1 -- 172.21.15.77:0/652242456 shutdown_connections 2024-03-16T10:22:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.088+0000 7fe77db37640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe7680a28b0 0x7fe7680a4d70 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.088+0000 7fe77db37640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe778101860 0x7fe7781a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.088+0000 7fe77db37640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe778100ea0 0x7fe77819a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.088+0000 7fe77db37640 1 --2- 172.21.15.77:0/652242456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7780ffcb0 0x7fe778199c10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.088+0000 7fe77db37640 1 -- 172.21.15.77:0/652242456 >> 172.21.15.77:0/652242456 conn(0x7fe7780fb220 msgr2=0x7fe7781907b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.088+0000 7fe77db37640 1 -- 172.21.15.77:0/652242456 shutdown_connections 2024-03-16T10:22:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:38.089+0000 7fe77db37640 1 -- 172.21.15.77:0/652242456 wait complete. 2024-03-16T10:22:38.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:37 smithi192 ceph-mon[36354]: pgmap v516: 97 pgs: 97 active+clean; 671 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.7 MiB/s wr, 546 op/s 2024-03-16T10:22:38.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:37 smithi077 ceph-mon[32400]: pgmap v516: 97 pgs: 97 active+clean; 671 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.7 MiB/s wr, 546 op/s 2024-03-16T10:22:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:37 smithi193 ceph-mon[36311]: pgmap v516: 97 pgs: 97 active+clean; 671 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.7 MiB/s wr, 546 op/s 2024-03-16T10:22:40.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:39 smithi192 ceph-mon[36354]: pgmap v517: 97 pgs: 97 active+clean; 675 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.1 MiB/s wr, 575 op/s 2024-03-16T10:22:40.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:39 smithi077 ceph-mon[32400]: pgmap v517: 97 pgs: 97 active+clean; 675 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.1 MiB/s wr, 575 op/s 2024-03-16T10:22:40.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:39 smithi193 ceph-mon[36311]: pgmap v517: 97 pgs: 97 active+clean; 675 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.1 MiB/s wr, 575 op/s 2024-03-16T10:22:41.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:22:41.825 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:22:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.966+0000 7f654aa81640 1 -- 172.21.15.77:0/643743999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65441022d0 msgr2=0x7f6544108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.966+0000 7f654aa81640 1 --2- 172.21.15.77:0/643743999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65441022d0 0x7f6544108b60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f653c009ca0 tx=0x7f653c02c120 comp rx=0 tx=0).stop 2024-03-16T10:22:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.966+0000 7f654aa81640 1 -- 172.21.15.77:0/643743999 shutdown_connections 2024-03-16T10:22:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.966+0000 7f654aa81640 1 --2- 172.21.15.77:0/643743999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65441022d0 0x7f6544108b60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.966+0000 7f654aa81640 1 --2- 172.21.15.77:0/643743999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6544101910 0x7f6544101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.966+0000 7f654aa81640 1 --2- 172.21.15.77:0/643743999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6544100720 0x7f6544100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.966+0000 7f654aa81640 1 -- 172.21.15.77:0/643743999 >> 172.21.15.77:0/643743999 conn(0x7f65440fbc90 msgr2=0x7f65440fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.967+0000 7f654aa81640 1 -- 172.21.15.77:0/643743999 shutdown_connections 2024-03-16T10:22:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.967+0000 7f654aa81640 1 -- 172.21.15.77:0/643743999 wait complete. 2024-03-16T10:22:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.968+0000 7f654aa81640 1 Processor -- start 2024-03-16T10:22:41.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.968+0000 7f654aa81640 1 -- start start 2024-03-16T10:22:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.968+0000 7f654aa81640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6544100720 0x7f6544196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:41.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.968+0000 7f654aa81640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6544101910 0x7f6544196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.969+0000 7f654881d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6544100720 0x7f6544196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.969+0000 7f6543fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6544101910 0x7f6544196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:41.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.969+0000 7f654881d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6544100720 0x7f6544196840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:51998/0 (socket says 172.21.15.77:51998) 2024-03-16T10:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.969+0000 7f6543fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6544101910 0x7f6544196d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:48416/0 (socket says 172.21.15.77:48416) 2024-03-16T10:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.969+0000 7f654aa81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65441022d0 0x7f654419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.969+0000 7f654901e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65441022d0 0x7f654419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.969+0000 7f654901e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65441022d0 0x7f654419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40750/0 (socket says 172.21.15.77:40750) 2024-03-16T10:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.969+0000 7f654881d640 1 -- 172.21.15.77:0/1894848854 learned_addr learned my addr 172.21.15.77:0/1894848854 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.970+0000 7f654aa81640 1 -- 172.21.15.77:0/1894848854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f654410c380 con 0x7f65441022d0 2024-03-16T10:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.970+0000 7f654aa81640 1 -- 172.21.15.77:0/1894848854 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f654410c200 con 0x7f6544101910 2024-03-16T10:22:41.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.970+0000 7f654aa81640 1 -- 172.21.15.77:0/1894848854 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f654410c500 con 0x7f6544100720 2024-03-16T10:22:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.970+0000 7f654881d640 1 -- 172.21.15.77:0/1894848854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6544101910 msgr2=0x7f6544196d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.970+0000 7f654881d640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6544101910 0x7f6544196d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.970+0000 7f654881d640 1 -- 172.21.15.77:0/1894848854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65441022d0 msgr2=0x7f654419ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:41.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.970+0000 7f654881d640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65441022d0 0x7f654419ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.970+0000 7f654881d640 1 -- 172.21.15.77:0/1894848854 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654419e460 con 0x7f6544100720 2024-03-16T10:22:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.970+0000 7f654901e640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65441022d0 0x7f654419ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.971+0000 7f654881d640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6544100720 0x7f6544196840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f653800e9c0 tx=0x7f653800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.971+0000 7f6541ffb640 1 -- 172.21.15.77:0/1894848854 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653800cd80 con 0x7f6544100720 2024-03-16T10:22:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.971+0000 7f6541ffb640 1 -- 172.21.15.77:0/1894848854 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65380035f0 con 0x7f6544100720 2024-03-16T10:22:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.971+0000 7f654aa81640 1 -- 172.21.15.77:0/1894848854 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6544069b60 con 0x7f6544100720 2024-03-16T10:22:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.971+0000 7f6541ffb640 1 -- 172.21.15.77:0/1894848854 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538010690 con 0x7f6544100720 2024-03-16T10:22:41.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.971+0000 7f654aa81640 1 -- 172.21.15.77:0/1894848854 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f654406a050 con 0x7f6544100720 2024-03-16T10:22:41.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.973+0000 7f6541ffb640 1 -- 172.21.15.77:0/1894848854 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f6538004020 con 0x7f6544100720 2024-03-16T10:22:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.974+0000 7f6541ffb640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65340a2800 0x7f65340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.974+0000 7f6541ffb640 1 -- 172.21.15.77:0/1894848854 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f65380cedd0 con 0x7f6544100720 2024-03-16T10:22:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f6543fff640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65340a2800 0x7f65340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f654d66e740 1 Processor -- start 2024-03-16T10:22:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f654d66e740 1 -- start start 2024-03-16T10:22:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f654d66e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8170 0x5622a07f8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f654d66e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8b60 0x5622a07fd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f654d66e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 0x5622a0800160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f654d66e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622a075d500 con 0x5622a07fdcd0 2024-03-16T10:22:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f654d66e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5622a0705be0 con 0x5622a07f8170 2024-03-16T10:22:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f654d66e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5622a07595a0 con 0x5622a07f8b60 2024-03-16T10:22:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f654881d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8170 0x5622a07f8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f654901e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 0x5622a0800160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f654901e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 0x5622a0800160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40754/0 (socket says 172.21.15.77:40754) 2024-03-16T10:22:41.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.975+0000 7f654901e640 1 -- 172.21.15.77:0/3518626941 learned_addr learned my addr 172.21.15.77:0/3518626941 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.976+0000 7f654881d640 1 -- 172.21.15.77:0/3518626941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8b60 msgr2=0x5622a07fd660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:22:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.976+0000 7f654881d640 1 --2- 172.21.15.77:0/3518626941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8b60 0x5622a07fd660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.976+0000 7f654881d640 1 -- 172.21.15.77:0/3518626941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 msgr2=0x5622a0800160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.976+0000 7f654881d640 1 --2- 172.21.15.77:0/3518626941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 0x5622a0800160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.976+0000 7f654881d640 1 -- 172.21.15.77:0/3518626941 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622a07194b0 con 0x5622a07f8170 2024-03-16T10:22:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.976+0000 7f654901e640 1 --2- 172.21.15.77:0/3518626941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 0x5622a0800160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.976+0000 7f6543fff640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65340a2800 0x7f65340a4cc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f653000d440 tx=0x7f653000d5a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.976+0000 7f654881d640 1 --2- 172.21.15.77:0/3518626941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8170 0x5622a07f8590 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f65380cc990 tx=0x7f653809b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e103b76ff207c7b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.976+0000 7f652effd640 1 -- 172.21.15.77:0/3518626941 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65380b4070 con 0x5622a07f8170 2024-03-16T10:22:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.976+0000 7f652effd640 1 -- 172.21.15.77:0/3518626941 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f653809f930 con 0x5622a07f8170 2024-03-16T10:22:41.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.976+0000 7f652effd640 1 -- 172.21.15.77:0/3518626941 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538097e50 con 0x5622a07f8170 2024-03-16T10:22:41.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.977+0000 7f654d66e740 1 -- 172.21.15.77:0/3518626941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8170 msgr2=0x5622a07f8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:41.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.977+0000 7f654d66e740 1 --2- 172.21.15.77:0/3518626941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8170 0x5622a07f8590 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f65380cc990 tx=0x7f653809b6e0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.978+0000 7f654d66e740 1 -- 172.21.15.77:0/3518626941 shutdown_connections 2024-03-16T10:22:41.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.978+0000 7f654d66e740 1 --2- 172.21.15.77:0/3518626941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 0x5622a0800160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.978+0000 7f654d66e740 1 --2- 172.21.15.77:0/3518626941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8b60 0x5622a07fd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.978+0000 7f654d66e740 1 --2- 172.21.15.77:0/3518626941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8170 0x5622a07f8590 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.978+0000 7f654d66e740 1 -- 172.21.15.77:0/3518626941 >> 172.21.15.77:0/3518626941 conn(0x5622a07f36e0 msgr2=0x5622a07f3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:41.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.978+0000 7f654d66e740 1 -- 172.21.15.77:0/3518626941 shutdown_connections 2024-03-16T10:22:41.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.978+0000 7f654d66e740 1 -- 172.21.15.77:0/3518626941 wait complete. 2024-03-16T10:22:41.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.978+0000 7f654d66e740 1 -- start start 2024-03-16T10:22:41.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.978+0000 7f654d66e740 1 Processor -- start 2024-03-16T10:22:41.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.979+0000 7f654d66e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07f8170 0x5622a080af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.979+0000 7f654d66e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8b60 0x5622a080b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.979+0000 7f654d66e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07fdcd0 0x5622a08124f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.979+0000 7f654d66e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622a0800c20 con 0x5622a07f8170 2024-03-16T10:22:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.979+0000 7f6543fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8b60 0x5622a080b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f6543fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8b60 0x5622a080b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52008/0 (socket says 172.21.15.77:52008) 2024-03-16T10:22:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f6543fff640 1 -- 172.21.15.77:0/368536548 learned_addr learned my addr 172.21.15.77:0/368536548 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f654901e640 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07fdcd0 0x5622a08124f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f654881d640 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07f8170 0x5622a080af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f654d66e740 1 -- 172.21.15.77:0/368536548 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5622a0719160 con 0x5622a07fdcd0 2024-03-16T10:22:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f654d66e740 1 -- 172.21.15.77:0/368536548 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5622a0800da0 con 0x5622a07f8b60 2024-03-16T10:22:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f654d66e740 10 client.0 _ll_register_callbacks cb 0x5622a06fd150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f6543fff640 1 -- 172.21.15.77:0/368536548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07fdcd0 msgr2=0x5622a08124f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f6543fff640 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07fdcd0 0x5622a08124f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f6543fff640 1 -- 172.21.15.77:0/368536548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07f8170 msgr2=0x5622a080af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f6543fff640 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07f8170 0x5622a080af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f6543fff640 1 -- 172.21.15.77:0/368536548 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622a053b7d0 con 0x5622a07f8b60 2024-03-16T10:22:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f654901e640 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07fdcd0 0x5622a08124f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.980+0000 7f654881d640 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07f8170 0x5622a080af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.981+0000 7f6543fff640 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8b60 0x5622a080b4b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f653001a8b0 tx=0x7f65300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c4064d99e3b20d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.981+0000 7f6523fff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:22:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.981+0000 7f6523fff640 1 -- 172.21.15.77:0/368536548 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653001ae30 con 0x5622a07f8b60 2024-03-16T10:22:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.981+0000 7f6523fff640 1 -- 172.21.15.77:0/368536548 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f6530005020 con 0x5622a07f8b60 2024-03-16T10:22:41.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.981+0000 7f654d66e740 1 -- 172.21.15.77:0/368536548 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f653809f930 con 0x5622a07f8b60 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.981+0000 7f6523fff640 1 -- 172.21.15.77:0/368536548 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6530005220 con 0x5622a07f8b60 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.981+0000 7f654d66e740 10 client.26737 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.981+0000 7f654d66e740 1 -- 172.21.15.77:0/368536548 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5622a04713f0 con 0x5622a07f8b60 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.982+0000 7f6523fff640 1 -- 172.21.15.77:0/368536548 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6530004c80 con 0x5622a07f8b60 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.982+0000 7f6523fff640 1 -- 172.21.15.77:0/368536548 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f6530017070 con 0x5622a07f8b60 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.982+0000 7f654d66e740 10 client.26737 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.982+0000 7f654d66e740 10 client.26737 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.982+0000 7f654d66e740 20 client.26737 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.982+0000 7f654d66e740 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5622a0816900 0x5622a0818d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.982+0000 7f654d66e740 4 client.26737 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.982+0000 7f654d66e740 1 -- 172.21.15.77:0/368536548 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f653809f930 con 0x5622a0816900 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.983+0000 7f654881d640 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5622a0816900 0x5622a0818d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:41.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.983+0000 7f6523fff640 10 client.26737 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.983+0000 7f654881d640 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5622a0816900 0x5622a0818d80 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44c21711b1e9ecaf server_cookie=3b8eef32dbb8fe22 in_seq=0 out_seq=0 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.984+0000 7f6523fff640 1 -- 172.21.15.77:0/368536548 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f653809f930 con 0x5622a0816900 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.984+0000 7f6523fff640 10 client.26737 handle_command_reply: tid=0 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.984+0000 7f654d66e740 1 client.26737 shutdown 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.984+0000 7f654d66e740 1 -- 172.21.15.77:0/368536548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8b60 msgr2=0x5622a080b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.984+0000 7f654d66e740 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8b60 0x5622a080b4b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f653001a8b0 tx=0x7f65300046c0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.985+0000 7f654d66e740 1 -- 172.21.15.77:0/368536548 shutdown_connections 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.985+0000 7f654d66e740 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07fdcd0 0x5622a08124f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.985+0000 7f654d66e740 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8b60 0x5622a080b4b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.985+0000 7f654d66e740 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5622a0816900 0x5622a0818d80 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.985+0000 7f6523fff640 0 client.26737 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.985+0000 7f654d66e740 1 --2- 172.21.15.77:0/368536548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07f8170 0x5622a080af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.985+0000 7f654d66e740 1 -- 172.21.15.77:0/368536548 >> 172.21.15.77:0/368536548 conn(0x5622a07f36e0 msgr2=0x5622a07f2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.985+0000 7f654d66e740 1 -- 172.21.15.77:0/368536548 shutdown_connections 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.985+0000 7f654d66e740 1 -- 172.21.15.77:0/368536548 wait complete. 2024-03-16T10:22:41.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:41.985+0000 7f654d66e740 20 client.26737 trim_cache size 0 max 16384 2024-03-16T10:22:42.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.007+0000 7f654d66e740 1 Processor -- start 2024-03-16T10:22:42.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.007+0000 7f654d66e740 1 -- start start 2024-03-16T10:22:42.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.007+0000 7f654d66e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07f8170 0x5622a08099d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:42.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.007+0000 7f654d66e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 0x5622a08124b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:42.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.007+0000 7f654d66e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07fdcd0 0x5622a07f2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:42.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f654d66e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f65441a6880 con 0x5622a07f8170 2024-03-16T10:22:42.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f654d66e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f65441a8450 con 0x5622a07f8b60 2024-03-16T10:22:42.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f654881d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07f8170 0x5622a08099d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:42.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f6543fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 0x5622a08124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:42.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f6543fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 0x5622a08124b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:48436/0 (socket says 172.21.15.77:48436) 2024-03-16T10:22:42.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f6543fff640 1 -- 172.21.15.77:0/2870283709 learned_addr learned my addr 172.21.15.77:0/2870283709 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:42.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f654d66e740 1 -- 172.21.15.77:0/2870283709 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f65441a6f50 con 0x5622a07fdcd0 2024-03-16T10:22:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f654901e640 1 --2- 172.21.15.77:0/2870283709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07fdcd0 0x5622a07f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f6543fff640 1 -- 172.21.15.77:0/2870283709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07fdcd0 msgr2=0x5622a07f2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f6543fff640 1 --2- 172.21.15.77:0/2870283709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07fdcd0 0x5622a07f2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f6543fff640 1 -- 172.21.15.77:0/2870283709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07f8170 msgr2=0x5622a08099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f6543fff640 1 --2- 172.21.15.77:0/2870283709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07f8170 0x5622a08099d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f6543fff640 1 -- 172.21.15.77:0/2870283709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622a04310f0 con 0x5622a07f8b60 2024-03-16T10:22:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.008+0000 7f6543fff640 1 --2- 172.21.15.77:0/2870283709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 0x5622a08124b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f653000da40 tx=0x7f653000da70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=619ae030777f0a3c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:42.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.009+0000 7f652e7fc640 1 -- 172.21.15.77:0/2870283709 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653003a070 con 0x5622a07f8b60 2024-03-16T10:22:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.009+0000 7f652e7fc640 1 -- 172.21.15.77:0/2870283709 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f653001ae30 con 0x5622a07f8b60 2024-03-16T10:22:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.009+0000 7f652e7fc640 1 -- 172.21.15.77:0/2870283709 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6530004960 con 0x5622a07f8b60 2024-03-16T10:22:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.010+0000 7f654d66e740 1 -- 172.21.15.77:0/2870283709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 msgr2=0x5622a08124b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.010+0000 7f654d66e740 1 --2- 172.21.15.77:0/2870283709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 0x5622a08124b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f653000da40 tx=0x7f653000da70 comp rx=0 tx=0).stop 2024-03-16T10:22:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.010+0000 7f654d66e740 1 -- 172.21.15.77:0/2870283709 shutdown_connections 2024-03-16T10:22:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.010+0000 7f654d66e740 1 --2- 172.21.15.77:0/2870283709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07fdcd0 0x5622a07f2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.010+0000 7f654d66e740 1 --2- 172.21.15.77:0/2870283709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 0x5622a08124b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.010+0000 7f654d66e740 1 --2- 172.21.15.77:0/2870283709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07f8170 0x5622a08099d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.010+0000 7f654d66e740 1 -- 172.21.15.77:0/2870283709 >> 172.21.15.77:0/2870283709 conn(0x5622a07f36e0 msgr2=0x5622a080af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.011+0000 7f654d66e740 1 -- 172.21.15.77:0/2870283709 shutdown_connections 2024-03-16T10:22:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.011+0000 7f654d66e740 1 -- 172.21.15.77:0/2870283709 wait complete. 2024-03-16T10:22:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.011+0000 7f654d66e740 1 -- start start 2024-03-16T10:22:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.011+0000 7f654d66e740 1 Processor -- start 2024-03-16T10:22:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.012+0000 7f654d66e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8170 0x5622a080f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.012+0000 7f654d66e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 0x5622a080f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.012+0000 7f654d66e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 0x5622a087acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.012+0000 7f654881d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8170 0x5622a080f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:42.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f654881d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8170 0x5622a080f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52026/0 (socket says 172.21.15.77:52026) 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f654d66e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f65441a7d80 con 0x5622a07fdcd0 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f6543fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 0x5622a080f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f6543fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 0x5622a080f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:48452/0 (socket says 172.21.15.77:48452) 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f6543fff640 1 -- 172.21.15.77:0/994894876 learned_addr learned my addr 172.21.15.77:0/994894876 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f654d66e740 1 -- 172.21.15.77:0/994894876 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f65441a76b0 con 0x5622a07f8b60 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f654d66e740 1 -- 172.21.15.77:0/994894876 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f65441a4860 con 0x5622a07f8170 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f654901e640 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 0x5622a087acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f654d66e740 10 client.0 _ll_register_callbacks cb 0x7f653000d8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f6543fff640 1 -- 172.21.15.77:0/994894876 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8170 msgr2=0x5622a080f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f6543fff640 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8170 0x5622a080f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f6543fff640 1 -- 172.21.15.77:0/994894876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 msgr2=0x5622a087acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f6543fff640 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 0x5622a087acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f6543fff640 1 -- 172.21.15.77:0/994894876 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622a07249d0 con 0x5622a07f8b60 2024-03-16T10:22:42.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.013+0000 7f654901e640 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 0x5622a087acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:42.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.014+0000 7f6543fff640 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 0x5622a080f700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f653000db30 tx=0x7f653000dfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40784393ad594bdd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:42.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.014+0000 7f6523fff640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:22:42.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.027+0000 7f6523fff640 1 -- 172.21.15.77:0/994894876 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653003a070 con 0x5622a07f8b60 2024-03-16T10:22:42.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.027+0000 7f6523fff640 1 -- 172.21.15.77:0/994894876 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f6530032040 con 0x5622a07f8b60 2024-03-16T10:22:42.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.028+0000 7f654d66e740 1 -- 172.21.15.77:0/994894876 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f653001ae30 con 0x5622a07f8b60 2024-03-16T10:22:42.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.034+0000 7f6523fff640 1 -- 172.21.15.77:0/994894876 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6530004ab0 con 0x5622a07f8b60 2024-03-16T10:22:42.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.034+0000 7f654d66e740 10 client.26762 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:42.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.034+0000 7f654d66e740 1 -- 172.21.15.77:0/994894876 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5622a087bfe0 con 0x5622a07f8b60 2024-03-16T10:22:42.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.034+0000 7f6523fff640 1 -- 172.21.15.77:0/994894876 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f653001ae30 con 0x5622a07f8b60 2024-03-16T10:22:42.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.034+0000 7f6523fff640 1 -- 172.21.15.77:0/994894876 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f6530017050 con 0x5622a07f8b60 2024-03-16T10:22:42.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.034+0000 7f654d66e740 10 client.26762 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:42.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.034+0000 7f654d66e740 10 client.26762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:42.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.034+0000 7f654d66e740 20 client.26762 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:42.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.034+0000 7f654d66e740 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5622a087c950 0x5622a087ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:42.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.034+0000 7f654d66e740 4 client.26762 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:22:42.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.034+0000 7f654d66e740 1 -- 172.21.15.77:0/994894876 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5622a076aa40 con 0x5622a087c950 2024-03-16T10:22:42.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.035+0000 7f654881d640 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5622a087c950 0x5622a087ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.035+0000 7f6523fff640 10 client.26762 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.035+0000 7f654881d640 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5622a087c950 0x5622a087ed70 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa030bc9b3ec4053 server_cookie=f00c5d3b374bec6b in_seq=0 out_seq=0 2024-03-16T10:22:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.035+0000 7f6523fff640 1 -- 172.21.15.77:0/994894876 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5622a076aa40 con 0x5622a087c950 2024-03-16T10:22:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.035+0000 7f6523fff640 10 client.26762 handle_command_reply: tid=0 2024-03-16T10:22:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f654d66e740 1 client.26762 shutdown 2024-03-16T10:22:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f654d66e740 1 -- 172.21.15.77:0/994894876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 msgr2=0x5622a080f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f654d66e740 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 0x5622a080f700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f653000db30 tx=0x7f653000dfa0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f654d66e740 1 -- 172.21.15.77:0/994894876 shutdown_connections 2024-03-16T10:22:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f654d66e740 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5622a087c950 0x5622a087ed70 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f654d66e740 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a07fdcd0 0x5622a087acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f654d66e740 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622a07f8b60 0x5622a080f700 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f654d66e740 1 --2- 172.21.15.77:0/994894876 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622a07f8170 0x5622a080f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f654d66e740 1 -- 172.21.15.77:0/994894876 >> 172.21.15.77:0/994894876 conn(0x5622a07f36e0 msgr2=0x5622a07f6940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f6523fff640 0 client.26762 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f654d66e740 1 -- 172.21.15.77:0/994894876 shutdown_connections 2024-03-16T10:22:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f654d66e740 1 -- 172.21.15.77:0/994894876 wait complete. 2024-03-16T10:22:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.036+0000 7f654d66e740 20 client.26762 trim_cache size 0 max 16384 2024-03-16T10:22:42.039 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:22:42.039 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:22:42.039 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:22:42.039 INFO:teuthology.orchestra.run.smithi077.stdout: "1f1adeef-2126-4ee3-aca7-819aadf3baf6": { 2024-03-16T10:22:42.039 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:22:42.039 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "1f1adeef-2126-4ee3-aca7-819aadf3baf6", 2024-03-16T10:22:42.039 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:22:42.039 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:22:42.039 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:22:42.039 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:22:42.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.037+0000 7f654aa81640 1 -- 172.21.15.77:0/1894848854 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65340a2800 msgr2=0x7f65340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:42.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.037+0000 7f654aa81640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65340a2800 0x7f65340a4cc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f653000d440 tx=0x7f653000d5a0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.038+0000 7f654aa81640 1 -- 172.21.15.77:0/1894848854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6544100720 msgr2=0x7f6544196840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:42.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.038+0000 7f654aa81640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6544100720 0x7f6544196840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f653800e9c0 tx=0x7f653800ee80 comp rx=0 tx=0).stop 2024-03-16T10:22:42.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.039+0000 7f654aa81640 1 -- 172.21.15.77:0/1894848854 shutdown_connections 2024-03-16T10:22:42.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.039+0000 7f654aa81640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65441022d0 0x7f654419ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.039+0000 7f654aa81640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6544101910 0x7f6544196d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.039+0000 7f654aa81640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f65340a2800 0x7f65340a4cc0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.039+0000 7f654aa81640 1 --2- 172.21.15.77:0/1894848854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6544100720 0x7f6544196840 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:42.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.039+0000 7f654aa81640 1 -- 172.21.15.77:0/1894848854 >> 172.21.15.77:0/1894848854 conn(0x7f65440fbc90 msgr2=0x7f6544104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:42.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.039+0000 7f654aa81640 1 -- 172.21.15.77:0/1894848854 shutdown_connections 2024-03-16T10:22:42.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:22:42.039+0000 7f654aa81640 1 -- 172.21.15.77:0/1894848854 wait complete. 2024-03-16T10:22:42.054 INFO:tasks.cephfs.filesystem:scrub status for tag:1f1adeef-2126-4ee3-aca7-819aadf3baf6 - {'path': '/', 'tag': '1f1adeef-2126-4ee3-aca7-819aadf3baf6', 'options': 'recursive,force'} 2024-03-16T10:22:42.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:41 smithi192 ceph-mon[36354]: pgmap v518: 97 pgs: 97 active+clean; 678 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 584 op/s 2024-03-16T10:22:42.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:41 smithi077 ceph-mon[32400]: pgmap v518: 97 pgs: 97 active+clean; 678 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 584 op/s 2024-03-16T10:22:42.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:41 smithi193 ceph-mon[36311]: pgmap v518: 97 pgs: 97 active+clean; 678 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 584 op/s 2024-03-16T10:22:44.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:43 smithi192 ceph-mon[36354]: pgmap v519: 97 pgs: 97 active+clean; 680 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 559 op/s 2024-03-16T10:22:44.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:43 smithi077 ceph-mon[32400]: pgmap v519: 97 pgs: 97 active+clean; 680 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 559 op/s 2024-03-16T10:22:44.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:43 smithi193 ceph-mon[36311]: pgmap v519: 97 pgs: 97 active+clean; 680 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 559 op/s 2024-03-16T10:22:46.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:45 smithi192 ceph-mon[36354]: pgmap v520: 97 pgs: 97 active+clean; 680 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 387 op/s 2024-03-16T10:22:46.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:45 smithi077 ceph-mon[32400]: pgmap v520: 97 pgs: 97 active+clean; 680 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 387 op/s 2024-03-16T10:22:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:45 smithi193 ceph-mon[36311]: pgmap v520: 97 pgs: 97 active+clean; 680 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 387 op/s 2024-03-16T10:22:47.105 INFO:tasks.workunit.client.0.smithi077.stdout: 144 2031311 292 1355901 464 1144955 996 2024-03-16T10:22:48.168 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:47 smithi192 ceph-mon[36354]: pgmap v521: 97 pgs: 97 active+clean; 678 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 315 op/s 2024-03-16T10:22:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:47 smithi193 ceph-mon[36311]: pgmap v521: 97 pgs: 97 active+clean; 678 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 315 op/s 2024-03-16T10:22:48.254 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:47 smithi077 ceph-mon[32400]: pgmap v521: 97 pgs: 97 active+clean; 678 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 315 op/s 2024-03-16T10:22:50.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:49 smithi192 ceph-mon[36354]: pgmap v522: 97 pgs: 97 active+clean; 683 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 282 op/s 2024-03-16T10:22:50.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:49 smithi077 ceph-mon[32400]: pgmap v522: 97 pgs: 97 active+clean; 683 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 282 op/s 2024-03-16T10:22:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:49 smithi193 ceph-mon[36311]: pgmap v522: 97 pgs: 97 active+clean; 683 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 282 op/s 2024-03-16T10:22:51.506 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.505+0000 7fcfa71cd640 1 -- 172.21.15.77:0/221130006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfa0109890 msgr2=0x7fcfa0109cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:51.506 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.505+0000 7fcfa71cd640 1 --2- 172.21.15.77:0/221130006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfa0109890 0x7fcfa0109cf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800adb0 tx=0x7fcf9802cff0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.506 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.505+0000 7fcfa71cd640 1 -- 172.21.15.77:0/221130006 shutdown_connections 2024-03-16T10:22:51.506 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.505+0000 7fcfa71cd640 1 --2- 172.21.15.77:0/221130006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa010a230 0x7fcfa0110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.505+0000 7fcfa71cd640 1 --2- 172.21.15.77:0/221130006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfa0109890 0x7fcfa0109cf0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.505+0000 7fcfa71cd640 1 --2- 172.21.15.77:0/221130006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa0108b80 0x7fcfa0108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.505+0000 7fcfa71cd640 1 -- 172.21.15.77:0/221130006 >> 172.21.15.77:0/221130006 conn(0x7fcfa006e820 msgr2=0x7fcfa0070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.518+0000 7fcfa71cd640 1 -- 172.21.15.77:0/221130006 shutdown_connections 2024-03-16T10:22:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.518+0000 7fcfa71cd640 1 -- 172.21.15.77:0/221130006 wait complete. 2024-03-16T10:22:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcfa71cd640 1 Processor -- start 2024-03-16T10:22:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcfa71cd640 1 -- start start 2024-03-16T10:22:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcfa71cd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfa0108b80 0x7fcfa01a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcfa71cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa0109890 0x7fcfa01a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcfa71cd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa010a230 0x7fcfa01a7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcfa71cd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa0113ac0 con 0x7fcfa0109890 2024-03-16T10:22:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcfa71cd640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa0113940 con 0x7fcfa0108b80 2024-03-16T10:22:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcfa71cd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa0113c40 con 0x7fcfa010a230 2024-03-16T10:22:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcf9ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa0109890 0x7fcfa01a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcfa4f69640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfa0108b80 0x7fcfa01a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcfa4f69640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfa0108b80 0x7fcfa01a02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:48476/0 (socket says 172.21.15.77:48476) 2024-03-16T10:22:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcfa4f69640 1 -- 172.21.15.77:0/1305134143 learned_addr learned my addr 172.21.15.77:0/1305134143 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.519+0000 7fcfa576a640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa010a230 0x7fcfa01a7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.520+0000 7fcfa576a640 1 -- 172.21.15.77:0/1305134143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfa0108b80 msgr2=0x7fcfa01a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.520+0000 7fcfa576a640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfa0108b80 0x7fcfa01a02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.520+0000 7fcfa576a640 1 -- 172.21.15.77:0/1305134143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa0109890 msgr2=0x7fcfa01a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.520+0000 7fcfa576a640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa0109890 0x7fcfa01a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.520+0000 7fcfa576a640 1 -- 172.21.15.77:0/1305134143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa019dcd0 con 0x7fcfa010a230 2024-03-16T10:22:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.520+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa0109890 0x7fcfa01a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.520+0000 7fcfa576a640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa010a230 0x7fcfa01a7850 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400d6b0 tx=0x7fcf9400db70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:51.526 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.526+0000 7fcf9dffb640 1 -- 172.21.15.77:0/1305134143 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf94017070 con 0x7fcfa010a230 2024-03-16T10:22:51.526 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.526+0000 7fcfa71cd640 1 -- 172.21.15.77:0/1305134143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa019dfc0 con 0x7fcfa010a230 2024-03-16T10:22:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.526+0000 7fcfa71cd640 1 -- 172.21.15.77:0/1305134143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa019e4e0 con 0x7fcfa010a230 2024-03-16T10:22:51.528 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.527+0000 7fcf9dffb640 1 -- 172.21.15.77:0/1305134143 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf94009d00 con 0x7fcfa010a230 2024-03-16T10:22:51.528 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.527+0000 7fcf9dffb640 1 -- 172.21.15.77:0/1305134143 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf94003d60 con 0x7fcfa010a230 2024-03-16T10:22:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.531+0000 7fcf9dffb640 1 -- 172.21.15.77:0/1305134143 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fcf94010460 con 0x7fcfa010a230 2024-03-16T10:22:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.531+0000 7fcfa9db0740 1 Processor -- start 2024-03-16T10:22:51.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.532+0000 7fcfa9db0740 1 -- start start 2024-03-16T10:22:51.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.532+0000 7fcfa9db0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 0x564a11550760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.532+0000 7fcfa9db0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a11550d30 0x564a11555830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.532+0000 7fcfa9db0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11555ea0 0x564a11558330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.532+0000 7fcfa9db0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564a114b5a40 con 0x564a11550340 2024-03-16T10:22:51.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.532+0000 7fcfa9db0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564a1145e920 con 0x564a11555ea0 2024-03-16T10:22:51.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.532+0000 7fcfa9db0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564a114b1ae0 con 0x564a11550d30 2024-03-16T10:22:51.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.533+0000 7fcf9ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 0x564a11550760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.533+0000 7fcfa4f69640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a11550d30 0x564a11555830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.533+0000 7fcfa4f69640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a11550d30 0x564a11555830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:52058/0 (socket says 172.21.15.77:52058) 2024-03-16T10:22:51.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.533+0000 7fcfa4f69640 1 -- 172.21.15.77:0/1218796360 learned_addr learned my addr 172.21.15.77:0/1218796360 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:51.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.533+0000 7fcfa4f69640 1 -- 172.21.15.77:0/1218796360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11555ea0 msgr2=0x564a11558330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:22:51.535 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.533+0000 7fcfa4f69640 1 --2- 172.21.15.77:0/1218796360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11555ea0 0x564a11558330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.535 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.533+0000 7fcfa4f69640 1 -- 172.21.15.77:0/1218796360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 msgr2=0x564a11550760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:51.535 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.533+0000 7fcfa4f69640 1 --2- 172.21.15.77:0/1218796360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 0x564a11550760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.535 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.533+0000 7fcfa4f69640 1 -- 172.21.15.77:0/1218796360 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a11476460 con 0x564a11550d30 2024-03-16T10:22:51.535 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.533+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/1218796360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 0x564a11550760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:22:51.535 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.534+0000 7fcf9dffb640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf880a2ae0 0x7fcf880a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.535+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf880a2ae0 0x7fcf880a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.536+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf880a2ae0 0x7fcf880a4fa0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcf98006010 tx=0x7fcf98000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.537+0000 7fcf9dffb640 1 -- 172.21.15.77:0/1305134143 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fcf940cf2b0 con 0x7fcfa010a230 2024-03-16T10:22:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.537+0000 7fcfa4f69640 1 --2- 172.21.15.77:0/1218796360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a11550d30 0x564a11555830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcf900098b0 tx=0x7fcf90009d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0b683a09a20840f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.540+0000 7fcf86ffd640 1 -- 172.21.15.77:0/1218796360 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90024070 con 0x564a11550d30 2024-03-16T10:22:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.541+0000 7fcf86ffd640 1 -- 172.21.15.77:0/1218796360 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf9001f400 con 0x564a11550d30 2024-03-16T10:22:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.545+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1218796360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a11550d30 msgr2=0x564a11555830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.545+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/1218796360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a11550d30 0x564a11555830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcf900098b0 tx=0x7fcf90009d70 comp rx=0 tx=0).stop 2024-03-16T10:22:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.547+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1218796360 shutdown_connections 2024-03-16T10:22:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.547+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/1218796360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11555ea0 0x564a11558330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.547+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/1218796360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a11550d30 0x564a11555830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.547+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/1218796360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 0x564a11550760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.548 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.547+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1218796360 >> 172.21.15.77:0/1218796360 conn(0x564a1154b8b0 msgr2=0x564a1154bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:51.588 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.588+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1218796360 shutdown_connections 2024-03-16T10:22:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.588+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1218796360 wait complete. 2024-03-16T10:22:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.588+0000 7fcfa9db0740 1 -- start start 2024-03-16T10:22:51.589 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.588+0000 7fcfa9db0740 1 Processor -- start 2024-03-16T10:22:51.591 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.589+0000 7fcfa9db0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11550340 0x564a1155a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.591 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.589+0000 7fcfa9db0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11555ea0 0x564a1155aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.591 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.589+0000 7fcfa9db0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a1155fb00 0x564a11561ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.591 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.589+0000 7fcfa9db0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564a11558ea0 con 0x564a11555ea0 2024-03-16T10:22:51.591 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.589+0000 7fcfa9db0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564a11471d70 con 0x564a11550340 2024-03-16T10:22:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.589+0000 7fcfa9db0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564a11559020 con 0x564a1155fb00 2024-03-16T10:22:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.589+0000 7fcfa9db0740 10 client.0 _ll_register_callbacks cb 0x564a111c9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.590+0000 7fcf9ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11555ea0 0x564a1155aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.590+0000 7fcf9ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11555ea0 0x564a1155aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56176/0 (socket says 172.21.15.77:56176) 2024-03-16T10:22:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.590+0000 7fcf9ffff640 1 -- 172.21.15.77:0/1619850739 learned_addr learned my addr 172.21.15.77:0/1619850739 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.591+0000 7fcf9ffff640 1 -- 172.21.15.77:0/1619850739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a1155fb00 msgr2=0x564a11561ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.591+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/1619850739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a1155fb00 0x564a11561ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.591+0000 7fcf9ffff640 1 -- 172.21.15.77:0/1619850739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11550340 msgr2=0x564a1155a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.591+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/1619850739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11550340 0x564a1155a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.591+0000 7fcf9ffff640 1 -- 172.21.15.77:0/1619850739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a111ca210 con 0x564a11555ea0 2024-03-16T10:22:51.593 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.592+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/1619850739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11555ea0 0x564a1155aad0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800adb0 tx=0x7fcf9802cff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22fe03a3194a1d4f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:51.593 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.592+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1619850739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564a114c2f80 con 0x564a11555ea0 2024-03-16T10:22:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.615+0000 7fcf7ffff640 10 client.16995 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:22:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.616+0000 7fcf7ffff640 1 -- 172.21.15.77:0/1619850739 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9803a040 con 0x564a11555ea0 2024-03-16T10:22:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.616+0000 7fcf7ffff640 1 -- 172.21.15.77:0/1619850739 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fcf98004180 con 0x564a11555ea0 2024-03-16T10:22:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.616+0000 7fcf7ffff640 1 -- 172.21.15.77:0/1619850739 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcf98004380 con 0x564a11555ea0 2024-03-16T10:22:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.616+0000 7fcfa9db0740 10 client.16995 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.616+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1619850739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=36}) -- 0x564a11296620 con 0x564a11555ea0 2024-03-16T10:22:51.618 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.617+0000 7fcf7ffff640 1 -- 172.21.15.77:0/1619850739 <== mon.0 v2:172.21.15.77:3300/0 4 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fcf98004520 con 0x564a11555ea0 2024-03-16T10:22:51.619 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.618+0000 7fcfa9db0740 10 client.16995 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:51.619 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.618+0000 7fcfa9db0740 10 client.16995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:51.619 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.618+0000 7fcfa9db0740 20 client.16995 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:51.619 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.618+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/1619850739 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564a11566cb0 0x564a11569170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.619 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.618+0000 7fcfa9db0740 4 client.16995 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:22:51.619 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.618+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1619850739 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a114bd2d0 con 0x564a11566cb0 2024-03-16T10:22:51.620 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.619+0000 7fcfa4f69640 1 --2- 172.21.15.77:0/1619850739 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564a11566cb0 0x564a11569170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.620 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.619+0000 7fcfa4f69640 1 --2- 172.21.15.77:0/1619850739 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564a11566cb0 0x564a11569170 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3814b5314eb571a2 server_cookie=979f9d6e81b57483 in_seq=0 out_seq=0 2024-03-16T10:22:51.620 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.620+0000 7fcf7ffff640 10 client.16995 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:51.621 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.620+0000 7fcf7ffff640 1 -- 172.21.15.77:0/1619850739 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564a114bd2d0 con 0x564a11566cb0 2024-03-16T10:22:51.621 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.620+0000 7fcf7ffff640 10 client.16995 handle_command_reply: tid=0 2024-03-16T10:22:51.621 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.621+0000 7fcfa9db0740 1 client.16995 shutdown 2024-03-16T10:22:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.668+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1619850739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11555ea0 msgr2=0x564a1155aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.668+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/1619850739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11555ea0 0x564a1155aad0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800adb0 tx=0x7fcf9802cff0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.708+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1619850739 shutdown_connections 2024-03-16T10:22:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.708+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/1619850739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a1155fb00 0x564a11561ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.708+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/1619850739 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564a11566cb0 0x564a11569170 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.708+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/1619850739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11555ea0 0x564a1155aad0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.708+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/1619850739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11550340 0x564a1155a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.708+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1619850739 >> 172.21.15.77:0/1619850739 conn(0x564a1154b8b0 msgr2=0x564a1154abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.708+0000 7fcf7ffff640 0 client.16995 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.708+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1619850739 shutdown_connections 2024-03-16T10:22:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.708+0000 7fcfa9db0740 1 -- 172.21.15.77:0/1619850739 wait complete. 2024-03-16T10:22:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.708+0000 7fcfa9db0740 20 client.16995 trim_cache size 0 max 16384 2024-03-16T10:22:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.827+0000 7fcfa9db0740 1 Processor -- start 2024-03-16T10:22:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.827+0000 7fcfa9db0740 1 -- start start 2024-03-16T10:22:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.829+0000 7fcfa9db0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11550340 0x564a11551e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.829+0000 7fcfa9db0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a11555ea0 0x564a11552390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa9db0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a1155fb00 0x564a11559860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa4f69640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11550340 0x564a11551e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa4f69640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11550340 0x564a11551e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:50130/0 (socket says 172.21.15.77:50130) 2024-03-16T10:22:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa4f69640 1 -- 172.21.15.77:0/2086244224 learned_addr learned my addr 172.21.15.77:0/2086244224 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa9db0740 1 -- 172.21.15.77:0/2086244224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c00e430 con 0x564a1155fb00 2024-03-16T10:22:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa9db0740 1 -- 172.21.15.77:0/2086244224 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c010000 con 0x564a11550340 2024-03-16T10:22:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa9db0740 1 -- 172.21.15.77:0/2086244224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c00eb00 con 0x564a11555ea0 2024-03-16T10:22:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa576a640 1 --2- 172.21.15.77:0/2086244224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a1155fb00 0x564a11559860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/2086244224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a11555ea0 0x564a11552390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa4f69640 1 -- 172.21.15.77:0/2086244224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a11555ea0 msgr2=0x564a11552390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa4f69640 1 --2- 172.21.15.77:0/2086244224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a11555ea0 0x564a11552390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa4f69640 1 -- 172.21.15.77:0/2086244224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a1155fb00 msgr2=0x564a11559860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa4f69640 1 --2- 172.21.15.77:0/2086244224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a1155fb00 0x564a11559860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.833 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.830+0000 7fcfa4f69640 1 -- 172.21.15.77:0/2086244224 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a11547310 con 0x564a11550340 2024-03-16T10:22:51.833 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.831+0000 7fcfa576a640 1 --2- 172.21.15.77:0/2086244224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a1155fb00 0x564a11559860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:22:51.833 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.831+0000 7fcfa4f69640 1 --2- 172.21.15.77:0/2086244224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11550340 0x564a11551e50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcf900098b0 tx=0x7fcf90014b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=122951c564ca07df server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:51.836 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.835+0000 7fcf867fc640 1 -- 172.21.15.77:0/2086244224 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90032070 con 0x564a11550340 2024-03-16T10:22:51.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.836+0000 7fcf867fc640 1 -- 172.21.15.77:0/2086244224 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf9002c170 con 0x564a11550340 2024-03-16T10:22:51.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.836+0000 7fcf867fc640 1 -- 172.21.15.77:0/2086244224 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90024040 con 0x564a11550340 2024-03-16T10:22:51.839 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.839+0000 7fcfa9db0740 1 -- 172.21.15.77:0/2086244224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11550340 msgr2=0x564a11551e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:51.840 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.839+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/2086244224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11550340 0x564a11551e50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcf900098b0 tx=0x7fcf90014b20 comp rx=0 tx=0).stop 2024-03-16T10:22:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.843+0000 7fcfa9db0740 1 -- 172.21.15.77:0/2086244224 shutdown_connections 2024-03-16T10:22:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.843+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/2086244224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a1155fb00 0x564a11559860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.843+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/2086244224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a11555ea0 0x564a11552390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.843+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/2086244224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11550340 0x564a11551e50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.843+0000 7fcfa9db0740 1 -- 172.21.15.77:0/2086244224 >> 172.21.15.77:0/2086244224 conn(0x564a1154b8b0 msgr2=0x564a1155a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.843+0000 7fcfa9db0740 1 -- 172.21.15.77:0/2086244224 shutdown_connections 2024-03-16T10:22:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.843+0000 7fcfa9db0740 1 -- 172.21.15.77:0/2086244224 wait complete. 2024-03-16T10:22:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.844+0000 7fcfa9db0740 1 -- start start 2024-03-16T10:22:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.844+0000 7fcfa9db0740 1 Processor -- start 2024-03-16T10:22:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.846+0000 7fcfa9db0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 0x564a115779f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.847+0000 7fcfa4f69640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 0x564a115779f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.847+0000 7fcfa4f69640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 0x564a115779f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56188/0 (socket says 172.21.15.77:56188) 2024-03-16T10:22:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.847+0000 7fcfa4f69640 1 -- 172.21.15.77:0/3691454047 learned_addr learned my addr 172.21.15.77:0/3691454047 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:22:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.847+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11555ea0 0x564a11577f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.847+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a1155fb00 0x564a11578f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.847+0000 7fcfa9db0740 1 -- 172.21.15.77:0/3691454047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c00f930 con 0x564a11550340 2024-03-16T10:22:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.847+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11555ea0 0x564a11577f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.847+0000 7fcfa9db0740 1 -- 172.21.15.77:0/3691454047 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c00f260 con 0x564a11555ea0 2024-03-16T10:22:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.847+0000 7fcfa9db0740 1 -- 172.21.15.77:0/3691454047 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c00c410 con 0x564a1155fb00 2024-03-16T10:22:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.847+0000 7fcfa576a640 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a1155fb00 0x564a11578f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.847+0000 7fcfa9db0740 10 client.0 _ll_register_callbacks cb 0x7fcf90005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:22:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.848+0000 7fcf9ffff640 1 -- 172.21.15.77:0/3691454047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a1155fb00 msgr2=0x564a11578f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.848+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a1155fb00 0x564a11578f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.848+0000 7fcf9ffff640 1 -- 172.21.15.77:0/3691454047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 msgr2=0x564a115779f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:22:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.848+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 0x564a115779f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.848+0000 7fcf9ffff640 1 -- 172.21.15.77:0/3691454047 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a11579530 con 0x564a11555ea0 2024-03-16T10:22:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.848+0000 7fcfa4f69640 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 0x564a115779f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:22:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.849+0000 7fcf9ffff640 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11555ea0 0x564a11577f30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcf980333b0 tx=0x7fcf98042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f30df09be42b2c8b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:22:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.859+0000 7fcf7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:22:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.859+0000 7fcf7ffff640 1 -- 172.21.15.77:0/3691454047 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98064070 con 0x564a11555ea0 2024-03-16T10:22:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.859+0000 7fcf7ffff640 1 -- 172.21.15.77:0/3691454047 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fcf9804c8c0 con 0x564a11555ea0 2024-03-16T10:22:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.864+0000 7fcfa9db0740 1 -- 172.21.15.77:0/3691454047 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf9002c170 con 0x564a11555ea0 2024-03-16T10:22:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.867+0000 7fcf7ffff640 1 -- 172.21.15.77:0/3691454047 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcf9806e050 con 0x564a11555ea0 2024-03-16T10:22:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.867+0000 7fcfa9db0740 10 client.26774 fetch_fsmap learned FSMap version 36 2024-03-16T10:22:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.867+0000 7fcfa9db0740 1 -- 172.21.15.77:0/3691454047 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564a115ccae0 con 0x564a11555ea0 2024-03-16T10:22:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.872+0000 7fcf7ffff640 1 -- 172.21.15.77:0/3691454047 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf9002c170 con 0x564a11555ea0 2024-03-16T10:22:51.873 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.872+0000 7fcf7ffff640 1 -- 172.21.15.77:0/3691454047 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fcf98035b20 con 0x564a11555ea0 2024-03-16T10:22:51.873 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.872+0000 7fcfa9db0740 10 client.26774 fetch_fsmap finished waiting for FSMap version 36 2024-03-16T10:22:51.873 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.872+0000 7fcfa9db0740 10 client.26774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:22:51.874 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.872+0000 7fcfa9db0740 20 client.26774 populate_metadata read hostname 'smithi077' 2024-03-16T10:22:51.874 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.872+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564a115cd410 0x564a115cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:22:51.874 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.872+0000 7fcfa9db0740 4 client.26774 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:22:51.874 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.872+0000 7fcfa9db0740 1 -- 172.21.15.77:0/3691454047 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564a1143b770 con 0x564a115cd410 2024-03-16T10:22:51.874 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.872+0000 7fcfa4f69640 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564a115cd410 0x564a115cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:22:51.874 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.873+0000 7fcf7ffff640 10 client.26774 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:51.874 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.873+0000 7fcfa4f69640 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564a115cd410 0x564a115cf830 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=198660202408c6bb server_cookie=143422030479b43e in_seq=0 out_seq=0 2024-03-16T10:22:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.889+0000 7fcf7ffff640 1 -- 172.21.15.77:0/3691454047 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564a1143b770 con 0x564a115cd410 2024-03-16T10:22:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.889+0000 7fcf7ffff640 10 client.26774 handle_command_reply: tid=0 2024-03-16T10:22:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.889+0000 7fcfa9db0740 1 client.26774 shutdown 2024-03-16T10:22:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.889+0000 7fcfa9db0740 1 -- 172.21.15.77:0/3691454047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11555ea0 msgr2=0x564a11577f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.889+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11555ea0 0x564a11577f30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcf980333b0 tx=0x7fcf98042000 comp rx=0 tx=0).stop 2024-03-16T10:22:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.890+0000 7fcfa9db0740 1 -- 172.21.15.77:0/3691454047 shutdown_connections 2024-03-16T10:22:51.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.890+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a1155fb00 0x564a11578f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.890+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a11555ea0 0x564a11577f30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.890+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564a115cd410 0x564a115cf830 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.890+0000 7fcfa9db0740 1 --2- 172.21.15.77:0/3691454047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a11550340 0x564a115779f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.890+0000 7fcfa9db0740 1 -- 172.21.15.77:0/3691454047 >> 172.21.15.77:0/3691454047 conn(0x564a1154b8b0 msgr2=0x564a11551e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.890+0000 7fcf7ffff640 0 client.26774 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:22:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.890+0000 7fcfa9db0740 1 -- 172.21.15.77:0/3691454047 shutdown_connections 2024-03-16T10:22:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.890+0000 7fcfa9db0740 1 -- 172.21.15.77:0/3691454047 wait complete. 2024-03-16T10:22:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.890+0000 7fcfa9db0740 20 client.26774 trim_cache size 0 max 16384 2024-03-16T10:22:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:22:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:22:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:22:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:22:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.905+0000 7fcfa71cd640 1 -- 172.21.15.77:0/1305134143 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf880a2ae0 msgr2=0x7fcf880a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.905+0000 7fcfa71cd640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf880a2ae0 0x7fcf880a4fa0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcf98006010 tx=0x7fcf98000f40 comp rx=0 tx=0).stop 2024-03-16T10:22:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.905+0000 7fcfa71cd640 1 -- 172.21.15.77:0/1305134143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa010a230 msgr2=0x7fcfa01a7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:22:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.905+0000 7fcfa71cd640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa010a230 0x7fcfa01a7850 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400d6b0 tx=0x7fcf9400db70 comp rx=0 tx=0).stop 2024-03-16T10:22:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.905+0000 7fcfa71cd640 1 -- 172.21.15.77:0/1305134143 shutdown_connections 2024-03-16T10:22:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.905+0000 7fcfa71cd640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcf880a2ae0 0x7fcf880a4fa0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.905+0000 7fcfa71cd640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa010a230 0x7fcfa01a7850 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.905+0000 7fcfa71cd640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfa0109890 0x7fcfa01a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.905+0000 7fcfa71cd640 1 --2- 172.21.15.77:0/1305134143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcfa0108b80 0x7fcfa01a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:22:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.905+0000 7fcfa71cd640 1 -- 172.21.15.77:0/1305134143 >> 172.21.15.77:0/1305134143 conn(0x7fcfa006e820 msgr2=0x7fcfa010cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:22:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.907+0000 7fcfa71cd640 1 -- 172.21.15.77:0/1305134143 shutdown_connections 2024-03-16T10:22:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:22:51.908+0000 7fcfa71cd640 1 -- 172.21.15.77:0/1305134143 wait complete. 2024-03-16T10:22:52.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:51 smithi192 ceph-mon[36354]: pgmap v523: 97 pgs: 97 active+clean; 682 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 253 op/s 2024-03-16T10:22:52.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:51 smithi077 ceph-mon[32400]: pgmap v523: 97 pgs: 97 active+clean; 682 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 253 op/s 2024-03-16T10:22:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:51 smithi193 ceph-mon[36311]: pgmap v523: 97 pgs: 97 active+clean; 682 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 253 op/s 2024-03-16T10:22:54.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:53 smithi077 ceph-mon[32400]: pgmap v524: 97 pgs: 97 active+clean; 664 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 282 op/s 2024-03-16T10:22:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:53 smithi193 ceph-mon[36311]: pgmap v524: 97 pgs: 97 active+clean; 664 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 282 op/s 2024-03-16T10:22:54.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:53 smithi192 ceph-mon[36354]: pgmap v524: 97 pgs: 97 active+clean; 664 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 282 op/s 2024-03-16T10:22:56.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:55 smithi077 ceph-mon[32400]: pgmap v525: 97 pgs: 97 active+clean; 665 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 273 op/s 2024-03-16T10:22:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:55 smithi193 ceph-mon[36311]: pgmap v525: 97 pgs: 97 active+clean; 665 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 273 op/s 2024-03-16T10:22:56.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:55 smithi192 ceph-mon[36354]: pgmap v525: 97 pgs: 97 active+clean; 665 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 273 op/s 2024-03-16T10:22:57.105 INFO:tasks.workunit.client.0.smithi077.stdout: 149 1896495 355 1281808 320 1100170 884 2024-03-16T10:22:57.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:56 smithi077 ceph-mon[32400]: pgmap v526: 97 pgs: 97 active+clean; 678 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 434 op/s 2024-03-16T10:22:57.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:56 smithi193 ceph-mon[36311]: pgmap v526: 97 pgs: 97 active+clean; 678 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 434 op/s 2024-03-16T10:22:57.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:56 smithi192 ceph-mon[36354]: pgmap v526: 97 pgs: 97 active+clean; 678 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 434 op/s 2024-03-16T10:22:59.667 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:22:59 smithi192 ceph-mon[36354]: pgmap v527: 97 pgs: 97 active+clean; 694 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 564 op/s 2024-03-16T10:22:59.747 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:22:59 smithi077 ceph-mon[32400]: pgmap v527: 97 pgs: 97 active+clean; 694 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 564 op/s 2024-03-16T10:22:59.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:22:59 smithi193 ceph-mon[36311]: pgmap v527: 97 pgs: 97 active+clean; 694 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 564 op/s 2024-03-16T10:23:02.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:01 smithi192 ceph-mon[36354]: pgmap v528: 97 pgs: 97 active+clean; 717 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.0 MiB/s wr, 723 op/s 2024-03-16T10:23:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:01 smithi077 ceph-mon[32400]: pgmap v528: 97 pgs: 97 active+clean; 717 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.0 MiB/s wr, 723 op/s 2024-03-16T10:23:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:01 smithi193 ceph-mon[36311]: pgmap v528: 97 pgs: 97 active+clean; 717 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.0 MiB/s wr, 723 op/s 2024-03-16T10:23:04.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:03 smithi192 ceph-mon[36354]: pgmap v529: 97 pgs: 97 active+clean; 727 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.0 MiB/s wr, 816 op/s 2024-03-16T10:23:04.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:03 smithi077 ceph-mon[32400]: pgmap v529: 97 pgs: 97 active+clean; 727 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.0 MiB/s wr, 816 op/s 2024-03-16T10:23:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:03 smithi193 ceph-mon[36311]: pgmap v529: 97 pgs: 97 active+clean; 727 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.0 MiB/s wr, 816 op/s 2024-03-16T10:23:06.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:05 smithi077 ceph-mon[32400]: pgmap v530: 97 pgs: 97 active+clean; 726 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.7 MiB/s wr, 791 op/s 2024-03-16T10:23:06.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:05 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:23:06.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:05 smithi193 ceph-mon[36311]: pgmap v530: 97 pgs: 97 active+clean; 726 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.7 MiB/s wr, 791 op/s 2024-03-16T10:23:06.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:05 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:23:06.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:05 smithi192 ceph-mon[36354]: pgmap v530: 97 pgs: 97 active+clean; 726 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.7 MiB/s wr, 791 op/s 2024-03-16T10:23:06.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:05 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:23:06.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.423+0000 7f43cde61640 1 -- 172.21.15.77:0/3849425232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43c8109510 msgr2=0x7f43c810e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:06.425 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.423+0000 7f43cde61640 1 --2- 172.21.15.77:0/3849425232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43c8109510 0x7f43c810e1a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f43b0009900 tx=0x7f43b002bc80 comp rx=0 tx=0).stop 2024-03-16T10:23:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.442+0000 7f43cde61640 1 -- 172.21.15.77:0/3849425232 shutdown_connections 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.442+0000 7f43cde61640 1 --2- 172.21.15.77:0/3849425232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43c8109510 0x7f43c810e1a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.442+0000 7f43cde61640 1 --2- 172.21.15.77:0/3849425232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c8073410 0x7f43c8108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.442+0000 7f43cde61640 1 --2- 172.21.15.77:0/3849425232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43c8072ab0 0x7f43c8072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.442+0000 7f43cde61640 1 -- 172.21.15.77:0/3849425232 >> 172.21.15.77:0/3849425232 conn(0x7f43c806e650 msgr2=0x7f43c8070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.444+0000 7f43cde61640 1 -- 172.21.15.77:0/3849425232 shutdown_connections 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.445+0000 7f43cde61640 1 -- 172.21.15.77:0/3849425232 wait complete. 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.446+0000 7f43cde61640 1 Processor -- start 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.446+0000 7f43cde61640 1 -- start start 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.446+0000 7f43cde61640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c8072ab0 0x7f43c8117da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.446+0000 7f43cde61640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43c8073410 0x7f43c81182e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.446+0000 7f43cde61640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43c8109510 0x7f43c8119320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.446+0000 7f43cde61640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f43c8110520 con 0x7f43c8072ab0 2024-03-16T10:23:06.454 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.446+0000 7f43cde61640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f43c81103a0 con 0x7f43c8073410 2024-03-16T10:23:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.446+0000 7f43cde61640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43c81106a0 con 0x7f43c8109510 2024-03-16T10:23:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.447+0000 7f43c77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c8072ab0 0x7f43c8117da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.447+0000 7f43c77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c8072ab0 0x7f43c8117da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47154/0 (socket says 172.21.15.77:47154) 2024-03-16T10:23:06.457 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.447+0000 7f43c77fe640 1 -- 172.21.15.77:0/1606248890 learned_addr learned my addr 172.21.15.77:0/1606248890 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.447+0000 7f43c77fe640 1 -- 172.21.15.77:0/1606248890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43c8109510 msgr2=0x7f43c8119320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:23:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.447+0000 7f43c77fe640 1 --2- 172.21.15.77:0/1606248890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43c8109510 0x7f43c8119320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.447+0000 7f43c77fe640 1 -- 172.21.15.77:0/1606248890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43c8073410 msgr2=0x7f43c81182e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:23:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.447+0000 7f43c77fe640 1 --2- 172.21.15.77:0/1606248890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43c8073410 0x7f43c81182e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.447+0000 7f43c77fe640 1 -- 172.21.15.77:0/1606248890 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c8119990 con 0x7f43c8072ab0 2024-03-16T10:23:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.449+0000 7f43c77fe640 1 --2- 172.21.15.77:0/1606248890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c8072ab0 0x7f43c8117da0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f43b800bd60 tx=0x7f43b8009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.456+0000 7f43c4ff9640 1 -- 172.21.15.77:0/1606248890 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8019070 con 0x7f43c8072ab0 2024-03-16T10:23:06.466 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.456+0000 7f43cde61640 1 -- 172.21.15.77:0/1606248890 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c8119c80 con 0x7f43c8072ab0 2024-03-16T10:23:06.466 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.456+0000 7f43cde61640 1 -- 172.21.15.77:0/1606248890 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c811a1d0 con 0x7f43c8072ab0 2024-03-16T10:23:06.466 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.464+0000 7f43d0a53740 1 Processor -- start 2024-03-16T10:23:06.466 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.464+0000 7f43d0a53740 1 -- start start 2024-03-16T10:23:06.466 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.464+0000 7f43d0a53740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa8719500 0x555aa8719920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.466 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.464+0000 7f43d0a53740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa871e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.466 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.464+0000 7f43d0a53740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa871f060 0x555aa87214f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.466 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.464+0000 7f43d0a53740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555aa867ea40 con 0x555aa871f060 2024-03-16T10:23:06.467 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.464+0000 7f43d0a53740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x555aa8627920 con 0x555aa8719ef0 2024-03-16T10:23:06.467 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.464+0000 7f43d0a53740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555aa867aae0 con 0x555aa8719500 2024-03-16T10:23:06.467 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.465+0000 7f43c7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa8719500 0x555aa8719920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.467 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.465+0000 7f43c6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa871f060 0x555aa87214f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.467 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.465+0000 7f43c6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa871f060 0x555aa87214f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47156/0 (socket says 172.21.15.77:47156) 2024-03-16T10:23:06.467 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.465+0000 7f43c6ffd640 1 -- 172.21.15.77:0/1147500288 learned_addr learned my addr 172.21.15.77:0/1147500288 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:06.467 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.465+0000 7f43c77fe640 1 --2- 172.21.15.77:0/1147500288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa871e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.466+0000 7f43c7fff640 1 -- 172.21.15.77:0/1147500288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 msgr2=0x555aa871e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.466+0000 7f43c7fff640 1 --2- 172.21.15.77:0/1147500288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa871e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.466+0000 7f43c7fff640 1 -- 172.21.15.77:0/1147500288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa871f060 msgr2=0x555aa87214f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.466+0000 7f43c7fff640 1 --2- 172.21.15.77:0/1147500288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa871f060 0x555aa87214f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.466+0000 7f43c7fff640 1 -- 172.21.15.77:0/1147500288 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555aa863f460 con 0x555aa8719500 2024-03-16T10:23:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.466+0000 7f43c7fff640 1 --2- 172.21.15.77:0/1147500288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa8719500 0x555aa8719920 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f43b0036040 tx=0x7f43b0005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=834abfa233ad9d0b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.466+0000 7f43c4ff9640 1 -- 172.21.15.77:0/1606248890 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43b800cc80 con 0x7f43c8072ab0 2024-03-16T10:23:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.466+0000 7f43c4ff9640 1 -- 172.21.15.77:0/1606248890 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8007770 con 0x7f43c8072ab0 2024-03-16T10:23:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.466+0000 7f43c4ff9640 1 -- 172.21.15.77:0/1606248890 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f43b801c070 con 0x7f43c8072ab0 2024-03-16T10:23:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.469+0000 7f43c4ff9640 1 --2- 172.21.15.77:0/1606248890 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f43b40a2a30 0x7f43b40a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.469+0000 7f43c4ff9640 1 -- 172.21.15.77:0/1606248890 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f43b80cf130 con 0x7f43c8072ab0 2024-03-16T10:23:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.470+0000 7f43c77fe640 1 --2- 172.21.15.77:0/1606248890 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f43b40a2a30 0x7f43b40a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.478 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.477+0000 7f43bdffb640 1 -- 172.21.15.77:0/1147500288 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0043020 con 0x555aa8719500 2024-03-16T10:23:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.480+0000 7f43c77fe640 1 --2- 172.21.15.77:0/1606248890 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f43b40a2a30 0x7f43b40a4ef0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f43b8000f80 tx=0x7f43b8097de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.481+0000 7f43bdffb640 1 -- 172.21.15.77:0/1147500288 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43b0009540 con 0x555aa8719500 2024-03-16T10:23:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.481+0000 7f43bdffb640 1 -- 172.21.15.77:0/1147500288 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0005510 con 0x555aa8719500 2024-03-16T10:23:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.482+0000 7f43d0a53740 1 -- 172.21.15.77:0/1147500288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa8719500 msgr2=0x555aa8719920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.482+0000 7f43d0a53740 1 --2- 172.21.15.77:0/1147500288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa8719500 0x555aa8719920 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f43b0036040 tx=0x7f43b0005bd0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.484+0000 7f43d0a53740 1 -- 172.21.15.77:0/1147500288 shutdown_connections 2024-03-16T10:23:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.484+0000 7f43d0a53740 1 --2- 172.21.15.77:0/1147500288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa871f060 0x555aa87214f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.484+0000 7f43d0a53740 1 --2- 172.21.15.77:0/1147500288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa871e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.484+0000 7f43d0a53740 1 --2- 172.21.15.77:0/1147500288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa8719500 0x555aa8719920 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.484+0000 7f43d0a53740 1 -- 172.21.15.77:0/1147500288 >> 172.21.15.77:0/1147500288 conn(0x555aa8714a70 msgr2=0x555aa8714e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:06.487 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.486+0000 7f43d0a53740 1 -- 172.21.15.77:0/1147500288 shutdown_connections 2024-03-16T10:23:06.487 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.486+0000 7f43d0a53740 1 -- 172.21.15.77:0/1147500288 wait complete. 2024-03-16T10:23:06.487 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.486+0000 7f43d0a53740 1 -- start start 2024-03-16T10:23:06.487 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.487+0000 7f43d0a53740 1 Processor -- start 2024-03-16T10:23:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.487+0000 7f43d0a53740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719500 0x555aa8745870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.491+0000 7f43d0a53740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719ef0 0x555aa8745db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.491+0000 7f43d0a53740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 0x555aa874cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.491+0000 7f43c7fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719500 0x555aa8745870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.491+0000 7f43c7fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719500 0x555aa8745870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:40938/0 (socket says 172.21.15.77:40938) 2024-03-16T10:23:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.491+0000 7f43c7fff640 1 -- 172.21.15.77:0/595025197 learned_addr learned my addr 172.21.15.77:0/595025197 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:06.492 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.491+0000 7f43c6ffd640 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719ef0 0x555aa8745db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.493 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.491+0000 7f43d0a53740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555aa8743fc0 con 0x555aa8719ef0 2024-03-16T10:23:06.493 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.491+0000 7f43d0a53740 1 -- 172.21.15.77:0/595025197 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x555aa863ad70 con 0x555aa8719500 2024-03-16T10:23:06.493 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.491+0000 7f43d0a53740 1 -- 172.21.15.77:0/595025197 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555aa8744140 con 0x555aa871f060 2024-03-16T10:23:06.493 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.491+0000 7f43d0a53740 10 client.0 _ll_register_callbacks cb 0x555aa8392a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:06.493 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.492+0000 7f43c7fff640 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 0x555aa874cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.494 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.493+0000 7f43c7fff640 1 -- 172.21.15.77:0/595025197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 msgr2=0x555aa874cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:06.494 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.493+0000 7f43c7fff640 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 0x555aa874cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.494 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.493+0000 7f43c7fff640 1 -- 172.21.15.77:0/595025197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719ef0 msgr2=0x555aa8745db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:06.494 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.493+0000 7f43c7fff640 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719ef0 0x555aa8745db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.494 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.493+0000 7f43c7fff640 1 -- 172.21.15.77:0/595025197 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555aa8393210 con 0x555aa8719500 2024-03-16T10:23:06.495 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.494+0000 7f43c7fff640 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719500 0x555aa8745870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f43b0006010 tx=0x7f43b002bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e390a208cac419b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:06.495 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.494+0000 7f43a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:23:06.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.495+0000 7f43a6ffd640 1 -- 172.21.15.77:0/595025197 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0043030 con 0x555aa8719500 2024-03-16T10:23:06.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.495+0000 7f43d0a53740 1 -- 172.21.15.77:0/595025197 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43b0009540 con 0x555aa8719500 2024-03-16T10:23:06.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.495+0000 7f43a6ffd640 1 -- 172.21.15.77:0/595025197 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f43b00048c0 con 0x555aa8719500 2024-03-16T10:23:06.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.495+0000 7f43a6ffd640 1 -- 172.21.15.77:0/595025197 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f43b0004ac0 con 0x555aa8719500 2024-03-16T10:23:06.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.495+0000 7f43d0a53740 10 client.26780 fetch_fsmap learned FSMap version 37 2024-03-16T10:23:06.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.495+0000 7f43d0a53740 1 -- 172.21.15.77:0/595025197 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x555aa845f620 con 0x555aa8719500 2024-03-16T10:23:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.496+0000 7f43a6ffd640 1 -- 172.21.15.77:0/595025197 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43b0009540 con 0x555aa8719500 2024-03-16T10:23:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.497+0000 7f43a6ffd640 1 -- 172.21.15.77:0/595025197 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f43b0034440 con 0x555aa8719500 2024-03-16T10:23:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.497+0000 7f43d0a53740 10 client.26780 fetch_fsmap finished waiting for FSMap version 37 2024-03-16T10:23:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.497+0000 7f43d0a53740 10 client.26780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.497+0000 7f43d0a53740 20 client.26780 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.497+0000 7f43d0a53740 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555aa8751820 0x555aa8753c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.497+0000 7f43d0a53740 4 client.26780 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.497+0000 7f43d0a53740 1 -- 172.21.15.77:0/595025197 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555aa868bf80 con 0x555aa8751820 2024-03-16T10:23:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.499+0000 7f43c6ffd640 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555aa8751820 0x555aa8753c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.500+0000 7f43c6ffd640 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555aa8751820 0x555aa8753c60 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ec8b4a55a76b2a2 server_cookie=8238d140b782cf43 in_seq=0 out_seq=0 2024-03-16T10:23:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.500+0000 7f43a6ffd640 10 client.26780 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.502+0000 7f43a6ffd640 1 -- 172.21.15.77:0/595025197 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555aa868bf80 con 0x555aa8751820 2024-03-16T10:23:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.502+0000 7f43a6ffd640 10 client.26780 handle_command_reply: tid=0 2024-03-16T10:23:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.502+0000 7f43d0a53740 1 client.26780 shutdown 2024-03-16T10:23:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.504+0000 7f43d0a53740 1 -- 172.21.15.77:0/595025197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719500 msgr2=0x555aa8745870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.504+0000 7f43d0a53740 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719500 0x555aa8745870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f43b0006010 tx=0x7f43b002bc80 comp rx=0 tx=0).stop 2024-03-16T10:23:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.506+0000 7f43d0a53740 1 -- 172.21.15.77:0/595025197 shutdown_connections 2024-03-16T10:23:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.506+0000 7f43d0a53740 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 0x555aa874cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.506+0000 7f43d0a53740 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555aa8751820 0x555aa8753c60 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.506+0000 7f43d0a53740 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719ef0 0x555aa8745db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.506+0000 7f43d0a53740 1 --2- 172.21.15.77:0/595025197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719500 0x555aa8745870 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.506+0000 7f43d0a53740 1 -- 172.21.15.77:0/595025197 >> 172.21.15.77:0/595025197 conn(0x555aa8714a70 msgr2=0x555aa8713c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.506+0000 7f43a6ffd640 0 client.26780 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.506+0000 7f43d0a53740 1 -- 172.21.15.77:0/595025197 shutdown_connections 2024-03-16T10:23:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.506+0000 7f43d0a53740 1 -- 172.21.15.77:0/595025197 wait complete. 2024-03-16T10:23:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.506+0000 7f43d0a53740 20 client.26780 trim_cache size 0 max 16384 2024-03-16T10:23:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.602+0000 7f43d0a53740 1 Processor -- start 2024-03-16T10:23:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.602+0000 7f43d0a53740 1 -- start start 2024-03-16T10:23:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.603+0000 7f43d0a53740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719500 0x555aa87183e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.603+0000 7f43d0a53740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa874cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.603+0000 7f43d0a53740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 0x555aa8745480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.603+0000 7f43d0a53740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f43a800e230 con 0x555aa8719500 2024-03-16T10:23:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.604+0000 7f43d0a53740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f43a800fe30 con 0x555aa8719ef0 2024-03-16T10:23:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.604+0000 7f43c6ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa874cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.604+0000 7f43c6ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa874cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:40952/0 (socket says 172.21.15.77:40952) 2024-03-16T10:23:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.604+0000 7f43d0a53740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43a800e900 con 0x555aa871f060 2024-03-16T10:23:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.604+0000 7f43c6ffd640 1 -- 172.21.15.77:0/2538620904 learned_addr learned my addr 172.21.15.77:0/2538620904 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.604+0000 7f43c6ffd640 1 -- 172.21.15.77:0/2538620904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 msgr2=0x555aa8745480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.604+0000 7f43c6ffd640 1 --2- 172.21.15.77:0/2538620904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 0x555aa8745480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.604+0000 7f43c6ffd640 1 -- 172.21.15.77:0/2538620904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719500 msgr2=0x555aa87183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.604+0000 7f43c6ffd640 1 --2- 172.21.15.77:0/2538620904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719500 0x555aa87183e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.604+0000 7f43c6ffd640 1 -- 172.21.15.77:0/2538620904 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555aa8719140 con 0x555aa8719ef0 2024-03-16T10:23:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.604+0000 7f43c6ffd640 1 --2- 172.21.15.77:0/2538620904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa874cd80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f43c00058c0 tx=0x7f43c0004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e56e4dddc3d7b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.610+0000 7f43bd7fa640 1 -- 172.21.15.77:0/2538620904 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c0023070 con 0x555aa8719ef0 2024-03-16T10:23:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.610+0000 7f43bd7fa640 1 -- 172.21.15.77:0/2538620904 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43c001d440 con 0x555aa8719ef0 2024-03-16T10:23:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.610+0000 7f43bd7fa640 1 -- 172.21.15.77:0/2538620904 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c0004990 con 0x555aa8719ef0 2024-03-16T10:23:06.612 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.611+0000 7f43d0a53740 1 -- 172.21.15.77:0/2538620904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 msgr2=0x555aa874cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.611+0000 7f43d0a53740 1 --2- 172.21.15.77:0/2538620904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa874cd80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f43c00058c0 tx=0x7f43c0004410 comp rx=0 tx=0).stop 2024-03-16T10:23:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.611+0000 7f43d0a53740 1 -- 172.21.15.77:0/2538620904 shutdown_connections 2024-03-16T10:23:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.612+0000 7f43d0a53740 1 --2- 172.21.15.77:0/2538620904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 0x555aa8745480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.612+0000 7f43d0a53740 1 --2- 172.21.15.77:0/2538620904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa874cd80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.612+0000 7f43d0a53740 1 --2- 172.21.15.77:0/2538620904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719500 0x555aa87183e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.612+0000 7f43d0a53740 1 -- 172.21.15.77:0/2538620904 >> 172.21.15.77:0/2538620904 conn(0x555aa8714a70 msgr2=0x555aa8717cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.612+0000 7f43d0a53740 1 -- 172.21.15.77:0/2538620904 shutdown_connections 2024-03-16T10:23:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.613+0000 7f43d0a53740 1 -- 172.21.15.77:0/2538620904 wait complete. 2024-03-16T10:23:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.613+0000 7f43d0a53740 1 -- start start 2024-03-16T10:23:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.614+0000 7f43d0a53740 1 Processor -- start 2024-03-16T10:23:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.614+0000 7f43d0a53740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719500 0x555aa8749f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.614+0000 7f43d0a53740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa874a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.614+0000 7f43d0a53740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 0x555aa874b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43d0a53740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f43a800f760 con 0x555aa8719500 2024-03-16T10:23:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43c7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719500 0x555aa8749f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43c6ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa874a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43c7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719500 0x555aa8749f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47188/0 (socket says 172.21.15.77:47188) 2024-03-16T10:23:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43c7fff640 1 -- 172.21.15.77:0/3311233586 learned_addr learned my addr 172.21.15.77:0/3311233586 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43d0a53740 1 -- 172.21.15.77:0/3311233586 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f43a800f090 con 0x555aa8719ef0 2024-03-16T10:23:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43d0a53740 1 -- 172.21.15.77:0/3311233586 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43a800c2c0 con 0x555aa871f060 2024-03-16T10:23:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43d0a53740 10 client.0 _ll_register_callbacks cb 0x7f43b0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43c7fff640 1 --2- 172.21.15.77:0/3311233586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 0x555aa874b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43c6ffd640 1 -- 172.21.15.77:0/3311233586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 msgr2=0x555aa874b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43c6ffd640 1 --2- 172.21.15.77:0/3311233586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 0x555aa874b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43c6ffd640 1 -- 172.21.15.77:0/3311233586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719500 msgr2=0x555aa8749f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43c6ffd640 1 --2- 172.21.15.77:0/3311233586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719500 0x555aa8749f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.615+0000 7f43c6ffd640 1 -- 172.21.15.77:0/3311233586 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555aa87442e0 con 0x555aa8719ef0 2024-03-16T10:23:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.616+0000 7f43c6ffd640 1 --2- 172.21.15.77:0/3311233586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa874a470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f43c0021040 tx=0x7f43c0009fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7afd4d152bac0f1e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.620+0000 7f43a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:23:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.620+0000 7f43a6ffd640 1 -- 172.21.15.77:0/3311233586 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c002f060 con 0x555aa8719ef0 2024-03-16T10:23:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.620+0000 7f43d0a53740 1 -- 172.21.15.77:0/3311233586 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43c001d440 con 0x555aa8719ef0 2024-03-16T10:23:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.621+0000 7f43a6ffd640 1 -- 172.21.15.77:0/3311233586 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f43c001dd90 con 0x555aa8719ef0 2024-03-16T10:23:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.624+0000 7f43a6ffd640 1 -- 172.21.15.77:0/3311233586 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f43c0023070 con 0x555aa8719ef0 2024-03-16T10:23:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.624+0000 7f43d0a53740 10 client.26786 fetch_fsmap learned FSMap version 37 2024-03-16T10:23:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.624+0000 7f43d0a53740 1 -- 172.21.15.77:0/3311233586 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x555aa874cb90 con 0x555aa8719ef0 2024-03-16T10:23:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.625+0000 7f43a6ffd640 1 -- 172.21.15.77:0/3311233586 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43c001d440 con 0x555aa8719ef0 2024-03-16T10:23:06.627 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.626+0000 7f43a6ffd640 1 -- 172.21.15.77:0/3311233586 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f43c0029070 con 0x555aa8719ef0 2024-03-16T10:23:06.627 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.626+0000 7f43d0a53740 10 client.26786 fetch_fsmap finished waiting for FSMap version 37 2024-03-16T10:23:06.627 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.626+0000 7f43d0a53740 10 client.26786 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:06.627 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.626+0000 7f43d0a53740 20 client.26786 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:06.627 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.626+0000 7f43d0a53740 1 --2- 172.21.15.77:0/3311233586 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555aa87b7c90 0x555aa87ba0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:06.627 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.626+0000 7f43d0a53740 4 client.26786 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:23:06.627 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.626+0000 7f43d0a53740 1 -- 172.21.15.77:0/3311233586 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555aa86862d0 con 0x555aa87b7c90 2024-03-16T10:23:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.627+0000 7f43c7fff640 1 --2- 172.21.15.77:0/3311233586 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555aa87b7c90 0x555aa87ba0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.627+0000 7f43c7fff640 1 --2- 172.21.15.77:0/3311233586 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555aa87b7c90 0x555aa87ba0b0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e018650a42c50ae4 server_cookie=a1d6a4211e0b07bc in_seq=0 out_seq=0 2024-03-16T10:23:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.627+0000 7f43a6ffd640 10 client.26786 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:06.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.829+0000 7f43a6ffd640 1 -- 172.21.15.77:0/3311233586 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555aa86862d0 con 0x555aa87b7c90 2024-03-16T10:23:06.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.829+0000 7f43a6ffd640 10 client.26786 handle_command_reply: tid=0 2024-03-16T10:23:06.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.829+0000 7f43d0a53740 1 client.26786 shutdown 2024-03-16T10:23:06.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.829+0000 7f43d0a53740 1 -- 172.21.15.77:0/3311233586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 msgr2=0x555aa874a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:06.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.829+0000 7f43d0a53740 1 --2- 172.21.15.77:0/3311233586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa874a470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f43c0021040 tx=0x7f43c0009fb0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.830+0000 7f43d0a53740 1 -- 172.21.15.77:0/3311233586 shutdown_connections 2024-03-16T10:23:06.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.830+0000 7f43d0a53740 1 --2- 172.21.15.77:0/3311233586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aa871f060 0x555aa874b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.830+0000 7f43d0a53740 1 --2- 172.21.15.77:0/3311233586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555aa8719ef0 0x555aa874a470 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.830+0000 7f43d0a53740 1 --2- 172.21.15.77:0/3311233586 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555aa87b7c90 0x555aa87ba0b0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.830+0000 7f43d0a53740 1 --2- 172.21.15.77:0/3311233586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555aa8719500 0x555aa8749f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.830+0000 7f43d0a53740 1 -- 172.21.15.77:0/3311233586 >> 172.21.15.77:0/3311233586 conn(0x555aa8714a70 msgr2=0x555aa8745480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:06.831 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.830+0000 7f43a6ffd640 0 client.26786 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:06.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.830+0000 7f43d0a53740 1 -- 172.21.15.77:0/3311233586 shutdown_connections 2024-03-16T10:23:06.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.830+0000 7f43d0a53740 1 -- 172.21.15.77:0/3311233586 wait complete. 2024-03-16T10:23:06.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.830+0000 7f43d0a53740 20 client.26786 trim_cache size 0 max 16384 2024-03-16T10:23:06.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:23:06.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:23:06.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:23:06.832 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:23:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.850+0000 7f43cde61640 1 -- 172.21.15.77:0/1606248890 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f43b40a2a30 msgr2=0x7f43b40a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:06.851 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.850+0000 7f43cde61640 1 --2- 172.21.15.77:0/1606248890 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f43b40a2a30 0x7f43b40a4ef0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f43b8000f80 tx=0x7f43b8097de0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.853 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.852+0000 7f43cde61640 1 -- 172.21.15.77:0/1606248890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c8072ab0 msgr2=0x7f43c8117da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:06.853 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.852+0000 7f43cde61640 1 --2- 172.21.15.77:0/1606248890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c8072ab0 0x7f43c8117da0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f43b800bd60 tx=0x7f43b8009f10 comp rx=0 tx=0).stop 2024-03-16T10:23:06.853 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.852+0000 7f43cde61640 1 -- 172.21.15.77:0/1606248890 shutdown_connections 2024-03-16T10:23:06.853 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.852+0000 7f43cde61640 1 --2- 172.21.15.77:0/1606248890 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f43b40a2a30 0x7f43b40a4ef0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.853 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.852+0000 7f43cde61640 1 --2- 172.21.15.77:0/1606248890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43c8109510 0x7f43c8119320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.854 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.852+0000 7f43cde61640 1 --2- 172.21.15.77:0/1606248890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43c8073410 0x7f43c81182e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.854 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.852+0000 7f43cde61640 1 --2- 172.21.15.77:0/1606248890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c8072ab0 0x7f43c8117da0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:06.854 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.853+0000 7f43cde61640 1 -- 172.21.15.77:0/1606248890 >> 172.21.15.77:0/1606248890 conn(0x7f43c806e650 msgr2=0x7f43c810c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:06.854 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.853+0000 7f43cde61640 1 -- 172.21.15.77:0/1606248890 shutdown_connections 2024-03-16T10:23:06.854 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:06.854+0000 7f43cde61640 1 -- 172.21.15.77:0/1606248890 wait complete. 2024-03-16T10:23:07.105 INFO:tasks.workunit.client.0.smithi077.stdout: 153 1790105 391 1225123 297 1022269 822 2024-03-16T10:23:08.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:07 smithi077 ceph-mon[32400]: pgmap v531: 97 pgs: 97 active+clean; 727 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.6 MiB/s wr, 812 op/s 2024-03-16T10:23:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:07 smithi193 ceph-mon[36311]: pgmap v531: 97 pgs: 97 active+clean; 727 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.6 MiB/s wr, 812 op/s 2024-03-16T10:23:08.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:07 smithi192 ceph-mon[36354]: pgmap v531: 97 pgs: 97 active+clean; 727 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.6 MiB/s wr, 812 op/s 2024-03-16T10:23:10.218 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:09 smithi192 ceph-mon[36354]: pgmap v532: 97 pgs: 97 active+clean; 727 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 700 op/s 2024-03-16T10:23:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:09 smithi077 ceph-mon[32400]: pgmap v532: 97 pgs: 97 active+clean; 727 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 700 op/s 2024-03-16T10:23:10.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:09 smithi193 ceph-mon[36311]: pgmap v532: 97 pgs: 97 active+clean; 727 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 700 op/s 2024-03-16T10:23:12.055 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:23:12.055 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:23:12.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:11 smithi077 ceph-mon[32400]: pgmap v533: 97 pgs: 97 active+clean; 726 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 611 op/s 2024-03-16T10:23:12.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:11 smithi193 ceph-mon[36311]: pgmap v533: 97 pgs: 97 active+clean; 726 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 611 op/s 2024-03-16T10:23:12.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:11 smithi192 ceph-mon[36354]: pgmap v533: 97 pgs: 97 active+clean; 726 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 611 op/s 2024-03-16T10:23:13.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:12 smithi077 ceph-mon[32400]: pgmap v534: 97 pgs: 97 active+clean; 722 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.2 MiB/s wr, 547 op/s 2024-03-16T10:23:13.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:12 smithi193 ceph-mon[36311]: pgmap v534: 97 pgs: 97 active+clean; 722 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.2 MiB/s wr, 547 op/s 2024-03-16T10:23:13.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:12 smithi192 ceph-mon[36354]: pgmap v534: 97 pgs: 97 active+clean; 722 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.2 MiB/s wr, 547 op/s 2024-03-16T10:23:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.079+0000 7f199b59e640 1 -- 172.21.15.77:0/3698899497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f199c109780 msgr2=0x7f199c10e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:15.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.079+0000 7f199b59e640 1 --2- 172.21.15.77:0/3698899497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f199c109780 0x7f199c10e040 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1994009be0 tx=0x7f199402bf10 comp rx=0 tx=0).stop 2024-03-16T10:23:15.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.100+0000 7f199b59e640 1 -- 172.21.15.77:0/3698899497 shutdown_connections 2024-03-16T10:23:15.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.100+0000 7f199b59e640 1 --2- 172.21.15.77:0/3698899497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f199c109780 0x7f199c10e040 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.100+0000 7f199b59e640 1 --2- 172.21.15.77:0/3698899497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199c108de0 0x7f199c109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.100+0000 7f199b59e640 1 --2- 172.21.15.77:0/3698899497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f199c072b30 0x7f199c072f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.100+0000 7f199b59e640 1 -- 172.21.15.77:0/3698899497 >> 172.21.15.77:0/3698899497 conn(0x7f199c06e600 msgr2=0x7f199c070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:15.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.100+0000 7f199b59e640 1 -- 172.21.15.77:0/3698899497 shutdown_connections 2024-03-16T10:23:15.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.100+0000 7f199b59e640 1 -- 172.21.15.77:0/3698899497 wait complete. 2024-03-16T10:23:15.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.101+0000 7f199b59e640 1 Processor -- start 2024-03-16T10:23:15.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.101+0000 7f199b59e640 1 -- start start 2024-03-16T10:23:15.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.101+0000 7f199b59e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f199c072b30 0x7f199c114d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.101+0000 7f199b59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199c108de0 0x7f199c115290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.101+0000 7f199b59e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f199c109780 0x7f199c1b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.101+0000 7f199b59e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f199c11c0e0 con 0x7f199c108de0 2024-03-16T10:23:15.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.101+0000 7f199b59e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f199c11bf60 con 0x7f199c109780 2024-03-16T10:23:15.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.101+0000 7f199b59e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f199c11c260 con 0x7f199c072b30 2024-03-16T10:23:15.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.102+0000 7f199ad9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f199c109780 0x7f199c1b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.102+0000 7f1999d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199c108de0 0x7f199c115290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.102+0000 7f199ad9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f199c109780 0x7f199c1b67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:36166/0 (socket says 172.21.15.77:36166) 2024-03-16T10:23:15.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.102+0000 7f199ad9d640 1 -- 172.21.15.77:0/1999125556 learned_addr learned my addr 172.21.15.77:0/1999125556 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:15.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.102+0000 7f199a59c640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f199c072b30 0x7f199c114d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.102+0000 7f199ad9d640 1 -- 172.21.15.77:0/1999125556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f199c072b30 msgr2=0x7f199c114d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:15.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.102+0000 7f199ad9d640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f199c072b30 0x7f199c114d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.102+0000 7f199ad9d640 1 -- 172.21.15.77:0/1999125556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199c108de0 msgr2=0x7f199c115290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:15.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.102+0000 7f199ad9d640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199c108de0 0x7f199c115290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.102+0000 7f199ad9d640 1 -- 172.21.15.77:0/1999125556 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199c118410 con 0x7f199c109780 2024-03-16T10:23:15.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.103+0000 7f1999d9b640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199c108de0 0x7f199c115290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:15.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.103+0000 7f199ad9d640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f199c109780 0x7f199c1b67b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1994009b30 tx=0x7f1994003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:15.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.113+0000 7f198b7fe640 1 -- 172.21.15.77:0/1999125556 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1994043020 con 0x7f199c109780 2024-03-16T10:23:15.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.113+0000 7f199b59e640 1 -- 172.21.15.77:0/1999125556 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199c1b6cf0 con 0x7f199c109780 2024-03-16T10:23:15.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.113+0000 7f199b59e640 1 -- 172.21.15.77:0/1999125556 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199c1b7190 con 0x7f199c109780 2024-03-16T10:23:15.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.118+0000 7f19a3a26740 1 Processor -- start 2024-03-16T10:23:15.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.119+0000 7f19a3a26740 1 -- start start 2024-03-16T10:23:15.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.119+0000 7f19a3a26740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5c380 0x5560aed5c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.119+0000 7f19a3a26740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed5cd70 0x5560aed61870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.119+0000 7f19a3a26740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed61ee0 0x5560aed64370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.119+0000 7f19a3a26740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5560aecc1500 con 0x5560aed5c380 2024-03-16T10:23:15.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.119+0000 7f19a3a26740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5560aec69be0 con 0x5560aed61ee0 2024-03-16T10:23:15.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.119+0000 7f19a3a26740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5560aecbd5a0 con 0x5560aed5cd70 2024-03-16T10:23:15.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.119+0000 7f198b7fe640 1 -- 172.21.15.77:0/1999125556 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1994003e50 con 0x7f199c109780 2024-03-16T10:23:15.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.119+0000 7f198b7fe640 1 -- 172.21.15.77:0/1999125556 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1994034cd0 con 0x7f199c109780 2024-03-16T10:23:15.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.120+0000 7f1999d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5c380 0x5560aed5c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.120+0000 7f1999d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5c380 0x5560aed5c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43256/0 (socket says 172.21.15.77:43256) 2024-03-16T10:23:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.120+0000 7f1999d9b640 1 -- 172.21.15.77:0/1520627779 learned_addr learned my addr 172.21.15.77:0/1520627779 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.120+0000 7f1999d9b640 1 -- 172.21.15.77:0/1520627779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed5cd70 msgr2=0x5560aed61870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:23:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.120+0000 7f1999d9b640 1 --2- 172.21.15.77:0/1520627779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed5cd70 0x5560aed61870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.120+0000 7f1999d9b640 1 -- 172.21.15.77:0/1520627779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed61ee0 msgr2=0x5560aed64370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:23:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.120+0000 7f1999d9b640 1 --2- 172.21.15.77:0/1520627779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed61ee0 0x5560aed64370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.121+0000 7f1999d9b640 1 -- 172.21.15.77:0/1520627779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560aec7d4b0 con 0x5560aed5c380 2024-03-16T10:23:15.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.121+0000 7f1999d9b640 1 --2- 172.21.15.77:0/1520627779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5c380 0x5560aed5c7a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f198c0114c0 tx=0x7f198c011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc70a31b5c251b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:15.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.123+0000 7f198b7fe640 1 -- 172.21.15.77:0/1999125556 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f1994046020 con 0x7f199c109780 2024-03-16T10:23:15.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.124+0000 7f198b7fe640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f19840a2a10 0x7f19840a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.124+0000 7f198b7fe640 1 -- 172.21.15.77:0/1999125556 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f19940f13e0 con 0x7f199c109780 2024-03-16T10:23:15.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.128+0000 7f199a59c640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f19840a2a10 0x7f19840a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.129+0000 7f199a59c640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f19840a2a10 0x7f19840a4ed0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1990005e10 tx=0x7f1990005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:15.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.134+0000 7f1988ff9640 1 -- 172.21.15.77:0/1520627779 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c004020 con 0x5560aed5c380 2024-03-16T10:23:15.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.136+0000 7f1988ff9640 1 -- 172.21.15.77:0/1520627779 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f198c005500 con 0x5560aed5c380 2024-03-16T10:23:15.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.138+0000 7f19a3a26740 1 -- 172.21.15.77:0/1520627779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5c380 msgr2=0x5560aed5c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:15.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.138+0000 7f19a3a26740 1 --2- 172.21.15.77:0/1520627779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5c380 0x5560aed5c7a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f198c0114c0 tx=0x7f198c011980 comp rx=0 tx=0).stop 2024-03-16T10:23:15.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.138+0000 7f19a3a26740 1 -- 172.21.15.77:0/1520627779 shutdown_connections 2024-03-16T10:23:15.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.138+0000 7f19a3a26740 1 --2- 172.21.15.77:0/1520627779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed61ee0 0x5560aed64370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.138+0000 7f19a3a26740 1 --2- 172.21.15.77:0/1520627779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed5cd70 0x5560aed61870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.138+0000 7f19a3a26740 1 --2- 172.21.15.77:0/1520627779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5c380 0x5560aed5c7a0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.138+0000 7f19a3a26740 1 -- 172.21.15.77:0/1520627779 >> 172.21.15.77:0/1520627779 conn(0x5560aed578f0 msgr2=0x5560aed57cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:15.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.138+0000 7f19a3a26740 1 -- 172.21.15.77:0/1520627779 shutdown_connections 2024-03-16T10:23:15.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.138+0000 7f19a3a26740 1 -- 172.21.15.77:0/1520627779 wait complete. 2024-03-16T10:23:15.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.139+0000 7f19a3a26740 1 -- start start 2024-03-16T10:23:15.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.139+0000 7f19a3a26740 1 Processor -- start 2024-03-16T10:23:15.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.139+0000 7f19a3a26740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed5cd70 0x5560aed66760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.139+0000 7f19a3a26740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 0x5560aed66ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.139+0000 7f19a3a26740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed6bcd0 0x5560aed6e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.139+0000 7f19a3a26740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5560aed64ee0 con 0x5560aed61ee0 2024-03-16T10:23:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.139+0000 7f19a3a26740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5560aec7d160 con 0x5560aed6bcd0 2024-03-16T10:23:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.139+0000 7f19a3a26740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5560aed65060 con 0x5560aed5cd70 2024-03-16T10:23:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.139+0000 7f19a3a26740 10 client.0 _ll_register_callbacks cb 0x5560aec61150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:15.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.140+0000 7f1999d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 0x5560aed66ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.140+0000 7f1999d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 0x5560aed66ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43268/0 (socket says 172.21.15.77:43268) 2024-03-16T10:23:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.140+0000 7f1999d9b640 1 -- 172.21.15.77:0/2436121482 learned_addr learned my addr 172.21.15.77:0/2436121482 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.140+0000 7f199a59c640 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed5cd70 0x5560aed66760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.140+0000 7f199a59c640 1 -- 172.21.15.77:0/2436121482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed6bcd0 msgr2=0x5560aed6e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.140+0000 7f199ad9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed6bcd0 0x5560aed6e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.141+0000 7f199a59c640 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed6bcd0 0x5560aed6e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.141+0000 7f199a59c640 1 -- 172.21.15.77:0/2436121482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 msgr2=0x5560aed66ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.141+0000 7f199a59c640 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 0x5560aed66ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.141+0000 7f199a59c640 1 -- 172.21.15.77:0/2436121482 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560aea9f7d0 con 0x5560aed5cd70 2024-03-16T10:23:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.141+0000 7f1999d9b640 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 0x5560aed66ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.141+0000 7f199ad9d640 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed6bcd0 0x5560aed6e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:23:15.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.141+0000 7f199a59c640 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed5cd70 0x5560aed66760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x5560aed5d170 tx=0x7f19900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b66bd45b68d8fd42 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.158+0000 7f1981ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:23:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.158+0000 7f1981ffb640 1 -- 172.21.15.77:0/2436121482 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1990021070 con 0x5560aed5cd70 2024-03-16T10:23:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.158+0000 7f19a3a26740 1 -- 172.21.15.77:0/2436121482 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5560aeccea40 con 0x5560aed5cd70 2024-03-16T10:23:15.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.158+0000 7f1981ffb640 1 -- 172.21.15.77:0/2436121482 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f199000fa90 con 0x5560aed5cd70 2024-03-16T10:23:15.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.166+0000 7f1981ffb640 1 -- 172.21.15.77:0/2436121482 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f199002d050 con 0x5560aed5cd70 2024-03-16T10:23:15.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.167+0000 7f19a3a26740 10 client.26773 fetch_fsmap learned FSMap version 37 2024-03-16T10:23:15.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.167+0000 7f19a3a26740 1 -- 172.21.15.77:0/2436121482 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5560ae9d53f0 con 0x5560aed5cd70 2024-03-16T10:23:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.171+0000 7f1981ffb640 1 -- 172.21.15.77:0/2436121482 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5560aeccea40 con 0x5560aed5cd70 2024-03-16T10:23:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.171+0000 7f1981ffb640 1 -- 172.21.15.77:0/2436121482 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f19900253b0 con 0x5560aed5cd70 2024-03-16T10:23:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.171+0000 7f19a3a26740 10 client.26773 fetch_fsmap finished waiting for FSMap version 37 2024-03-16T10:23:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.171+0000 7f19a3a26740 10 client.26773 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.171+0000 7f19a3a26740 20 client.26773 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.171+0000 7f19a3a26740 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5560aed726d0 0x5560aed74b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.171+0000 7f19a3a26740 4 client.26773 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.171+0000 7f19a3a26740 1 -- 172.21.15.77:0/2436121482 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560aecc8d90 con 0x5560aed726d0 2024-03-16T10:23:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.171+0000 7f1999d9b640 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5560aed726d0 0x5560aed74b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.172+0000 7f1999d9b640 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5560aed726d0 0x5560aed74b70 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58f5f48d133e399b server_cookie=55f635e056bed32b in_seq=0 out_seq=0 2024-03-16T10:23:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.174+0000 7f1981ffb640 10 client.26773 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.174+0000 7f1981ffb640 1 -- 172.21.15.77:0/2436121482 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5560aecc8d90 con 0x5560aed726d0 2024-03-16T10:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.174+0000 7f1981ffb640 10 client.26773 handle_command_reply: tid=0 2024-03-16T10:23:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.174+0000 7f19a3a26740 1 client.26773 shutdown 2024-03-16T10:23:15.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.175+0000 7f19a3a26740 1 -- 172.21.15.77:0/2436121482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed5cd70 msgr2=0x5560aed66760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:15.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.175+0000 7f19a3a26740 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed5cd70 0x5560aed66760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x5560aed5d170 tx=0x7f19900046c0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.175+0000 7f19a3a26740 1 -- 172.21.15.77:0/2436121482 shutdown_connections 2024-03-16T10:23:15.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.175+0000 7f19a3a26740 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed6bcd0 0x5560aed6e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.175+0000 7f19a3a26740 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5560aed726d0 0x5560aed74b70 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.175+0000 7f19a3a26740 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 0x5560aed66ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.175+0000 7f19a3a26740 1 --2- 172.21.15.77:0/2436121482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed5cd70 0x5560aed66760 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.175+0000 7f19a3a26740 1 -- 172.21.15.77:0/2436121482 >> 172.21.15.77:0/2436121482 conn(0x5560aed578f0 msgr2=0x5560aed5c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.176+0000 7f1981ffb640 0 client.26773 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.178+0000 7f19a3a26740 1 -- 172.21.15.77:0/2436121482 shutdown_connections 2024-03-16T10:23:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.178+0000 7f19a3a26740 1 -- 172.21.15.77:0/2436121482 wait complete. 2024-03-16T10:23:15.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.178+0000 7f19a3a26740 20 client.26773 trim_cache size 0 max 16384 2024-03-16T10:23:15.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.277+0000 7f19a3a26740 1 Processor -- start 2024-03-16T10:23:15.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.277+0000 7f19a3a26740 1 -- start start 2024-03-16T10:23:15.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.278+0000 7f19a3a26740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed578f0 0x5560aed7a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.278+0000 7f19a3a26740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5cd70 0x5560aed5ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.278+0000 7f199a59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed578f0 0x5560aed7a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.278+0000 7f199a59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed578f0 0x5560aed7a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:36182/0 (socket says 172.21.15.77:36182) 2024-03-16T10:23:15.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.278+0000 7f1999d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5cd70 0x5560aed5ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.278+0000 7f1999d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5cd70 0x5560aed5ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43276/0 (socket says 172.21.15.77:43276) 2024-03-16T10:23:15.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.278+0000 7f199a59c640 1 -- 172.21.15.77:0/4249380527 learned_addr learned my addr 172.21.15.77:0/4249380527 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:15.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.279+0000 7f19a3a26740 1 --2- 172.21.15.77:0/4249380527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed61ee0 0x5560aed5b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.279+0000 7f19a3a26740 1 -- 172.21.15.77:0/4249380527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f197c00e470 con 0x5560aed5cd70 2024-03-16T10:23:15.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.279+0000 7f19a3a26740 1 -- 172.21.15.77:0/4249380527 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f197c010070 con 0x5560aed578f0 2024-03-16T10:23:15.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.279+0000 7f19a3a26740 1 -- 172.21.15.77:0/4249380527 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f197c00eb40 con 0x5560aed61ee0 2024-03-16T10:23:15.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.279+0000 7f199ad9d640 1 --2- 172.21.15.77:0/4249380527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed61ee0 0x5560aed5b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.279+0000 7f1999d9b640 1 -- 172.21.15.77:0/4249380527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed61ee0 msgr2=0x5560aed5b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:15.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.279+0000 7f1999d9b640 1 --2- 172.21.15.77:0/4249380527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed61ee0 0x5560aed5b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.279+0000 7f1999d9b640 1 -- 172.21.15.77:0/4249380527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed578f0 msgr2=0x5560aed7a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:15.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.279+0000 7f1999d9b640 1 --2- 172.21.15.77:0/4249380527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed578f0 0x5560aed7a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.279+0000 7f1999d9b640 1 -- 172.21.15.77:0/4249380527 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560ae9950f0 con 0x5560aed5cd70 2024-03-16T10:23:15.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.280+0000 7f1999d9b640 1 --2- 172.21.15.77:0/4249380527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5cd70 0x5560aed5ab50 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f198c011490 tx=0x7f198c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f11c26651c9c65 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:15.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.287+0000 7f1983fff640 1 -- 172.21.15.77:0/4249380527 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c01d040 con 0x5560aed5cd70 2024-03-16T10:23:15.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.287+0000 7f1983fff640 1 -- 172.21.15.77:0/4249380527 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f198c005100 con 0x5560aed5cd70 2024-03-16T10:23:15.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.288+0000 7f19a3a26740 1 -- 172.21.15.77:0/4249380527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5cd70 msgr2=0x5560aed5ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:15.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.288+0000 7f19a3a26740 1 --2- 172.21.15.77:0/4249380527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5cd70 0x5560aed5ab50 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f198c011490 tx=0x7f198c005850 comp rx=0 tx=0).stop 2024-03-16T10:23:15.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.289+0000 7f19a3a26740 1 -- 172.21.15.77:0/4249380527 shutdown_connections 2024-03-16T10:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.289+0000 7f19a3a26740 1 --2- 172.21.15.77:0/4249380527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed61ee0 0x5560aed5b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.289+0000 7f19a3a26740 1 --2- 172.21.15.77:0/4249380527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed5cd70 0x5560aed5ab50 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.289+0000 7f19a3a26740 1 --2- 172.21.15.77:0/4249380527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed578f0 0x5560aed7a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.289+0000 7f19a3a26740 1 -- 172.21.15.77:0/4249380527 >> 172.21.15.77:0/4249380527 conn(0x5560aed58040 msgr2=0x5560aed746e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.289+0000 7f19a3a26740 1 -- 172.21.15.77:0/4249380527 shutdown_connections 2024-03-16T10:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.289+0000 7f19a3a26740 1 -- 172.21.15.77:0/4249380527 wait complete. 2024-03-16T10:23:15.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.290+0000 7f19a3a26740 1 -- start start 2024-03-16T10:23:15.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.290+0000 7f19a3a26740 1 Processor -- start 2024-03-16T10:23:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.291+0000 7f19a3a26740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed578f0 0x5560aed83110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.291+0000 7f19a3a26740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed5cd70 0x5560aed83650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.292+0000 7f199a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed578f0 0x5560aed83110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.292+0000 7f199a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed578f0 0x5560aed83110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:60122/0 (socket says 172.21.15.77:60122) 2024-03-16T10:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.292+0000 7f19a3a26740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 0x5560aed84660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.292+0000 7f1999d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed5cd70 0x5560aed83650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.292+0000 7f1999d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed5cd70 0x5560aed83650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:36196/0 (socket says 172.21.15.77:36196) 2024-03-16T10:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.292+0000 7f199ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 0x5560aed84660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.292+0000 7f199ad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 0x5560aed84660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43290/0 (socket says 172.21.15.77:43290) 2024-03-16T10:23:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.292+0000 7f199a59c640 1 -- 172.21.15.77:0/2599988108 learned_addr learned my addr 172.21.15.77:0/2599988108 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.292+0000 7f19a3a26740 1 -- 172.21.15.77:0/2599988108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f197c00f9a0 con 0x5560aed61ee0 2024-03-16T10:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.292+0000 7f19a3a26740 1 -- 172.21.15.77:0/2599988108 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f197c00f2d0 con 0x5560aed5cd70 2024-03-16T10:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.292+0000 7f19a3a26740 1 -- 172.21.15.77:0/2599988108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f197c00c450 con 0x5560aed578f0 2024-03-16T10:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.293+0000 7f1999d9b640 1 -- 172.21.15.77:0/2599988108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed578f0 msgr2=0x5560aed83110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.293+0000 7f1999d9b640 1 --2- 172.21.15.77:0/2599988108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed578f0 0x5560aed83110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.293+0000 7f1999d9b640 1 -- 172.21.15.77:0/2599988108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 msgr2=0x5560aed84660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.292+0000 7f19a3a26740 10 client.0 _ll_register_callbacks cb 0x7f1990014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:15.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.293+0000 7f1999d9b640 1 --2- 172.21.15.77:0/2599988108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 0x5560aed84660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.293+0000 7f1999d9b640 1 -- 172.21.15.77:0/2599988108 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560aec889d0 con 0x5560aed5cd70 2024-03-16T10:23:15.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.293+0000 7f199ad9d640 1 --2- 172.21.15.77:0/2599988108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 0x5560aed84660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:15.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.293+0000 7f19a3a26740 1 -- 172.21.15.77:0/2599988108 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f198c005100 con 0x5560aed5cd70 2024-03-16T10:23:15.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.293+0000 7f1999d9b640 1 --2- 172.21.15.77:0/2599988108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed5cd70 0x5560aed83650 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f198c0058c0 tx=0x7f198c02a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cbf9339d7b79be8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:15.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.296+0000 7f1981ffb640 10 client.26795 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:23:15.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.296+0000 7f1981ffb640 1 -- 172.21.15.77:0/2599988108 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c01d030 con 0x5560aed5cd70 2024-03-16T10:23:15.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.297+0000 7f1981ffb640 1 -- 172.21.15.77:0/2599988108 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7f198c0052b0 con 0x5560aed5cd70 2024-03-16T10:23:15.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.297+0000 7f1981ffb640 1 -- 172.21.15.77:0/2599988108 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f198c02f070 con 0x5560aed5cd70 2024-03-16T10:23:15.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.297+0000 7f19a3a26740 10 client.26795 fetch_fsmap learned FSMap version 37 2024-03-16T10:23:15.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.297+0000 7f19a3a26740 1 -- 172.21.15.77:0/2599988108 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({fsmap=37}) -- 0x5560aed85c80 con 0x5560aed5cd70 2024-03-16T10:23:15.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.298+0000 7f1981ffb640 1 -- 172.21.15.77:0/2599988108 <== mon.2 v2:172.21.15.192:3300/0 4 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7f198c02a6c0 con 0x5560aed5cd70 2024-03-16T10:23:15.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.298+0000 7f19a3a26740 10 client.26795 fetch_fsmap finished waiting for FSMap version 37 2024-03-16T10:23:15.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.298+0000 7f19a3a26740 10 client.26795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:15.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.298+0000 7f19a3a26740 20 client.26795 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:15.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.298+0000 7f19a3a26740 1 --2- 172.21.15.77:0/2599988108 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5560aed6bcd0 0x5560aedda6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:15.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.298+0000 7f19a3a26740 4 client.26795 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:23:15.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.298+0000 7f19a3a26740 1 -- 172.21.15.77:0/2599988108 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5560aec46540 con 0x5560aed6bcd0 2024-03-16T10:23:15.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.298+0000 7f199a59c640 1 --2- 172.21.15.77:0/2599988108 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5560aed6bcd0 0x5560aedda6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:15.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.299+0000 7f199a59c640 1 --2- 172.21.15.77:0/2599988108 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5560aed6bcd0 0x5560aedda6e0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71c7a43c8cc22bf1 server_cookie=fa18e02785e8d497 in_seq=0 out_seq=0 2024-03-16T10:23:15.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.299+0000 7f1981ffb640 10 client.26795 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:15.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.299+0000 7f1981ffb640 1 -- 172.21.15.77:0/2599988108 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5560aec46540 con 0x5560aed6bcd0 2024-03-16T10:23:15.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.299+0000 7f1981ffb640 10 client.26795 handle_command_reply: tid=0 2024-03-16T10:23:15.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.299+0000 7f19a3a26740 1 client.26795 shutdown 2024-03-16T10:23:15.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.300+0000 7f19a3a26740 1 -- 172.21.15.77:0/2599988108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed5cd70 msgr2=0x5560aed83650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:15.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.300+0000 7f19a3a26740 1 --2- 172.21.15.77:0/2599988108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed5cd70 0x5560aed83650 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f198c0058c0 tx=0x7f198c02a3f0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.312+0000 7f19a3a26740 1 -- 172.21.15.77:0/2599988108 shutdown_connections 2024-03-16T10:23:15.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.312+0000 7f19a3a26740 1 --2- 172.21.15.77:0/2599988108 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5560aed6bcd0 0x5560aedda6e0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.312+0000 7f19a3a26740 1 --2- 172.21.15.77:0/2599988108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5560aed61ee0 0x5560aed84660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.312+0000 7f19a3a26740 1 --2- 172.21.15.77:0/2599988108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560aed5cd70 0x5560aed83650 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.312+0000 7f19a3a26740 1 --2- 172.21.15.77:0/2599988108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560aed578f0 0x5560aed83110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.312+0000 7f19a3a26740 1 -- 172.21.15.77:0/2599988108 >> 172.21.15.77:0/2599988108 conn(0x5560aed58040 msgr2=0x5560aed5ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:15.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.312+0000 7f1981ffb640 0 client.26795 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.312+0000 7f19a3a26740 1 -- 172.21.15.77:0/2599988108 shutdown_connections 2024-03-16T10:23:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.312+0000 7f19a3a26740 1 -- 172.21.15.77:0/2599988108 wait complete. 2024-03-16T10:23:15.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.313+0000 7f19a3a26740 20 client.26795 trim_cache size 0 max 16384 2024-03-16T10:23:15.314 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:23:15.314 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:23:15.314 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:23:15.314 INFO:teuthology.orchestra.run.smithi077.stdout: "1f1adeef-2126-4ee3-aca7-819aadf3baf6": { 2024-03-16T10:23:15.314 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:23:15.315 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "1f1adeef-2126-4ee3-aca7-819aadf3baf6", 2024-03-16T10:23:15.315 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:23:15.315 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:23:15.315 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:23:15.315 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:23:15.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.325+0000 7f199b59e640 1 -- 172.21.15.77:0/1999125556 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f19840a2a10 msgr2=0x7f19840a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:15.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.325+0000 7f199b59e640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f19840a2a10 0x7f19840a4ed0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1990005e10 tx=0x7f1990005da0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.325+0000 7f199b59e640 1 -- 172.21.15.77:0/1999125556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f199c109780 msgr2=0x7f199c1b67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:15.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.325+0000 7f199b59e640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f199c109780 0x7f199c1b67b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1994009b30 tx=0x7f1994003ce0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.326+0000 7f199b59e640 1 -- 172.21.15.77:0/1999125556 shutdown_connections 2024-03-16T10:23:15.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.326+0000 7f199b59e640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f199c109780 0x7f199c1b67b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.326+0000 7f199b59e640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f199c108de0 0x7f199c115290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.326+0000 7f199b59e640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f19840a2a10 0x7f19840a4ed0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.326+0000 7f199b59e640 1 --2- 172.21.15.77:0/1999125556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f199c072b30 0x7f199c114d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:15.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.326+0000 7f199b59e640 1 -- 172.21.15.77:0/1999125556 >> 172.21.15.77:0/1999125556 conn(0x7f199c06e600 msgr2=0x7f199c10a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:15.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.329+0000 7f199b59e640 1 -- 172.21.15.77:0/1999125556 shutdown_connections 2024-03-16T10:23:15.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:15.330+0000 7f199b59e640 1 -- 172.21.15.77:0/1999125556 wait complete. 2024-03-16T10:23:15.523 INFO:tasks.cephfs.filesystem:scrub status for tag:1f1adeef-2126-4ee3-aca7-819aadf3baf6 - {'path': '/', 'tag': '1f1adeef-2126-4ee3-aca7-819aadf3baf6', 'options': 'recursive,force'} 2024-03-16T10:23:16.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:15 smithi192 ceph-mon[36354]: pgmap v535: 97 pgs: 97 active+clean; 714 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 458 op/s 2024-03-16T10:23:16.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:15 smithi077 ceph-mon[32400]: pgmap v535: 97 pgs: 97 active+clean; 714 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 458 op/s 2024-03-16T10:23:16.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:15 smithi193 ceph-mon[36311]: pgmap v535: 97 pgs: 97 active+clean; 714 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 458 op/s 2024-03-16T10:23:17.105 INFO:tasks.workunit.client.0.smithi077.stdout: 160 1986477 507 1354067 348 1118117 1299 2024-03-16T10:23:18.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:17 smithi192 ceph-mon[36354]: pgmap v536: 97 pgs: 97 active+clean; 701 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 483 op/s 2024-03-16T10:23:18.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:17 smithi077 ceph-mon[32400]: pgmap v536: 97 pgs: 97 active+clean; 701 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 483 op/s 2024-03-16T10:23:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:17 smithi193 ceph-mon[36311]: pgmap v536: 97 pgs: 97 active+clean; 701 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.3 MiB/s wr, 483 op/s 2024-03-16T10:23:20.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:19 smithi192 ceph-mon[36354]: pgmap v537: 97 pgs: 97 active+clean; 705 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 450 op/s 2024-03-16T10:23:20.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:19 smithi077 ceph-mon[32400]: pgmap v537: 97 pgs: 97 active+clean; 705 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 450 op/s 2024-03-16T10:23:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:19 smithi193 ceph-mon[36311]: pgmap v537: 97 pgs: 97 active+clean; 705 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 450 op/s 2024-03-16T10:23:21.780 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.779+0000 7fa0ed7df640 1 -- 172.21.15.77:0/2435337085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e8072ab0 msgr2=0x7fa0e8072ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:21.780 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.779+0000 7fa0ed7df640 1 --2- 172.21.15.77:0/2435337085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e8072ab0 0x7fa0e8072ed0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa0d801a910 tx=0x7fa0d803cca0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.782+0000 7fa0ed7df640 1 -- 172.21.15.77:0/2435337085 shutdown_connections 2024-03-16T10:23:21.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.782+0000 7fa0ed7df640 1 --2- 172.21.15.77:0/2435337085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e8109510 0x7fa0e810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.782+0000 7fa0ed7df640 1 --2- 172.21.15.77:0/2435337085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0e8073410 0x7fa0e8108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.782+0000 7fa0ed7df640 1 --2- 172.21.15.77:0/2435337085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e8072ab0 0x7fa0e8072ed0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.782+0000 7fa0ed7df640 1 -- 172.21.15.77:0/2435337085 >> 172.21.15.77:0/2435337085 conn(0x7fa0e806e650 msgr2=0x7fa0e8070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.802+0000 7fa0ed7df640 1 -- 172.21.15.77:0/2435337085 shutdown_connections 2024-03-16T10:23:21.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.802+0000 7fa0ed7df640 1 -- 172.21.15.77:0/2435337085 wait complete. 2024-03-16T10:23:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.802+0000 7fa0ed7df640 1 Processor -- start 2024-03-16T10:23:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.803+0000 7fa0ed7df640 1 -- start start 2024-03-16T10:23:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.803+0000 7fa0ed7df640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0e8072ab0 0x7fa0e819e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.803+0000 7fa0ed7df640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e8073410 0x7fa0e819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.803+0000 7fa0ed7df640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e8109510 0x7fa0e81a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.803+0000 7fa0ed7df640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e8111a70 con 0x7fa0e8073410 2024-03-16T10:23:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.803+0000 7fa0ed7df640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e81118f0 con 0x7fa0e8109510 2024-03-16T10:23:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.803+0000 7fa0ed7df640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e8111bf0 con 0x7fa0e8072ab0 2024-03-16T10:23:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.803+0000 7fa0e77fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e8109510 0x7fa0e81a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.803+0000 7fa0e77fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e8109510 0x7fa0e81a56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:35682/0 (socket says 172.21.15.77:35682) 2024-03-16T10:23:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.803+0000 7fa0e77fe640 1 -- 172.21.15.77:0/1021559617 learned_addr learned my addr 172.21.15.77:0/1021559617 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.804+0000 7fa0e77fe640 1 -- 172.21.15.77:0/1021559617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0e8072ab0 msgr2=0x7fa0e819e130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:23:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.804+0000 7fa0e77fe640 1 --2- 172.21.15.77:0/1021559617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0e8072ab0 0x7fa0e819e130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.804+0000 7fa0e77fe640 1 -- 172.21.15.77:0/1021559617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e8073410 msgr2=0x7fa0e819e670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:23:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.804+0000 7fa0e77fe640 1 --2- 172.21.15.77:0/1021559617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e8073410 0x7fa0e819e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.804+0000 7fa0e77fe640 1 -- 172.21.15.77:0/1021559617 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e81a5de0 con 0x7fa0e8109510 2024-03-16T10:23:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.804+0000 7fa0e77fe640 1 --2- 172.21.15.77:0/1021559617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e8109510 0x7fa0e81a56e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00b3f0 tx=0x7fa0dc00b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:21.827 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.826+0000 7fa0cffff640 1 -- 172.21.15.77:0/1021559617 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc015020 con 0x7fa0e8109510 2024-03-16T10:23:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.828+0000 7fa0ed7df640 1 -- 172.21.15.77:0/1021559617 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e81a60d0 con 0x7fa0e8109510 2024-03-16T10:23:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.828+0000 7fa0ed7df640 1 -- 172.21.15.77:0/1021559617 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e81a6620 con 0x7fa0e8109510 2024-03-16T10:23:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.829+0000 7fa0cffff640 1 -- 172.21.15.77:0/1021559617 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0dc011d10 con 0x7fa0e8109510 2024-03-16T10:23:21.829 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.829+0000 7fa0cffff640 1 -- 172.21.15.77:0/1021559617 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc00f2e0 con 0x7fa0e8109510 2024-03-16T10:23:21.833 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.832+0000 7fa0cffff640 1 -- 172.21.15.77:0/1021559617 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fa0dc018020 con 0x7fa0e8109510 2024-03-16T10:23:21.837 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.837+0000 7fa0cffff640 1 --2- 172.21.15.77:0/1021559617 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0d00a2a00 0x7fa0d00a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:21.845 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.837+0000 7fa0cffff640 1 -- 172.21.15.77:0/1021559617 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa0dc0ce140 con 0x7fa0e8109510 2024-03-16T10:23:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.845+0000 7fa0f03c6740 1 Processor -- start 2024-03-16T10:23:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.845+0000 7fa0e6ffd640 1 --2- 172.21.15.77:0/1021559617 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0d00a2a00 0x7fa0d00a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.845+0000 7fa0f03c6740 1 -- start start 2024-03-16T10:23:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.845+0000 7fa0f03c6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864500 0x5559a9864920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:21.846 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.845+0000 7fa0f03c6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864ef0 0x5559a98699f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.845+0000 7fa0f03c6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a986a060 0x5559a986c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.845+0000 7fa0f03c6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5559a97c9a40 con 0x5559a986a060 2024-03-16T10:23:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.845+0000 7fa0f03c6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5559a9772920 con 0x5559a9864500 2024-03-16T10:23:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.845+0000 7fa0f03c6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5559a97c5ae0 con 0x5559a9864ef0 2024-03-16T10:23:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.846+0000 7fa0e67fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864ef0 0x5559a98699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.846+0000 7fa0e67fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864ef0 0x5559a98699f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:45306/0 (socket says 172.21.15.77:45306) 2024-03-16T10:23:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.846+0000 7fa0e67fc640 1 -- 172.21.15.77:0/3307248403 learned_addr learned my addr 172.21.15.77:0/3307248403 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.846+0000 7fa0e6ffd640 1 --2- 172.21.15.77:0/3307248403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864500 0x5559a9864920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:21.847 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.846+0000 7fa0e67fc640 1 -- 172.21.15.77:0/3307248403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864500 msgr2=0x5559a9864920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:21.848 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.846+0000 7fa0e67fc640 1 --2- 172.21.15.77:0/3307248403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864500 0x5559a9864920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.848 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.846+0000 7fa0e67fc640 1 -- 172.21.15.77:0/3307248403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a986a060 msgr2=0x5559a986c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:21.848 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.846+0000 7fa0e67fc640 1 --2- 172.21.15.77:0/3307248403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a986a060 0x5559a986c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.848 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.846+0000 7fa0e67fc640 1 -- 172.21.15.77:0/3307248403 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559a978a460 con 0x5559a9864ef0 2024-03-16T10:23:21.848 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.846+0000 7fa0e67fc640 1 --2- 172.21.15.77:0/3307248403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864ef0 0x5559a98699f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009dd0 tx=0x7fa0e000e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53f7879deff46a66 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.873+0000 7fa0cd7fa640 1 -- 172.21.15.77:0/3307248403 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0004b20 con 0x5559a9864ef0 2024-03-16T10:23:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.879+0000 7fa0cd7fa640 1 -- 172.21.15.77:0/3307248403 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0e0003e50 con 0x5559a9864ef0 2024-03-16T10:23:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.880+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3307248403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864ef0 msgr2=0x5559a98699f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.880+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/3307248403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864ef0 0x5559a98699f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009dd0 tx=0x7fa0e000e720 comp rx=0 tx=0).stop 2024-03-16T10:23:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.878+0000 7fa0e6ffd640 1 --2- 172.21.15.77:0/1021559617 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0d00a2a00 0x7fa0d00a4ec0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x5559a9865750 tx=0x7fa0d8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.885+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3307248403 shutdown_connections 2024-03-16T10:23:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.885+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/3307248403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a986a060 0x5559a986c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.885+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/3307248403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864ef0 0x5559a98699f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.885+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/3307248403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864500 0x5559a9864920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.885+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3307248403 >> 172.21.15.77:0/3307248403 conn(0x5559a985fa70 msgr2=0x5559a985fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:21.900 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.900+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3307248403 shutdown_connections 2024-03-16T10:23:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.904+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3307248403 wait complete. 2024-03-16T10:23:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.904+0000 7fa0f03c6740 1 -- start start 2024-03-16T10:23:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.904+0000 7fa0f03c6740 1 Processor -- start 2024-03-16T10:23:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.908+0000 7fa0f03c6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 0x5559a986eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.908+0000 7fa0f03c6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a986f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.908+0000 7fa0f03c6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a986a060 0x5559a9876050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.908+0000 7fa0f03c6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5559a986d060 con 0x5559a986a060 2024-03-16T10:23:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.908+0000 7fa0f03c6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5559a9785d70 con 0x5559a9864ef0 2024-03-16T10:23:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.908+0000 7fa0f03c6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5559a986d1e0 con 0x5559a9864500 2024-03-16T10:23:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.908+0000 7fa0f03c6740 10 client.0 _ll_register_callbacks cb 0x5559a94dda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.910+0000 7fa0e6ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 0x5559a986eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.910+0000 7fa0e6ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 0x5559a986eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:45308/0 (socket says 172.21.15.77:45308) 2024-03-16T10:23:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.910+0000 7fa0e6ffd640 1 -- 172.21.15.77:0/3229421983 learned_addr learned my addr 172.21.15.77:0/3229421983 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.910+0000 7fa0e6ffd640 1 -- 172.21.15.77:0/3229421983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 msgr2=0x5559a986f080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:23:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.910+0000 7fa0e6ffd640 1 --2- 172.21.15.77:0/3229421983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a986f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.910+0000 7fa0e6ffd640 1 -- 172.21.15.77:0/3229421983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a986a060 msgr2=0x5559a9876050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:23:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.910+0000 7fa0e6ffd640 1 --2- 172.21.15.77:0/3229421983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a986a060 0x5559a9876050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.910+0000 7fa0e6ffd640 1 -- 172.21.15.77:0/3229421983 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559a94de210 con 0x5559a9864500 2024-03-16T10:23:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.910+0000 7fa0e6ffd640 1 --2- 172.21.15.77:0/3229421983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 0x5559a986eb40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8045000 tx=0x7fa0d803cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4f1cc0909505c65 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.936+0000 7fa0caff5640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:23:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.936+0000 7fa0caff5640 1 -- 172.21.15.77:0/3229421983 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8004020 con 0x5559a9864500 2024-03-16T10:23:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.936+0000 7fa0caff5640 1 -- 172.21.15.77:0/3229421983 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa0d80041c0 con 0x5559a9864500 2024-03-16T10:23:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.936+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3229421983 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5559a97d6f80 con 0x5559a9864500 2024-03-16T10:23:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.955+0000 7fa0caff5640 1 -- 172.21.15.77:0/3229421983 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa0d80043c0 con 0x5559a9864500 2024-03-16T10:23:21.956 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.956+0000 7fa0f03c6740 10 client.26788 fetch_fsmap learned FSMap version 37 2024-03-16T10:23:21.966 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.956+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3229421983 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5559a95aa620 con 0x5559a9864500 2024-03-16T10:23:21.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.966+0000 7fa0caff5640 1 -- 172.21.15.77:0/3229421983 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5559a97d6f80 con 0x5559a9864500 2024-03-16T10:23:21.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.966+0000 7fa0caff5640 1 -- 172.21.15.77:0/3229421983 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fa0d806a020 con 0x5559a9864500 2024-03-16T10:23:21.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.966+0000 7fa0f03c6740 10 client.26788 fetch_fsmap finished waiting for FSMap version 37 2024-03-16T10:23:21.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.966+0000 7fa0f03c6740 10 client.26788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:21.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.966+0000 7fa0f03c6740 20 client.26788 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:21.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.966+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/3229421983 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5559a987a910 0x5559a987cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:21.967 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.966+0000 7fa0f03c6740 4 client.26788 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:21.968 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.966+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3229421983 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559a97d12d0 con 0x5559a987a910 2024-03-16T10:23:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.970+0000 7fa0e67fc640 1 --2- 172.21.15.77:0/3229421983 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5559a987a910 0x5559a987cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.970+0000 7fa0e67fc640 1 --2- 172.21.15.77:0/3229421983 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5559a987a910 0x5559a987cdb0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ec3999e1c26f4f0 server_cookie=d2d8cb8b62d6a197 in_seq=0 out_seq=0 2024-03-16T10:23:21.971 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.970+0000 7fa0caff5640 10 client.26788 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.971+0000 7fa0caff5640 1 -- 172.21.15.77:0/3229421983 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5559a97d12d0 con 0x5559a987a910 2024-03-16T10:23:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.971+0000 7fa0caff5640 10 client.26788 handle_command_reply: tid=0 2024-03-16T10:23:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.971+0000 7fa0f03c6740 1 client.26788 shutdown 2024-03-16T10:23:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.971+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3229421983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 msgr2=0x5559a986eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:21.972 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.971+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/3229421983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 0x5559a986eb40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8045000 tx=0x7fa0d803cca0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.978+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3229421983 shutdown_connections 2024-03-16T10:23:21.978 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.978+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/3229421983 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5559a987a910 0x5559a987cdb0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.978+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/3229421983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a986a060 0x5559a9876050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.978+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/3229421983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a986f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.978+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/3229421983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 0x5559a986eb40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:21.979 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.978+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3229421983 >> 172.21.15.77:0/3229421983 conn(0x5559a985fa70 msgr2=0x5559a985ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.985+0000 7fa0caff5640 0 client.26788 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.985+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3229421983 shutdown_connections 2024-03-16T10:23:21.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.986+0000 7fa0f03c6740 1 -- 172.21.15.77:0/3229421983 wait complete. 2024-03-16T10:23:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:21.986+0000 7fa0f03c6740 20 client.26788 trim_cache size 0 max 16384 2024-03-16T10:23:22.169 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.160+0000 7fa0f03c6740 1 Processor -- start 2024-03-16T10:23:22.172 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.172+0000 7fa0f03c6740 1 -- start start 2024-03-16T10:23:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.172+0000 7fa0f03c6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a9864500 0x5559a986e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0f03c6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a9876010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0f03c6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a986a060 0x5559a986ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0f03c6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d400e490 con 0x5559a9864500 2024-03-16T10:23:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0f03c6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d4010090 con 0x5559a9864ef0 2024-03-16T10:23:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0f03c6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d400eb60 con 0x5559a986a060 2024-03-16T10:23:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0e67fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a9876010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0e67fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a9876010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:35706/0 (socket says 172.21.15.77:35706) 2024-03-16T10:23:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0e67fc640 1 -- 172.21.15.77:0/1458572757 learned_addr learned my addr 172.21.15.77:0/1458572757 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0e67fc640 1 -- 172.21.15.77:0/1458572757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a986a060 msgr2=0x5559a986ec90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:23:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0e67fc640 1 --2- 172.21.15.77:0/1458572757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a986a060 0x5559a986ec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0e67fc640 1 -- 172.21.15.77:0/1458572757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a9864500 msgr2=0x5559a986e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0e67fc640 1 --2- 172.21.15.77:0/1458572757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a9864500 0x5559a986e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.173+0000 7fa0e67fc640 1 -- 172.21.15.77:0/1458572757 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559a98607e0 con 0x5559a9864ef0 2024-03-16T10:23:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.174+0000 7fa0e67fc640 1 --2- 172.21.15.77:0/1458572757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a9876010 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0000f80 tx=0x7fa0e00099b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb731b889660e4aa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.182+0000 7fa0ccff9640 1 -- 172.21.15.77:0/1458572757 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e002a070 con 0x5559a9864ef0 2024-03-16T10:23:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.182+0000 7fa0ccff9640 1 -- 172.21.15.77:0/1458572757 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0e0025bb0 con 0x5559a9864ef0 2024-03-16T10:23:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.182+0000 7fa0ccff9640 1 -- 172.21.15.77:0/1458572757 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e00054b0 con 0x5559a9864ef0 2024-03-16T10:23:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.183+0000 7fa0f03c6740 1 -- 172.21.15.77:0/1458572757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 msgr2=0x5559a9876010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.183+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/1458572757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a9876010 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0000f80 tx=0x7fa0e00099b0 comp rx=0 tx=0).stop 2024-03-16T10:23:22.188 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.187+0000 7fa0f03c6740 1 -- 172.21.15.77:0/1458572757 shutdown_connections 2024-03-16T10:23:22.188 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.187+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/1458572757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a986a060 0x5559a986ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:22.188 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.187+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/1458572757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a9876010 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:22.188 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.187+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/1458572757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a9864500 0x5559a986e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:22.188 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.187+0000 7fa0f03c6740 1 -- 172.21.15.77:0/1458572757 >> 172.21.15.77:0/1458572757 conn(0x5559a985fa70 msgr2=0x5559a98663f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:22.188 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.187+0000 7fa0f03c6740 1 -- 172.21.15.77:0/1458572757 shutdown_connections 2024-03-16T10:23:22.189 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.187+0000 7fa0f03c6740 1 -- 172.21.15.77:0/1458572757 wait complete. 2024-03-16T10:23:22.189 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.188+0000 7fa0f03c6740 1 -- start start 2024-03-16T10:23:22.189 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.188+0000 7fa0f03c6740 1 Processor -- start 2024-03-16T10:23:22.189 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0f03c6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 0x5559a9872dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:22.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0f03c6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a9873310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:22.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0f03c6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a986a060 0x5559a98dece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:22.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0e6ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 0x5559a9872dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:22.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0e6ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 0x5559a9872dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:45324/0 (socket says 172.21.15.77:45324) 2024-03-16T10:23:22.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0e67fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a9873310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:22.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0f03c6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d400f9c0 con 0x5559a986a060 2024-03-16T10:23:22.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0f03c6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d400f2f0 con 0x5559a9864ef0 2024-03-16T10:23:22.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0f03c6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d400c470 con 0x5559a9864500 2024-03-16T10:23:22.190 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0e67fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a9873310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:35718/0 (socket says 172.21.15.77:35718) 2024-03-16T10:23:22.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0e67fc640 1 -- 172.21.15.77:0/827999093 learned_addr learned my addr 172.21.15.77:0/827999093 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:22.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0f03c6740 10 client.0 _ll_register_callbacks cb 0x7fa0e00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:22.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0e67fc640 1 -- 172.21.15.77:0/827999093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 msgr2=0x5559a9872dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:22.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.189+0000 7fa0e67fc640 1 --2- 172.21.15.77:0/827999093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 0x5559a9872dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:22.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.190+0000 7fa0e67fc640 1 -- 172.21.15.77:0/827999093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a986a060 msgr2=0x5559a98dece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:22.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.190+0000 7fa0e67fc640 1 --2- 172.21.15.77:0/827999093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a986a060 0x5559a98dece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:22.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.190+0000 7fa0e67fc640 1 -- 172.21.15.77:0/827999093 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559a98df220 con 0x5559a9864ef0 2024-03-16T10:23:22.191 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.190+0000 7fa0e67fc640 1 --2- 172.21.15.77:0/827999093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a9873310 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009690 tx=0x7fa0e0025f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9ca8584b6e02127 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.195+0000 7fa0caff5640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:23:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.195+0000 7fa0caff5640 1 -- 172.21.15.77:0/827999093 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e002a070 con 0x5559a9864ef0 2024-03-16T10:23:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.196+0000 7fa0f03c6740 1 -- 172.21.15.77:0/827999093 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0e0025bb0 con 0x5559a9864ef0 2024-03-16T10:23:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.196+0000 7fa0caff5640 1 -- 172.21.15.77:0/827999093 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7fa0e0025480 con 0x5559a9864ef0 2024-03-16T10:23:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.198+0000 7fa0caff5640 1 -- 172.21.15.77:0/827999093 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa0e0025680 con 0x5559a9864ef0 2024-03-16T10:23:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.198+0000 7fa0f03c6740 10 client.26810 fetch_fsmap learned FSMap version 37 2024-03-16T10:23:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.198+0000 7fa0f03c6740 1 -- 172.21.15.77:0/827999093 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5559a98e0390 con 0x5559a9864ef0 2024-03-16T10:23:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.202+0000 7fa0caff5640 1 -- 172.21.15.77:0/827999093 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0e0025bb0 con 0x5559a9864ef0 2024-03-16T10:23:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.203+0000 7fa0caff5640 1 -- 172.21.15.77:0/827999093 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fa0e0018880 con 0x5559a9864ef0 2024-03-16T10:23:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.203+0000 7fa0f03c6740 10 client.26810 fetch_fsmap finished waiting for FSMap version 37 2024-03-16T10:23:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.203+0000 7fa0f03c6740 10 client.26810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.203+0000 7fa0f03c6740 20 client.26810 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.203+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/827999093 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5559a98e0a00 0x5559a98e2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:22.204 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.203+0000 7fa0f03c6740 4 client.26810 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:23:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.203+0000 7fa0f03c6740 1 -- 172.21.15.77:0/827999093 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5559a974f770 con 0x5559a98e0a00 2024-03-16T10:23:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.204+0000 7fa0e6ffd640 1 --2- 172.21.15.77:0/827999093 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5559a98e0a00 0x5559a98e2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.204+0000 7fa0caff5640 10 client.26810 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:22.204+0000 7fa0e6ffd640 1 --2- 172.21.15.77:0/827999093 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5559a98e0a00 0x5559a98e2e20 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a09711623992de3e server_cookie=f6264ace3f5d19d1 in_seq=0 out_seq=0 2024-03-16T10:23:22.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:21 smithi077 ceph-mon[32400]: pgmap v538: 97 pgs: 97 active+clean; 705 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 430 op/s 2024-03-16T10:23:22.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:21 smithi193 ceph-mon[36311]: pgmap v538: 97 pgs: 97 active+clean; 705 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 430 op/s 2024-03-16T10:23:22.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:21 smithi192 ceph-mon[36354]: pgmap v538: 97 pgs: 97 active+clean; 705 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.3 MiB/s wr, 430 op/s 2024-03-16T10:23:23.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.230+0000 7fa0caff5640 1 -- 172.21.15.77:0/827999093 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5559a974f770 con 0x5559a98e0a00 2024-03-16T10:23:23.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0caff5640 10 client.26810 handle_command_reply: tid=0 2024-03-16T10:23:23.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0f03c6740 1 client.26810 shutdown 2024-03-16T10:23:23.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0f03c6740 1 -- 172.21.15.77:0/827999093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 msgr2=0x5559a9873310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:23.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/827999093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a9873310 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009690 tx=0x7fa0e0025f90 comp rx=0 tx=0).stop 2024-03-16T10:23:23.233 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0f03c6740 1 -- 172.21.15.77:0/827999093 shutdown_connections 2024-03-16T10:23:23.233 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/827999093 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5559a98e0a00 0x5559a98e2e20 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:23.233 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/827999093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a986a060 0x5559a98dece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:23.233 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0caff5640 0 client.26810 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:23.233 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/827999093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a9864ef0 0x5559a9873310 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:23.233 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0f03c6740 1 --2- 172.21.15.77:0/827999093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a9864500 0x5559a9872dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:23.233 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0f03c6740 1 -- 172.21.15.77:0/827999093 >> 172.21.15.77:0/827999093 conn(0x5559a985fa70 msgr2=0x5559a986e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:23.233 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0f03c6740 1 -- 172.21.15.77:0/827999093 shutdown_connections 2024-03-16T10:23:23.233 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0f03c6740 1 -- 172.21.15.77:0/827999093 wait complete. 2024-03-16T10:23:23.233 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.231+0000 7fa0f03c6740 20 client.26810 trim_cache size 0 max 16384 2024-03-16T10:23:23.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:23:23.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:23:23.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:23:23.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:23:23.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.232+0000 7fa0ed7df640 1 -- 172.21.15.77:0/1021559617 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0d00a2a00 msgr2=0x7fa0d00a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:23.234 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.232+0000 7fa0ed7df640 1 --2- 172.21.15.77:0/1021559617 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0d00a2a00 0x7fa0d00a4ec0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x5559a9865750 tx=0x7fa0d8000bc0 comp rx=0 tx=0).stop 2024-03-16T10:23:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.232+0000 7fa0ed7df640 1 -- 172.21.15.77:0/1021559617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e8109510 msgr2=0x7fa0e81a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.232+0000 7fa0ed7df640 1 --2- 172.21.15.77:0/1021559617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e8109510 0x7fa0e81a56e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00b3f0 tx=0x7fa0dc00b8b0 comp rx=0 tx=0).stop 2024-03-16T10:23:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.232+0000 7fa0ed7df640 1 -- 172.21.15.77:0/1021559617 shutdown_connections 2024-03-16T10:23:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.232+0000 7fa0ed7df640 1 --2- 172.21.15.77:0/1021559617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e8109510 0x7fa0e81a56e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.232+0000 7fa0ed7df640 1 --2- 172.21.15.77:0/1021559617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0e8073410 0x7fa0e819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.232+0000 7fa0ed7df640 1 --2- 172.21.15.77:0/1021559617 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa0d00a2a00 0x7fa0d00a4ec0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.232+0000 7fa0ed7df640 1 --2- 172.21.15.77:0/1021559617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa0e8072ab0 0x7fa0e819e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.232+0000 7fa0ed7df640 1 -- 172.21.15.77:0/1021559617 >> 172.21.15.77:0/1021559617 conn(0x7fa0e806e650 msgr2=0x7fa0e810c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.232+0000 7fa0ed7df640 1 -- 172.21.15.77:0/1021559617 shutdown_connections 2024-03-16T10:23:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:23.232+0000 7fa0ed7df640 1 -- 172.21.15.77:0/1021559617 wait complete. 2024-03-16T10:23:23.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:22 smithi077 ceph-mon[32400]: pgmap v539: 97 pgs: 97 active+clean; 721 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 545 op/s 2024-03-16T10:23:23.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:22 smithi193 ceph-mon[36311]: pgmap v539: 97 pgs: 97 active+clean; 721 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 545 op/s 2024-03-16T10:23:23.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:22 smithi192 ceph-mon[36354]: pgmap v539: 97 pgs: 97 active+clean; 721 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 545 op/s 2024-03-16T10:23:26.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:25 smithi192 ceph-mon[36354]: pgmap v540: 97 pgs: 97 active+clean; 746 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.5 MiB/s wr, 604 op/s 2024-03-16T10:23:26.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:25 smithi077 ceph-mon[32400]: pgmap v540: 97 pgs: 97 active+clean; 746 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.5 MiB/s wr, 604 op/s 2024-03-16T10:23:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:25 smithi193 ceph-mon[36311]: pgmap v540: 97 pgs: 97 active+clean; 746 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.5 MiB/s wr, 604 op/s 2024-03-16T10:23:27.105 INFO:tasks.workunit.client.0.smithi077.stdout: 163 1174013 204 795363 221 671478 674 2024-03-16T10:23:27.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:26 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:26 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:27.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:26 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:27 smithi077 ceph-mon[32400]: pgmap v541: 97 pgs: 97 active+clean; 764 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.0 MiB/s wr, 784 op/s 2024-03-16T10:23:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:27 smithi193 ceph-mon[36311]: pgmap v541: 97 pgs: 97 active+clean; 764 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.0 MiB/s wr, 784 op/s 2024-03-16T10:23:28.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:27 smithi192 ceph-mon[36354]: pgmap v541: 97 pgs: 97 active+clean; 764 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.0 MiB/s wr, 784 op/s 2024-03-16T10:23:30.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:29 smithi077 ceph-mon[32400]: pgmap v542: 97 pgs: 97 active+clean; 773 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.0 MiB/s wr, 882 op/s 2024-03-16T10:23:30.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:29 smithi193 ceph-mon[36311]: pgmap v542: 97 pgs: 97 active+clean; 773 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.0 MiB/s wr, 882 op/s 2024-03-16T10:23:30.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:29 smithi192 ceph-mon[36354]: pgmap v542: 97 pgs: 97 active+clean; 773 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.0 MiB/s wr, 882 op/s 2024-03-16T10:23:32.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:31 smithi077 ceph-mon[32400]: pgmap v543: 97 pgs: 97 active+clean; 770 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.5 MiB/s wr, 896 op/s 2024-03-16T10:23:32.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:31 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:23:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:31 smithi193 ceph-mon[36311]: pgmap v543: 97 pgs: 97 active+clean; 770 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.5 MiB/s wr, 896 op/s 2024-03-16T10:23:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:31 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:23:32.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:31 smithi192 ceph-mon[36354]: pgmap v543: 97 pgs: 97 active+clean; 770 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.5 MiB/s wr, 896 op/s 2024-03-16T10:23:32.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:31 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:23:33.222 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:32 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:23:33.222 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:32 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:23:33.222 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:32 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:23:33.222 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:32 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:23:33.222 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:32 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:23:33.222 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:32 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:23:33.222 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:32 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:23:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:32 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:23:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:32 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:23:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:32 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:23:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:32 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:23:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:32 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:23:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:32 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:23:33.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:32 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:23:33.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:32 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:23:33.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:32 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:23:33.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:32 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:23:33.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:32 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:23:33.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:32 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:23:33.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:32 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:23:33.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:32 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:23:33.502 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:33 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f[66870]: *** Caught signal (Segmentation fault) ** 2024-03-16T10:23:33.502 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:33 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f[66870]: in thread 7fd8dc9f5700 thread_name:MR_Finisher 2024-03-16T10:23:33.503 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:33 smithi193 podman[70710]: 2024-03-16 10:23:33.271656711 +0000 UTC m=+0.021675533 container died f91bda9fc43cc9cab5d245d6ed80a398a480fa7bdee115cc63cba6fb9664cee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T10:23:34.207 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:33 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-03-16T10:23:34.207 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:33 smithi193 ceph-mon[36311]: Unable to set osd_memory_target on smithi077 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-16T10:23:34.207 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:33 smithi193 ceph-mon[36311]: pgmap v544: 97 pgs: 97 active+clean; 774 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.2 MiB/s wr, 919 op/s 2024-03-16T10:23:34.207 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:33 smithi193 podman[70710]: 2024-03-16 10:23:33.95112308 +0000 UTC m=+0.701141883 container remove f91bda9fc43cc9cab5d245d6ed80a398a480fa7bdee115cc63cba6fb9664cee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:23:34.208 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:33 smithi193 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.f.service: Main process exited, code=exited, status=139/n/a 2024-03-16T10:23:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:33 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-03-16T10:23:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:33 smithi077 ceph-mon[32400]: Unable to set osd_memory_target on smithi077 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-16T10:23:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:33 smithi077 ceph-mon[32400]: pgmap v544: 97 pgs: 97 active+clean; 774 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.2 MiB/s wr, 919 op/s 2024-03-16T10:23:34.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:33 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-03-16T10:23:34.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:33 smithi192 ceph-mon[36354]: Unable to set osd_memory_target on smithi077 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-16T10:23:34.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:33 smithi192 ceph-mon[36354]: pgmap v544: 97 pgs: 97 active+clean; 774 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.2 MiB/s wr, 919 op/s 2024-03-16T10:23:34.502 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:34 smithi193 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-16T10:23:34.503 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:34 smithi193 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.f.service: Consumed 1.833s CPU time. 2024-03-16T10:23:35.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:34 smithi077 ceph-mon[32400]: pgmap v545: 97 pgs: 97 active+clean; 766 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.7 MiB/s wr, 767 op/s 2024-03-16T10:23:35.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:34 smithi193 ceph-mon[36311]: pgmap v545: 97 pgs: 97 active+clean; 766 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.7 MiB/s wr, 767 op/s 2024-03-16T10:23:35.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:34 smithi192 ceph-mon[36354]: pgmap v545: 97 pgs: 97 active+clean; 766 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.7 MiB/s wr, 767 op/s 2024-03-16T10:23:36.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:36 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:23:36.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:36 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:23:36.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:36 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:23:37.105 INFO:tasks.workunit.client.0.smithi077.stdout: 168 1882892 573 1283642 290 1115043 1195 2024-03-16T10:23:37.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:37 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:23:37.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:37 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:23:37.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:37 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:23:37.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:37 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:23:37.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:37 smithi192 ceph-mon[36354]: pgmap v546: 97 pgs: 97 active+clean; 753 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 622 op/s 2024-03-16T10:23:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.447+0000 7ff05a68f640 1 -- 172.21.15.77:0/3803810559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0540733f0 msgr2=0x7ff054108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.447+0000 7ff05a68f640 1 --2- 172.21.15.77:0/3803810559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0540733f0 0x7ff054108e10 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff04c009ca0 tx=0x7ff04c02c0f0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.447+0000 7ff05a68f640 1 -- 172.21.15.77:0/3803810559 shutdown_connections 2024-03-16T10:23:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.447+0000 7ff05a68f640 1 --2- 172.21.15.77:0/3803810559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff0541094d0 0x7ff05410a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.447+0000 7ff05a68f640 1 --2- 172.21.15.77:0/3803810559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0540733f0 0x7ff054108e10 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.447+0000 7ff05a68f640 1 --2- 172.21.15.77:0/3803810559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff054072a90 0x7ff054072eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.447+0000 7ff05a68f640 1 -- 172.21.15.77:0/3803810559 >> 172.21.15.77:0/3803810559 conn(0x7ff05406e650 msgr2=0x7ff054070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.448+0000 7ff05a68f640 1 -- 172.21.15.77:0/3803810559 shutdown_connections 2024-03-16T10:23:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.448+0000 7ff05a68f640 1 -- 172.21.15.77:0/3803810559 wait complete. 2024-03-16T10:23:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.449+0000 7ff05a68f640 1 Processor -- start 2024-03-16T10:23:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.449+0000 7ff05a68f640 1 -- start start 2024-03-16T10:23:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.449+0000 7ff05a68f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff054072a90 0x7ff0541a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.449+0000 7ff05a68f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0541094d0 0x7ff0541a6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.449+0000 7ff05a68f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0541abee0 0x7ff0541ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.449+0000 7ff05a68f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff054117d30 con 0x7ff0541094d0 2024-03-16T10:23:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.449+0000 7ff05a68f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff054117bb0 con 0x7ff054072a90 2024-03-16T10:23:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.449+0000 7ff05a68f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff054117eb0 con 0x7ff0541abee0 2024-03-16T10:23:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.449+0000 7ff059e8e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0541abee0 0x7ff0541ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.449+0000 7ff059e8e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0541abee0 0x7ff0541ae2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58644/0 (socket says 172.21.15.77:58644) 2024-03-16T10:23:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.449+0000 7ff059e8e640 1 -- 172.21.15.77:0/4106803264 learned_addr learned my addr 172.21.15.77:0/4106803264 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.450+0000 7ff059e8e640 1 -- 172.21.15.77:0/4106803264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff054072a90 msgr2=0x7ff0541a6940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T10:23:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.450+0000 7ff058e8c640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0541094d0 0x7ff0541a6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.450+0000 7ff059e8e640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff054072a90 0x7ff0541a6940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.450+0000 7ff059e8e640 1 -- 172.21.15.77:0/4106803264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0541094d0 msgr2=0x7ff0541a6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.450+0000 7ff059e8e640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0541094d0 0x7ff0541a6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.450+0000 7ff059e8e640 1 -- 172.21.15.77:0/4106803264 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0541ae940 con 0x7ff0541abee0 2024-03-16T10:23:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.450+0000 7ff058e8c640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0541094d0 0x7ff0541a6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:23:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.450+0000 7ff059e8e640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0541abee0 0x7ff0541ae2d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff05000c970 tx=0x7ff05000ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.451+0000 7ff04a7fc640 1 -- 172.21.15.77:0/4106803264 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff050007bb0 con 0x7ff0541abee0 2024-03-16T10:23:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.451+0000 7ff04a7fc640 1 -- 172.21.15.77:0/4106803264 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff050007d50 con 0x7ff0541abee0 2024-03-16T10:23:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.451+0000 7ff05a68f640 1 -- 172.21.15.77:0/4106803264 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0541aec30 con 0x7ff0541abee0 2024-03-16T10:23:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.451+0000 7ff04a7fc640 1 -- 172.21.15.77:0/4106803264 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff050018880 con 0x7ff0541abee0 2024-03-16T10:23:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.451+0000 7ff05a68f640 1 -- 172.21.15.77:0/4106803264 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0541af180 con 0x7ff0541abee0 2024-03-16T10:23:37.455 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.454+0000 7ff04a7fc640 1 -- 172.21.15.77:0/4106803264 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7ff05000f450 con 0x7ff0541abee0 2024-03-16T10:23:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.455+0000 7ff05e5bb740 1 Processor -- start 2024-03-16T10:23:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.456+0000 7ff05e5bb740 1 -- start start 2024-03-16T10:23:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.456+0000 7ff05e5bb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 0x55ad990328f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.456+0000 7ff058e8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 0x55ad990328f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.456+0000 7ff058e8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 0x55ad990328f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34388/0 (socket says 172.21.15.77:34388) 2024-03-16T10:23:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.456+0000 7ff058e8c640 1 -- 172.21.15.77:0/780639213 learned_addr learned my addr 172.21.15.77:0/780639213 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.456+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/780639213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad990379c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.457+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/780639213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad99038030 0x55ad9903a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.457+0000 7ff05e5bb740 1 -- 172.21.15.77:0/780639213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ad98f97a40 con 0x55ad990324d0 2024-03-16T10:23:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.457+0000 7ff05e5bb740 1 -- 172.21.15.77:0/780639213 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ad98f40920 con 0x55ad99032ec0 2024-03-16T10:23:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.457+0000 7ff05e5bb740 1 -- 172.21.15.77:0/780639213 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ad98f93ae0 con 0x55ad99038030 2024-03-16T10:23:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.457+0000 7ff05968d640 1 --2- 172.21.15.77:0/780639213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad990379c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.457+0000 7ff058e8c640 1 -- 172.21.15.77:0/780639213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad99038030 msgr2=0x55ad9903a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.457+0000 7ff058e8c640 1 --2- 172.21.15.77:0/780639213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad99038030 0x55ad9903a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.457+0000 7ff058e8c640 1 -- 172.21.15.77:0/780639213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 msgr2=0x55ad990379c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.457+0000 7ff058e8c640 1 --2- 172.21.15.77:0/780639213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad990379c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.457+0000 7ff058e8c640 1 -- 172.21.15.77:0/780639213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad98f58460 con 0x55ad990324d0 2024-03-16T10:23:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.458+0000 7ff058e8c640 1 --2- 172.21.15.77:0/780639213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 0x55ad990328f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff04c037040 tx=0x7ff04c009980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae062ae4f5be54e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.458+0000 7ff03f7fe640 1 -- 172.21.15.77:0/780639213 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c039050 con 0x55ad990324d0 2024-03-16T10:23:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.458+0000 7ff03f7fe640 1 -- 172.21.15.77:0/780639213 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff04c045070 con 0x55ad990324d0 2024-03-16T10:23:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.458+0000 7ff03f7fe640 1 -- 172.21.15.77:0/780639213 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c004280 con 0x55ad990324d0 2024-03-16T10:23:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.460+0000 7ff05e5bb740 1 -- 172.21.15.77:0/780639213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 msgr2=0x55ad990328f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.460+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/780639213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 0x55ad990328f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff04c037040 tx=0x7ff04c009980 comp rx=0 tx=0).stop 2024-03-16T10:23:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.461+0000 7ff04a7fc640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff0400a2930 0x7ff0400a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.461+0000 7ff058e8c640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff0400a2930 0x7ff0400a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.462+0000 7ff058e8c640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff0400a2930 0x7ff0400a4df0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0049d0 tx=0x7ff04c004420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.462+0000 7ff04a7fc640 1 -- 172.21.15.77:0/4106803264 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ff0500ceb20 con 0x7ff0541abee0 2024-03-16T10:23:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.463+0000 7ff05e5bb740 1 -- 172.21.15.77:0/780639213 shutdown_connections 2024-03-16T10:23:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.463+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/780639213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad99038030 0x55ad9903a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.463+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/780639213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad990379c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.463+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/780639213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 0x55ad990328f0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.463+0000 7ff05e5bb740 1 -- 172.21.15.77:0/780639213 >> 172.21.15.77:0/780639213 conn(0x55ad9902da40 msgr2=0x55ad9902de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.463+0000 7ff05e5bb740 1 -- 172.21.15.77:0/780639213 shutdown_connections 2024-03-16T10:23:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.463+0000 7ff05e5bb740 1 -- 172.21.15.77:0/780639213 wait complete. 2024-03-16T10:23:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.463+0000 7ff05e5bb740 1 -- start start 2024-03-16T10:23:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.464+0000 7ff05e5bb740 1 Processor -- start 2024-03-16T10:23:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.467+0000 7ff05e5bb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 0x55ad9903c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.468+0000 7ff05e5bb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad9903cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.468+0000 7ff05e5bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad99038030 0x55ad99043d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.468+0000 7ff05e5bb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ad9903af80 con 0x55ad990324d0 2024-03-16T10:23:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.468+0000 7ff05e5bb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ad98f53d70 con 0x55ad99032ec0 2024-03-16T10:23:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.468+0000 7ff05e5bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ad9903b100 con 0x55ad99038030 2024-03-16T10:23:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.468+0000 7ff05968d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 0x55ad9903c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.468+0000 7ff05e5bb740 10 client.0 _ll_register_callbacks cb 0x55ad98caba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.469+0000 7ff05968d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 0x55ad9903c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34392/0 (socket says 172.21.15.77:34392) 2024-03-16T10:23:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.469+0000 7ff05968d640 1 -- 172.21.15.77:0/2650151555 learned_addr learned my addr 172.21.15.77:0/2650151555 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.470+0000 7ff059e8e640 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad99038030 0x55ad99043d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.470+0000 7ff05968d640 1 -- 172.21.15.77:0/2650151555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad99038030 msgr2=0x55ad99043d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.470+0000 7ff058e8c640 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad9903cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.470+0000 7ff05968d640 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad99038030 0x55ad99043d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.470+0000 7ff05968d640 1 -- 172.21.15.77:0/2650151555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 msgr2=0x55ad9903cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.470+0000 7ff05968d640 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad9903cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.470+0000 7ff05968d640 1 -- 172.21.15.77:0/2650151555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad98cac210 con 0x55ad990324d0 2024-03-16T10:23:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.471+0000 7ff059e8e640 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad99038030 0x55ad99043d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:23:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.472+0000 7ff05968d640 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 0x55ad9903c7f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff04400f670 tx=0x7ff04400fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe76c529d53d534b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.488+0000 7ff03cff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:23:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.488+0000 7ff03cff9640 1 -- 172.21.15.77:0/2650151555 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff044005220 con 0x55ad990324d0 2024-03-16T10:23:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.488+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2650151555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff04c045070 con 0x55ad990324d0 2024-03-16T10:23:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.489+0000 7ff03cff9640 1 -- 172.21.15.77:0/2650151555 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ff0440053c0 con 0x55ad990324d0 2024-03-16T10:23:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.489+0000 7ff03cff9640 1 -- 172.21.15.77:0/2650151555 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff0440055c0 con 0x55ad990324d0 2024-03-16T10:23:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.489+0000 7ff05e5bb740 10 client.17040 fetch_fsmap learned FSMap version 37 2024-03-16T10:23:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.489+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2650151555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ad98d78620 con 0x55ad990324d0 2024-03-16T10:23:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.493+0000 7ff03cff9640 1 -- 172.21.15.77:0/2650151555 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff04c045070 con 0x55ad990324d0 2024-03-16T10:23:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.493+0000 7ff03cff9640 1 -- 172.21.15.77:0/2650151555 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7ff044004c90 con 0x55ad990324d0 2024-03-16T10:23:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:37 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:23:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:37 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:23:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:37 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:23:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:37 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:23:37.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:37 smithi077 ceph-mon[32400]: pgmap v546: 97 pgs: 97 active+clean; 753 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 622 op/s 2024-03-16T10:23:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.494+0000 7ff05e5bb740 10 client.17040 fetch_fsmap finished waiting for FSMap version 37 2024-03-16T10:23:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.494+0000 7ff05e5bb740 10 client.17040 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.494+0000 7ff05e5bb740 20 client.17040 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.494+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ad990486d0 0x55ad9904ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.494+0000 7ff05e5bb740 4 client.17040 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.494+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2650151555 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad98fa4f80 con 0x55ad990486d0 2024-03-16T10:23:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.494+0000 7ff058e8c640 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ad990486d0 0x55ad9904ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.494+0000 7ff03cff9640 10 client.17040 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.495+0000 7ff058e8c640 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ad990486d0 0x55ad9904ab60 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f1e37f6b5605671 server_cookie=5845ccf53c10409c in_seq=0 out_seq=0 2024-03-16T10:23:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.495+0000 7ff03cff9640 1 -- 172.21.15.77:0/2650151555 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ad98fa4f80 con 0x55ad990486d0 2024-03-16T10:23:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.495+0000 7ff03cff9640 10 client.17040 handle_command_reply: tid=0 2024-03-16T10:23:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.495+0000 7ff05e5bb740 1 client.17040 shutdown 2024-03-16T10:23:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.496+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2650151555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 msgr2=0x55ad9903c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.496+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 0x55ad9903c7f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff04400f670 tx=0x7ff04400fb30 comp rx=0 tx=0).stop 2024-03-16T10:23:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.496+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2650151555 shutdown_connections 2024-03-16T10:23:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.496+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad99038030 0x55ad99043d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.496+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad9903cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.496+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ad990486d0 0x55ad9904ab60 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.496+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/2650151555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad990324d0 0x55ad9903c7f0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.496+0000 7ff03cff9640 0 client.17040 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.496+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2650151555 >> 172.21.15.77:0/2650151555 conn(0x55ad9902da40 msgr2=0x55ad9902cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.496+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2650151555 shutdown_connections 2024-03-16T10:23:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.496+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2650151555 wait complete. 2024-03-16T10:23:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.496+0000 7ff05e5bb740 20 client.17040 trim_cache size 0 max 16384 2024-03-16T10:23:37.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:37 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:23:37.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:37 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:23:37.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:37 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:23:37.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:37 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:23:37.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:37 smithi193 ceph-mon[36311]: pgmap v546: 97 pgs: 97 active+clean; 753 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 622 op/s 2024-03-16T10:23:37.528 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.527+0000 7ff05e5bb740 1 Processor -- start 2024-03-16T10:23:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.527+0000 7ff05e5bb740 1 -- start start 2024-03-16T10:23:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff05e5bb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad99032ec0 0x55ad9903be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff05e5bb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99038030 0x55ad99043d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff05e5bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 0x55ad9903c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff05e5bb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541b7dd0 con 0x55ad99032ec0 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff05e5bb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541b99d0 con 0x55ad99038030 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff05e5bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541b84a0 con 0x55ad990313b0 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff058e8c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99038030 0x55ad99043d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff058e8c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99038030 0x55ad99043d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:56222/0 (socket says 172.21.15.77:56222) 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff058e8c640 1 -- 172.21.15.77:0/2250157821 learned_addr learned my addr 172.21.15.77:0/2250157821 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff05968d640 1 --2- 172.21.15.77:0/2250157821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad99032ec0 0x55ad9903be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff058e8c640 1 -- 172.21.15.77:0/2250157821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 msgr2=0x55ad9903c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff059e8e640 1 --2- 172.21.15.77:0/2250157821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 0x55ad9903c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff058e8c640 1 --2- 172.21.15.77:0/2250157821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 0x55ad9903c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff058e8c640 1 -- 172.21.15.77:0/2250157821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad99032ec0 msgr2=0x55ad9903be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff058e8c640 1 --2- 172.21.15.77:0/2250157821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad99032ec0 0x55ad9903be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff058e8c640 1 -- 172.21.15.77:0/2250157821 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad9903b280 con 0x55ad99038030 2024-03-16T10:23:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff05968d640 1 --2- 172.21.15.77:0/2250157821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad99032ec0 0x55ad9903be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:23:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.528+0000 7ff059e8e640 1 --2- 172.21.15.77:0/2250157821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 0x55ad9903c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:23:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.529+0000 7ff058e8c640 1 --2- 172.21.15.77:0/2250157821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99038030 0x55ad99043d30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff04c000f80 tx=0x7ff04c04c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88e6e746bb1a45d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.529+0000 7ff03effd640 1 -- 172.21.15.77:0/2250157821 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c039040 con 0x55ad99038030 2024-03-16T10:23:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.529+0000 7ff03effd640 1 -- 172.21.15.77:0/2250157821 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff04c045070 con 0x55ad99038030 2024-03-16T10:23:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.529+0000 7ff03effd640 1 -- 172.21.15.77:0/2250157821 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c050070 con 0x55ad99038030 2024-03-16T10:23:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.530+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2250157821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99038030 msgr2=0x55ad99043d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.530+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/2250157821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99038030 0x55ad99043d30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff04c000f80 tx=0x7ff04c04c000 comp rx=0 tx=0).stop 2024-03-16T10:23:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.530+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2250157821 shutdown_connections 2024-03-16T10:23:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.530+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/2250157821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 0x55ad9903c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.530+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/2250157821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99038030 0x55ad99043d30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.530+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/2250157821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad99032ec0 0x55ad9903be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.530+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2250157821 >> 172.21.15.77:0/2250157821 conn(0x55ad9902da40 msgr2=0x55ad99030ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.530+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2250157821 shutdown_connections 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.530+0000 7ff05e5bb740 1 -- 172.21.15.77:0/2250157821 wait complete. 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.530+0000 7ff05e5bb740 1 -- start start 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.531+0000 7ff05e5bb740 1 Processor -- start 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.531+0000 7ff05e5bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 0x55ad99041cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.531+0000 7ff05e5bb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad99042200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.531+0000 7ff05e5bb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad99038030 0x55ad99043210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.531+0000 7ff058e8c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad99042200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.531+0000 7ff05968d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 0x55ad99041cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.531+0000 7ff058e8c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad99042200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:56234/0 (socket says 172.21.15.77:56234) 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.531+0000 7ff058e8c640 1 -- 172.21.15.77:0/1750560509 learned_addr learned my addr 172.21.15.77:0/1750560509 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.531+0000 7ff05e5bb740 1 -- 172.21.15.77:0/1750560509 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541b9300 con 0x55ad99038030 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.531+0000 7ff05e5bb740 1 -- 172.21.15.77:0/1750560509 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541b8c30 con 0x55ad99032ec0 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.531+0000 7ff05e5bb740 1 -- 172.21.15.77:0/1750560509 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541b5db0 con 0x55ad990313b0 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff059e8e640 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad99038030 0x55ad99043210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff05e5bb740 10 client.0 _ll_register_callbacks cb 0x7ff04400a440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff058e8c640 1 -- 172.21.15.77:0/1750560509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 msgr2=0x55ad99041cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.531+0000 7ff05968d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 0x55ad99041cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:58674/0 (socket says 172.21.15.77:58674) 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff058e8c640 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 0x55ad99041cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff058e8c640 1 -- 172.21.15.77:0/1750560509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad99038030 msgr2=0x55ad99043210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff058e8c640 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad99038030 0x55ad99043210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff058e8c640 1 -- 172.21.15.77:0/1750560509 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad99043800 con 0x55ad99032ec0 2024-03-16T10:23:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff059e8e640 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad99038030 0x55ad99043210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff05968d640 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 0x55ad99041cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff058e8c640 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad99042200 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff04c002230 tx=0x7ff04c032e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=915d3097b4cff2f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff03cff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff03cff9640 1 -- 172.21.15.77:0/1750560509 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c06a060 con 0x55ad99032ec0 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.532+0000 7ff05e5bb740 1 -- 172.21.15.77:0/1750560509 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff04c045070 con 0x55ad99032ec0 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff03cff9640 1 -- 172.21.15.77:0/1750560509 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8161+0+0 (secure 0 0 0) 0x7ff04c032690 con 0x55ad99032ec0 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff03cff9640 1 -- 172.21.15.77:0/1750560509 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff04c039040 con 0x55ad99032ec0 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff05e5bb740 10 client.26819 fetch_fsmap learned FSMap version 37 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff05e5bb740 1 -- 172.21.15.77:0/1750560509 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ad990af240 con 0x55ad99032ec0 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff03cff9640 1 -- 172.21.15.77:0/1750560509 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff04c045070 con 0x55ad99032ec0 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff03cff9640 1 -- 172.21.15.77:0/1750560509 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7ff04c05c6a0 con 0x55ad99032ec0 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff05e5bb740 10 client.26819 fetch_fsmap finished waiting for FSMap version 37 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff05e5bb740 10 client.26819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff05e5bb740 20 client.26819 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:37.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ad990af7c0 0x55ad990b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:37.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff05e5bb740 4 client.26819 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:23:37.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff05e5bb740 1 -- 172.21.15.77:0/1750560509 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ad98f9f2d0 con 0x55ad990af7c0 2024-03-16T10:23:37.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.533+0000 7ff05968d640 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ad990af7c0 0x55ad990b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:37.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.534+0000 7ff03cff9640 10 client.26819 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:37.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:37.534+0000 7ff05968d640 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ad990af7c0 0x55ad990b1be0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8463c891547c8041 server_cookie=10808c838d103b17 in_seq=0 out_seq=0 2024-03-16T10:23:38.369 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff03cff9640 1 -- 172.21.15.77:0/1750560509 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ad98f9f2d0 con 0x55ad990af7c0 2024-03-16T10:23:38.369 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff03cff9640 10 client.26819 handle_command_reply: tid=0 2024-03-16T10:23:38.369 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff05e5bb740 1 client.26819 shutdown 2024-03-16T10:23:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff05e5bb740 1 -- 172.21.15.77:0/1750560509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 msgr2=0x55ad99042200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad99042200 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff04c002230 tx=0x7ff04c032e60 comp rx=0 tx=0).stop 2024-03-16T10:23:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff05e5bb740 1 -- 172.21.15.77:0/1750560509 shutdown_connections 2024-03-16T10:23:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55ad990af7c0 0x55ad990b1be0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ad99038030 0x55ad99043210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad99032ec0 0x55ad99042200 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff05e5bb740 1 --2- 172.21.15.77:0/1750560509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad990313b0 0x55ad99041cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff03cff9640 0 client.26819 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff05e5bb740 1 -- 172.21.15.77:0/1750560509 >> 172.21.15.77:0/1750560509 conn(0x55ad9902da40 msgr2=0x55ad9902c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff05e5bb740 1 -- 172.21.15.77:0/1750560509 shutdown_connections 2024-03-16T10:23:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff05e5bb740 1 -- 172.21.15.77:0/1750560509 wait complete. 2024-03-16T10:23:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.368+0000 7ff05e5bb740 20 client.26819 trim_cache size 0 max 16384 2024-03-16T10:23:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:23:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:23:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:23:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:23:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.369+0000 7ff05a68f640 1 -- 172.21.15.77:0/4106803264 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff0400a2930 msgr2=0x7ff0400a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.369+0000 7ff05a68f640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff0400a2930 0x7ff0400a4df0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0049d0 tx=0x7ff04c004420 comp rx=0 tx=0).stop 2024-03-16T10:23:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.369+0000 7ff05a68f640 1 -- 172.21.15.77:0/4106803264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0541abee0 msgr2=0x7ff0541ae2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.369+0000 7ff05a68f640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0541abee0 0x7ff0541ae2d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff05000c970 tx=0x7ff05000ce30 comp rx=0 tx=0).stop 2024-03-16T10:23:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.369+0000 7ff05a68f640 1 -- 172.21.15.77:0/4106803264 shutdown_connections 2024-03-16T10:23:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.369+0000 7ff05a68f640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ff0400a2930 0x7ff0400a4df0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.369+0000 7ff05a68f640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0541abee0 0x7ff0541ae2d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.369+0000 7ff05a68f640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff0541094d0 0x7ff0541a6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.369+0000 7ff05a68f640 1 --2- 172.21.15.77:0/4106803264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff054072a90 0x7ff0541a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.369+0000 7ff05a68f640 1 -- 172.21.15.77:0/4106803264 >> 172.21.15.77:0/4106803264 conn(0x7ff05406e650 msgr2=0x7ff054070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.369+0000 7ff05a68f640 1 -- 172.21.15.77:0/4106803264 shutdown_connections 2024-03-16T10:23:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:38.369+0000 7ff05a68f640 1 -- 172.21.15.77:0/4106803264 wait complete. 2024-03-16T10:23:40.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:39 smithi077 ceph-mon[32400]: pgmap v547: 97 pgs: 97 active+clean; 746 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 448 op/s 2024-03-16T10:23:40.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:39 smithi193 ceph-mon[36311]: pgmap v547: 97 pgs: 97 active+clean; 746 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 448 op/s 2024-03-16T10:23:40.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:39 smithi192 ceph-mon[36354]: pgmap v547: 97 pgs: 97 active+clean; 746 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 448 op/s 2024-03-16T10:23:42.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:41 smithi077 ceph-mon[32400]: pgmap v548: 97 pgs: 97 active+clean; 748 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 313 op/s 2024-03-16T10:23:42.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:41 smithi193 ceph-mon[36311]: pgmap v548: 97 pgs: 97 active+clean; 748 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 313 op/s 2024-03-16T10:23:42.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:41 smithi192 ceph-mon[36354]: pgmap v548: 97 pgs: 97 active+clean; 748 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 313 op/s 2024-03-16T10:23:44.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:43 smithi077 ceph-mon[32400]: pgmap v549: 97 pgs: 97 active+clean; 748 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 324 op/s 2024-03-16T10:23:44.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:43 smithi193 ceph-mon[36311]: pgmap v549: 97 pgs: 97 active+clean; 748 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 324 op/s 2024-03-16T10:23:44.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:43 smithi192 ceph-mon[36354]: pgmap v549: 97 pgs: 97 active+clean; 748 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 324 op/s 2024-03-16T10:23:44.526 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:44 smithi193 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-03-16T10:23:44.527 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:44 smithi193 systemd[1]: Stopped Ceph mds.f for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:23:44.527 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:44 smithi193 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.f.service: Consumed 1.833s CPU time. 2024-03-16T10:23:44.527 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:44 smithi193 systemd[1]: Starting Ceph mds.f for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:23:44.527 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:44 smithi193 podman[70821]: 2024-03-16 10:23:44.395223773 +0000 UTC m=+0.027280063 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:23:44.527 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:44 smithi193 podman[70821]: 2024-03-16 10:23:44.518539807 +0000 UTC m=+0.150596083 container create f7a8a18435c45a99939187ad87731ffb5a1d2ce589d33cdecc5e79f4820a80cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-16T10:23:45.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:44 smithi077 ceph-mon[32400]: daemon mds.f restarted 2024-03-16T10:23:45.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:44 smithi193 ceph-mon[36311]: daemon mds.f restarted 2024-03-16T10:23:45.252 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:44 smithi193 podman[70821]: 2024-03-16 10:23:44.77191069 +0000 UTC m=+0.403966972 container init f7a8a18435c45a99939187ad87731ffb5a1d2ce589d33cdecc5e79f4820a80cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-16T10:23:45.253 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:44 smithi193 podman[70821]: 2024-03-16 10:23:44.775806006 +0000 UTC m=+0.407862280 container start f7a8a18435c45a99939187ad87731ffb5a1d2ce589d33cdecc5e79f4820a80cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-16T10:23:45.253 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:44 smithi193 ceph-mds[70839]: -- 172.21.15.193:0/213087796 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55899af6da40 con 0x55899af41000 2024-03-16T10:23:45.253 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:44 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f[70835]: starting mds.f at 2024-03-16T10:23:45.253 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:44 smithi193 bash[70821]: f7a8a18435c45a99939187ad87731ffb5a1d2ce589d33cdecc5e79f4820a80cd 2024-03-16T10:23:45.253 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:23:44 smithi193 systemd[1]: Started Ceph mds.f for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:23:45.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:44 smithi192 ceph-mon[36354]: daemon mds.f restarted 2024-03-16T10:23:45.524 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:23:45.525 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:23:45.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.696+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/896862816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb98101390 msgr2=0x7fcb98101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:45.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.696+0000 7fcb9e4ae640 1 --2- 172.21.15.77:0/896862816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb98101390 0x7fcb98101810 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009ca0 tx=0x7fcb9002c100 comp rx=0 tx=0).stop 2024-03-16T10:23:45.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.696+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/896862816 shutdown_connections 2024-03-16T10:23:45.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.696+0000 7fcb9e4ae640 1 --2- 172.21.15.77:0/896862816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb98101390 0x7fcb98101810 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.696+0000 7fcb9e4ae640 1 --2- 172.21.15.77:0/896862816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb981009f0 0x7fcb98100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.696+0000 7fcb9e4ae640 1 --2- 172.21.15.77:0/896862816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb98100090 0x7fcb981004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.696+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/896862816 >> 172.21.15.77:0/896862816 conn(0x7fcb980fbc70 msgr2=0x7fcb980fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:45.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.696+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/896862816 shutdown_connections 2024-03-16T10:23:45.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.696+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/896862816 wait complete. 2024-03-16T10:23:45.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.697+0000 7fcb9e4ae640 1 Processor -- start 2024-03-16T10:23:45.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.697+0000 7fcb9e4ae640 1 -- start start 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.697+0000 7fcb9e4ae640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb98100090 0x7fcb98199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb9e4ae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb981009f0 0x7fcb9819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb9e4ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb98101390 0x7fcb981a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb9e4ae640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9810f600 con 0x7fcb981009f0 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb9e4ae640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9810f480 con 0x7fcb98101390 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb9e4ae640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9810f780 con 0x7fcb98100090 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb981009f0 0x7fcb9819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb981009f0 0x7fcb9819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37238/0 (socket says 172.21.15.77:37238) 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb977fe640 1 -- 172.21.15.77:0/1153658618 learned_addr learned my addr 172.21.15.77:0/1153658618 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb97fff640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb98100090 0x7fcb98199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb9ca4b640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb98101390 0x7fcb981a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb97fff640 1 -- 172.21.15.77:0/1153658618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb98101390 msgr2=0x7fcb981a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb97fff640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb98101390 0x7fcb981a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb97fff640 1 -- 172.21.15.77:0/1153658618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb981009f0 msgr2=0x7fcb9819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:45.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb97fff640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb981009f0 0x7fcb9819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb97fff640 1 -- 172.21.15.77:0/1153658618 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb981a1910 con 0x7fcb98100090 2024-03-16T10:23:45.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.698+0000 7fcb977fe640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb981009f0 0x7fcb9819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:45.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.699+0000 7fcb97fff640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb98100090 0x7fcb98199d00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800e9f0 tx=0x7fcb8800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:45.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.699+0000 7fcb957fa640 1 -- 172.21.15.77:0/1153658618 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8800ccc0 con 0x7fcb98100090 2024-03-16T10:23:45.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.699+0000 7fcb957fa640 1 -- 172.21.15.77:0/1153658618 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb88010910 con 0x7fcb98100090 2024-03-16T10:23:45.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.699+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/1153658618 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb981a1c00 con 0x7fcb98100090 2024-03-16T10:23:45.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.699+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/1153658618 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb981a21a0 con 0x7fcb98100090 2024-03-16T10:23:45.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.699+0000 7fcb957fa640 1 -- 172.21.15.77:0/1153658618 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88018960 con 0x7fcb98100090 2024-03-16T10:23:45.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.702+0000 7fcb957fa640 1 -- 172.21.15.77:0/1153658618 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fcb88010ac0 con 0x7fcb98100090 2024-03-16T10:23:45.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.702+0000 7fcb957fa640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcb800a2880 0x7fcb800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.703+0000 7fcb957fa640 1 -- 172.21.15.77:0/1153658618 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fcb880ced50 con 0x7fcb98100090 2024-03-16T10:23:45.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.703+0000 7fcb977fe640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcb800a2880 0x7fcb800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.703+0000 7fcba1170740 1 Processor -- start 2024-03-16T10:23:45.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.703+0000 7fcba1170740 1 -- start start 2024-03-16T10:23:45.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.703+0000 7fcb977fe640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcb800a2880 0x7fcb800a4d40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c011500 tx=0x7fcb8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:45.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.703+0000 7fcba1170740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c4969e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.703+0000 7fcba1170740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c4969eb60 0x559c496a3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.703+0000 7fcba1170740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 0x559c496a6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.703+0000 7fcba1170740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559c49603500 con 0x559c496a3cd0 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.703+0000 7fcba1170740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559c495abbe0 con 0x559c4969eb60 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.703+0000 7fcba1170740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559c495ff5a0 con 0x559c4969e170 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb977fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c4969eb60 0x559c496a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb97fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c4969e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb97fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c4969e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44822/0 (socket says 172.21.15.77:44822) 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb977fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c4969eb60 0x559c496a3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44514/0 (socket says 172.21.15.77:44514) 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb97fff640 1 -- 172.21.15.77:0/3087824737 learned_addr learned my addr 172.21.15.77:0/3087824737 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb9ca4b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 0x559c496a6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb97fff640 1 -- 172.21.15.77:0/3087824737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c4969eb60 msgr2=0x559c496a3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb97fff640 1 --2- 172.21.15.77:0/3087824737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c4969eb60 0x559c496a3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb97fff640 1 -- 172.21.15.77:0/3087824737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 msgr2=0x559c496a6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:45.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb97fff640 1 --2- 172.21.15.77:0/3087824737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 0x559c496a6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb97fff640 1 -- 172.21.15.77:0/3087824737 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c495bf4b0 con 0x559c4969e170 2024-03-16T10:23:45.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb9ca4b640 1 --2- 172.21.15.77:0/3087824737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 0x559c496a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:45.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb97fff640 1 --2- 172.21.15.77:0/3087824737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c4969e590 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcb880a39f0 tx=0x7fcb88099970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df3d7d63fef3e062 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:45.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.704+0000 7fcb7e7fc640 1 -- 172.21.15.77:0/3087824737 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb880a3f20 con 0x559c4969e170 2024-03-16T10:23:45.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.705+0000 7fcb7e7fc640 1 -- 172.21.15.77:0/3087824737 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb88086070 con 0x559c4969e170 2024-03-16T10:23:45.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.705+0000 7fcb7e7fc640 1 -- 172.21.15.77:0/3087824737 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8809afb0 con 0x559c4969e170 2024-03-16T10:23:45.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.705+0000 7fcba1170740 1 -- 172.21.15.77:0/3087824737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 msgr2=0x559c4969e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:45.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.705+0000 7fcba1170740 1 --2- 172.21.15.77:0/3087824737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c4969e590 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcb880a39f0 tx=0x7fcb88099970 comp rx=0 tx=0).stop 2024-03-16T10:23:45.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.706+0000 7fcba1170740 1 -- 172.21.15.77:0/3087824737 shutdown_connections 2024-03-16T10:23:45.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.706+0000 7fcba1170740 1 --2- 172.21.15.77:0/3087824737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 0x559c496a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.706+0000 7fcba1170740 1 --2- 172.21.15.77:0/3087824737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c4969eb60 0x559c496a3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.706+0000 7fcba1170740 1 --2- 172.21.15.77:0/3087824737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c4969e590 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.706+0000 7fcba1170740 1 -- 172.21.15.77:0/3087824737 >> 172.21.15.77:0/3087824737 conn(0x559c496996e0 msgr2=0x559c49699ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:45.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.706+0000 7fcba1170740 1 -- 172.21.15.77:0/3087824737 shutdown_connections 2024-03-16T10:23:45.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.706+0000 7fcba1170740 1 -- 172.21.15.77:0/3087824737 wait complete. 2024-03-16T10:23:45.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.706+0000 7fcba1170740 1 -- start start 2024-03-16T10:23:45.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.706+0000 7fcba1170740 1 Processor -- start 2024-03-16T10:23:45.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.707+0000 7fcba1170740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c496a8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.707+0000 7fcba1170740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 0x559c496a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.707+0000 7fcb97fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c496a8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.707+0000 7fcb97fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c496a8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44838/0 (socket says 172.21.15.77:44838) 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 0x559c496a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 0x559c496a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37248/0 (socket says 172.21.15.77:37248) 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcba1170740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c496a3cd0 0x559c496afc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb97fff640 1 -- 172.21.15.77:0/2037100637 learned_addr learned my addr 172.21.15.77:0/2037100637 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcba1170740 1 -- 172.21.15.77:0/2037100637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559c496a6c20 con 0x559c4969eb60 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcba1170740 1 -- 172.21.15.77:0/2037100637 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559c495bf160 con 0x559c496a3cd0 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcba1170740 1 -- 172.21.15.77:0/2037100637 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559c496a6da0 con 0x559c4969e170 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb9ca4b640 1 --2- 172.21.15.77:0/2037100637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c496a3cd0 0x559c496afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcba1170740 10 client.0 _ll_register_callbacks cb 0x559c495a3150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb97fff640 1 -- 172.21.15.77:0/2037100637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c496a3cd0 msgr2=0x559c496afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb97fff640 1 --2- 172.21.15.77:0/2037100637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c496a3cd0 0x559c496afc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb97fff640 1 -- 172.21.15.77:0/2037100637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 msgr2=0x559c496a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:45.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb97fff640 1 --2- 172.21.15.77:0/2037100637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 0x559c496a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb97fff640 1 -- 172.21.15.77:0/2037100637 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c493e17d0 con 0x559c4969e170 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcba1170740 1 -- 172.21.15.77:0/2037100637 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb88086070 con 0x559c4969e170 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb97fff640 1 --2- 172.21.15.77:0/2037100637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c496a8720 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcb88097c00 tx=0x7fcb8809b9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e22ccbe3cd4c1465 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb777fe640 10 client.26812 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb777fe640 1 -- 172.21.15.77:0/2037100637 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb880b9060 con 0x559c4969e170 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb777fe640 1 -- 172.21.15.77:0/2037100637 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fcb880afd70 con 0x559c4969e170 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcb777fe640 1 -- 172.21.15.77:0/2037100637 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fcb88097fa0 con 0x559c4969e170 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcba1170740 10 client.26812 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.708+0000 7fcba1170740 1 -- 172.21.15.77:0/2037100637 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=39}) -- 0x559c493173f0 con 0x559c4969e170 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.709+0000 7fcb777fe640 1 -- 172.21.15.77:0/2037100637 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fcb88018b00 con 0x559c4969e170 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.709+0000 7fcba1170740 10 client.26812 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.709+0000 7fcba1170740 10 client.26812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.709+0000 7fcba1170740 20 client.26812 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:45.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.709+0000 7fcba1170740 1 --2- 172.21.15.77:0/2037100637 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559c496b3c70 0x559c496b6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.709+0000 7fcba1170740 4 client.26812 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:45.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.709+0000 7fcba1170740 1 -- 172.21.15.77:0/2037100637 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c49610a40 con 0x559c496b3c70 2024-03-16T10:23:45.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.709+0000 7fcb977fe640 1 --2- 172.21.15.77:0/2037100637 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559c496b3c70 0x559c496b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.709+0000 7fcb777fe640 10 client.26812 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:45.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.709+0000 7fcb977fe640 1 --2- 172.21.15.77:0/2037100637 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559c496b3c70 0x559c496b6060 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96b0f27b1b3d6eab server_cookie=9a33e353a0561c00 in_seq=0 out_seq=0 2024-03-16T10:23:45.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.710+0000 7fcb777fe640 1 -- 172.21.15.77:0/2037100637 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559c49610a40 con 0x559c496b3c70 2024-03-16T10:23:45.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.710+0000 7fcb777fe640 10 client.26812 handle_command_reply: tid=0 2024-03-16T10:23:45.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.710+0000 7fcba1170740 1 client.26812 shutdown 2024-03-16T10:23:45.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.711+0000 7fcba1170740 1 -- 172.21.15.77:0/2037100637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 msgr2=0x559c496a8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:45.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.711+0000 7fcba1170740 1 --2- 172.21.15.77:0/2037100637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c496a8720 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcb88097c00 tx=0x7fcb8809b9b0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.711+0000 7fcba1170740 1 -- 172.21.15.77:0/2037100637 shutdown_connections 2024-03-16T10:23:45.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.711+0000 7fcba1170740 1 --2- 172.21.15.77:0/2037100637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c496a3cd0 0x559c496afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.711+0000 7fcba1170740 1 --2- 172.21.15.77:0/2037100637 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559c496b3c70 0x559c496b6060 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.711+0000 7fcba1170740 1 --2- 172.21.15.77:0/2037100637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 0x559c496a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.711+0000 7fcb777fe640 0 client.26812 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:45.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.711+0000 7fcba1170740 1 --2- 172.21.15.77:0/2037100637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c496a8720 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.711+0000 7fcba1170740 1 -- 172.21.15.77:0/2037100637 >> 172.21.15.77:0/2037100637 conn(0x559c496996e0 msgr2=0x559c49698900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:45.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.711+0000 7fcba1170740 1 -- 172.21.15.77:0/2037100637 shutdown_connections 2024-03-16T10:23:45.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.711+0000 7fcba1170740 1 -- 172.21.15.77:0/2037100637 wait complete. 2024-03-16T10:23:45.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.711+0000 7fcba1170740 20 client.26812 trim_cache size 0 max 16384 2024-03-16T10:23:45.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.739+0000 7fcba1170740 1 Processor -- start 2024-03-16T10:23:45.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.739+0000 7fcba1170740 1 -- start start 2024-03-16T10:23:45.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.739+0000 7fcba1170740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c4969e170 0x559c496a8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.739+0000 7fcba1170740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969eb60 0x559c496afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.739+0000 7fcba1170740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 0x559c496a8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.739+0000 7fcba1170740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8400e250 con 0x559c496a3cd0 2024-03-16T10:23:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.739+0000 7fcba1170740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8400fd90 con 0x559c4969e170 2024-03-16T10:23:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.739+0000 7fcba1170740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8400e920 con 0x559c4969eb60 2024-03-16T10:23:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb97fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c4969e170 0x559c496a8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb977fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969eb60 0x559c496afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb9ca4b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 0x559c496a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb977fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969eb60 0x559c496afbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44850/0 (socket says 172.21.15.77:44850) 2024-03-16T10:23:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb9ca4b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 0x559c496a8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37250/0 (socket says 172.21.15.77:37250) 2024-03-16T10:23:45.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb977fe640 1 -- 172.21.15.77:0/1402052962 learned_addr learned my addr 172.21.15.77:0/1402052962 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb977fe640 1 -- 172.21.15.77:0/1402052962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c4969e170 msgr2=0x559c496a8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb977fe640 1 --2- 172.21.15.77:0/1402052962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c4969e170 0x559c496a8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb977fe640 1 -- 172.21.15.77:0/1402052962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 msgr2=0x559c496a8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb977fe640 1 --2- 172.21.15.77:0/1402052962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 0x559c496a8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb977fe640 1 -- 172.21.15.77:0/1402052962 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c492d70f0 con 0x559c4969eb60 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb9ca4b640 1 --2- 172.21.15.77:0/1402052962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 0x559c496a8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb977fe640 1 --2- 172.21.15.77:0/1402052962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969eb60 0x559c496afbc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x559c4969f5d0 tx=0x7fcb8c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4eb2f2b951f06d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb7dffb640 1 -- 172.21.15.77:0/1402052962 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c01b950 con 0x559c4969eb60 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb7dffb640 1 -- 172.21.15.77:0/1402052962 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb8c005020 con 0x559c4969eb60 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.740+0000 7fcb7dffb640 1 -- 172.21.15.77:0/1402052962 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c004bd0 con 0x559c4969eb60 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.741+0000 7fcba1170740 1 -- 172.21.15.77:0/1402052962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969eb60 msgr2=0x559c496afbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.741+0000 7fcba1170740 1 --2- 172.21.15.77:0/1402052962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969eb60 0x559c496afbc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x559c4969f5d0 tx=0x7fcb8c014000 comp rx=0 tx=0).stop 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.741+0000 7fcba1170740 1 -- 172.21.15.77:0/1402052962 shutdown_connections 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.741+0000 7fcba1170740 1 --2- 172.21.15.77:0/1402052962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c496a3cd0 0x559c496a8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.748 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.741+0000 7fcba1170740 1 --2- 172.21.15.77:0/1402052962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969eb60 0x559c496afbc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.741+0000 7fcba1170740 1 --2- 172.21.15.77:0/1402052962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c4969e170 0x559c496a8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.741+0000 7fcba1170740 1 -- 172.21.15.77:0/1402052962 >> 172.21.15.77:0/1402052962 conn(0x559c496996e0 msgr2=0x559c49699ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.741+0000 7fcba1170740 1 -- 172.21.15.77:0/1402052962 shutdown_connections 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.741+0000 7fcba1170740 1 -- 172.21.15.77:0/1402052962 wait complete. 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcba1170740 1 -- start start 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcba1170740 1 Processor -- start 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcba1170740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c496ad190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcba1170740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 0x559c496ad6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcba1170740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c496a3cd0 0x559c496ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcba1170740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8400f6c0 con 0x559c4969eb60 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcba1170740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8400eff0 con 0x559c496a3cd0 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcba1170740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8400c230 con 0x559c4969e170 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcba1170740 10 client.0 _ll_register_callbacks cb 0x7fcb88005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:45.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcb977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 0x559c496ad6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcb97fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c496ad190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcb977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 0x559c496ad6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37258/0 (socket says 172.21.15.77:37258) 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcb977fe640 1 -- 172.21.15.77:0/3238483014 learned_addr learned my addr 172.21.15.77:0/3238483014 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcb9ca4b640 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c496a3cd0 0x559c496ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcb97fff640 1 -- 172.21.15.77:0/3238483014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c496a3cd0 msgr2=0x559c496ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcb97fff640 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c496a3cd0 0x559c496ae6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcb97fff640 1 -- 172.21.15.77:0/3238483014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 msgr2=0x559c496ad6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcb97fff640 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 0x559c496ad6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.742+0000 7fcb97fff640 1 -- 172.21.15.77:0/3238483014 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c495ca9d0 con 0x559c4969e170 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcb977fe640 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 0x559c496ad6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcb97fff640 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c496ad190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcb8809b7c0 tx=0x7fcb8809fa50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a06c73975e826064 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcb777fe640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:23:45.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcb777fe640 1 -- 172.21.15.77:0/3238483014 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88010740 con 0x559c4969e170 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcb777fe640 1 -- 172.21.15.77:0/3238483014 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fcb88097e10 con 0x559c4969e170 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcba1170740 1 -- 172.21.15.77:0/3238483014 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb8c005020 con 0x559c4969e170 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcb777fe640 1 -- 172.21.15.77:0/3238483014 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fcb880a5070 con 0x559c4969e170 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcba1170740 10 client.26818 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcba1170740 1 -- 172.21.15.77:0/3238483014 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x559c495ca9d0 con 0x559c4969e170 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcb777fe640 1 -- 172.21.15.77:0/3238483014 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb8c005020 con 0x559c4969e170 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcb777fe640 1 -- 172.21.15.77:0/3238483014 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fcb8809a6f0 con 0x559c4969e170 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcba1170740 10 client.26818 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcba1170740 10 client.26818 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.743+0000 7fcba1170740 20 client.26818 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.744+0000 7fcba1170740 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559c496b3c70 0x559c4971c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.744+0000 7fcba1170740 4 client.26818 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.744+0000 7fcba1170740 1 -- 172.21.15.77:0/3238483014 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559c4960ad90 con 0x559c496b3c70 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.744+0000 7fcb977fe640 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559c496b3c70 0x559c4971c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:45.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.744+0000 7fcb777fe640 10 client.26818 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.744+0000 7fcb977fe640 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559c496b3c70 0x559c4971c120 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6221a979177594e1 server_cookie=1acb301738584a6e in_seq=0 out_seq=0 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcb777fe640 1 -- 172.21.15.77:0/3238483014 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559c4960ad90 con 0x559c496b3c70 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcb777fe640 10 client.26818 handle_command_reply: tid=0 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcba1170740 1 client.26818 shutdown 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcba1170740 1 -- 172.21.15.77:0/3238483014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 msgr2=0x559c496ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcba1170740 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c496ad190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcb8809b7c0 tx=0x7fcb8809fa50 comp rx=0 tx=0).stop 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcba1170740 1 -- 172.21.15.77:0/3238483014 shutdown_connections 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcba1170740 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559c496a3cd0 0x559c496ae6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcba1170740 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559c496b3c70 0x559c4971c120 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcba1170740 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559c4969eb60 0x559c496ad6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcba1170740 1 --2- 172.21.15.77:0/3238483014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c4969e170 0x559c496ad190 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcb777fe640 0 client.26818 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcba1170740 1 -- 172.21.15.77:0/3238483014 >> 172.21.15.77:0/3238483014 conn(0x559c496996e0 msgr2=0x559c496a82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:45.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcba1170740 1 -- 172.21.15.77:0/3238483014 shutdown_connections 2024-03-16T10:23:45.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcba1170740 1 -- 172.21.15.77:0/3238483014 wait complete. 2024-03-16T10:23:45.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.745+0000 7fcba1170740 20 client.26818 trim_cache size 0 max 16384 2024-03-16T10:23:45.753 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:23:45.753 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T10:23:45.753 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T10:23:45.753 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:23:45.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.746+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/1153658618 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcb800a2880 msgr2=0x7fcb800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:45.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.746+0000 7fcb9e4ae640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcb800a2880 0x7fcb800a4d40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c011500 tx=0x7fcb8c005f90 comp rx=0 tx=0).stop 2024-03-16T10:23:45.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.746+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/1153658618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb98100090 msgr2=0x7fcb98199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:45.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.746+0000 7fcb9e4ae640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb98100090 0x7fcb98199d00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800e9f0 tx=0x7fcb8800eeb0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.746+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/1153658618 shutdown_connections 2024-03-16T10:23:45.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.746+0000 7fcb9e4ae640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb98101390 0x7fcb981a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.746+0000 7fcb9e4ae640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb981009f0 0x7fcb9819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.746+0000 7fcb9e4ae640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcb800a2880 0x7fcb800a4d40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.746+0000 7fcb9e4ae640 1 --2- 172.21.15.77:0/1153658618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb98100090 0x7fcb98199d00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:45.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.746+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/1153658618 >> 172.21.15.77:0/1153658618 conn(0x7fcb980fbc70 msgr2=0x7fcb980fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:45.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.746+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/1153658618 shutdown_connections 2024-03-16T10:23:45.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:45.746+0000 7fcb9e4ae640 1 -- 172.21.15.77:0/1153658618 wait complete. 2024-03-16T10:23:45.759 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T10:23:45.759 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:23:45.931 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:45 smithi077 ceph-mon[32400]: pgmap v550: 97 pgs: 97 active+clean; 751 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 305 op/s 2024-03-16T10:23:45.931 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:45 smithi077 ceph-mon[32400]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T10:23:45.931 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:45 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.193:6834/946505511,v1:172.21.15.193:6835/946505511] up:boot 2024-03-16T10:23:45.931 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:45 smithi077 ceph-mon[32400]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-16T10:23:45.931 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:45 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T10:23:45.931 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:45 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:23:46.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:45 smithi193 ceph-mon[36311]: pgmap v550: 97 pgs: 97 active+clean; 751 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 305 op/s 2024-03-16T10:23:46.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:45 smithi193 ceph-mon[36311]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T10:23:46.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:45 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.193:6834/946505511,v1:172.21.15.193:6835/946505511] up:boot 2024-03-16T10:23:46.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:45 smithi193 ceph-mon[36311]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-16T10:23:46.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:45 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T10:23:46.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:45 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:23:46.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:45 smithi192 ceph-mon[36354]: pgmap v550: 97 pgs: 97 active+clean; 751 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 305 op/s 2024-03-16T10:23:46.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:45 smithi192 ceph-mon[36354]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T10:23:46.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:45 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.193:6834/946505511,v1:172.21.15.193:6835/946505511] up:boot 2024-03-16T10:23:46.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:45 smithi192 ceph-mon[36354]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-16T10:23:46.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:45 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T10:23:46.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:45 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:23:47.107 INFO:tasks.workunit.client.0.smithi077.stdout: 173 1904431 439 1276654 258 1132896 1086 2024-03-16T10:23:47.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.227+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1095504624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6ec10db20 msgr2=0x7fc6ec10dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:47.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.227+0000 7fc6f23a9640 1 --2- 172.21.15.77:0/1095504624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6ec10db20 0x7fc6ec10dfd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009860 tx=0x7fc6e002bb90 comp rx=0 tx=0).stop 2024-03-16T10:23:47.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.230+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1095504624 shutdown_connections 2024-03-16T10:23:47.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.230+0000 7fc6f23a9640 1 --2- 172.21.15.77:0/1095504624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6ec10db20 0x7fc6ec10dfd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.230+0000 7fc6f23a9640 1 --2- 172.21.15.77:0/1095504624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6ec10b150 0x7fc6ec10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.230+0000 7fc6f23a9640 1 --2- 172.21.15.77:0/1095504624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec1087e0 0x7fc6ec10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.230+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1095504624 >> 172.21.15.77:0/1095504624 conn(0x7fc6ec06e650 msgr2=0x7fc6ec070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.232+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1095504624 shutdown_connections 2024-03-16T10:23:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.232+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1095504624 wait complete. 2024-03-16T10:23:47.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.233+0000 7fc6f23a9640 1 Processor -- start 2024-03-16T10:23:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.236+0000 7fc6f23a9640 1 -- start start 2024-03-16T10:23:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.236+0000 7fc6f23a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6ec1087e0 0x7fc6ec114f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.236+0000 7fc6f23a9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec10b150 0x7fc6ec115450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.236+0000 7fc6f23a9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6ec10db20 0x7fc6ec1b6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.236+0000 7fc6f23a9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec11c2d0 con 0x7fc6ec1087e0 2024-03-16T10:23:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.237+0000 7fc6f23a9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec11c150 con 0x7fc6ec10b150 2024-03-16T10:23:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.237+0000 7fc6f23a9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec11c450 con 0x7fc6ec10db20 2024-03-16T10:23:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.237+0000 7fc6ebfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6ec1087e0 0x7fc6ec114f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.237+0000 7fc6ebfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6ec1087e0 0x7fc6ec114f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37280/0 (socket says 172.21.15.77:37280) 2024-03-16T10:23:47.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.237+0000 7fc6ebfff640 1 -- 172.21.15.77:0/1378675605 learned_addr learned my addr 172.21.15.77:0/1378675605 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:47.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.237+0000 7fc6ebfff640 1 -- 172.21.15.77:0/1378675605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6ec10db20 msgr2=0x7fc6ec1b6980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T10:23:47.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.237+0000 7fc6ebfff640 1 --2- 172.21.15.77:0/1378675605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6ec10db20 0x7fc6ec1b6980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.237+0000 7fc6ebfff640 1 -- 172.21.15.77:0/1378675605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec10b150 msgr2=0x7fc6ec115450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T10:23:47.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.237+0000 7fc6ebfff640 1 --2- 172.21.15.77:0/1378675605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec10b150 0x7fc6ec115450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.237+0000 7fc6ebfff640 1 -- 172.21.15.77:0/1378675605 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ec1b6ec0 con 0x7fc6ec1087e0 2024-03-16T10:23:47.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.238+0000 7fc6ebfff640 1 --2- 172.21.15.77:0/1378675605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6ec1087e0 0x7fc6ec114f10 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400c3f0 tx=0x7fc6e4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:47.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.238+0000 7fc6e97fa640 1 -- 172.21.15.77:0/1378675605 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e401b070 con 0x7fc6ec1087e0 2024-03-16T10:23:47.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.238+0000 7fc6e97fa640 1 -- 172.21.15.77:0/1378675605 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6e400ba50 con 0x7fc6ec1087e0 2024-03-16T10:23:47.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.238+0000 7fc6e97fa640 1 -- 172.21.15.77:0/1378675605 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4016d20 con 0x7fc6ec1087e0 2024-03-16T10:23:47.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.239+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1378675605 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ec1b7090 con 0x7fc6ec1087e0 2024-03-16T10:23:47.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.239+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1378675605 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ec1b7590 con 0x7fc6ec1087e0 2024-03-16T10:23:47.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.241+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1378675605 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ec0685d0 con 0x7fc6ec1087e0 2024-03-16T10:23:47.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.243+0000 7fc6e97fa640 1 -- 172.21.15.77:0/1378675605 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fc6e401e070 con 0x7fc6ec1087e0 2024-03-16T10:23:47.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.244+0000 7fc6e97fa640 1 --2- 172.21.15.77:0/1378675605 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc6d40a2980 0x7fc6d40a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.244+0000 7fc6e97fa640 1 -- 172.21.15.77:0/1378675605 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fc6e40cef00 con 0x7fc6ec1087e0 2024-03-16T10:23:47.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.245+0000 7fc6eb7fe640 1 --2- 172.21.15.77:0/1378675605 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc6d40a2980 0x7fc6d40a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.247+0000 7fc6eb7fe640 1 --2- 172.21.15.77:0/1378675605 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc6d40a2980 0x7fc6d40a4e40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec116420 tx=0x7fc6dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:47.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:47 smithi193 ceph-mon[36311]: pgmap v552: 97 pgs: 97 active+clean; 752 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 303 op/s 2024-03-16T10:23:47.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.251+0000 7fc6e97fa640 1 -- 172.21.15.77:0/1378675605 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc6e4025080 con 0x7fc6ec1087e0 2024-03-16T10:23:47.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:47 smithi192 ceph-mon[36354]: pgmap v552: 97 pgs: 97 active+clean; 752 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 303 op/s 2024-03-16T10:23:47.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:47 smithi077 ceph-mon[32400]: pgmap v552: 97 pgs: 97 active+clean; 752 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 303 op/s 2024-03-16T10:23:47.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.683+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1378675605 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6ec02d3e0 con 0x7fc6ec1087e0 2024-03-16T10:23:47.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.684+0000 7fc6e97fa640 1 -- 172.21.15.77:0/1378675605 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11942 (secure 0 0 0) 0x7fc6e409c010 con 0x7fc6ec1087e0 2024-03-16T10:23:47.685 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:23:47.685 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:23:44.952104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26800":{"gid":26800,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/946505511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":946505511},{"type":"v1","addr":"172.21.15.193:6835","nonce":946505511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24475,24454,24482,14658,24455]},"id":1}]} 2024-03-16T10:23:47.686 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 39 2024-03-16T10:23:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.687+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1378675605 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc6d40a2980 msgr2=0x7fc6d40a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.687+0000 7fc6f23a9640 1 --2- 172.21.15.77:0/1378675605 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc6d40a2980 0x7fc6d40a4e40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec116420 tx=0x7fc6dc005f90 comp rx=0 tx=0).stop 2024-03-16T10:23:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.687+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1378675605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6ec1087e0 msgr2=0x7fc6ec114f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.687+0000 7fc6f23a9640 1 --2- 172.21.15.77:0/1378675605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6ec1087e0 0x7fc6ec114f10 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400c3f0 tx=0x7fc6e4009f90 comp rx=0 tx=0).stop 2024-03-16T10:23:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.687+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1378675605 shutdown_connections 2024-03-16T10:23:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.687+0000 7fc6f23a9640 1 --2- 172.21.15.77:0/1378675605 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc6d40a2980 0x7fc6d40a4e40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.687+0000 7fc6f23a9640 1 --2- 172.21.15.77:0/1378675605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6ec10db20 0x7fc6ec1b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.687+0000 7fc6f23a9640 1 --2- 172.21.15.77:0/1378675605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6ec10b150 0x7fc6ec115450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.687+0000 7fc6f23a9640 1 --2- 172.21.15.77:0/1378675605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6ec1087e0 0x7fc6ec114f10 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.688+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1378675605 >> 172.21.15.77:0/1378675605 conn(0x7fc6ec06e650 msgr2=0x7fc6ec109da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:47.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.690+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1378675605 shutdown_connections 2024-03-16T10:23:47.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.690+0000 7fc6f23a9640 1 -- 172.21.15.77:0/1378675605 wait complete. 2024-03-16T10:23:47.708 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-16T10:23:47.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.874+0000 7f4d4d189640 1 -- 172.21.15.77:0/1516016746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d48100050 msgr2=0x7f4d48102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:47.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.874+0000 7f4d4d189640 1 --2- 172.21.15.77:0/1516016746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d48100050 0x7f4d48102480 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009be0 tx=0x7f4d4002bef0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.875+0000 7f4d4d189640 1 -- 172.21.15.77:0/1516016746 shutdown_connections 2024-03-16T10:23:47.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.875+0000 7f4d4d189640 1 --2- 172.21.15.77:0/1516016746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d48105370 0x7f4d481057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.875+0000 7f4d4d189640 1 --2- 172.21.15.77:0/1516016746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d481029c0 0x7f4d48104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.875+0000 7f4d4d189640 1 --2- 172.21.15.77:0/1516016746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d48100050 0x7f4d48102480 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.875+0000 7f4d4d189640 1 -- 172.21.15.77:0/1516016746 >> 172.21.15.77:0/1516016746 conn(0x7f4d480fbc70 msgr2=0x7f4d480fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:47.876 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.875+0000 7f4d4d189640 1 -- 172.21.15.77:0/1516016746 shutdown_connections 2024-03-16T10:23:47.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.875+0000 7f4d4d189640 1 -- 172.21.15.77:0/1516016746 wait complete. 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.876+0000 7f4d4d189640 1 Processor -- start 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.876+0000 7f4d4d189640 1 -- start start 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.876+0000 7f4d4d189640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d48100050 0x7f4d480701b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.876+0000 7f4d4d189640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d481029c0 0x7f4d480706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4d189640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d48105370 0x7f4d481ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4d189640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4d48113a80 con 0x7f4d481029c0 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4d189640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4d48113900 con 0x7f4d48105370 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4d189640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4d48113c00 con 0x7f4d48100050 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d481029c0 0x7f4d480706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d481029c0 0x7f4d480706f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37296/0 (socket says 172.21.15.77:37296) 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4659c640 1 -- 172.21.15.77:0/1079620145 learned_addr learned my addr 172.21.15.77:0/1079620145 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d46d9d640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d48100050 0x7f4d480701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4759e640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d48105370 0x7f4d481ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4659c640 1 -- 172.21.15.77:0/1079620145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d48100050 msgr2=0x7f4d480701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4659c640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d48100050 0x7f4d480701b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4659c640 1 -- 172.21.15.77:0/1079620145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d48105370 msgr2=0x7f4d481ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:47.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4659c640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d48105370 0x7f4d481ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.877+0000 7f4d4659c640 1 -- 172.21.15.77:0/1079620145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d481ae6d0 con 0x7f4d481029c0 2024-03-16T10:23:47.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.878+0000 7f4d4659c640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d481029c0 0x7f4d480706f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800e9c0 tx=0x7f4d3800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:47.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.878+0000 7f4d2ffff640 1 -- 172.21.15.77:0/1079620145 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3800ccc0 con 0x7f4d481029c0 2024-03-16T10:23:47.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.878+0000 7f4d4d189640 1 -- 172.21.15.77:0/1079620145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d481ae8a0 con 0x7f4d481029c0 2024-03-16T10:23:47.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.878+0000 7f4d2ffff640 1 -- 172.21.15.77:0/1079620145 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d38005500 con 0x7f4d481029c0 2024-03-16T10:23:47.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.878+0000 7f4d2ffff640 1 -- 172.21.15.77:0/1079620145 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d38010430 con 0x7f4d481029c0 2024-03-16T10:23:47.880 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.878+0000 7f4d4d189640 1 -- 172.21.15.77:0/1079620145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d481aede0 con 0x7f4d481029c0 2024-03-16T10:23:47.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.881+0000 7f4d2ffff640 1 -- 172.21.15.77:0/1079620145 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f4d380046c0 con 0x7f4d481029c0 2024-03-16T10:23:47.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.882+0000 7f4d4fd79740 1 Processor -- start 2024-03-16T10:23:47.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.882+0000 7f4d4fd79740 1 -- start start 2024-03-16T10:23:47.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.882+0000 7f4d4fd79740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c04cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.882+0000 7f4d2ffff640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d300a2900 0x7f4d300a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.882+0000 7f4d4fd79740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c04d240 0x555b4c051d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.882+0000 7f4d4fd79740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c0523b0 0x555b4c054840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.882+0000 7f4d4fd79740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555b4bfb1490 con 0x555b4c0523b0 2024-03-16T10:23:47.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.882+0000 7f4d4fd79740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x555b4bf59c90 con 0x555b4c04c850 2024-03-16T10:23:47.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.882+0000 7f4d4fd79740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555b4bfad530 con 0x555b4c04d240 2024-03-16T10:23:47.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.883+0000 7f4d2ffff640 1 -- 172.21.15.77:0/1079620145 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f4d380cecf0 con 0x7f4d481029c0 2024-03-16T10:23:47.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.883+0000 7f4d4759e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c04cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.883+0000 7f4d4759e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c04cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44576/0 (socket says 172.21.15.77:44576) 2024-03-16T10:23:47.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.883+0000 7f4d4759e640 1 -- 172.21.15.77:0/2928681061 learned_addr learned my addr 172.21.15.77:0/2928681061 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:47.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.883+0000 7f4d4659c640 1 --2- 172.21.15.77:0/2928681061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c04d240 0x555b4c051d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.883+0000 7f4d4659c640 1 -- 172.21.15.77:0/2928681061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 msgr2=0x555b4c04cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:47.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.883+0000 7f4d4659c640 1 --2- 172.21.15.77:0/2928681061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c04cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.883+0000 7f4d46d9d640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d300a2900 0x7f4d300a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.883+0000 7f4d4659c640 1 -- 172.21.15.77:0/2928681061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c0523b0 msgr2=0x555b4c054840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:47.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.883+0000 7f4d4659c640 1 --2- 172.21.15.77:0/2928681061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c0523b0 0x555b4c054840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.883+0000 7f4d4659c640 1 -- 172.21.15.77:0/2928681061 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b4c014500 con 0x555b4c04d240 2024-03-16T10:23:47.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.883+0000 7f4d4759e640 1 --2- 172.21.15.77:0/2928681061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c04cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:47.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.884+0000 7f4d4659c640 1 --2- 172.21.15.77:0/2928681061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c04d240 0x555b4c051d40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4d3809b380 tx=0x7f4d38004a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31615b2317c7f711 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:47.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.884+0000 7f4d2d7fa640 1 -- 172.21.15.77:0/2928681061 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d380b4070 con 0x555b4c04d240 2024-03-16T10:23:47.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.884+0000 7f4d46d9d640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d300a2900 0x7f4d300a4dc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4d40006010 tx=0x7f4d40000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:47.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.884+0000 7f4d2d7fa640 1 -- 172.21.15.77:0/2928681061 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d380bb070 con 0x555b4c04d240 2024-03-16T10:23:47.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.884+0000 7f4d2d7fa640 1 -- 172.21.15.77:0/2928681061 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d38097cf0 con 0x555b4c04d240 2024-03-16T10:23:47.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.885+0000 7f4d4fd79740 1 -- 172.21.15.77:0/2928681061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c04d240 msgr2=0x555b4c051d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:47.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.885+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/2928681061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c04d240 0x555b4c051d40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4d3809b380 tx=0x7f4d38004a60 comp rx=0 tx=0).stop 2024-03-16T10:23:47.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.885+0000 7f4d4fd79740 1 -- 172.21.15.77:0/2928681061 shutdown_connections 2024-03-16T10:23:47.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.885+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/2928681061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c0523b0 0x555b4c054840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.885+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/2928681061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c04d240 0x555b4c051d40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.885+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/2928681061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c04cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.886+0000 7f4d4fd79740 1 -- 172.21.15.77:0/2928681061 >> 172.21.15.77:0/2928681061 conn(0x555b4c047dc0 msgr2=0x555b4c0481c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:47.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.886+0000 7f4d4fd79740 1 -- 172.21.15.77:0/2928681061 shutdown_connections 2024-03-16T10:23:47.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.886+0000 7f4d4fd79740 1 -- 172.21.15.77:0/2928681061 wait complete. 2024-03-16T10:23:47.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.886+0000 7f4d4fd79740 1 -- start start 2024-03-16T10:23:47.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.886+0000 7f4d4fd79740 1 Processor -- start 2024-03-16T10:23:47.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.886+0000 7f4d4fd79740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c04c850 0x555b4c056df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4fd79740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04d240 0x555b4c057330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4fd79740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c0523b0 0x555b4c05e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4fd79740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555b4c055360 con 0x555b4c04c850 2024-03-16T10:23:47.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4fd79740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x555b4bf6d230 con 0x555b4c04d240 2024-03-16T10:23:47.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4fd79740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555b4c0554e0 con 0x555b4c0523b0 2024-03-16T10:23:47.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04d240 0x555b4c057330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4759e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c0523b0 0x555b4c05e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d46d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c04c850 0x555b4c056df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04d240 0x555b4c057330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44578/0 (socket says 172.21.15.77:44578) 2024-03-16T10:23:47.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4759e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c0523b0 0x555b4c05e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44924/0 (socket says 172.21.15.77:44924) 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4659c640 1 -- 172.21.15.77:0/1142480501 learned_addr learned my addr 172.21.15.77:0/1142480501 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4fd79740 10 client.0 _ll_register_callbacks cb 0x555b4bfa9010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4759e640 1 -- 172.21.15.77:0/1142480501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04d240 msgr2=0x555b4c057330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4759e640 1 --2- 172.21.15.77:0/1142480501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04d240 0x555b4c057330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4759e640 1 -- 172.21.15.77:0/1142480501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c04c850 msgr2=0x555b4c056df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4759e640 1 --2- 172.21.15.77:0/1142480501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c04c850 0x555b4c056df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4759e640 1 -- 172.21.15.77:0/1142480501 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b4bf7c100 con 0x555b4c0523b0 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4659c640 1 --2- 172.21.15.77:0/1142480501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04d240 0x555b4c057330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d4759e640 1 --2- 172.21.15.77:0/1142480501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c0523b0 0x555b4c05e330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c010660 tx=0x7f4d3c010b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=127b965b95aa9d8f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.887+0000 7f4d2aff5640 1 -- 172.21.15.77:0/1142480501 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c004320 con 0x555b4c0523b0 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1142480501 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d380bb070 con 0x555b4c0523b0 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d2aff5640 1 -- 172.21.15.77:0/1142480501 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f4d3c005020 con 0x555b4c0523b0 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d2aff5640 1 -- 172.21.15.77:0/1142480501 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4d3c024050 con 0x555b4c0523b0 2024-03-16T10:23:47.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d4fd79740 10 client.26833 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:47.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1142480501 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x555b4bcc53f0 con 0x555b4c0523b0 2024-03-16T10:23:47.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d2aff5640 1 -- 172.21.15.77:0/1142480501 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d380bb070 con 0x555b4c0523b0 2024-03-16T10:23:47.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d2aff5640 1 -- 172.21.15.77:0/1142480501 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f4d3c0047c0 con 0x555b4c0523b0 2024-03-16T10:23:47.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d4fd79740 10 client.26833 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:47.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d4fd79740 10 client.26833 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:23:47.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d4fd79740 20 client.26833 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:47.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/1142480501 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x555b4c062ba0 0x555b4c065040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d4fd79740 4 client.26833 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:47.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1142480501 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b4bfbe9d0 con 0x555b4c062ba0 2024-03-16T10:23:47.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.888+0000 7f4d46d9d640 1 --2- 172.21.15.77:0/1142480501 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x555b4c062ba0 0x555b4c065040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.889+0000 7f4d2aff5640 10 client.26833 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:23:47.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.889+0000 7f4d46d9d640 1 --2- 172.21.15.77:0/1142480501 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x555b4c062ba0 0x555b4c065040 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a423e86952519ee8 server_cookie=e44f34e306f92ef1 in_seq=0 out_seq=0 2024-03-16T10:23:47.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d2aff5640 1 -- 172.21.15.77:0/1142480501 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555b4bfbe9d0 con 0x555b4c062ba0 2024-03-16T10:23:47.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d2aff5640 10 client.26833 handle_command_reply: tid=0 2024-03-16T10:23:47.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d4fd79740 1 client.26833 shutdown 2024-03-16T10:23:47.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1142480501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c0523b0 msgr2=0x555b4c05e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:47.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/1142480501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c0523b0 0x555b4c05e330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c010660 tx=0x7f4d3c010b20 comp rx=0 tx=0).stop 2024-03-16T10:23:47.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1142480501 shutdown_connections 2024-03-16T10:23:47.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/1142480501 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x555b4c062ba0 0x555b4c065040 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/1142480501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c0523b0 0x555b4c05e330 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d2aff5640 0 client.26833 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:23:47.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/1142480501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04d240 0x555b4c057330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/1142480501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c04c850 0x555b4c056df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1142480501 >> 172.21.15.77:0/1142480501 conn(0x555b4c047dc0 msgr2=0x555b4c046fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:47.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1142480501 shutdown_connections 2024-03-16T10:23:47.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1142480501 wait complete. 2024-03-16T10:23:47.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.890+0000 7f4d4fd79740 20 client.26833 trim_cache size 0 max 16384 2024-03-16T10:23:47.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.908+0000 7f4d4fd79740 1 Processor -- start 2024-03-16T10:23:47.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.908+0000 7f4d4fd79740 1 -- start start 2024-03-16T10:23:47.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d4fd79740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c046e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d4fd79740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c04d240 0x555b4c05e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d4fd79740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c0523b0 0x555b4c069900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d4fd79740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4d481b7a50 con 0x555b4c0523b0 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d4fd79740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4d481b9590 con 0x555b4c04c850 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d4fd79740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4d481b8120 con 0x555b4c04d240 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d46d9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c046e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d46d9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c046e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44580/0 (socket says 172.21.15.77:44580) 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d46d9d640 1 -- 172.21.15.77:0/194344746 learned_addr learned my addr 172.21.15.77:0/194344746 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d4659c640 1 --2- 172.21.15.77:0/194344746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c04d240 0x555b4c05e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d4759e640 1 --2- 172.21.15.77:0/194344746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c0523b0 0x555b4c069900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d46d9d640 1 -- 172.21.15.77:0/194344746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c04d240 msgr2=0x555b4c05e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d46d9d640 1 --2- 172.21.15.77:0/194344746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c04d240 0x555b4c05e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d46d9d640 1 -- 172.21.15.77:0/194344746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c0523b0 msgr2=0x555b4c069900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:47.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d46d9d640 1 --2- 172.21.15.77:0/194344746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c0523b0 0x555b4c069900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d46d9d640 1 -- 172.21.15.77:0/194344746 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b4bd8f7d0 con 0x555b4c04c850 2024-03-16T10:23:47.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.909+0000 7f4d4759e640 1 --2- 172.21.15.77:0/194344746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c0523b0 0x555b4c069900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:23:47.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.910+0000 7f4d46d9d640 1 --2- 172.21.15.77:0/194344746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c046e60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009be0 tx=0x7f4d4003f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52790c39c84091f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:47.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.910+0000 7f4d2cff9640 1 -- 172.21.15.77:0/194344746 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4002da20 con 0x555b4c04c850 2024-03-16T10:23:47.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.910+0000 7f4d2cff9640 1 -- 172.21.15.77:0/194344746 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d40003e50 con 0x555b4c04c850 2024-03-16T10:23:47.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.910+0000 7f4d2cff9640 1 -- 172.21.15.77:0/194344746 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40051730 con 0x555b4c04c850 2024-03-16T10:23:47.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.911+0000 7f4d4fd79740 1 -- 172.21.15.77:0/194344746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 msgr2=0x555b4c046e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:47.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.911+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/194344746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c046e60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009be0 tx=0x7f4d4003f000 comp rx=0 tx=0).stop 2024-03-16T10:23:47.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.911+0000 7f4d4fd79740 1 -- 172.21.15.77:0/194344746 shutdown_connections 2024-03-16T10:23:47.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.911+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/194344746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c0523b0 0x555b4c069900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.911+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/194344746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c04d240 0x555b4c05e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.911+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/194344746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c046e60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.911+0000 7f4d4fd79740 1 -- 172.21.15.77:0/194344746 >> 172.21.15.77:0/194344746 conn(0x555b4c047dc0 msgr2=0x555b4c046750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:47.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.911+0000 7f4d4fd79740 1 -- 172.21.15.77:0/194344746 shutdown_connections 2024-03-16T10:23:47.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.911+0000 7f4d4fd79740 1 -- 172.21.15.77:0/194344746 wait complete. 2024-03-16T10:23:47.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.911+0000 7f4d4fd79740 1 -- start start 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d4fd79740 1 Processor -- start 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d4fd79740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c05b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d4fd79740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c04d240 0x555b4c05bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d4fd79740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c0523b0 0x555b4c05cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d4fd79740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4d481b8ec0 con 0x555b4c04d240 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d4fd79740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4d481b87f0 con 0x555b4c04c850 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d4fd79740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4d481b5a30 con 0x555b4c0523b0 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d46d9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c05b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d4fd79740 10 client.0 _ll_register_callbacks cb 0x7f4d3c009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d46d9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c05b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44584/0 (socket says 172.21.15.77:44584) 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d46d9d640 1 -- 172.21.15.77:0/1899685430 learned_addr learned my addr 172.21.15.77:0/1899685430 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d4759e640 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c0523b0 0x555b4c05cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.912+0000 7f4d4659c640 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c04d240 0x555b4c05bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.913+0000 7f4d46d9d640 1 -- 172.21.15.77:0/1899685430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c0523b0 msgr2=0x555b4c05cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.913+0000 7f4d46d9d640 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c0523b0 0x555b4c05cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.913+0000 7f4d46d9d640 1 -- 172.21.15.77:0/1899685430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c04d240 msgr2=0x555b4c05bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.913+0000 7f4d46d9d640 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c04d240 0x555b4c05bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.913+0000 7f4d46d9d640 1 -- 172.21.15.77:0/1899685430 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b4c05d1d0 con 0x555b4c04c850 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.913+0000 7f4d46d9d640 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c05b690 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4d4002c430 tx=0x7f4d40009480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae738c2b8e238735 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.913+0000 7f4d2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.913+0000 7f4d2aff5640 1 -- 172.21.15.77:0/1899685430 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4005e020 con 0x555b4c04c850 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.913+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1899685430 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d40003e50 con 0x555b4c04c850 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.913+0000 7f4d2aff5640 1 -- 172.21.15.77:0/1899685430 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f4d40051770 con 0x555b4c04c850 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.913+0000 7f4d2aff5640 1 -- 172.21.15.77:0/1899685430 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4d4002da20 con 0x555b4c04c850 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.913+0000 7f4d4fd79740 10 client.26864 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.914+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1899685430 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x555b4c05e520 con 0x555b4c04c850 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.914+0000 7f4d2aff5640 1 -- 172.21.15.77:0/1899685430 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d40003e50 con 0x555b4c04c850 2024-03-16T10:23:47.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.914+0000 7f4d2aff5640 1 -- 172.21.15.77:0/1899685430 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f4d40034430 con 0x555b4c04c850 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.914+0000 7f4d4fd79740 10 client.26864 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.914+0000 7f4d4fd79740 10 client.26864 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.914+0000 7f4d4fd79740 20 client.26864 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.914+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x555b4c0c9600 0x555b4c0cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.915+0000 7f4d4fd79740 4 client.26864 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.915+0000 7f4d4659c640 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x555b4c0c9600 0x555b4c0cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.915+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1899685430 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555b4bfb8d20 con 0x555b4c0c9600 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.915+0000 7f4d4659c640 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x555b4c0c9600 0x555b4c0cba20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fdc5d381966e8006 server_cookie=2d66944e1d8b7a6a in_seq=0 out_seq=0 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.915+0000 7f4d2aff5640 10 client.26864 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.915+0000 7f4d2aff5640 1 -- 172.21.15.77:0/1899685430 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555b4bfb8d20 con 0x555b4c0c9600 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d2aff5640 10 client.26864 handle_command_reply: tid=0 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d4fd79740 1 client.26864 shutdown 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1899685430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 msgr2=0x555b4c05b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c05b690 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4d4002c430 tx=0x7f4d40009480 comp rx=0 tx=0).stop 2024-03-16T10:23:47.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1899685430 shutdown_connections 2024-03-16T10:23:47.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x555b4c0c9600 0x555b4c0cba20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4c0523b0 0x555b4c05cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b4c04d240 0x555b4c05bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d2aff5640 0 client.26864 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:23:47.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d4fd79740 1 --2- 172.21.15.77:0/1899685430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555b4c04c850 0x555b4c05b690 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1899685430 >> 172.21.15.77:0/1899685430 conn(0x555b4c047dc0 msgr2=0x555b4c04b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:47.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1899685430 shutdown_connections 2024-03-16T10:23:47.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.916+0000 7f4d4fd79740 1 -- 172.21.15.77:0/1899685430 wait complete. 2024-03-16T10:23:47.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.917+0000 7f4d4fd79740 20 client.26864 trim_cache size 0 max 16384 2024-03-16T10:23:47.922 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:23:47.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.918+0000 7f4d4d189640 1 -- 172.21.15.77:0/1079620145 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d300a2900 msgr2=0x7f4d300a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:47.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.918+0000 7f4d4d189640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d300a2900 0x7f4d300a4dc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4d40006010 tx=0x7f4d40000f40 comp rx=0 tx=0).stop 2024-03-16T10:23:47.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.918+0000 7f4d4d189640 1 -- 172.21.15.77:0/1079620145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d481029c0 msgr2=0x7f4d480706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:47.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.918+0000 7f4d4d189640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d481029c0 0x7f4d480706f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800e9c0 tx=0x7f4d3800edc0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.918+0000 7f4d4d189640 1 -- 172.21.15.77:0/1079620145 shutdown_connections 2024-03-16T10:23:47.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.918+0000 7f4d4d189640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d48105370 0x7f4d481ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.918+0000 7f4d4d189640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d481029c0 0x7f4d480706f0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.918+0000 7f4d4d189640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4d300a2900 0x7f4d300a4dc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.918+0000 7f4d4d189640 1 --2- 172.21.15.77:0/1079620145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d48100050 0x7f4d480701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:47.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.918+0000 7f4d4d189640 1 -- 172.21.15.77:0/1079620145 >> 172.21.15.77:0/1079620145 conn(0x7f4d480fbc70 msgr2=0x7f4d481032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:47.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.918+0000 7f4d4d189640 1 -- 172.21.15.77:0/1079620145 shutdown_connections 2024-03-16T10:23:47.923 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:47.918+0000 7f4d4d189640 1 -- 172.21.15.77:0/1079620145 wait complete. 2024-03-16T10:23:47.933 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-16T10:23:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.040+0000 7ffb6db59640 1 -- 172.21.15.77:0/2539030780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb681006e0 msgr2=0x7ffb68100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.040+0000 7ffb6db59640 1 --2- 172.21.15.77:0/2539030780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb681006e0 0x7ffb68100b00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ffb58009900 tx=0x7ffb5802bc30 comp rx=0 tx=0).stop 2024-03-16T10:23:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.040+0000 7ffb6db59640 1 -- 172.21.15.77:0/2539030780 shutdown_connections 2024-03-16T10:23:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.040+0000 7ffb6db59640 1 --2- 172.21.15.77:0/2539030780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102290 0x7ffb68108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.040+0000 7ffb6db59640 1 --2- 172.21.15.77:0/2539030780 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb681018d0 0x7ffb68101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.040+0000 7ffb6db59640 1 --2- 172.21.15.77:0/2539030780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb681006e0 0x7ffb68100b00 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.040+0000 7ffb6db59640 1 -- 172.21.15.77:0/2539030780 >> 172.21.15.77:0/2539030780 conn(0x7ffb680fbc70 msgr2=0x7ffb680fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.040+0000 7ffb6db59640 1 -- 172.21.15.77:0/2539030780 shutdown_connections 2024-03-16T10:23:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.040+0000 7ffb6db59640 1 -- 172.21.15.77:0/2539030780 wait complete. 2024-03-16T10:23:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.040+0000 7ffb6db59640 1 Processor -- start 2024-03-16T10:23:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb6db59640 1 -- start start 2024-03-16T10:23:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb6db59640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb681006e0 0x7ffb68196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb6db59640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb681018d0 0x7ffb68196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb6db59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102290 0x7ffb6819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb6db59640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6810c350 con 0x7ffb68102290 2024-03-16T10:23:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb6db59640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6810c1d0 con 0x7ffb681006e0 2024-03-16T10:23:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb6db59640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6810c4d0 con 0x7ffb681018d0 2024-03-16T10:23:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102290 0x7ffb6819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb66ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb681018d0 0x7ffb68196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb677fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb681006e0 0x7ffb68196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb677fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb681006e0 0x7ffb68196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44594/0 (socket says 172.21.15.77:44594) 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb66ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb681018d0 0x7ffb68196f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44964/0 (socket says 172.21.15.77:44964) 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb677fe640 1 -- 172.21.15.77:0/2257809749 learned_addr learned my addr 172.21.15.77:0/2257809749 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb67fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102290 0x7ffb6819e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37332/0 (socket says 172.21.15.77:37332) 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb67fff640 1 -- 172.21.15.77:0/2257809749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb681018d0 msgr2=0x7ffb68196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb67fff640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb681018d0 0x7ffb68196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb67fff640 1 -- 172.21.15.77:0/2257809749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb681006e0 msgr2=0x7ffb68196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb67fff640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb681006e0 0x7ffb68196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb67fff640 1 -- 172.21.15.77:0/2257809749 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6819e670 con 0x7ffb68102290 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.041+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb681018d0 0x7ffb68196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.042+0000 7ffb67fff640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102290 0x7ffb6819e000 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009fd0 tx=0x7ffb6000b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.042+0000 7ffb64ff9640 1 -- 172.21.15.77:0/2257809749 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6000be40 con 0x7ffb68102290 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.042+0000 7ffb6db59640 1 -- 172.21.15.77:0/2257809749 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb680689a0 con 0x7ffb68102290 2024-03-16T10:23:48.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.042+0000 7ffb64ff9640 1 -- 172.21.15.77:0/2257809749 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb60017450 con 0x7ffb68102290 2024-03-16T10:23:48.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.042+0000 7ffb64ff9640 1 -- 172.21.15.77:0/2257809749 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60005280 con 0x7ffb68102290 2024-03-16T10:23:48.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.042+0000 7ffb6db59640 1 -- 172.21.15.77:0/2257809749 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb68068ef0 con 0x7ffb68102290 2024-03-16T10:23:48.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.044+0000 7ffb64ff9640 1 -- 172.21.15.77:0/2257809749 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7ffb60004a40 con 0x7ffb68102290 2024-03-16T10:23:48.052 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.044+0000 7ffb70813740 1 Processor -- start 2024-03-16T10:23:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.044+0000 7ffb70813740 1 -- start start 2024-03-16T10:23:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb70813740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6c2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb70813740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb70813740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 0x56549e6ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb70813740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56549e627490 con 0x56549e6c2850 2024-03-16T10:23:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb70813740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56549e5cfc90 con 0x56549e6c3240 2024-03-16T10:23:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb70813740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56549e623530 con 0x56549e6c83b0 2024-03-16T10:23:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb66ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6c2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb66ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6c2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37342/0 (socket says 172.21.15.77:37342) 2024-03-16T10:23:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb66ffd640 1 -- 172.21.15.77:0/2630457635 learned_addr learned my addr 172.21.15.77:0/2630457635 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb677fe640 1 --2- 172.21.15.77:0/2630457635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/2630457635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 0x56549e6ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb64ff9640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ffb500a2800 0x7ffb500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb67fff640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ffb500a2800 0x7ffb500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb66ffd640 1 -- 172.21.15.77:0/2630457635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 msgr2=0x56549e6ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/2630457635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 0x56549e6ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb66ffd640 1 -- 172.21.15.77:0/2630457635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 msgr2=0x56549e6c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/2630457635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb66ffd640 1 -- 172.21.15.77:0/2630457635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56549e68a500 con 0x56549e6c2850 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb64ff9640 1 -- 172.21.15.77:0/2257809749 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7ffb600d0650 con 0x7ffb68102290 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.045+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/2630457635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6c2c70 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c013a10 tx=0x7ffb5c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaab1191df738675 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb4dffb640 1 -- 172.21.15.77:0/2630457635 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c004050 con 0x56549e6c2850 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb4dffb640 1 -- 172.21.15.77:0/2630457635 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb5c00fe60 con 0x56549e6c2850 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb4dffb640 1 -- 172.21.15.77:0/2630457635 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c003b50 con 0x56549e6c2850 2024-03-16T10:23:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb67fff640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ffb500a2800 0x7ffb500a4cc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ffb60013390 tx=0x7ffb600bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb70813740 1 -- 172.21.15.77:0/2630457635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 msgr2=0x56549e6c2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb70813740 1 --2- 172.21.15.77:0/2630457635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6c2c70 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c013a10 tx=0x7ffb5c009b30 comp rx=0 tx=0).stop 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb70813740 1 -- 172.21.15.77:0/2630457635 shutdown_connections 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb70813740 1 --2- 172.21.15.77:0/2630457635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 0x56549e6ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb70813740 1 --2- 172.21.15.77:0/2630457635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb70813740 1 --2- 172.21.15.77:0/2630457635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6c2c70 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb70813740 1 -- 172.21.15.77:0/2630457635 >> 172.21.15.77:0/2630457635 conn(0x56549e6bddc0 msgr2=0x56549e6be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb70813740 1 -- 172.21.15.77:0/2630457635 shutdown_connections 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.046+0000 7ffb70813740 1 -- 172.21.15.77:0/2630457635 wait complete. 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb70813740 1 -- start start 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb70813740 1 Processor -- start 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb70813740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb70813740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb70813740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 0x56549e6d4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb70813740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56549e6cb360 con 0x56549e6c2850 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb70813740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56549e5e3230 con 0x56549e6c3240 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb70813740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56549e6cb4e0 con 0x56549e6c83b0 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb70813740 10 client.0 _ll_register_callbacks cb 0x56549e61f010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb66ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb66ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37354/0 (socket says 172.21.15.77:37354) 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb66ffd640 1 -- 172.21.15.77:0/1128122245 learned_addr learned my addr 172.21.15.77:0/1128122245 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb677fe640 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.047+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 0x56549e6d4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb677fe640 1 -- 172.21.15.77:0/1128122245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 msgr2=0x56549e6d4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb677fe640 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 0x56549e6d4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb677fe640 1 -- 172.21.15.77:0/1128122245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 msgr2=0x56549e6ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb677fe640 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb677fe640 1 -- 172.21.15.77:0/1128122245 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56549e5f2100 con 0x56549e6c3240 2024-03-16T10:23:48.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb677fe640 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6cd2d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffb5802c7b0 tx=0x7ffb58004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75838260c3bd13d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb46ffd640 1 -- 172.21.15.77:0/1128122245 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb58005d50 con 0x56549e6c3240 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb70813740 1 -- 172.21.15.77:0/1128122245 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb5c00fe60 con 0x56549e6c3240 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb46ffd640 1 -- 172.21.15.77:0/1128122245 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7ffb580046c0 con 0x56549e6c3240 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb46ffd640 1 -- 172.21.15.77:0/1128122245 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ffb580048c0 con 0x56549e6c3240 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb70813740 10 client.26876 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb70813740 1 -- 172.21.15.77:0/1128122245 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56549e33b3f0 con 0x56549e6c3240 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.048+0000 7ffb46ffd640 1 -- 172.21.15.77:0/1128122245 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb5c00fe60 con 0x56549e6c3240 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.049+0000 7ffb46ffd640 1 -- 172.21.15.77:0/1128122245 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7ffb58003680 con 0x56549e6c3240 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.049+0000 7ffb70813740 10 client.26876 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.049+0000 7ffb70813740 10 client.26876 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.049+0000 7ffb70813740 20 client.26876 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:48.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.049+0000 7ffb70813740 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x56549e6d8b10 0x56549e6daf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.049+0000 7ffb70813740 4 client.26876 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.049+0000 7ffb70813740 1 -- 172.21.15.77:0/1128122245 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56549e6349d0 con 0x56549e6d8b10 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.049+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x56549e6d8b10 0x56549e6daf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.049+0000 7ffb46ffd640 10 client.26876 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.049+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x56549e6d8b10 0x56549e6daf90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dc5153d44d6b6b68 server_cookie=32eeeaa87b24c5c0 in_seq=0 out_seq=0 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb46ffd640 1 -- 172.21.15.77:0/1128122245 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56549e6349d0 con 0x56549e6d8b10 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb46ffd640 10 client.26876 handle_command_reply: tid=0 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb70813740 1 client.26876 shutdown 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb70813740 1 -- 172.21.15.77:0/1128122245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 msgr2=0x56549e6cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb70813740 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6cd2d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffb5802c7b0 tx=0x7ffb58004270 comp rx=0 tx=0).stop 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb70813740 1 -- 172.21.15.77:0/1128122245 shutdown_connections 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb70813740 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x56549e6d8b10 0x56549e6daf90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb70813740 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 0x56549e6d4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb70813740 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6cd2d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb46ffd640 0 client.26876 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:23:48.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb70813740 1 --2- 172.21.15.77:0/1128122245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb70813740 1 -- 172.21.15.77:0/1128122245 >> 172.21.15.77:0/1128122245 conn(0x56549e6bddc0 msgr2=0x56549e6bcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb70813740 1 -- 172.21.15.77:0/1128122245 shutdown_connections 2024-03-16T10:23:48.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb70813740 1 -- 172.21.15.77:0/1128122245 wait complete. 2024-03-16T10:23:48.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.050+0000 7ffb70813740 20 client.26876 trim_cache size 0 max 16384 2024-03-16T10:23:48.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb70813740 1 Processor -- start 2024-03-16T10:23:48.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb70813740 1 -- start start 2024-03-16T10:23:48.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb70813740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6be1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb70813740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6d42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb70813740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 0x56549e6c4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb70813740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffb681a6980 con 0x56549e6c2850 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb70813740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ffb681a84c0 con 0x56549e6c3240 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb70813740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb681a7050 con 0x56549e6c83b0 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb66ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6be1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb66ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6be1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37368/0 (socket says 172.21.15.77:37368) 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb66ffd640 1 -- 172.21.15.77:0/1927148759 learned_addr learned my addr 172.21.15.77:0/1927148759 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb677fe640 1 --2- 172.21.15.77:0/1927148759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6d42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.064+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/1927148759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 0x56549e6c4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb66ffd640 1 -- 172.21.15.77:0/1927148759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 msgr2=0x56549e6c4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/1927148759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 0x56549e6c4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb66ffd640 1 -- 172.21.15.77:0/1927148759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 msgr2=0x56549e6d42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/1927148759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6d42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb66ffd640 1 -- 172.21.15.77:0/1927148759 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56549e4057d0 con 0x56549e6c2850 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/1927148759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6be1a0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c015010 tx=0x7ffb5c013f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12474fa5c65f4c63 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb4d7fa640 1 -- 172.21.15.77:0/1927148759 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c00fdb0 con 0x56549e6c2850 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb4d7fa640 1 -- 172.21.15.77:0/1927148759 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb5c02f380 con 0x56549e6c2850 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb4d7fa640 1 -- 172.21.15.77:0/1927148759 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c0037e0 con 0x56549e6c2850 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb70813740 1 -- 172.21.15.77:0/1927148759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 msgr2=0x56549e6be1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb70813740 1 --2- 172.21.15.77:0/1927148759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6be1a0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c015010 tx=0x7ffb5c013f90 comp rx=0 tx=0).stop 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb70813740 1 -- 172.21.15.77:0/1927148759 shutdown_connections 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb70813740 1 --2- 172.21.15.77:0/1927148759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c83b0 0x56549e6c4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb70813740 1 --2- 172.21.15.77:0/1927148759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c3240 0x56549e6d42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb70813740 1 --2- 172.21.15.77:0/1927148759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c2850 0x56549e6be1a0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.065+0000 7ffb70813740 1 -- 172.21.15.77:0/1927148759 >> 172.21.15.77:0/1927148759 conn(0x56549e6bddc0 msgr2=0x56549e6c3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.072 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb70813740 1 -- 172.21.15.77:0/1927148759 shutdown_connections 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb70813740 1 -- 172.21.15.77:0/1927148759 wait complete. 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb70813740 1 -- start start 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb70813740 1 Processor -- start 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb70813740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c2850 0x56549e6d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb70813740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c3240 0x56549e6d1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb70813740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c83b0 0x56549e73ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb70813740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffb681a7df0 con 0x56549e6c83b0 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb70813740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ffb681a7720 con 0x56549e6c2850 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb70813740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb681a4960 con 0x56549e6c3240 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb70813740 10 client.0 _ll_register_callbacks cb 0x7ffb58009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb677fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c3240 0x56549e6d1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb66ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c83b0 0x56549e73ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb677fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c3240 0x56549e6d1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:44990/0 (socket says 172.21.15.77:44990) 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb677fe640 1 -- 172.21.15.77:0/871664091 learned_addr learned my addr 172.21.15.77:0/871664091 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.066+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/871664091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c2850 0x56549e6d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb677fe640 1 -- 172.21.15.77:0/871664091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c2850 msgr2=0x56549e6d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb677fe640 1 --2- 172.21.15.77:0/871664091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c2850 0x56549e6d0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb677fe640 1 -- 172.21.15.77:0/871664091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c83b0 msgr2=0x56549e73ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb677fe640 1 --2- 172.21.15.77:0/871664091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c83b0 0x56549e73ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb677fe640 1 -- 172.21.15.77:0/871664091 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56549e6d44d0 con 0x56549e6c3240 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb677fe640 1 --2- 172.21.15.77:0/871664091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c3240 0x56549e6d1400 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ffb58003a50 tx=0x7ffb5802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67a183301ad32e6b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb46ffd640 1 -- 172.21.15.77:0/871664091 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb58058070 con 0x56549e6c3240 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb70813740 1 -- 172.21.15.77:0/871664091 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb5c02f380 con 0x56549e6c3240 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb46ffd640 1 -- 172.21.15.77:0/871664091 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7ffb580035f0 con 0x56549e6c3240 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb46ffd640 1 -- 172.21.15.77:0/871664091 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ffb58009520 con 0x56549e6c3240 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb70813740 10 client.26857 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:48.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb70813740 1 -- 172.21.15.77:0/871664091 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56549e73e330 con 0x56549e6c3240 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.067+0000 7ffb46ffd640 1 -- 172.21.15.77:0/871664091 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb5c02f380 con 0x56549e6c3240 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb46ffd640 1 -- 172.21.15.77:0/871664091 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7ffb58004a60 con 0x56549e6c3240 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb70813740 10 client.26857 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb70813740 10 client.26857 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb70813740 20 client.26857 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb70813740 1 --2- 172.21.15.77:0/871664091 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x56549e73ed50 0x56549e741170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb70813740 4 client.26857 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb70813740 1 -- 172.21.15.77:0/871664091 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56549e62ed20 con 0x56549e73ed50 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/871664091 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x56549e73ed50 0x56549e741170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb46ffd640 10 client.26857 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb66ffd640 1 --2- 172.21.15.77:0/871664091 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x56549e73ed50 0x56549e741170 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b6bf8db2d8e45763 server_cookie=a7d59202cd763591 in_seq=0 out_seq=0 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb46ffd640 1 -- 172.21.15.77:0/871664091 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56549e62ed20 con 0x56549e73ed50 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb46ffd640 10 client.26857 handle_command_reply: tid=0 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.068+0000 7ffb70813740 1 client.26857 shutdown 2024-03-16T10:23:48.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb70813740 1 -- 172.21.15.77:0/871664091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c3240 msgr2=0x56549e6d1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb70813740 1 --2- 172.21.15.77:0/871664091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c3240 0x56549e6d1400 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ffb58003a50 tx=0x7ffb5802bc30 comp rx=0 tx=0).stop 2024-03-16T10:23:48.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb70813740 1 -- 172.21.15.77:0/871664091 shutdown_connections 2024-03-16T10:23:48.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb70813740 1 --2- 172.21.15.77:0/871664091 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x56549e73ed50 0x56549e741170 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb70813740 1 --2- 172.21.15.77:0/871664091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56549e6c83b0 0x56549e73ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb46ffd640 0 client.26857 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:23:48.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb70813740 1 --2- 172.21.15.77:0/871664091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56549e6c3240 0x56549e6d1400 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb70813740 1 --2- 172.21.15.77:0/871664091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56549e6c2850 0x56549e6d0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb70813740 1 -- 172.21.15.77:0/871664091 >> 172.21.15.77:0/871664091 conn(0x56549e6bddc0 msgr2=0x56549e6cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb70813740 1 -- 172.21.15.77:0/871664091 shutdown_connections 2024-03-16T10:23:48.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb70813740 1 -- 172.21.15.77:0/871664091 wait complete. 2024-03-16T10:23:48.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb70813740 20 client.26857 trim_cache size 0 max 16384 2024-03-16T10:23:48.077 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:23:48.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb6db59640 1 -- 172.21.15.77:0/2257809749 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ffb500a2800 msgr2=0x7ffb500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb6db59640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ffb500a2800 0x7ffb500a4cc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ffb60013390 tx=0x7ffb600bd040 comp rx=0 tx=0).stop 2024-03-16T10:23:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb6db59640 1 -- 172.21.15.77:0/2257809749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102290 msgr2=0x7ffb6819e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb6db59640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102290 0x7ffb6819e000 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009fd0 tx=0x7ffb6000b690 comp rx=0 tx=0).stop 2024-03-16T10:23:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb6db59640 1 -- 172.21.15.77:0/2257809749 shutdown_connections 2024-03-16T10:23:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb6db59640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102290 0x7ffb6819e000 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb6db59640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7ffb500a2800 0x7ffb500a4cc0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb6db59640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb681018d0 0x7ffb68196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb6db59640 1 --2- 172.21.15.77:0/2257809749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb681006e0 0x7ffb68196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb6db59640 1 -- 172.21.15.77:0/2257809749 >> 172.21.15.77:0/2257809749 conn(0x7ffb680fbc70 msgr2=0x7ffb68104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb6db59640 1 -- 172.21.15.77:0/2257809749 shutdown_connections 2024-03-16T10:23:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.069+0000 7ffb6db59640 1 -- 172.21.15.77:0/2257809749 wait complete. 2024-03-16T10:23:48.081 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T10:23:48.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.190+0000 7f6b3a990640 1 -- 172.21.15.77:0/910084246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b341006e0 msgr2=0x7f6b34100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.190+0000 7f6b3a990640 1 --2- 172.21.15.77:0/910084246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b341006e0 0x7f6b34100b00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009900 tx=0x7f6b2802bc30 comp rx=0 tx=0).stop 2024-03-16T10:23:48.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.190+0000 7f6b3a990640 1 -- 172.21.15.77:0/910084246 shutdown_connections 2024-03-16T10:23:48.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.190+0000 7f6b3a990640 1 --2- 172.21.15.77:0/910084246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b34102290 0x7f6b34108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.190+0000 7f6b3a990640 1 --2- 172.21.15.77:0/910084246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b341018d0 0x7f6b34101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.190+0000 7f6b3a990640 1 --2- 172.21.15.77:0/910084246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b341006e0 0x7f6b34100b00 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.190+0000 7f6b3a990640 1 -- 172.21.15.77:0/910084246 >> 172.21.15.77:0/910084246 conn(0x7f6b340fbc70 msgr2=0x7f6b340fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.190+0000 7f6b3a990640 1 -- 172.21.15.77:0/910084246 shutdown_connections 2024-03-16T10:23:48.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.190+0000 7f6b3a990640 1 -- 172.21.15.77:0/910084246 wait complete. 2024-03-16T10:23:48.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.190+0000 7f6b3a990640 1 Processor -- start 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.190+0000 7f6b3a990640 1 -- start start 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b3a990640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b341006e0 0x7f6b340694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b3a990640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b341018d0 0x7f6b3419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b3a990640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b34102290 0x7f6b3419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b3a990640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3410c350 con 0x7f6b341018d0 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b3a990640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3410c1d0 con 0x7f6b341006e0 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b3a990640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3410c4d0 con 0x7f6b34102290 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b341018d0 0x7f6b3419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b341006e0 0x7f6b340694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b341018d0 0x7f6b3419c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37386/0 (socket says 172.21.15.77:37386) 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b337fe640 1 -- 172.21.15.77:0/3509453483 learned_addr learned my addr 172.21.15.77:0/3509453483 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b38f2d640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b34102290 0x7f6b3419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b337fe640 1 -- 172.21.15.77:0/3509453483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b34102290 msgr2=0x7f6b3419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b337fe640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b34102290 0x7f6b3419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b337fe640 1 -- 172.21.15.77:0/3509453483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b341006e0 msgr2=0x7f6b340694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b337fe640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b341006e0 0x7f6b340694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b337fe640 1 -- 172.21.15.77:0/3509453483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3419f800 con 0x7f6b341018d0 2024-03-16T10:23:48.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.191+0000 7f6b337fe640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b341018d0 0x7f6b3419c8d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400e9f0 tx=0x7f6b2400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.192+0000 7f6b317fa640 1 -- 172.21.15.77:0/3509453483 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2400cd80 con 0x7f6b341018d0 2024-03-16T10:23:48.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.192+0000 7f6b317fa640 1 -- 172.21.15.77:0/3509453483 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b24005500 con 0x7f6b341018d0 2024-03-16T10:23:48.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.192+0000 7f6b3a990640 1 -- 172.21.15.77:0/3509453483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b3419fa90 con 0x7f6b341018d0 2024-03-16T10:23:48.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.192+0000 7f6b317fa640 1 -- 172.21.15.77:0/3509453483 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b24010690 con 0x7f6b341018d0 2024-03-16T10:23:48.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.192+0000 7f6b3a990640 1 -- 172.21.15.77:0/3509453483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b3419ff90 con 0x7f6b341018d0 2024-03-16T10:23:48.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.194+0000 7f6b317fa640 1 -- 172.21.15.77:0/3509453483 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f6b240046c0 con 0x7f6b341018d0 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.194+0000 7f6b3d57d740 1 Processor -- start 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.194+0000 7f6b3d57d740 1 -- start start 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.194+0000 7f6b3d57d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 0x56230c7eec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.194+0000 7f6b3d57d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7ef240 0x56230c7f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.194+0000 7f6b3d57d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7f43b0 0x56230c7f6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.194+0000 7f6b3d57d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56230c753490 con 0x56230c7ee850 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.194+0000 7f6b3d57d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56230c6fbc90 con 0x56230c7ef240 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.194+0000 7f6b3d57d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56230c74f530 con 0x56230c7f43b0 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b38f2d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 0x56230c7eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7ef240 0x56230c7f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b337fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7f43b0 0x56230c7f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7ef240 0x56230c7f3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44646/0 (socket says 172.21.15.77:44646) 2024-03-16T10:23:48.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b33fff640 1 -- 172.21.15.77:0/1355947646 learned_addr learned my addr 172.21.15.77:0/1355947646 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b38f2d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 0x56230c7eec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37390/0 (socket says 172.21.15.77:37390) 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b317fa640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b1c0a2800 0x7f6b1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b38f2d640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b1c0a2800 0x7f6b1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b33fff640 1 -- 172.21.15.77:0/1355947646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7f43b0 msgr2=0x56230c7f6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b33fff640 1 --2- 172.21.15.77:0/1355947646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7f43b0 0x56230c7f6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b33fff640 1 -- 172.21.15.77:0/1355947646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 msgr2=0x56230c7eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b33fff640 1 --2- 172.21.15.77:0/1355947646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 0x56230c7eec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b33fff640 1 -- 172.21.15.77:0/1355947646 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56230c7b6500 con 0x56230c7ef240 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b317fa640 1 -- 172.21.15.77:0/3509453483 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6b240cedc0 con 0x7f6b341018d0 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b33fff640 1 --2- 172.21.15.77:0/1355947646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7ef240 0x56230c7f3d40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009900 tx=0x7f6b280095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=192fb4d93155eedc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b38f2d640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b1c0a2800 0x7f6b1c0a4cc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x56230c7efaa0 tx=0x7f6b2c004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b1a7fc640 1 -- 172.21.15.77:0/1355947646 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2802cb80 con 0x56230c7ef240 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b1a7fc640 1 -- 172.21.15.77:0/1355947646 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b28005040 con 0x56230c7ef240 2024-03-16T10:23:48.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.195+0000 7f6b1a7fc640 1 -- 172.21.15.77:0/1355947646 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b28004920 con 0x56230c7ef240 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.196+0000 7f6b3d57d740 1 -- 172.21.15.77:0/1355947646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7ef240 msgr2=0x56230c7f3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.196+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/1355947646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7ef240 0x56230c7f3d40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009900 tx=0x7f6b280095e0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.196+0000 7f6b3d57d740 1 -- 172.21.15.77:0/1355947646 shutdown_connections 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.196+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/1355947646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7f43b0 0x56230c7f6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.196+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/1355947646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7ef240 0x56230c7f3d40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.196+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/1355947646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 0x56230c7eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.196+0000 7f6b3d57d740 1 -- 172.21.15.77:0/1355947646 >> 172.21.15.77:0/1355947646 conn(0x56230c7e9dc0 msgr2=0x56230c7ea1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.196+0000 7f6b3d57d740 1 -- 172.21.15.77:0/1355947646 shutdown_connections 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.196+0000 7f6b3d57d740 1 -- 172.21.15.77:0/1355947646 wait complete. 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.196+0000 7f6b3d57d740 1 -- start start 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b3d57d740 1 Processor -- start 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b3d57d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 0x56230c7f8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b3d57d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ef240 0x56230c7f92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b3d57d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7f43b0 0x56230c8002a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b3d57d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56230c7f7300 con 0x56230c7ee850 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b3d57d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56230c70f230 con 0x56230c7f43b0 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b3d57d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56230c7f7480 con 0x56230c7ef240 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b3d57d740 10 client.0 _ll_register_callbacks cb 0x56230c74b010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 0x56230c7f8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 0x56230c7f8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37400/0 (socket says 172.21.15.77:37400) 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b38f2d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7f43b0 0x56230c8002a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b337fe640 1 --2- 172.21.15.77:0/862848043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ef240 0x56230c7f92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b33fff640 1 -- 172.21.15.77:0/862848043 learned_addr learned my addr 172.21.15.77:0/862848043 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b33fff640 1 -- 172.21.15.77:0/862848043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ef240 msgr2=0x56230c7f92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b33fff640 1 --2- 172.21.15.77:0/862848043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ef240 0x56230c7f92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b33fff640 1 -- 172.21.15.77:0/862848043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7f43b0 msgr2=0x56230c8002a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b33fff640 1 --2- 172.21.15.77:0/862848043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7f43b0 0x56230c8002a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b33fff640 1 -- 172.21.15.77:0/862848043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56230c71e100 con 0x56230c7ee850 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b33fff640 1 --2- 172.21.15.77:0/862848043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 0x56230c7f8d60 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6b2802cfd0 tx=0x7f6b280054a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becc3919a843a06b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.197+0000 7f6b137fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b137fe640 1 -- 172.21.15.77:0/862848043 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b28004ad0 con 0x56230c7ee850 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b137fe640 1 -- 172.21.15.77:0/862848043 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6b28004c70 con 0x56230c7ee850 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b3d57d740 1 -- 172.21.15.77:0/862848043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b28005040 con 0x56230c7ee850 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b137fe640 1 -- 172.21.15.77:0/862848043 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6b28056050 con 0x56230c7ee850 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b3d57d740 10 client.17115 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b3d57d740 1 -- 172.21.15.77:0/862848043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56230c4673f0 con 0x56230c7ee850 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b137fe640 1 -- 172.21.15.77:0/862848043 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b28005040 con 0x56230c7ee850 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b137fe640 1 -- 172.21.15.77:0/862848043 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f6b2804d070 con 0x56230c7ee850 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b3d57d740 10 client.17115 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b3d57d740 10 client.17115 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b3d57d740 20 client.17115 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/862848043 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x56230c8047b0 0x56230c806ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b3d57d740 4 client.17115 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b3d57d740 1 -- 172.21.15.77:0/862848043 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56230c7609d0 con 0x56230c8047b0 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.198+0000 7f6b337fe640 1 --2- 172.21.15.77:0/862848043 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x56230c8047b0 0x56230c806ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.199+0000 7f6b137fe640 10 client.17115 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:23:48.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.199+0000 7f6b337fe640 1 --2- 172.21.15.77:0/862848043 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x56230c8047b0 0x56230c806ba0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5a661b1ee1661749 server_cookie=b24224f3600a0ced in_seq=0 out_seq=0 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b137fe640 1 -- 172.21.15.77:0/862848043 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56230c7609d0 con 0x56230c8047b0 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b137fe640 10 client.17115 handle_command_reply: tid=0 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b3d57d740 1 client.17115 shutdown 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b3d57d740 1 -- 172.21.15.77:0/862848043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 msgr2=0x56230c7f8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/862848043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 0x56230c7f8d60 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6b2802cfd0 tx=0x7f6b280054a0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b3d57d740 1 -- 172.21.15.77:0/862848043 shutdown_connections 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/862848043 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x56230c8047b0 0x56230c806ba0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/862848043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7f43b0 0x56230c8002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b137fe640 0 client.17115 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/862848043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ef240 0x56230c7f92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/862848043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ee850 0x56230c7f8d60 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b3d57d740 1 -- 172.21.15.77:0/862848043 >> 172.21.15.77:0/862848043 conn(0x56230c7e9dc0 msgr2=0x56230c7e8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b3d57d740 1 -- 172.21.15.77:0/862848043 shutdown_connections 2024-03-16T10:23:48.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.201+0000 7f6b3d57d740 1 -- 172.21.15.77:0/862848043 wait complete. 2024-03-16T10:23:48.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.202+0000 7f6b3d57d740 20 client.17115 trim_cache size 0 max 16384 2024-03-16T10:23:48.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b3d57d740 1 Processor -- start 2024-03-16T10:23:48.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b3d57d740 1 -- start start 2024-03-16T10:23:48.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b3d57d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ee850 0x56230c7ed730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b3d57d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7ef240 0x56230c800260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b3d57d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7f43b0 0x56230c80b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b3d57d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b341a8ad0 con 0x56230c7f43b0 2024-03-16T10:23:48.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b3d57d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6b341aa610 con 0x56230c7ef240 2024-03-16T10:23:48.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b3d57d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6b341a91a0 con 0x56230c7ee850 2024-03-16T10:23:48.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b38f2d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7f43b0 0x56230c80b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b38f2d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7f43b0 0x56230c80b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37408/0 (socket says 172.21.15.77:37408) 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b38f2d640 1 -- 172.21.15.77:0/4204546165 learned_addr learned my addr 172.21.15.77:0/4204546165 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b33fff640 1 --2- 172.21.15.77:0/4204546165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ee850 0x56230c7ed730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b337fe640 1 --2- 172.21.15.77:0/4204546165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7ef240 0x56230c800260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b38f2d640 1 -- 172.21.15.77:0/4204546165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ee850 msgr2=0x56230c7ed730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.215+0000 7f6b38f2d640 1 --2- 172.21.15.77:0/4204546165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ee850 0x56230c7ed730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b38f2d640 1 -- 172.21.15.77:0/4204546165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7ef240 msgr2=0x56230c800260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b38f2d640 1 --2- 172.21.15.77:0/4204546165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7ef240 0x56230c800260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b38f2d640 1 -- 172.21.15.77:0/4204546165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56230c5317d0 con 0x56230c7f43b0 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b38f2d640 1 --2- 172.21.15.77:0/4204546165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7f43b0 0x56230c80b360 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c012610 tx=0x7f6b2c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af40113a716efad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b19ffb640 1 -- 172.21.15.77:0/4204546165 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c0235c0 con 0x56230c7f43b0 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b19ffb640 1 -- 172.21.15.77:0/4204546165 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b2c028020 con 0x56230c7f43b0 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b19ffb640 1 -- 172.21.15.77:0/4204546165 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c0109c0 con 0x56230c7f43b0 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b3d57d740 1 -- 172.21.15.77:0/4204546165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7f43b0 msgr2=0x56230c80b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.221 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/4204546165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7f43b0 0x56230c80b360 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c012610 tx=0x7f6b2c009920 comp rx=0 tx=0).stop 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b3d57d740 1 -- 172.21.15.77:0/4204546165 shutdown_connections 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/4204546165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7f43b0 0x56230c80b360 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/4204546165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7ef240 0x56230c800260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/4204546165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ee850 0x56230c7ed730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b3d57d740 1 -- 172.21.15.77:0/4204546165 >> 172.21.15.77:0/4204546165 conn(0x56230c7e9dc0 msgr2=0x56230c7ed020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b3d57d740 1 -- 172.21.15.77:0/4204546165 shutdown_connections 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.216+0000 7f6b3d57d740 1 -- 172.21.15.77:0/4204546165 wait complete. 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b3d57d740 1 -- start start 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b3d57d740 1 Processor -- start 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b3d57d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ee850 0x56230c7fd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b3d57d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ef240 0x56230c7fdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b3d57d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7f43b0 0x56230c7fece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b3d57d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b341a9f40 con 0x56230c7ef240 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b3d57d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6b341a9870 con 0x56230c7f43b0 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b3d57d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6b341a6b20 con 0x56230c7ee850 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b3d57d740 10 client.0 _ll_register_callbacks cb 0x7f6b2c009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ef240 0x56230c7fdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ef240 0x56230c7fdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37420/0 (socket says 172.21.15.77:37420) 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b337fe640 1 -- 172.21.15.77:0/761576095 learned_addr learned my addr 172.21.15.77:0/761576095 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b33fff640 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ee850 0x56230c7fd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b38f2d640 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7f43b0 0x56230c7fece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b337fe640 1 -- 172.21.15.77:0/761576095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ee850 msgr2=0x56230c7fd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.217+0000 7f6b337fe640 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ee850 0x56230c7fd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b337fe640 1 -- 172.21.15.77:0/761576095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7f43b0 msgr2=0x56230c7fece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b337fe640 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7f43b0 0x56230c7fece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b337fe640 1 -- 172.21.15.77:0/761576095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56230c7ff2d0 con 0x56230c7ef240 2024-03-16T10:23:48.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b337fe640 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ef240 0x56230c7fdcd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6b240cc950 tx=0x7f6b240cc820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6f82103dccd7b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b137fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b137fe640 1 -- 172.21.15.77:0/761576095 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2409b150 con 0x56230c7ef240 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b137fe640 1 -- 172.21.15.77:0/761576095 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6b24099af0 con 0x56230c7ef240 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b3d57d740 1 -- 172.21.15.77:0/761576095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b2c028020 con 0x56230c7ef240 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b137fe640 1 -- 172.21.15.77:0/761576095 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6b2409bef0 con 0x56230c7ef240 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b3d57d740 10 client.17121 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b3d57d740 1 -- 172.21.15.77:0/761576095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56230c86a9a0 con 0x56230c7ef240 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b137fe640 1 -- 172.21.15.77:0/761576095 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b2c028020 con 0x56230c7ef240 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b137fe640 1 -- 172.21.15.77:0/761576095 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f6b240bb040 con 0x56230c7ef240 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b3d57d740 10 client.17121 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b3d57d740 10 client.17121 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b3d57d740 20 client.17121 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x56230c8047b0 0x56230c86d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b3d57d740 4 client.17121 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:23:48.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.218+0000 7f6b3d57d740 1 -- 172.21.15.77:0/761576095 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56230c75ad20 con 0x56230c8047b0 2024-03-16T10:23:48.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.219+0000 7f6b33fff640 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x56230c8047b0 0x56230c86d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.219+0000 7f6b137fe640 10 client.17121 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:23:48.225 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.219+0000 7f6b33fff640 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x56230c8047b0 0x56230c86d2a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fb05295ab3ab58ed server_cookie=e8586cd86947073 in_seq=0 out_seq=0 2024-03-16T10:23:48.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.221+0000 7f6b137fe640 1 -- 172.21.15.77:0/761576095 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56230c75ad20 con 0x56230c8047b0 2024-03-16T10:23:48.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.221+0000 7f6b137fe640 10 client.17121 handle_command_reply: tid=0 2024-03-16T10:23:48.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.221+0000 7f6b3d57d740 1 client.17121 shutdown 2024-03-16T10:23:48.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.221+0000 7f6b3d57d740 1 -- 172.21.15.77:0/761576095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ef240 msgr2=0x56230c7fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.221+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ef240 0x56230c7fdcd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6b240cc950 tx=0x7f6b240cc820 comp rx=0 tx=0).stop 2024-03-16T10:23:48.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.221+0000 7f6b3d57d740 1 -- 172.21.15.77:0/761576095 shutdown_connections 2024-03-16T10:23:48.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.221+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x56230c8047b0 0x56230c86d2a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.221+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56230c7f43b0 0x56230c7fece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.221+0000 7f6b137fe640 0 client.17121 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:23:48.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.221+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56230c7ef240 0x56230c7fdcd0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.221+0000 7f6b3d57d740 1 --2- 172.21.15.77:0/761576095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56230c7ee850 0x56230c7fd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.221+0000 7f6b3d57d740 1 -- 172.21.15.77:0/761576095 >> 172.21.15.77:0/761576095 conn(0x56230c7e9dc0 msgr2=0x56230c7ed8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3d57d740 1 -- 172.21.15.77:0/761576095 shutdown_connections 2024-03-16T10:23:48.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3d57d740 1 -- 172.21.15.77:0/761576095 wait complete. 2024-03-16T10:23:48.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3d57d740 20 client.17121 trim_cache size 0 max 16384 2024-03-16T10:23:48.227 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:23:48.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3a990640 1 -- 172.21.15.77:0/3509453483 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b1c0a2800 msgr2=0x7f6b1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3a990640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b1c0a2800 0x7f6b1c0a4cc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x56230c7efaa0 tx=0x7f6b2c004ab0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3a990640 1 -- 172.21.15.77:0/3509453483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b341018d0 msgr2=0x7f6b3419c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3a990640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b341018d0 0x7f6b3419c8d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400e9f0 tx=0x7f6b2400edf0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3a990640 1 -- 172.21.15.77:0/3509453483 shutdown_connections 2024-03-16T10:23:48.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3a990640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6b1c0a2800 0x7f6b1c0a4cc0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3a990640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b34102290 0x7f6b3419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3a990640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b341018d0 0x7f6b3419c8d0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.228 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3a990640 1 --2- 172.21.15.77:0/3509453483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b341006e0 0x7f6b340694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3a990640 1 -- 172.21.15.77:0/3509453483 >> 172.21.15.77:0/3509453483 conn(0x7f6b340fbc70 msgr2=0x7f6b34104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3a990640 1 -- 172.21.15.77:0/3509453483 shutdown_connections 2024-03-16T10:23:48.229 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.222+0000 7f6b3a990640 1 -- 172.21.15.77:0/3509453483 wait complete. 2024-03-16T10:23:48.234 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:48 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1378675605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:23:48.235 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T10:23:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:48 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1378675605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:23:48.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.344+0000 7f0639d6f640 1 -- 172.21.15.77:0/2002489967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0634100700 msgr2=0x7f0634100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.344+0000 7f0639d6f640 1 --2- 172.21.15.77:0/2002489967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0634100700 0x7f0634100b20 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0624009900 tx=0x7f062402bc30 comp rx=0 tx=0).stop 2024-03-16T10:23:48.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.344+0000 7f0639d6f640 1 -- 172.21.15.77:0/2002489967 shutdown_connections 2024-03-16T10:23:48.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.344+0000 7f0639d6f640 1 --2- 172.21.15.77:0/2002489967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06341022b0 0x7f0634108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.344+0000 7f0639d6f640 1 --2- 172.21.15.77:0/2002489967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06341018f0 0x7f0634101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.344+0000 7f0639d6f640 1 --2- 172.21.15.77:0/2002489967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0634100700 0x7f0634100b20 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.344+0000 7f0639d6f640 1 -- 172.21.15.77:0/2002489967 >> 172.21.15.77:0/2002489967 conn(0x7f06340fbc90 msgr2=0x7f06340fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.344+0000 7f0639d6f640 1 -- 172.21.15.77:0/2002489967 shutdown_connections 2024-03-16T10:23:48.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.344+0000 7f0639d6f640 1 -- 172.21.15.77:0/2002489967 wait complete. 2024-03-16T10:23:48.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f0639d6f640 1 Processor -- start 2024-03-16T10:23:48.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f0639d6f640 1 -- start start 2024-03-16T10:23:48.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f0639d6f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0634100700 0x7f0634197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f0639d6f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06341018f0 0x7f0634198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f0639d6f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06341022b0 0x7f063419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f0639d6f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f063410c380 con 0x7f0634100700 2024-03-16T10:23:48.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f0639d6f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f063410c200 con 0x7f06341022b0 2024-03-16T10:23:48.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f0639d6f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f063410c500 con 0x7f06341018f0 2024-03-16T10:23:48.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f06337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0634100700 0x7f0634197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f06337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0634100700 0x7f0634197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37440/0 (socket says 172.21.15.77:37440) 2024-03-16T10:23:48.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f06337fe640 1 -- 172.21.15.77:0/421219930 learned_addr learned my addr 172.21.15.77:0/421219930 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f0633fff640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06341022b0 0x7f063419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.345+0000 7f0632ffd640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06341018f0 0x7f0634198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.346+0000 7f06337fe640 1 -- 172.21.15.77:0/421219930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06341018f0 msgr2=0x7f0634198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.346+0000 7f06337fe640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06341018f0 0x7f0634198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.346+0000 7f06337fe640 1 -- 172.21.15.77:0/421219930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06341022b0 msgr2=0x7f063419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.346+0000 7f06337fe640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06341022b0 0x7f063419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.346+0000 7f06337fe640 1 -- 172.21.15.77:0/421219930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f063419f700 con 0x7f0634100700 2024-03-16T10:23:48.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.346+0000 7f06337fe640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0634100700 0x7f0634197ae0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0624000c00 tx=0x7f062402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.346+0000 7f0630ff9640 1 -- 172.21.15.77:0/421219930 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624041070 con 0x7f0634100700 2024-03-16T10:23:48.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.346+0000 7f0639d6f640 1 -- 172.21.15.77:0/421219930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f063419f990 con 0x7f0634100700 2024-03-16T10:23:48.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.346+0000 7f0630ff9640 1 -- 172.21.15.77:0/421219930 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0624004450 con 0x7f0634100700 2024-03-16T10:23:48.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.346+0000 7f0630ff9640 1 -- 172.21.15.77:0/421219930 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624034440 con 0x7f0634100700 2024-03-16T10:23:48.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.346+0000 7f0639d6f640 1 -- 172.21.15.77:0/421219930 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f063419fed0 con 0x7f0634100700 2024-03-16T10:23:48.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.348+0000 7f0630ff9640 1 -- 172.21.15.77:0/421219930 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f0624044070 con 0x7f0634100700 2024-03-16T10:23:48.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f063ca31740 1 Processor -- start 2024-03-16T10:23:48.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f063ca31740 1 -- start start 2024-03-16T10:23:48.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f063ca31740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a895850 0x55bf0a895c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f063ca31740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 0x55bf0a89ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f063ca31740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a89b3b0 0x55bf0a89d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f063ca31740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bf0a7fa490 con 0x55bf0a896240 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f063ca31740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55bf0a7a2c90 con 0x55bf0a89b3b0 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f063ca31740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bf0a7f6530 con 0x55bf0a895850 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f06337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 0x55bf0a89ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f0633fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a895850 0x55bf0a895c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f06337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 0x55bf0a89ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37442/0 (socket says 172.21.15.77:37442) 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f06337fe640 1 -- 172.21.15.77:0/4069387140 learned_addr learned my addr 172.21.15.77:0/4069387140 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f0632ffd640 1 --2- 172.21.15.77:0/4069387140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a89b3b0 0x55bf0a89d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f0630ff9640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f061c0a2800 0x7f061c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f0633fff640 1 -- 172.21.15.77:0/4069387140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a89b3b0 msgr2=0x55bf0a89d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f0633fff640 1 --2- 172.21.15.77:0/4069387140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a89b3b0 0x55bf0a89d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f0633fff640 1 -- 172.21.15.77:0/4069387140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 msgr2=0x55bf0a89ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f0633fff640 1 --2- 172.21.15.77:0/4069387140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 0x55bf0a89ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f0633fff640 1 -- 172.21.15.77:0/4069387140 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf0a85d500 con 0x55bf0a895850 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f0630ff9640 1 -- 172.21.15.77:0/421219930 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f06240f0490 con 0x7f0634100700 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f0633fff640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f061c0a2800 0x7f061c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.349+0000 7f0633fff640 1 --2- 172.21.15.77:0/4069387140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a895850 0x55bf0a895c70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f062c009fd0 tx=0x7f062c011920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc9ae78549441256 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.350+0000 7f0619ffb640 1 -- 172.21.15.77:0/4069387140 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c017070 con 0x55bf0a895850 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.350+0000 7f0633fff640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f061c0a2800 0x7f061c0a4cc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f062c022000 tx=0x7f062c023000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.350+0000 7f0619ffb640 1 -- 172.21.15.77:0/4069387140 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f062c00fce0 con 0x55bf0a895850 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.350+0000 7f0619ffb640 1 -- 172.21.15.77:0/4069387140 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c00eac0 con 0x55bf0a895850 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.350+0000 7f063ca31740 1 -- 172.21.15.77:0/4069387140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a895850 msgr2=0x55bf0a895c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.350+0000 7f063ca31740 1 --2- 172.21.15.77:0/4069387140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a895850 0x55bf0a895c70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f062c009fd0 tx=0x7f062c011920 comp rx=0 tx=0).stop 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.350+0000 7f063ca31740 1 -- 172.21.15.77:0/4069387140 shutdown_connections 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.350+0000 7f063ca31740 1 --2- 172.21.15.77:0/4069387140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a89b3b0 0x55bf0a89d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.350+0000 7f063ca31740 1 --2- 172.21.15.77:0/4069387140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 0x55bf0a89ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.350+0000 7f063ca31740 1 --2- 172.21.15.77:0/4069387140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a895850 0x55bf0a895c70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.350+0000 7f063ca31740 1 -- 172.21.15.77:0/4069387140 >> 172.21.15.77:0/4069387140 conn(0x55bf0a890dc0 msgr2=0x55bf0a8911c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.350+0000 7f063ca31740 1 -- 172.21.15.77:0/4069387140 shutdown_connections 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f063ca31740 1 -- 172.21.15.77:0/4069387140 wait complete. 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f063ca31740 1 -- start start 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f063ca31740 1 Processor -- start 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f063ca31740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a895850 0x55bf0a89fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f063ca31740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a896240 0x55bf0a8a0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f063ca31740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a89b3b0 0x55bf0a8a7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f063ca31740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bf0a89e360 con 0x55bf0a895850 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f063ca31740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55bf0a7b6230 con 0x55bf0a896240 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f063ca31740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bf0a89e4e0 con 0x55bf0a89b3b0 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f063ca31740 10 client.0 _ll_register_callbacks cb 0x55bf0a7f2010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f06337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a895850 0x55bf0a89fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f06337fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a895850 0x55bf0a89fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37446/0 (socket says 172.21.15.77:37446) 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f06337fe640 1 -- 172.21.15.77:0/3842296697 learned_addr learned my addr 172.21.15.77:0/3842296697 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f0632ffd640 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a896240 0x55bf0a8a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.351+0000 7f0633fff640 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a89b3b0 0x55bf0a8a7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f06337fe640 1 -- 172.21.15.77:0/3842296697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a89b3b0 msgr2=0x55bf0a8a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f06337fe640 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a89b3b0 0x55bf0a8a7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f06337fe640 1 -- 172.21.15.77:0/3842296697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a896240 msgr2=0x55bf0a8a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f06337fe640 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a896240 0x55bf0a8a0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f06337fe640 1 -- 172.21.15.77:0/3842296697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf0a7c5100 con 0x55bf0a895850 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f06337fe640 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a895850 0x55bf0a89fde0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0624005ab0 tx=0x7f0624005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c009d3788abcaf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f0612ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f0612ffd640 1 -- 172.21.15.77:0/3842296697 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062404b040 con 0x55bf0a895850 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f0612ffd640 1 -- 172.21.15.77:0/3842296697 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f06240f0080 con 0x55bf0a895850 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f063ca31740 1 -- 172.21.15.77:0/3842296697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f062c00fce0 con 0x55bf0a895850 2024-03-16T10:23:48.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f0612ffd640 1 -- 172.21.15.77:0/3842296697 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f06240bcdd0 con 0x55bf0a895850 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f063ca31740 10 client.17133 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f063ca31740 1 -- 172.21.15.77:0/3842296697 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55bf0a50e3f0 con 0x55bf0a895850 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f0612ffd640 1 -- 172.21.15.77:0/3842296697 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f062c00fce0 con 0x55bf0a895850 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f0612ffd640 1 -- 172.21.15.77:0/3842296697 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f06240d3050 con 0x55bf0a895850 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f063ca31740 10 client.17133 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.352+0000 7f063ca31740 10 client.17133 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.353+0000 7f063ca31740 20 client.17133 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.353+0000 7f063ca31740 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55bf0a8abae0 0x55bf0a8adf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.353+0000 7f063ca31740 4 client.17133 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.353+0000 7f063ca31740 1 -- 172.21.15.77:0/3842296697 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf0a8079d0 con 0x55bf0a8abae0 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.353+0000 7f0632ffd640 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55bf0a8abae0 0x55bf0a8adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.353+0000 7f0632ffd640 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55bf0a8abae0 0x55bf0a8adf80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=980c06011a8ea6cb server_cookie=8a2200b65433afd7 in_seq=0 out_seq=0 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.353+0000 7f0612ffd640 10 client.17133 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f0612ffd640 1 -- 172.21.15.77:0/3842296697 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bf0a8079d0 con 0x55bf0a8abae0 2024-03-16T10:23:48.363 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f0612ffd640 10 client.17133 handle_command_reply: tid=0 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f063ca31740 1 client.17133 shutdown 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f063ca31740 1 -- 172.21.15.77:0/3842296697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a895850 msgr2=0x55bf0a89fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f063ca31740 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a895850 0x55bf0a89fde0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0624005ab0 tx=0x7f0624005bf0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f063ca31740 1 -- 172.21.15.77:0/3842296697 shutdown_connections 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f063ca31740 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55bf0a8abae0 0x55bf0a8adf80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f063ca31740 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a89b3b0 0x55bf0a8a7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f0612ffd640 0 client.17133 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f063ca31740 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a896240 0x55bf0a8a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f063ca31740 1 --2- 172.21.15.77:0/3842296697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a895850 0x55bf0a89fde0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f063ca31740 1 -- 172.21.15.77:0/3842296697 >> 172.21.15.77:0/3842296697 conn(0x55bf0a890dc0 msgr2=0x55bf0a88ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f063ca31740 1 -- 172.21.15.77:0/3842296697 shutdown_connections 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f063ca31740 1 -- 172.21.15.77:0/3842296697 wait complete. 2024-03-16T10:23:48.364 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.354+0000 7f063ca31740 20 client.17133 trim_cache size 0 max 16384 2024-03-16T10:23:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f063ca31740 1 Processor -- start 2024-03-16T10:23:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f063ca31740 1 -- start start 2024-03-16T10:23:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f063ca31740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a895850 0x55bf0a896f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f063ca31740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 0x55bf0a8a72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f063ca31740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a89b3b0 0x55bf0a894730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f063ca31740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f06341a8bc0 con 0x55bf0a896240 2024-03-16T10:23:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f063ca31740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f06341aa700 con 0x55bf0a89b3b0 2024-03-16T10:23:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f063ca31740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f06341a9290 con 0x55bf0a895850 2024-03-16T10:23:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f0632ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 0x55bf0a8a72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f06337fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a895850 0x55bf0a896f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f0632ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 0x55bf0a8a72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37458/0 (socket says 172.21.15.77:37458) 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f0632ffd640 1 -- 172.21.15.77:0/2945310606 learned_addr learned my addr 172.21.15.77:0/2945310606 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f0633fff640 1 --2- 172.21.15.77:0/2945310606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a89b3b0 0x55bf0a894730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f0632ffd640 1 -- 172.21.15.77:0/2945310606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a895850 msgr2=0x55bf0a896f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f0632ffd640 1 --2- 172.21.15.77:0/2945310606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a895850 0x55bf0a896f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f0632ffd640 1 -- 172.21.15.77:0/2945310606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a89b3b0 msgr2=0x55bf0a894730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f0632ffd640 1 --2- 172.21.15.77:0/2945310606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a89b3b0 0x55bf0a894730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f0632ffd640 1 -- 172.21.15.77:0/2945310606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf0a5d87d0 con 0x55bf0a896240 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f0633fff640 1 --2- 172.21.15.77:0/2945310606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a89b3b0 0x55bf0a894730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f0632ffd640 1 --2- 172.21.15.77:0/2945310606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 0x55bf0a8a72e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f0628009dd0 tx=0x7f0628005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac35e371daf9d2ca server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.368+0000 7f06197fa640 1 -- 172.21.15.77:0/2945310606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0628024070 con 0x55bf0a896240 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.369+0000 7f06197fa640 1 -- 172.21.15.77:0/2945310606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0628003e40 con 0x55bf0a896240 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.369+0000 7f06197fa640 1 -- 172.21.15.77:0/2945310606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0628010b40 con 0x55bf0a896240 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.369+0000 7f063ca31740 1 -- 172.21.15.77:0/2945310606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 msgr2=0x55bf0a8a72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.369+0000 7f063ca31740 1 --2- 172.21.15.77:0/2945310606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 0x55bf0a8a72e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f0628009dd0 tx=0x7f0628005dc0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.369+0000 7f063ca31740 1 -- 172.21.15.77:0/2945310606 shutdown_connections 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.369+0000 7f063ca31740 1 --2- 172.21.15.77:0/2945310606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a89b3b0 0x55bf0a894730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.369+0000 7f063ca31740 1 --2- 172.21.15.77:0/2945310606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a896240 0x55bf0a8a72e0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.369+0000 7f063ca31740 1 --2- 172.21.15.77:0/2945310606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a895850 0x55bf0a896f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.369+0000 7f063ca31740 1 -- 172.21.15.77:0/2945310606 >> 172.21.15.77:0/2945310606 conn(0x55bf0a890dc0 msgr2=0x55bf0a894020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.369+0000 7f063ca31740 1 -- 172.21.15.77:0/2945310606 shutdown_connections 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.369+0000 7f063ca31740 1 -- 172.21.15.77:0/2945310606 wait complete. 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.369+0000 7f063ca31740 1 -- start start 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f063ca31740 1 Processor -- start 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f063ca31740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a895850 0x55bf0a8a4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f063ca31740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a896240 0x55bf0a8a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f063ca31740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a89b3b0 0x55bf0a8a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f063ca31740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f06341aa030 con 0x55bf0a89b3b0 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f063ca31740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f06341a9960 con 0x55bf0a895850 2024-03-16T10:23:48.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f063ca31740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f06341a6ba0 con 0x55bf0a896240 2024-03-16T10:23:48.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f063ca31740 10 client.0 _ll_register_callbacks cb 0x7f0624009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:48.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f0633fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a89b3b0 0x55bf0a8a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f0633fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a89b3b0 0x55bf0a8a5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37460/0 (socket says 172.21.15.77:37460) 2024-03-16T10:23:48.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f0633fff640 1 -- 172.21.15.77:0/261652224 learned_addr learned my addr 172.21.15.77:0/261652224 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f06337fe640 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a895850 0x55bf0a8a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f0632ffd640 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a896240 0x55bf0a8a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f0633fff640 1 -- 172.21.15.77:0/261652224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a896240 msgr2=0x55bf0a8a4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f0633fff640 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a896240 0x55bf0a8a4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f0633fff640 1 -- 172.21.15.77:0/261652224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a895850 msgr2=0x55bf0a8a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f0633fff640 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a895850 0x55bf0a8a4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.370+0000 7f0633fff640 1 -- 172.21.15.77:0/261652224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf0a8a6350 con 0x55bf0a89b3b0 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f0633fff640 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a89b3b0 0x55bf0a8a5d60 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x55bf0a896aa0 tx=0x7f062c00fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73961d48ba6290e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f0612ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f0612ffd640 1 -- 172.21.15.77:0/261652224 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c017040 con 0x55bf0a89b3b0 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f0612ffd640 1 -- 172.21.15.77:0/261652224 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f062c00f540 con 0x55bf0a89b3b0 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f063ca31740 1 -- 172.21.15.77:0/261652224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bf0a801d20 con 0x55bf0a89b3b0 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f0612ffd640 1 -- 172.21.15.77:0/261652224 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f062c00fe30 con 0x55bf0a89b3b0 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f063ca31740 10 client.17139 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f063ca31740 1 -- 172.21.15.77:0/261652224 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55bf0a8a76f0 con 0x55bf0a89b3b0 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f0612ffd640 1 -- 172.21.15.77:0/261652224 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bf0a801d20 con 0x55bf0a89b3b0 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f0612ffd640 1 -- 172.21.15.77:0/261652224 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f062c028430 con 0x55bf0a89b3b0 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f063ca31740 10 client.17139 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f063ca31740 10 client.17139 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:23:48.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f063ca31740 20 client.17139 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f063ca31740 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55bf0a912640 0x55bf0a914a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f063ca31740 4 client.17139 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.371+0000 7f063ca31740 1 -- 172.21.15.77:0/261652224 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bf0a77f6a0 con 0x55bf0a912640 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f06337fe640 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55bf0a912640 0x55bf0a914a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f0612ffd640 10 client.17139 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f06337fe640 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55bf0a912640 0x55bf0a914a60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9b7b5d1e1251f159 server_cookie=2c4976ece518583c in_seq=0 out_seq=0 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f0612ffd640 1 -- 172.21.15.77:0/261652224 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bf0a77f6a0 con 0x55bf0a912640 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f0612ffd640 10 client.17139 handle_command_reply: tid=0 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f063ca31740 1 client.17139 shutdown 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f063ca31740 1 -- 172.21.15.77:0/261652224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a89b3b0 msgr2=0x55bf0a8a5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f063ca31740 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a89b3b0 0x55bf0a8a5d60 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x55bf0a896aa0 tx=0x7f062c00fa30 comp rx=0 tx=0).stop 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f063ca31740 1 -- 172.21.15.77:0/261652224 shutdown_connections 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f063ca31740 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55bf0a912640 0x55bf0a914a60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f063ca31740 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bf0a89b3b0 0x55bf0a8a5d60 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f063ca31740 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf0a896240 0x55bf0a8a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f063ca31740 1 --2- 172.21.15.77:0/261652224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bf0a895850 0x55bf0a8a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f0612ffd640 0 client.17139 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:23:48.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f063ca31740 1 -- 172.21.15.77:0/261652224 >> 172.21.15.77:0/261652224 conn(0x55bf0a890dc0 msgr2=0x55bf0a8948b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f063ca31740 1 -- 172.21.15.77:0/261652224 shutdown_connections 2024-03-16T10:23:48.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f063ca31740 1 -- 172.21.15.77:0/261652224 wait complete. 2024-03-16T10:23:48.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.372+0000 7f063ca31740 20 client.17139 trim_cache size 0 max 16384 2024-03-16T10:23:48.380 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:23:48.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.373+0000 7f0639d6f640 1 -- 172.21.15.77:0/421219930 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f061c0a2800 msgr2=0x7f061c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.373+0000 7f0639d6f640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f061c0a2800 0x7f061c0a4cc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f062c022000 tx=0x7f062c023000 comp rx=0 tx=0).stop 2024-03-16T10:23:48.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.373+0000 7f0639d6f640 1 -- 172.21.15.77:0/421219930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0634100700 msgr2=0x7f0634197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.373+0000 7f0639d6f640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0634100700 0x7f0634197ae0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0624000c00 tx=0x7f062402c640 comp rx=0 tx=0).stop 2024-03-16T10:23:48.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.373+0000 7f0639d6f640 1 -- 172.21.15.77:0/421219930 shutdown_connections 2024-03-16T10:23:48.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.373+0000 7f0639d6f640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06341022b0 0x7f063419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.373+0000 7f0639d6f640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f061c0a2800 0x7f061c0a4cc0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.373+0000 7f0639d6f640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06341018f0 0x7f0634198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.373+0000 7f0639d6f640 1 --2- 172.21.15.77:0/421219930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0634100700 0x7f0634197ae0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.373+0000 7f0639d6f640 1 -- 172.21.15.77:0/421219930 >> 172.21.15.77:0/421219930 conn(0x7f06340fbc90 msgr2=0x7f06340fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.373+0000 7f0639d6f640 1 -- 172.21.15.77:0/421219930 shutdown_connections 2024-03-16T10:23:48.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.373+0000 7f0639d6f640 1 -- 172.21.15.77:0/421219930 wait complete. 2024-03-16T10:23:48.384 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T10:23:48.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:48 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1378675605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:23:48.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.497+0000 7f509deff640 1 -- 172.21.15.77:0/832227296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50981017f0 msgr2=0x7f5098101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.497+0000 7f509deff640 1 --2- 172.21.15.77:0/832227296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50981017f0 0x7f5098101c10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5088009900 tx=0x7f508802bc30 comp rx=0 tx=0).stop 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.497+0000 7f509deff640 1 -- 172.21.15.77:0/832227296 shutdown_connections 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.497+0000 7f509deff640 1 --2- 172.21.15.77:0/832227296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50981033a0 0x7f5098109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.497+0000 7f509deff640 1 --2- 172.21.15.77:0/832227296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50981029e0 0x7f5098102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.497+0000 7f509deff640 1 --2- 172.21.15.77:0/832227296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50981017f0 0x7f5098101c10 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.497+0000 7f509deff640 1 -- 172.21.15.77:0/832227296 >> 172.21.15.77:0/832227296 conn(0x7f50980fcd60 msgr2=0x7f50980ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.497+0000 7f509deff640 1 -- 172.21.15.77:0/832227296 shutdown_connections 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.497+0000 7f509deff640 1 -- 172.21.15.77:0/832227296 wait complete. 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.497+0000 7f509deff640 1 Processor -- start 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f509deff640 1 -- start start 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f509deff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50981017f0 0x7f5098197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f509deff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50981029e0 0x7f5098198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f509deff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50981033a0 0x7f509819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f509deff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f509810d470 con 0x7f50981029e0 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f509deff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f509810d2f0 con 0x7f50981017f0 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f509deff640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f509810d5f0 con 0x7f50981033a0 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5096ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50981029e0 0x7f5098198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5096ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50981029e0 0x7f5098198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37474/0 (socket says 172.21.15.77:37474) 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5096ffd640 1 -- 172.21.15.77:0/3435865077 learned_addr learned my addr 172.21.15.77:0/3435865077 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f50977fe640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50981017f0 0x7f5098197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5097fff640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50981033a0 0x7f509819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5096ffd640 1 -- 172.21.15.77:0/3435865077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50981033a0 msgr2=0x7f509819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50981033a0 0x7f509819f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5096ffd640 1 -- 172.21.15.77:0/3435865077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50981017f0 msgr2=0x7f5098197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50981017f0 0x7f5098197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5096ffd640 1 -- 172.21.15.77:0/3435865077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509819f760 con 0x7f50981029e0 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50981029e0 0x7f5098198080 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f508c00e9f0 tx=0x7f508c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5094ff9640 1 -- 172.21.15.77:0/3435865077 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508c00cd80 con 0x7f50981029e0 2024-03-16T10:23:48.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f509deff640 1 -- 172.21.15.77:0/3435865077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f509819fa50 con 0x7f50981029e0 2024-03-16T10:23:48.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5094ff9640 1 -- 172.21.15.77:0/3435865077 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f508c010910 con 0x7f50981029e0 2024-03-16T10:23:48.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.498+0000 7f5094ff9640 1 -- 172.21.15.77:0/3435865077 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508c0188f0 con 0x7f50981029e0 2024-03-16T10:23:48.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.499+0000 7f509deff640 1 -- 172.21.15.77:0/3435865077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f509819ffa0 con 0x7f50981029e0 2024-03-16T10:23:48.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.501+0000 7f5094ff9640 1 -- 172.21.15.77:0/3435865077 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f508c010ac0 con 0x7f50981029e0 2024-03-16T10:23:48.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.501+0000 7f5094ff9640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50800a2800 0x7f50800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.501+0000 7f5094ff9640 1 -- 172.21.15.77:0/3435865077 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f508c0ced80 con 0x7f50981029e0 2024-03-16T10:23:48.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.501+0000 7f50977fe640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50800a2800 0x7f50800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f50a0bc9740 1 Processor -- start 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f50a0bc9740 1 -- start start 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f50977fe640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50800a2800 0x7f50800a4cc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5088000c00 tx=0x7f5088005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f50a0bc9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e9850 0x564d436e9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f50a0bc9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ea240 0x564d436eed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f50a0bc9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ef3b0 0x564d436f1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f50a0bc9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564d4364e490 con 0x564d436ef3b0 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f50a0bc9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564d435f6c90 con 0x564d436e9850 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f50a0bc9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564d4364a530 con 0x564d436ea240 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f5097fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ef3b0 0x564d436f1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f50977fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e9850 0x564d436e9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f50977fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e9850 0x564d436e9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44746/0 (socket says 172.21.15.77:44746) 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f5097fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ef3b0 0x564d436f1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37490/0 (socket says 172.21.15.77:37490) 2024-03-16T10:23:48.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f50977fe640 1 -- 172.21.15.77:0/3822087524 learned_addr learned my addr 172.21.15.77:0/3822087524 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3822087524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ea240 0x564d436eed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f5097fff640 1 -- 172.21.15.77:0/3822087524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ea240 msgr2=0x564d436eed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f5097fff640 1 --2- 172.21.15.77:0/3822087524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ea240 0x564d436eed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f5097fff640 1 -- 172.21.15.77:0/3822087524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e9850 msgr2=0x564d436e9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.502+0000 7f5097fff640 1 --2- 172.21.15.77:0/3822087524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e9850 0x564d436e9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.503+0000 7f5097fff640 1 -- 172.21.15.77:0/3822087524 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d436b1500 con 0x564d436ef3b0 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.503+0000 7f50977fe640 1 --2- 172.21.15.77:0/3822087524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e9850 0x564d436e9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.503+0000 7f5097fff640 1 --2- 172.21.15.77:0/3822087524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ef3b0 0x564d436f1840 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5090009f00 tx=0x7f5090011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8e5fd0a6316e73 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.503+0000 7f507dffb640 1 -- 172.21.15.77:0/3822087524 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5090004d20 con 0x564d436ef3b0 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.503+0000 7f507dffb640 1 -- 172.21.15.77:0/3822087524 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5090004030 con 0x564d436ef3b0 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.503+0000 7f507dffb640 1 -- 172.21.15.77:0/3822087524 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5090005220 con 0x564d436ef3b0 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.503+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3822087524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ef3b0 msgr2=0x564d436f1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.503+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3822087524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ef3b0 0x564d436f1840 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5090009f00 tx=0x7f5090011810 comp rx=0 tx=0).stop 2024-03-16T10:23:48.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3822087524 shutdown_connections 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3822087524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ef3b0 0x564d436f1840 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3822087524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ea240 0x564d436eed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3822087524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e9850 0x564d436e9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3822087524 >> 172.21.15.77:0/3822087524 conn(0x564d436e4dc0 msgr2=0x564d436e51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3822087524 shutdown_connections 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3822087524 wait complete. 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 -- start start 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 Processor -- start 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e9850 0x564d436f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ea240 0x564d436f4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ef3b0 0x564d436fb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564d436f2360 con 0x564d436ea240 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564d4360a230 con 0x564d436e9850 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564d436f24e0 con 0x564d436ef3b0 2024-03-16T10:23:48.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50a0bc9740 10 client.0 _ll_register_callbacks cb 0x564d43646010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f5096ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ea240 0x564d436f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f5096ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ea240 0x564d436f4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37494/0 (socket says 172.21.15.77:37494) 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f5096ffd640 1 -- 172.21.15.77:0/3673692369 learned_addr learned my addr 172.21.15.77:0/3673692369 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.504+0000 7f50977fe640 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e9850 0x564d436f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f5097fff640 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ef3b0 0x564d436fb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f5096ffd640 1 -- 172.21.15.77:0/3673692369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ef3b0 msgr2=0x564d436fb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ef3b0 0x564d436fb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f5096ffd640 1 -- 172.21.15.77:0/3673692369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e9850 msgr2=0x564d436f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e9850 0x564d436f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f5096ffd640 1 -- 172.21.15.77:0/3673692369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d43619100 con 0x564d436ea240 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ea240 0x564d436f4290 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f508c098230 tx=0x7f508c09b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef73a81d66ad45d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f5076ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:23:48.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f5076ffd640 1 -- 172.21.15.77:0/3673692369 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508c0b4060 con 0x564d436ea240 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f5076ffd640 1 -- 172.21.15.77:0/3673692369 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f508c0afb90 con 0x564d436ea240 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3673692369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5090004030 con 0x564d436ea240 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f5076ffd640 1 -- 172.21.15.77:0/3673692369 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f508c0862b0 con 0x564d436ea240 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f50a0bc9740 10 client.17151 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3673692369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564d433623f0 con 0x564d436ea240 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.505+0000 7f5076ffd640 1 -- 172.21.15.77:0/3673692369 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5090004030 con 0x564d436ea240 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.506+0000 7f5076ffd640 1 -- 172.21.15.77:0/3673692369 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f508c09cae0 con 0x564d436ea240 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.506+0000 7f50a0bc9740 10 client.17151 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.506+0000 7f50a0bc9740 10 client.17151 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.506+0000 7f50a0bc9740 20 client.17151 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.506+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d436ffac0 0x564d43701f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.506+0000 7f50a0bc9740 4 client.17151 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.506+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3673692369 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d4365b9d0 con 0x564d436ffac0 2024-03-16T10:23:48.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.506+0000 7f50977fe640 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d436ffac0 0x564d43701f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.506+0000 7f5076ffd640 10 client.17151 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.506+0000 7f50977fe640 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d436ffac0 0x564d43701f30 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96b4181ab8672eef server_cookie=fd6dede782881182 in_seq=0 out_seq=0 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f5076ffd640 1 -- 172.21.15.77:0/3673692369 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564d4365b9d0 con 0x564d436ffac0 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f5076ffd640 10 client.17151 handle_command_reply: tid=0 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f50a0bc9740 1 client.17151 shutdown 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3673692369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ea240 msgr2=0x564d436f4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ea240 0x564d436f4290 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f508c098230 tx=0x7f508c09b9a0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3673692369 shutdown_connections 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ef3b0 0x564d436fb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d436ffac0 0x564d43701f30 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436ea240 0x564d436f4290 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3673692369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e9850 0x564d436f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f5076ffd640 0 client.17151 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:48.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3673692369 >> 172.21.15.77:0/3673692369 conn(0x564d436e4dc0 msgr2=0x564d436e3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3673692369 shutdown_connections 2024-03-16T10:23:48.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3673692369 wait complete. 2024-03-16T10:23:48.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.507+0000 7f50a0bc9740 20 client.17151 trim_cache size 0 max 16384 2024-03-16T10:23:48.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f50a0bc9740 1 Processor -- start 2024-03-16T10:23:48.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f50a0bc9740 1 -- start start 2024-03-16T10:23:48.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f50a0bc9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e4dc0 0x564d436eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f50a0bc9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 0x564d436e3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f50a0bc9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ea240 0x564d436e3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f50a0bc9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f50981a8c50 con 0x564d436e9850 2024-03-16T10:23:48.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f50a0bc9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f50981aa790 con 0x564d436e4dc0 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f50a0bc9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50981a9320 con 0x564d436ea240 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f5096ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 0x564d436e3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f5096ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 0x564d436e3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37510/0 (socket says 172.21.15.77:37510) 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f5096ffd640 1 -- 172.21.15.77:0/3843136878 learned_addr learned my addr 172.21.15.77:0/3843136878 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f50977fe640 1 --2- 172.21.15.77:0/3843136878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e4dc0 0x564d436eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.521+0000 7f5097fff640 1 --2- 172.21.15.77:0/3843136878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ea240 0x564d436e3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f5096ffd640 1 -- 172.21.15.77:0/3843136878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ea240 msgr2=0x564d436e3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3843136878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ea240 0x564d436e3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f5096ffd640 1 -- 172.21.15.77:0/3843136878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e4dc0 msgr2=0x564d436eaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3843136878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e4dc0 0x564d436eaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f5096ffd640 1 -- 172.21.15.77:0/3843136878 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d4342c7d0 con 0x564d436e9850 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3843136878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 0x564d436e3750 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f508c0a3a20 tx=0x7f508c099d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4977d99a9100481e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f507d7fa640 1 -- 172.21.15.77:0/3843136878 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508c097890 con 0x564d436e9850 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f507d7fa640 1 -- 172.21.15.77:0/3843136878 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f508c09cae0 con 0x564d436e9850 2024-03-16T10:23:48.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f507d7fa640 1 -- 172.21.15.77:0/3843136878 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508c09af10 con 0x564d436e9850 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3843136878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 msgr2=0x564d436e3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3843136878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 0x564d436e3750 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f508c0a3a20 tx=0x7f508c099d70 comp rx=0 tx=0).stop 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3843136878 shutdown_connections 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3843136878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436ea240 0x564d436e3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3843136878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 0x564d436e3750 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3843136878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436e4dc0 0x564d436eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.522+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3843136878 >> 172.21.15.77:0/3843136878 conn(0x564d436f2f60 msgr2=0x564d436f3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.523+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3843136878 shutdown_connections 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.523+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3843136878 wait complete. 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.523+0000 7f50a0bc9740 1 -- start start 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.523+0000 7f50a0bc9740 1 Processor -- start 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.523+0000 7f50a0bc9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436e4dc0 0x564d437610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.523+0000 7f50a0bc9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 0x564d43761630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.523+0000 7f50a0bc9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436ea240 0x564d43768670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.523+0000 7f50a0bc9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f50981aa0c0 con 0x564d436e9850 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.523+0000 7f50a0bc9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f50981a99f0 con 0x564d436ea240 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.523+0000 7f50a0bc9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50981a6c50 con 0x564d436e4dc0 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.523+0000 7f50a0bc9740 10 client.0 _ll_register_callbacks cb 0x7f508c00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f50977fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436e4dc0 0x564d437610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f5096ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 0x564d43761630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f50977fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436e4dc0 0x564d437610f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:45140/0 (socket says 172.21.15.77:45140) 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f5096ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 0x564d43761630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37518/0 (socket says 172.21.15.77:37518) 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f50977fe640 1 -- 172.21.15.77:0/3778536473 learned_addr learned my addr 172.21.15.77:0/3778536473 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f5097fff640 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436ea240 0x564d43768670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f50977fe640 1 -- 172.21.15.77:0/3778536473 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436ea240 msgr2=0x564d43768670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f50977fe640 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436ea240 0x564d43768670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f50977fe640 1 -- 172.21.15.77:0/3778536473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 msgr2=0x564d43761630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f50977fe640 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 0x564d43761630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f50977fe640 1 -- 172.21.15.77:0/3778536473 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d43768c60 con 0x564d436e4dc0 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 0x564d43761630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f50977fe640 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436e4dc0 0x564d437610f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5088036040 tx=0x7f508802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4be2bb38f5599dea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f5076ffd640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f5076ffd640 1 -- 172.21.15.77:0/3778536473 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088056070 con 0x564d436e4dc0 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3778536473 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f508c09cae0 con 0x564d436e4dc0 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f5076ffd640 1 -- 172.21.15.77:0/3778536473 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f5088052290 con 0x564d436e4dc0 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f5076ffd640 1 -- 172.21.15.77:0/3778536473 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f5088052490 con 0x564d436e4dc0 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f50a0bc9740 10 client.26911 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.524+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3778536473 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564d43769fd0 con 0x564d436e4dc0 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.525+0000 7f5076ffd640 1 -- 172.21.15.77:0/3778536473 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f508c09cae0 con 0x564d436e4dc0 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.525+0000 7f5076ffd640 1 -- 172.21.15.77:0/3778536473 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f50880048f0 con 0x564d436e4dc0 2024-03-16T10:23:48.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.525+0000 7f50a0bc9740 10 client.26911 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.525+0000 7f50a0bc9740 10 client.26911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.525+0000 7f50a0bc9740 20 client.26911 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.525+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d436ef3b0 0x564d4376ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.525+0000 7f50a0bc9740 4 client.26911 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.525+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3778536473 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564d43655d20 con 0x564d436ef3b0 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.525+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d436ef3b0 0x564d4376ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.525+0000 7f5076ffd640 10 client.26911 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.525+0000 7f5096ffd640 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d436ef3b0 0x564d4376ca10 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcb26e09d9e5bba3 server_cookie=61f305b8ac2ebd8c in_seq=0 out_seq=0 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f5076ffd640 1 -- 172.21.15.77:0/3778536473 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564d43655d20 con 0x564d436ef3b0 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f5076ffd640 10 client.26911 handle_command_reply: tid=0 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f50a0bc9740 1 client.26911 shutdown 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3778536473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436e4dc0 msgr2=0x564d437610f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436e4dc0 0x564d437610f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5088036040 tx=0x7f508802bc30 comp rx=0 tx=0).stop 2024-03-16T10:23:48.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3778536473 shutdown_connections 2024-03-16T10:23:48.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d436ea240 0x564d43768670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564d436ef3b0 0x564d4376ca10 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564d436e9850 0x564d43761630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f5076ffd640 0 client.26911 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:48.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f50a0bc9740 1 --2- 172.21.15.77:0/3778536473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d436e4dc0 0x564d437610f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3778536473 >> 172.21.15.77:0/3778536473 conn(0x564d436f2f60 msgr2=0x564d436e8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3778536473 shutdown_connections 2024-03-16T10:23:48.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f50a0bc9740 1 -- 172.21.15.77:0/3778536473 wait complete. 2024-03-16T10:23:48.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f50a0bc9740 20 client.26911 trim_cache size 0 max 16384 2024-03-16T10:23:48.533 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:23:48.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f509deff640 1 -- 172.21.15.77:0/3435865077 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50800a2800 msgr2=0x7f50800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f509deff640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50800a2800 0x7f50800a4cc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5088000c00 tx=0x7f5088005fd0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f509deff640 1 -- 172.21.15.77:0/3435865077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50981029e0 msgr2=0x7f5098198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:48.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f509deff640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50981029e0 0x7f5098198080 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f508c00e9f0 tx=0x7f508c00eeb0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f509deff640 1 -- 172.21.15.77:0/3435865077 shutdown_connections 2024-03-16T10:23:48.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f509deff640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50800a2800 0x7f50800a4cc0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f509deff640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50981033a0 0x7f509819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.526+0000 7f509deff640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50981029e0 0x7f5098198080 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.527+0000 7f509deff640 1 --2- 172.21.15.77:0/3435865077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50981017f0 0x7f5098197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:48.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.527+0000 7f509deff640 1 -- 172.21.15.77:0/3435865077 >> 172.21.15.77:0/3435865077 conn(0x7f50980fcd60 msgr2=0x7f5098105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:48.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.527+0000 7f509deff640 1 -- 172.21.15.77:0/3435865077 shutdown_connections 2024-03-16T10:23:48.535 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:48.527+0000 7f509deff640 1 -- 172.21.15.77:0/3435865077 wait complete. 2024-03-16T10:23:49.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:49 smithi192 ceph-mon[36354]: pgmap v553: 97 pgs: 97 active+clean; 776 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 504 op/s 2024-03-16T10:23:49.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:49 smithi077 ceph-mon[32400]: pgmap v553: 97 pgs: 97 active+clean; 776 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 504 op/s 2024-03-16T10:23:49.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:49 smithi193 ceph-mon[36311]: pgmap v553: 97 pgs: 97 active+clean; 776 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 504 op/s 2024-03-16T10:23:49.539 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T10:23:49.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:23:49.539 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T10:23:49.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.675+0000 7f9975589640 1 -- 172.21.15.77:0/3307367384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99701087e0 msgr2=0x7f997010ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.675+0000 7f9975589640 1 --2- 172.21.15.77:0/3307367384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99701087e0 0x7f997010ac10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9964009900 tx=0x7f996402bc60 comp rx=0 tx=0).stop 2024-03-16T10:23:49.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.677+0000 7f9975589640 1 -- 172.21.15.77:0/3307367384 shutdown_connections 2024-03-16T10:23:49.679 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.677+0000 7f9975589640 1 --2- 172.21.15.77:0/3307367384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f997010db00 0x7f997010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.677+0000 7f9975589640 1 --2- 172.21.15.77:0/3307367384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f997010b150 0x7f997010d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.677+0000 7f9975589640 1 --2- 172.21.15.77:0/3307367384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99701087e0 0x7f997010ac10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.677+0000 7f9975589640 1 -- 172.21.15.77:0/3307367384 >> 172.21.15.77:0/3307367384 conn(0x7f997006e640 msgr2=0x7f9970070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:49.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.680+0000 7f9975589640 1 -- 172.21.15.77:0/3307367384 shutdown_connections 2024-03-16T10:23:49.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.681+0000 7f9975589640 1 -- 172.21.15.77:0/3307367384 wait complete. 2024-03-16T10:23:49.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.682+0000 7f9975589640 1 Processor -- start 2024-03-16T10:23:49.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.683+0000 7f9975589640 1 -- start start 2024-03-16T10:23:49.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.683+0000 7f9975589640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99701087e0 0x7f99701a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.683+0000 7f9975589640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f997010b150 0x7f99701a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.683+0000 7f9975589640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f997010db00 0x7f99701adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.683+0000 7f9975589640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f997011c300 con 0x7f997010db00 2024-03-16T10:23:49.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.683+0000 7f9975589640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f997011c180 con 0x7f99701087e0 2024-03-16T10:23:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.683+0000 7f9975589640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f997011c480 con 0x7f997010b150 2024-03-16T10:23:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.684+0000 7f996f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f997010db00 0x7f99701adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.684+0000 7f996f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f997010db00 0x7f99701adef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37532/0 (socket says 172.21.15.77:37532) 2024-03-16T10:23:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.684+0000 7f996f7fe640 1 -- 172.21.15.77:0/40061634 learned_addr learned my addr 172.21.15.77:0/40061634 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.684+0000 7f996effd640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99701087e0 0x7f99701a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.684+0000 7f996f7fe640 1 -- 172.21.15.77:0/40061634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f997010b150 msgr2=0x7f99701a6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.684+0000 7f996f7fe640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f997010b150 0x7f99701a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.684+0000 7f996f7fe640 1 -- 172.21.15.77:0/40061634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99701087e0 msgr2=0x7f99701a6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.684+0000 7f996f7fe640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99701087e0 0x7f99701a6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.684+0000 7f996f7fe640 1 -- 172.21.15.77:0/40061634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99701ae560 con 0x7f997010db00 2024-03-16T10:23:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.684+0000 7f996effd640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99701087e0 0x7f99701a6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:23:49.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.685+0000 7f996f7fe640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f997010db00 0x7f99701adef0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f996800b9e0 tx=0x7f996800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:49.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.686+0000 7f995ffff640 1 -- 172.21.15.77:0/40061634 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9968019070 con 0x7f997010db00 2024-03-16T10:23:49.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.686+0000 7f995ffff640 1 -- 172.21.15.77:0/40061634 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9968014440 con 0x7f997010db00 2024-03-16T10:23:49.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.686+0000 7f995ffff640 1 -- 172.21.15.77:0/40061634 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99680125e0 con 0x7f997010db00 2024-03-16T10:23:49.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.686+0000 7f9975589640 1 -- 172.21.15.77:0/40061634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99701ae850 con 0x7f997010db00 2024-03-16T10:23:49.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.686+0000 7f9975589640 1 -- 172.21.15.77:0/40061634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99701aed50 con 0x7f997010db00 2024-03-16T10:23:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.689+0000 7f995ffff640 1 -- 172.21.15.77:0/40061634 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f996801c070 con 0x7f997010db00 2024-03-16T10:23:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.690+0000 7f997817b740 1 Processor -- start 2024-03-16T10:23:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.690+0000 7f997817b740 1 -- start start 2024-03-16T10:23:49.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.690+0000 7f995ffff640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f99580a28c0 0x7f99580a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.690+0000 7f997817b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc4f990 0x56443dc4fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.690+0000 7f997817b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc50380 0x56443dc54e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.690+0000 7f997817b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc554f0 0x56443dc57980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.690+0000 7f997817b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56443dbb40d0 con 0x56443dc4f990 2024-03-16T10:23:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.690+0000 7f997817b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56443db5cc70 con 0x56443dc554f0 2024-03-16T10:23:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.690+0000 7f997817b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56443dbb0170 con 0x56443dc50380 2024-03-16T10:23:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.690+0000 7f995ffff640 1 -- 172.21.15.77:0/40061634 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f99680cdfd0 con 0x7f997010db00 2024-03-16T10:23:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.690+0000 7f996e7fc640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f99580a28c0 0x7f99580a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.691+0000 7f996f7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc554f0 0x56443dc57980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.691+0000 7f996e7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc50380 0x56443dc54e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.691+0000 7f996f7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc554f0 0x56443dc57980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44802/0 (socket says 172.21.15.77:44802) 2024-03-16T10:23:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.691+0000 7f996f7fe640 1 -- 172.21.15.77:0/1050065692 learned_addr learned my addr 172.21.15.77:0/1050065692 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:49.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.691+0000 7f996effd640 1 --2- 172.21.15.77:0/1050065692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc4f990 0x56443dc4fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.691+0000 7f996f7fe640 1 -- 172.21.15.77:0/1050065692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc50380 msgr2=0x56443dc54e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.691+0000 7f996f7fe640 1 --2- 172.21.15.77:0/1050065692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc50380 0x56443dc54e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.691+0000 7f996f7fe640 1 -- 172.21.15.77:0/1050065692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc4f990 msgr2=0x56443dc4fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.691+0000 7f996f7fe640 1 --2- 172.21.15.77:0/1050065692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc4f990 0x56443dc4fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.691+0000 7f996f7fe640 1 -- 172.21.15.77:0/1050065692 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56443d9907e0 con 0x56443dc554f0 2024-03-16T10:23:49.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.692+0000 7f996e7fc640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f99580a28c0 0x7f99580a4d80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f996000afd0 tx=0x7f996000aa80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:49.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.692+0000 7f996f7fe640 1 --2- 172.21.15.77:0/1050065692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc554f0 0x56443dc57980 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f99680a2c70 tx=0x7f9968004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d05603123c727f64 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:49.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.692+0000 7f995d7fa640 1 -- 172.21.15.77:0/1050065692 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99680a31a0 con 0x56443dc554f0 2024-03-16T10:23:49.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.692+0000 7f995d7fa640 1 -- 172.21.15.77:0/1050065692 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9968014960 con 0x56443dc554f0 2024-03-16T10:23:49.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.692+0000 7f995d7fa640 1 -- 172.21.15.77:0/1050065692 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99680a2f70 con 0x56443dc554f0 2024-03-16T10:23:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.694+0000 7f997817b740 1 -- 172.21.15.77:0/1050065692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc554f0 msgr2=0x56443dc57980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.694+0000 7f997817b740 1 --2- 172.21.15.77:0/1050065692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc554f0 0x56443dc57980 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f99680a2c70 tx=0x7f9968004b50 comp rx=0 tx=0).stop 2024-03-16T10:23:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.694+0000 7f997817b740 1 -- 172.21.15.77:0/1050065692 shutdown_connections 2024-03-16T10:23:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.694+0000 7f997817b740 1 --2- 172.21.15.77:0/1050065692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc554f0 0x56443dc57980 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.694+0000 7f997817b740 1 --2- 172.21.15.77:0/1050065692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc50380 0x56443dc54e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.694+0000 7f997817b740 1 --2- 172.21.15.77:0/1050065692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc4f990 0x56443dc4fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.694+0000 7f997817b740 1 -- 172.21.15.77:0/1050065692 >> 172.21.15.77:0/1050065692 conn(0x56443dc17c50 msgr2=0x56443dc4b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.694+0000 7f997817b740 1 -- 172.21.15.77:0/1050065692 shutdown_connections 2024-03-16T10:23:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.694+0000 7f997817b740 1 -- 172.21.15.77:0/1050065692 wait complete. 2024-03-16T10:23:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.695+0000 7f997817b740 1 -- start start 2024-03-16T10:23:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.695+0000 7f997817b740 1 Processor -- start 2024-03-16T10:23:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.695+0000 7f997817b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc4f990 0x56443dc59cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.695+0000 7f997817b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc50380 0x56443dc5a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.695+0000 7f997817b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 0x56443dc61270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.695+0000 7f997817b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56443dc584a0 con 0x56443dc554f0 2024-03-16T10:23:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.695+0000 7f997817b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56443db700d0 con 0x56443dc50380 2024-03-16T10:23:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.695+0000 7f997817b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56443dc58620 con 0x56443dc4f990 2024-03-16T10:23:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.695+0000 7f996f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 0x56443dc61270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.695+0000 7f997817b740 10 client.0 _ll_register_callbacks cb 0x56443db39760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.695+0000 7f996f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 0x56443dc61270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37558/0 (socket says 172.21.15.77:37558) 2024-03-16T10:23:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.695+0000 7f996f7fe640 1 -- 172.21.15.77:0/2875491231 learned_addr learned my addr 172.21.15.77:0/2875491231 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:49.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.696+0000 7f996e7fc640 1 --2- 172.21.15.77:0/2875491231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc50380 0x56443dc5a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.696+0000 7f996f7fe640 1 -- 172.21.15.77:0/2875491231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc4f990 msgr2=0x56443dc59cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.696+0000 7f996f7fe640 1 --2- 172.21.15.77:0/2875491231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc4f990 0x56443dc59cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.696+0000 7f996f7fe640 1 -- 172.21.15.77:0/2875491231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc50380 msgr2=0x56443dc5a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.696+0000 7f996f7fe640 1 --2- 172.21.15.77:0/2875491231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc50380 0x56443dc5a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.696+0000 7f996f7fe640 1 -- 172.21.15.77:0/2875491231 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56443d9f0f70 con 0x56443dc554f0 2024-03-16T10:23:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.696+0000 7f996f7fe640 1 --2- 172.21.15.77:0/2875491231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 0x56443dc61270 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f996809a700 tx=0x7f99680a3060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306235febe058bfe server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.696+0000 7f99527fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:23:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.696+0000 7f99527fc640 1 -- 172.21.15.77:0/2875491231 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9968099e50 con 0x56443dc554f0 2024-03-16T10:23:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.696+0000 7f99527fc640 1 -- 172.21.15.77:0/2875491231 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f9968099ff0 con 0x56443dc554f0 2024-03-16T10:23:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.696+0000 7f997817b740 1 -- 172.21.15.77:0/2875491231 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9968014960 con 0x56443dc554f0 2024-03-16T10:23:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.697+0000 7f99527fc640 1 -- 172.21.15.77:0/2875491231 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f99680b9280 con 0x56443dc554f0 2024-03-16T10:23:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.697+0000 7f997817b740 10 client.17166 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.697+0000 7f997817b740 1 -- 172.21.15.77:0/2875491231 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56443d888170 con 0x56443dc554f0 2024-03-16T10:23:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.697+0000 7f99527fc640 1 -- 172.21.15.77:0/2875491231 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9968014960 con 0x56443dc554f0 2024-03-16T10:23:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.697+0000 7f99527fc640 1 -- 172.21.15.77:0/2875491231 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f99680850b0 con 0x56443dc554f0 2024-03-16T10:23:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.697+0000 7f997817b740 10 client.17166 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.697+0000 7f997817b740 10 client.17166 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.697+0000 7f997817b740 20 client.17166 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.697+0000 7f997817b740 1 --2- 172.21.15.77:0/2875491231 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56443dc653d0 0x56443dc677c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.697+0000 7f997817b740 4 client.17166 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.697+0000 7f997817b740 1 -- 172.21.15.77:0/2875491231 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56443dbc1610 con 0x56443dc653d0 2024-03-16T10:23:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.698+0000 7f996effd640 1 --2- 172.21.15.77:0/2875491231 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56443dc653d0 0x56443dc677c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.698+0000 7f996effd640 1 --2- 172.21.15.77:0/2875491231 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56443dc653d0 0x56443dc677c0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b86edc5b0ad10dad server_cookie=11895185d6697660 in_seq=0 out_seq=0 2024-03-16T10:23:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.698+0000 7f99527fc640 10 client.17166 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:49.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f99527fc640 1 -- 172.21.15.77:0/2875491231 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56443dbc1610 con 0x56443dc653d0 2024-03-16T10:23:49.701 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f99527fc640 10 client.17166 handle_command_reply: tid=0 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f997817b740 1 client.17166 shutdown 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f997817b740 1 -- 172.21.15.77:0/2875491231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 msgr2=0x56443dc61270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f997817b740 1 --2- 172.21.15.77:0/2875491231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 0x56443dc61270 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f996809a700 tx=0x7f99680a3060 comp rx=0 tx=0).stop 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f997817b740 1 -- 172.21.15.77:0/2875491231 shutdown_connections 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f997817b740 1 --2- 172.21.15.77:0/2875491231 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56443dc653d0 0x56443dc677c0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f997817b740 1 --2- 172.21.15.77:0/2875491231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 0x56443dc61270 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f99527fc640 0 client.17166 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f997817b740 1 --2- 172.21.15.77:0/2875491231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc50380 0x56443dc5a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f997817b740 1 --2- 172.21.15.77:0/2875491231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc4f990 0x56443dc59cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f997817b740 1 -- 172.21.15.77:0/2875491231 >> 172.21.15.77:0/2875491231 conn(0x56443dc17c50 msgr2=0x56443dc4a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f997817b740 1 -- 172.21.15.77:0/2875491231 shutdown_connections 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.699+0000 7f997817b740 1 -- 172.21.15.77:0/2875491231 wait complete. 2024-03-16T10:23:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.700+0000 7f997817b740 20 client.17166 trim_cache size 0 max 16384 2024-03-16T10:23:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.717+0000 7f997817b740 1 Processor -- start 2024-03-16T10:23:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.717+0000 7f997817b740 1 -- start start 2024-03-16T10:23:49.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.717+0000 7f997817b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc4f990 0x56443dc4ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.717+0000 7f997817b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc50380 0x56443dc61230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.717+0000 7f997817b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc554f0 0x56443dc6ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.717+0000 7f997817b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f99701b7a00 con 0x56443dc50380 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.717+0000 7f997817b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f99701b9540 con 0x56443dc4f990 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.717+0000 7f997817b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99701b80d0 con 0x56443dc554f0 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f996e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc50380 0x56443dc61230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f996e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc50380 0x56443dc61230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37572/0 (socket says 172.21.15.77:37572) 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f996e7fc640 1 -- 172.21.15.77:0/1673985978 learned_addr learned my addr 172.21.15.77:0/1673985978 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f996effd640 1 --2- 172.21.15.77:0/1673985978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc4f990 0x56443dc4ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f996f7fe640 1 --2- 172.21.15.77:0/1673985978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc554f0 0x56443dc6ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f996e7fc640 1 -- 172.21.15.77:0/1673985978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc554f0 msgr2=0x56443dc6ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f996e7fc640 1 --2- 172.21.15.77:0/1673985978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc554f0 0x56443dc6ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f996e7fc640 1 -- 172.21.15.77:0/1673985978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc4f990 msgr2=0x56443dc4ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f996e7fc640 1 --2- 172.21.15.77:0/1673985978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc4f990 0x56443dc4ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f996e7fc640 1 -- 172.21.15.77:0/1673985978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56443dc51d80 con 0x56443dc50380 2024-03-16T10:23:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f996e7fc640 1 --2- 172.21.15.77:0/1673985978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc50380 0x56443dc61230 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f99701a7e50 tx=0x7f9960009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df61ae1f8e5bb4d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f995cff9640 1 -- 172.21.15.77:0/1673985978 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9960004400 con 0x56443dc50380 2024-03-16T10:23:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f995cff9640 1 -- 172.21.15.77:0/1673985978 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f996000fc50 con 0x56443dc50380 2024-03-16T10:23:49.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.718+0000 7f995cff9640 1 -- 172.21.15.77:0/1673985978 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996001f670 con 0x56443dc50380 2024-03-16T10:23:49.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.719+0000 7f997817b740 1 -- 172.21.15.77:0/1673985978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc50380 msgr2=0x56443dc61230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.719+0000 7f997817b740 1 --2- 172.21.15.77:0/1673985978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc50380 0x56443dc61230 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f99701a7e50 tx=0x7f9960009000 comp rx=0 tx=0).stop 2024-03-16T10:23:49.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.719+0000 7f997817b740 1 -- 172.21.15.77:0/1673985978 shutdown_connections 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.719+0000 7f997817b740 1 --2- 172.21.15.77:0/1673985978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc554f0 0x56443dc6ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.719+0000 7f997817b740 1 --2- 172.21.15.77:0/1673985978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc50380 0x56443dc61230 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.719+0000 7f997817b740 1 --2- 172.21.15.77:0/1673985978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc4f990 0x56443dc4ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.719+0000 7f997817b740 1 -- 172.21.15.77:0/1673985978 >> 172.21.15.77:0/1673985978 conn(0x56443dc17c50 msgr2=0x56443dc4e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.719+0000 7f997817b740 1 -- 172.21.15.77:0/1673985978 shutdown_connections 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.719+0000 7f997817b740 1 -- 172.21.15.77:0/1673985978 wait complete. 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.719+0000 7f997817b740 1 -- start start 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f997817b740 1 Processor -- start 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f997817b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc4f990 0x56443dc5efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f997817b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc50380 0x56443dc5f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f997817b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 0x56443dc60530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f997817b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f99701b8e70 con 0x56443dc554f0 2024-03-16T10:23:49.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f997817b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f99701b87a0 con 0x56443dc50380 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f997817b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99701b5a00 con 0x56443dc4f990 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f996f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 0x56443dc60530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f997817b740 10 client.0 _ll_register_callbacks cb 0x7f9964009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f996f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 0x56443dc60530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37578/0 (socket says 172.21.15.77:37578) 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f996effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc4f990 0x56443dc5efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f996f7fe640 1 -- 172.21.15.77:0/1504536885 learned_addr learned my addr 172.21.15.77:0/1504536885 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f996e7fc640 1 --2- 172.21.15.77:0/1504536885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc50380 0x56443dc5f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f996f7fe640 1 -- 172.21.15.77:0/1504536885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc4f990 msgr2=0x56443dc5efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f996f7fe640 1 --2- 172.21.15.77:0/1504536885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc4f990 0x56443dc5efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f996f7fe640 1 -- 172.21.15.77:0/1504536885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc50380 msgr2=0x56443dc5f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f996f7fe640 1 --2- 172.21.15.77:0/1504536885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc50380 0x56443dc5f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f996f7fe640 1 -- 172.21.15.77:0/1504536885 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56443dc60b20 con 0x56443dc554f0 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f996f7fe640 1 --2- 172.21.15.77:0/1504536885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 0x56443dc60530 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f99680a2c70 tx=0x7f99680a30d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a862e6ea48c3378d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:49.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.720+0000 7f99527fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f99527fc640 1 -- 172.21.15.77:0/1504536885 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996807e080 con 0x56443dc554f0 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f99527fc640 1 -- 172.21.15.77:0/1504536885 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f99680b3ac0 con 0x56443dc554f0 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f997817b740 1 -- 172.21.15.77:0/1504536885 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f996000fc50 con 0x56443dc554f0 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f99527fc640 1 -- 172.21.15.77:0/1504536885 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f9968014960 con 0x56443dc554f0 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f997817b740 10 client.17172 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f997817b740 1 -- 172.21.15.77:0/1504536885 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56443dccbb20 con 0x56443dc554f0 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f99527fc640 1 -- 172.21.15.77:0/1504536885 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f996000fc50 con 0x56443dc554f0 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f99527fc640 1 -- 172.21.15.77:0/1504536885 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f9968099ff0 con 0x56443dc554f0 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f997817b740 10 client.17172 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f997817b740 10 client.17172 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f997817b740 20 client.17172 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f997817b740 1 --2- 172.21.15.77:0/1504536885 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56443dccc2e0 0x56443dcce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f997817b740 4 client.17172 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f997817b740 1 -- 172.21.15.77:0/1504536885 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56443dc59ae0 con 0x56443dccc2e0 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.721+0000 7f996effd640 1 --2- 172.21.15.77:0/1504536885 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56443dccc2e0 0x56443dcce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.722+0000 7f99527fc640 10 client.17172 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.722+0000 7f996effd640 1 --2- 172.21.15.77:0/1504536885 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56443dccc2e0 0x56443dcce700 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65e41bb62e1fd55e server_cookie=e492202158b1e921 in_seq=0 out_seq=0 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.723+0000 7f99527fc640 1 -- 172.21.15.77:0/1504536885 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56443dc59ae0 con 0x56443dccc2e0 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.723+0000 7f99527fc640 10 client.17172 handle_command_reply: tid=0 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.723+0000 7f997817b740 1 client.17172 shutdown 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.723+0000 7f997817b740 1 -- 172.21.15.77:0/1504536885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 msgr2=0x56443dc60530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.723+0000 7f997817b740 1 --2- 172.21.15.77:0/1504536885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 0x56443dc60530 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f99680a2c70 tx=0x7f99680a30d0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.723+0000 7f997817b740 1 -- 172.21.15.77:0/1504536885 shutdown_connections 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.723+0000 7f997817b740 1 --2- 172.21.15.77:0/1504536885 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56443dccc2e0 0x56443dcce700 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.723+0000 7f997817b740 1 --2- 172.21.15.77:0/1504536885 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56443dc554f0 0x56443dc60530 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.723+0000 7f997817b740 1 --2- 172.21.15.77:0/1504536885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56443dc50380 0x56443dc5f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.723+0000 7f997817b740 1 --2- 172.21.15.77:0/1504536885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56443dc4f990 0x56443dc5efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.723+0000 7f997817b740 1 -- 172.21.15.77:0/1504536885 >> 172.21.15.77:0/1504536885 conn(0x56443dc17c50 msgr2=0x56443dc6be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:49.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.723+0000 7f99527fc640 0 client.17172 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:49.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f997817b740 1 -- 172.21.15.77:0/1504536885 shutdown_connections 2024-03-16T10:23:49.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f997817b740 1 -- 172.21.15.77:0/1504536885 wait complete. 2024-03-16T10:23:49.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f997817b740 20 client.17172 trim_cache size 0 max 16384 2024-03-16T10:23:49.730 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:23:49.730 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T10:23:49.730 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "be0cd9e9-c385-4436-9dce-770010f9b142", 2024-03-16T10:23:49.730 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T10:23:49.731 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:23:49.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f9975589640 1 -- 172.21.15.77:0/40061634 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f99580a28c0 msgr2=0x7f99580a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f9975589640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f99580a28c0 0x7f99580a4d80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f996000afd0 tx=0x7f996000aa80 comp rx=0 tx=0).stop 2024-03-16T10:23:49.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f9975589640 1 -- 172.21.15.77:0/40061634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f997010db00 msgr2=0x7f99701adef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f9975589640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f997010db00 0x7f99701adef0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f996800b9e0 tx=0x7f996800bea0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f9975589640 1 -- 172.21.15.77:0/40061634 shutdown_connections 2024-03-16T10:23:49.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f9975589640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f997010db00 0x7f99701adef0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f9975589640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f99580a28c0 0x7f99580a4d80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f9975589640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f997010b150 0x7f99701a6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f9975589640 1 --2- 172.21.15.77:0/40061634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99701087e0 0x7f99701a6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f9975589640 1 -- 172.21.15.77:0/40061634 >> 172.21.15.77:0/40061634 conn(0x7f997006e640 msgr2=0x7f997010c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:49.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f9975589640 1 -- 172.21.15.77:0/40061634 shutdown_connections 2024-03-16T10:23:49.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.724+0000 7f9975589640 1 -- 172.21.15.77:0/40061634 wait complete. 2024-03-16T10:23:49.739 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:23:49.739 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:23:49.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.849+0000 7f504bfc4640 1 -- 172.21.15.77:0/4182266053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50441018d0 msgr2=0x7f5044101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.849+0000 7f504bfc4640 1 --2- 172.21.15.77:0/4182266053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50441018d0 0x7f5044101d50 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5034009900 tx=0x7f503402bc30 comp rx=0 tx=0).stop 2024-03-16T10:23:49.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.849+0000 7f504bfc4640 1 -- 172.21.15.77:0/4182266053 shutdown_connections 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.849+0000 7f504bfc4640 1 --2- 172.21.15.77:0/4182266053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5044102290 0x7f5044108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.849+0000 7f504bfc4640 1 --2- 172.21.15.77:0/4182266053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50441018d0 0x7f5044101d50 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.849+0000 7f504bfc4640 1 --2- 172.21.15.77:0/4182266053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50441006e0 0x7f5044100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.849+0000 7f504bfc4640 1 -- 172.21.15.77:0/4182266053 >> 172.21.15.77:0/4182266053 conn(0x7f50440fbc70 msgr2=0x7f50440fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.849+0000 7f504bfc4640 1 -- 172.21.15.77:0/4182266053 shutdown_connections 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.849+0000 7f504bfc4640 1 -- 172.21.15.77:0/4182266053 wait complete. 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504bfc4640 1 Processor -- start 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504bfc4640 1 -- start start 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504bfc4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50441006e0 0x7f5044197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504bfc4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50441018d0 0x7f5044198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504bfc4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044102290 0x7f504419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504bfc4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f504410d480 con 0x7f5044102290 2024-03-16T10:23:49.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504bfc4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f504410d300 con 0x7f50441018d0 2024-03-16T10:23:49.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504bfc4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f504410d600 con 0x7f50441006e0 2024-03-16T10:23:49.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504a561640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044102290 0x7f504419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504a561640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044102290 0x7f504419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37600/0 (socket says 172.21.15.77:37600) 2024-03-16T10:23:49.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f5049d60640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50441006e0 0x7f5044197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504a561640 1 -- 172.21.15.77:0/2639525503 learned_addr learned my addr 172.21.15.77:0/2639525503 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:49.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504955f640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50441018d0 0x7f5044198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504a561640 1 -- 172.21.15.77:0/2639525503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50441006e0 msgr2=0x7f5044197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504a561640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50441006e0 0x7f5044197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504a561640 1 -- 172.21.15.77:0/2639525503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50441018d0 msgr2=0x7f5044198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504a561640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50441018d0 0x7f5044198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.850+0000 7f504a561640 1 -- 172.21.15.77:0/2639525503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f504419f700 con 0x7f5044102290 2024-03-16T10:23:49.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.851+0000 7f504a561640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044102290 0x7f504419f090 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f503c00eb10 tx=0x7f503c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:49.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.851+0000 7f503affd640 1 -- 172.21.15.77:0/2639525503 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c004b20 con 0x7f5044102290 2024-03-16T10:23:49.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.851+0000 7f504bfc4640 1 -- 172.21.15.77:0/2639525503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f504419f9f0 con 0x7f5044102290 2024-03-16T10:23:49.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.851+0000 7f503affd640 1 -- 172.21.15.77:0/2639525503 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f503c004cc0 con 0x7f5044102290 2024-03-16T10:23:49.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.851+0000 7f503affd640 1 -- 172.21.15.77:0/2639525503 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c010760 con 0x7f5044102290 2024-03-16T10:23:49.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.851+0000 7f504bfc4640 1 -- 172.21.15.77:0/2639525503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f504419fec0 con 0x7f5044102290 2024-03-16T10:23:49.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.853+0000 7f503affd640 1 -- 172.21.15.77:0/2639525503 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f503c005020 con 0x7f5044102290 2024-03-16T10:23:49.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.853+0000 7f504ebac740 1 Processor -- start 2024-03-16T10:23:49.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.853+0000 7f504ebac740 1 -- start start 2024-03-16T10:23:49.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.853+0000 7f504ebac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330b170 0x564f3330b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.853+0000 7f504ebac740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330bb60 0x564f33310660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.853+0000 7f504ebac740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f33310cd0 0x564f33313160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.853+0000 7f504ebac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564f33270500 con 0x564f3330b170 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.853+0000 7f504ebac740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564f33218be0 con 0x564f3330bb60 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.853+0000 7f504ebac740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564f3326c5a0 con 0x564f33310cd0 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f504955f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330b170 0x564f3330b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f5049d60640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330bb60 0x564f33310660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f503affd640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50300a2800 0x7f50300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f504955f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330b170 0x564f3330b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37606/0 (socket says 172.21.15.77:37606) 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f5049d60640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330bb60 0x564f33310660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44850/0 (socket says 172.21.15.77:44850) 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f504955f640 1 -- 172.21.15.77:0/830730163 learned_addr learned my addr 172.21.15.77:0/830730163 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f504955f640 1 --2- 172.21.15.77:0/830730163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f33310cd0 0x564f33313160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f504a561640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50300a2800 0x7f50300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f503affd640 1 -- 172.21.15.77:0/2639525503 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f503c0cef40 con 0x7f5044102290 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f5049d60640 1 -- 172.21.15.77:0/830730163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f33310cd0 msgr2=0x564f33313160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f5049d60640 1 --2- 172.21.15.77:0/830730163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f33310cd0 0x564f33313160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f5049d60640 1 -- 172.21.15.77:0/830730163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330b170 msgr2=0x564f3330b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f5049d60640 1 --2- 172.21.15.77:0/830730163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330b170 0x564f3330b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f5049d60640 1 -- 172.21.15.77:0/830730163 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f3322c4b0 con 0x564f3330bb60 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f504955f640 1 --2- 172.21.15.77:0/830730163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330b170 0x564f3330b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f5049d60640 1 --2- 172.21.15.77:0/830730163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330bb60 0x564f33310660 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5040013ba0 tx=0x7f5040009b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2826ed7f06c95032 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f504a561640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50300a2800 0x7f50300a4cc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f503c0a3be0 tx=0x7f503c01bb30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.854+0000 7f502ffff640 1 -- 172.21.15.77:0/830730163 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040009e20 con 0x564f3330bb60 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.855+0000 7f502ffff640 1 -- 172.21.15.77:0/830730163 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f504000e990 con 0x564f3330bb60 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.855+0000 7f502ffff640 1 -- 172.21.15.77:0/830730163 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040003b90 con 0x564f3330bb60 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.855+0000 7f504ebac740 1 -- 172.21.15.77:0/830730163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330bb60 msgr2=0x564f33310660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.855+0000 7f504ebac740 1 --2- 172.21.15.77:0/830730163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330bb60 0x564f33310660 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5040013ba0 tx=0x7f5040009b70 comp rx=0 tx=0).stop 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.856+0000 7f504ebac740 1 -- 172.21.15.77:0/830730163 shutdown_connections 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.856+0000 7f504ebac740 1 --2- 172.21.15.77:0/830730163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f33310cd0 0x564f33313160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.856+0000 7f504ebac740 1 --2- 172.21.15.77:0/830730163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330bb60 0x564f33310660 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.856+0000 7f504ebac740 1 --2- 172.21.15.77:0/830730163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330b170 0x564f3330b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.856+0000 7f504ebac740 1 -- 172.21.15.77:0/830730163 >> 172.21.15.77:0/830730163 conn(0x564f333066e0 msgr2=0x564f33306ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.856+0000 7f504ebac740 1 -- 172.21.15.77:0/830730163 shutdown_connections 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.856+0000 7f504ebac740 1 -- 172.21.15.77:0/830730163 wait complete. 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.856+0000 7f504ebac740 1 -- start start 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.856+0000 7f504ebac740 1 Processor -- start 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504ebac740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 0x564f333156b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504ebac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330bb60 0x564f33315bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504ebac740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f33310cd0 0x564f3331cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504ebac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564f33313c80 con 0x564f3330bb60 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504ebac740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564f3322c160 con 0x564f3330b170 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504ebac740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564f33313e00 con 0x564f33310cd0 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504ebac740 10 client.0 _ll_register_callbacks cb 0x564f33210150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504955f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 0x564f333156b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f5049d60640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330bb60 0x564f33315bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504955f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 0x564f333156b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44862/0 (socket says 172.21.15.77:44862) 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504955f640 1 -- 172.21.15.77:0/2489124522 learned_addr learned my addr 172.21.15.77:0/2489124522 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504955f640 1 --2- 172.21.15.77:0/2489124522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f33310cd0 0x564f3331cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504955f640 1 -- 172.21.15.77:0/2489124522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f33310cd0 msgr2=0x564f3331cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504955f640 1 --2- 172.21.15.77:0/2489124522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f33310cd0 0x564f3331cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504955f640 1 -- 172.21.15.77:0/2489124522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330bb60 msgr2=0x564f33315bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504955f640 1 --2- 172.21.15.77:0/2489124522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330bb60 0x564f33315bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504955f640 1 -- 172.21.15.77:0/2489124522 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f3304e7d0 con 0x564f3330b170 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504955f640 1 --2- 172.21.15.77:0/2489124522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 0x564f333156b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5034009230 tx=0x7f5034005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aab3b126cb9aa4ea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f502d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f502d7fa640 1 -- 172.21.15.77:0/2489124522 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5034004460 con 0x564f3330b170 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f502d7fa640 1 -- 172.21.15.77:0/2489124522 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f50340048f0 con 0x564f3330b170 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.857+0000 7f504ebac740 1 -- 172.21.15.77:0/2489124522 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f504000e990 con 0x564f3330b170 2024-03-16T10:23:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.858+0000 7f502d7fa640 1 -- 172.21.15.77:0/2489124522 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f5034004af0 con 0x564f3330b170 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.858+0000 7f504ebac740 10 client.26963 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.858+0000 7f504ebac740 1 -- 172.21.15.77:0/2489124522 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564f32f843f0 con 0x564f3330b170 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.858+0000 7f502d7fa640 1 -- 172.21.15.77:0/2489124522 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f504000e990 con 0x564f3330b170 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.858+0000 7f502d7fa640 1 -- 172.21.15.77:0/2489124522 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f5034003c60 con 0x564f3330b170 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.858+0000 7f504ebac740 10 client.26963 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.858+0000 7f504ebac740 10 client.26963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.858+0000 7f504ebac740 20 client.26963 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.858+0000 7f504ebac740 1 --2- 172.21.15.77:0/2489124522 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564f333210e0 0x564f33323530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.858+0000 7f504ebac740 4 client.26963 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.858+0000 7f504ebac740 1 -- 172.21.15.77:0/2489124522 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f3327da40 con 0x564f333210e0 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.858+0000 7f5049d60640 1 --2- 172.21.15.77:0/2489124522 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564f333210e0 0x564f33323530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.859+0000 7f502d7fa640 10 client.26963 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.859+0000 7f5049d60640 1 --2- 172.21.15.77:0/2489124522 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564f333210e0 0x564f33323530 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e7336674225114e server_cookie=c5b2344c56495d3e in_seq=0 out_seq=0 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.859+0000 7f502d7fa640 1 -- 172.21.15.77:0/2489124522 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564f3327da40 con 0x564f333210e0 2024-03-16T10:23:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.859+0000 7f502d7fa640 10 client.26963 handle_command_reply: tid=0 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.859+0000 7f504ebac740 1 client.26963 shutdown 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.860+0000 7f504ebac740 1 -- 172.21.15.77:0/2489124522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 msgr2=0x564f333156b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.860+0000 7f504ebac740 1 --2- 172.21.15.77:0/2489124522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 0x564f333156b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5034009230 tx=0x7f5034005fd0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.860+0000 7f504ebac740 1 -- 172.21.15.77:0/2489124522 shutdown_connections 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.860+0000 7f504ebac740 1 --2- 172.21.15.77:0/2489124522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f33310cd0 0x564f3331cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.860+0000 7f504ebac740 1 --2- 172.21.15.77:0/2489124522 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564f333210e0 0x564f33323530 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.860+0000 7f504ebac740 1 --2- 172.21.15.77:0/2489124522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330bb60 0x564f33315bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.860+0000 7f504ebac740 1 --2- 172.21.15.77:0/2489124522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 0x564f333156b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.860+0000 7f504ebac740 1 -- 172.21.15.77:0/2489124522 >> 172.21.15.77:0/2489124522 conn(0x564f333066e0 msgr2=0x564f33305900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.860+0000 7f502d7fa640 0 client.26963 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.860+0000 7f504ebac740 1 -- 172.21.15.77:0/2489124522 shutdown_connections 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.860+0000 7f504ebac740 1 -- 172.21.15.77:0/2489124522 wait complete. 2024-03-16T10:23:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.860+0000 7f504ebac740 20 client.26963 trim_cache size 0 max 16384 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.877+0000 7f504ebac740 1 Processor -- start 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.877+0000 7f504ebac740 1 -- start start 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.877+0000 7f504ebac740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f3330b170 0x564f33315b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.877+0000 7f504ebac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330bb60 0x564f33305070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.877+0000 7f504ebac740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f33310cd0 0x564f333055b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.877+0000 7f504ebac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f50441a6900 con 0x564f3330bb60 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.877+0000 7f504ebac740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f50441a8440 con 0x564f33310cd0 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.877+0000 7f504ebac740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50441a6fd0 con 0x564f3330b170 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.877+0000 7f5049d60640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330bb60 0x564f33305070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.877+0000 7f5049d60640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330bb60 0x564f33305070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37624/0 (socket says 172.21.15.77:37624) 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.877+0000 7f5049d60640 1 -- 172.21.15.77:0/2448412287 learned_addr learned my addr 172.21.15.77:0/2448412287 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.878+0000 7f504955f640 1 --2- 172.21.15.77:0/2448412287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f3330b170 0x564f33315b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.878+0000 7f504955f640 1 --2- 172.21.15.77:0/2448412287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f33310cd0 0x564f333055b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.878+0000 7f5049d60640 1 -- 172.21.15.77:0/2448412287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f3330b170 msgr2=0x564f33315b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.878+0000 7f5049d60640 1 --2- 172.21.15.77:0/2448412287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f3330b170 0x564f33315b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.878+0000 7f5049d60640 1 -- 172.21.15.77:0/2448412287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f33310cd0 msgr2=0x564f333055b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.878+0000 7f5049d60640 1 --2- 172.21.15.77:0/2448412287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f33310cd0 0x564f333055b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.878+0000 7f5049d60640 1 -- 172.21.15.77:0/2448412287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f32f440f0 con 0x564f3330bb60 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.878+0000 7f504955f640 1 --2- 172.21.15.77:0/2448412287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f3330b170 0x564f33315b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.878+0000 7f5049d60640 1 --2- 172.21.15.77:0/2448412287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330bb60 0x564f33305070 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5040005fd0 tx=0x7f504000e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d371e69267ed073a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.878+0000 7f502f7fe640 1 -- 172.21.15.77:0/2448412287 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504002b070 con 0x564f3330bb60 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.878+0000 7f502f7fe640 1 -- 172.21.15.77:0/2448412287 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50400055a0 con 0x564f3330bb60 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.878+0000 7f502f7fe640 1 -- 172.21.15.77:0/2448412287 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504002c260 con 0x564f3330bb60 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 -- 172.21.15.77:0/2448412287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330bb60 msgr2=0x564f33305070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 --2- 172.21.15.77:0/2448412287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330bb60 0x564f33305070 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5040005fd0 tx=0x7f504000e830 comp rx=0 tx=0).stop 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 -- 172.21.15.77:0/2448412287 shutdown_connections 2024-03-16T10:23:49.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 --2- 172.21.15.77:0/2448412287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f33310cd0 0x564f333055b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 --2- 172.21.15.77:0/2448412287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f3330bb60 0x564f33305070 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 --2- 172.21.15.77:0/2448412287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f3330b170 0x564f33315b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 -- 172.21.15.77:0/2448412287 >> 172.21.15.77:0/2448412287 conn(0x564f333066e0 msgr2=0x564f33306ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 -- 172.21.15.77:0/2448412287 shutdown_connections 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 -- 172.21.15.77:0/2448412287 wait complete. 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 -- start start 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 Processor -- start 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 0x564f3331a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f3330bb60 0x564f3331a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.879+0000 7f504ebac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f33310cd0 0x564f3331b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504ebac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f50441a7d70 con 0x564f33310cd0 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504ebac740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f50441a76a0 con 0x564f3330b170 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504ebac740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50441a4950 con 0x564f3330bb60 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504ebac740 10 client.0 _ll_register_callbacks cb 0x7f503402cd50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:49.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504955f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 0x564f3331a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504955f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 0x564f3331a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44890/0 (socket says 172.21.15.77:44890) 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504955f640 1 -- 172.21.15.77:0/269437754 learned_addr learned my addr 172.21.15.77:0/269437754 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f5049d60640 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f3330bb60 0x564f3331a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504955f640 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f33310cd0 0x564f3331b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504955f640 1 -- 172.21.15.77:0/269437754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f3330bb60 msgr2=0x564f3331a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504955f640 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f3330bb60 0x564f3331a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504955f640 1 -- 172.21.15.77:0/269437754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f33310cd0 msgr2=0x564f3331b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504955f640 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f33310cd0 0x564f3331b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504955f640 1 -- 172.21.15.77:0/269437754 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f332379d0 con 0x564f3330b170 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504955f640 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 0x564f3331a1c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5034034580 tx=0x7f5034005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90dab6cdcac6a1ad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f502d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f502d7fa640 1 -- 172.21.15.77:0/269437754 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5034004460 con 0x564f3330b170 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504ebac740 1 -- 172.21.15.77:0/269437754 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50400055a0 con 0x564f3330b170 2024-03-16T10:23:49.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f502d7fa640 1 -- 172.21.15.77:0/269437754 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f50340035f0 con 0x564f3330b170 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f502d7fa640 1 -- 172.21.15.77:0/269437754 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f50340037f0 con 0x564f3330b170 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.880+0000 7f504ebac740 10 client.26969 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.881+0000 7f504ebac740 1 -- 172.21.15.77:0/269437754 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x564f33387400 con 0x564f3330b170 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.881+0000 7f502d7fa640 1 -- 172.21.15.77:0/269437754 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50400055a0 con 0x564f3330b170 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.881+0000 7f502d7fa640 1 -- 172.21.15.77:0/269437754 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f5034034be0 con 0x564f3330b170 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.881+0000 7f504ebac740 10 client.26969 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.881+0000 7f504ebac740 10 client.26969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.881+0000 7f504ebac740 20 client.26969 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.881+0000 7f504ebac740 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564f33387d00 0x564f3338a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.881+0000 7f504ebac740 4 client.26969 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.881+0000 7f504ebac740 1 -- 172.21.15.77:0/269437754 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f33277d90 con 0x564f33387d00 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.881+0000 7f5049d60640 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564f33387d00 0x564f3338a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f5049d60640 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564f33387d00 0x564f3338a120 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97326687ff0ba1f2 server_cookie=aa7d1b9f4d4f1507 in_seq=0 out_seq=0 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f502d7fa640 10 client.26969 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f502d7fa640 1 -- 172.21.15.77:0/269437754 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x564f33277d90 con 0x564f33387d00 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f502d7fa640 10 client.26969 handle_command_reply: tid=0 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f504ebac740 1 client.26969 shutdown 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f504ebac740 1 -- 172.21.15.77:0/269437754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 msgr2=0x564f3331a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f504ebac740 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 0x564f3331a1c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5034034580 tx=0x7f5034005fd0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f504ebac740 1 -- 172.21.15.77:0/269437754 shutdown_connections 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f504ebac740 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564f33387d00 0x564f3338a120 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f504ebac740 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f33310cd0 0x564f3331b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f504ebac740 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f3330bb60 0x564f3331a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f504ebac740 1 --2- 172.21.15.77:0/269437754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f3330b170 0x564f3331a1c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f502d7fa640 0 client.26969 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f504ebac740 1 -- 172.21.15.77:0/269437754 >> 172.21.15.77:0/269437754 conn(0x564f333066e0 msgr2=0x564f33309940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f504ebac740 1 -- 172.21.15.77:0/269437754 shutdown_connections 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f504ebac740 1 -- 172.21.15.77:0/269437754 wait complete. 2024-03-16T10:23:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.882+0000 7f504ebac740 20 client.26969 trim_cache size 0 max 16384 2024-03-16T10:23:49.890 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:23:49.890 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-16T10:23:49.890 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:23:49.890 INFO:teuthology.orchestra.run.smithi077.stdout: "be0cd9e9-c385-4436-9dce-770010f9b142": { 2024-03-16T10:23:49.890 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:23:49.890 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "be0cd9e9-c385-4436-9dce-770010f9b142", 2024-03-16T10:23:49.890 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:23:49.890 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:23:49.890 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:23:49.890 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:23:49.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.883+0000 7f504bfc4640 1 -- 172.21.15.77:0/2639525503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50300a2800 msgr2=0x7f50300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.883+0000 7f504bfc4640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50300a2800 0x7f50300a4cc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f503c0a3be0 tx=0x7f503c01bb30 comp rx=0 tx=0).stop 2024-03-16T10:23:49.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.883+0000 7f504bfc4640 1 -- 172.21.15.77:0/2639525503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044102290 msgr2=0x7f504419f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:49.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.883+0000 7f504bfc4640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044102290 0x7f504419f090 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f503c00eb10 tx=0x7f503c00c2d0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.883+0000 7f504bfc4640 1 -- 172.21.15.77:0/2639525503 shutdown_connections 2024-03-16T10:23:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.883+0000 7f504bfc4640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5044102290 0x7f504419f090 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.883+0000 7f504bfc4640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50441018d0 0x7f5044198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.883+0000 7f504bfc4640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f50300a2800 0x7f50300a4cc0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.883+0000 7f504bfc4640 1 --2- 172.21.15.77:0/2639525503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50441006e0 0x7f5044197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.883+0000 7f504bfc4640 1 -- 172.21.15.77:0/2639525503 >> 172.21.15.77:0/2639525503 conn(0x7f50440fbc70 msgr2=0x7f5044104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.883+0000 7f504bfc4640 1 -- 172.21.15.77:0/2639525503 shutdown_connections 2024-03-16T10:23:49.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:23:49.883+0000 7f504bfc4640 1 -- 172.21.15.77:0/2639525503 wait complete. 2024-03-16T10:23:49.894 INFO:tasks.cephfs.filesystem:scrub status for tag:be0cd9e9-c385-4436-9dce-770010f9b142 - {'path': '/', 'tag': 'be0cd9e9-c385-4436-9dce-770010f9b142', 'options': 'recursive,force'} 2024-03-16T10:23:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:50 smithi077 ceph-mon[32400]: scrub queued for path: / 2024-03-16T10:23:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:50 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:50 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:50.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:50 smithi192 ceph-mon[36354]: scrub queued for path: / 2024-03-16T10:23:50.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:50 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:50.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:50 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:50.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:50 smithi193 ceph-mon[36311]: scrub queued for path: / 2024-03-16T10:23:50.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:50 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:50.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:50 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:51.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:51 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:51.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:51 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:51.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:51 smithi192 ceph-mon[36354]: pgmap v554: 97 pgs: 97 active+clean; 790 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.2 MiB/s wr, 607 op/s 2024-03-16T10:23:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.465+0000 7fc801337640 1 -- 172.21.15.77:0/1856428192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7fc0ff890 msgr2=0x7fc7fc0ffd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.465+0000 7fc801337640 1 --2- 172.21.15.77:0/1856428192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7fc0ff890 0x7fc7fc0ffd10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009860 tx=0x7fc7f002bb70 comp rx=0 tx=0).stop 2024-03-16T10:23:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.466+0000 7fc801337640 1 -- 172.21.15.77:0/1856428192 shutdown_connections 2024-03-16T10:23:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.466+0000 7fc801337640 1 --2- 172.21.15.77:0/1856428192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7fc100250 0x7fc7fc109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.466+0000 7fc801337640 1 --2- 172.21.15.77:0/1856428192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7fc0ff890 0x7fc7fc0ffd10 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.466+0000 7fc801337640 1 --2- 172.21.15.77:0/1856428192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7fc102e20 0x7fc7fc103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.466+0000 7fc801337640 1 -- 172.21.15.77:0/1856428192 >> 172.21.15.77:0/1856428192 conn(0x7fc7fc0fb410 msgr2=0x7fc7fc0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.466+0000 7fc801337640 1 -- 172.21.15.77:0/1856428192 shutdown_connections 2024-03-16T10:23:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.466+0000 7fc801337640 1 -- 172.21.15.77:0/1856428192 wait complete. 2024-03-16T10:23:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.466+0000 7fc801337640 1 Processor -- start 2024-03-16T10:23:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc801337640 1 -- start start 2024-03-16T10:23:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc801337640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7fc0ff890 0x7fc7fc197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc801337640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7fc100250 0x7fc7fc197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc801337640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7fc102e20 0x7fc7fc19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc801337640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc10d4d0 con 0x7fc7fc0ff890 2024-03-16T10:23:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc801337640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc10d350 con 0x7fc7fc102e20 2024-03-16T10:23:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc801337640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc10d650 con 0x7fc7fc100250 2024-03-16T10:23:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc7faffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7fc0ff890 0x7fc7fc197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc7faffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7fc0ff890 0x7fc7fc197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37650/0 (socket says 172.21.15.77:37650) 2024-03-16T10:23:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc7faffd640 1 -- 172.21.15.77:0/1271417242 learned_addr learned my addr 172.21.15.77:0/1271417242 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc7fb7fe640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7fc102e20 0x7fc7fc19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc7fa7fc640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7fc100250 0x7fc7fc197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc7faffd640 1 -- 172.21.15.77:0/1271417242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7fc100250 msgr2=0x7fc7fc197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc7faffd640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7fc100250 0x7fc7fc197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc7faffd640 1 -- 172.21.15.77:0/1271417242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7fc102e20 msgr2=0x7fc7fc19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc7faffd640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7fc102e20 0x7fc7fc19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.467+0000 7fc7faffd640 1 -- 172.21.15.77:0/1271417242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7fc19f680 con 0x7fc7fc0ff890 2024-03-16T10:23:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.468+0000 7fc7faffd640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7fc0ff890 0x7fc7fc197a60 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec00e9f0 tx=0x7fc7ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.468+0000 7fc7e3fff640 1 -- 172.21.15.77:0/1271417242 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec00ccc0 con 0x7fc7fc0ff890 2024-03-16T10:23:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.468+0000 7fc7e3fff640 1 -- 172.21.15.77:0/1271417242 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ec0035f0 con 0x7fc7fc0ff890 2024-03-16T10:23:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.468+0000 7fc801337640 1 -- 172.21.15.77:0/1271417242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7fc19f970 con 0x7fc7fc0ff890 2024-03-16T10:23:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.468+0000 7fc7e3fff640 1 -- 172.21.15.77:0/1271417242 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec00ce60 con 0x7fc7fc0ff890 2024-03-16T10:23:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.468+0000 7fc801337640 1 -- 172.21.15.77:0/1271417242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7fc19fee0 con 0x7fc7fc0ff890 2024-03-16T10:23:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc803ff8740 1 Processor -- start 2024-03-16T10:23:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc803ff8740 1 -- start start 2024-03-16T10:23:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc7e3fff640 1 -- 172.21.15.77:0/1271417242 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fc7ec0037a0 con 0x7fc7fc0ff890 2024-03-16T10:23:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc803ff8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b678731340 0x55b678731760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc803ff8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678731d30 0x55b678736830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc803ff8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678736ea0 0x55b678739330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc803ff8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b678696a40 con 0x55b678731d30 2024-03-16T10:23:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc803ff8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55b67863f920 con 0x55b678731340 2024-03-16T10:23:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc803ff8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b678692ae0 con 0x55b678736ea0 2024-03-16T10:23:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc7faffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678731d30 0x55b678736830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc7fb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b678731340 0x55b678731760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc7fb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b678731340 0x55b678731760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44922/0 (socket says 172.21.15.77:44922) 2024-03-16T10:23:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc7fb7fe640 1 -- 172.21.15.77:0/3337152900 learned_addr learned my addr 172.21.15.77:0/3337152900 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.471+0000 7fc7fa7fc640 1 --2- 172.21.15.77:0/3337152900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678736ea0 0x55b678739330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7faffd640 1 -- 172.21.15.77:0/3337152900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678736ea0 msgr2=0x55b678739330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7faffd640 1 --2- 172.21.15.77:0/3337152900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678736ea0 0x55b678739330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7faffd640 1 -- 172.21.15.77:0/3337152900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b678731340 msgr2=0x55b678731760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7faffd640 1 --2- 172.21.15.77:0/3337152900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b678731340 0x55b678731760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7faffd640 1 -- 172.21.15.77:0/3337152900 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b678657460 con 0x55b678731d30 2024-03-16T10:23:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7fb7fe640 1 --2- 172.21.15.77:0/3337152900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b678731340 0x55b678731760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7faffd640 1 --2- 172.21.15.77:0/3337152900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678731d30 0x55b678736830 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0cc710 tx=0x7fc7ec0d2cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2348ada2641191f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7e3fff640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7e40a2920 0x7fc7e40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7e3fff640 1 -- 172.21.15.77:0/1271417242 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fc7ec0cebb0 con 0x7fc7fc0ff890 2024-03-16T10:23:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7fb7fe640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7e40a2920 0x7fc7e40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7e17fa640 1 -- 172.21.15.77:0/3337152900 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec097880 con 0x55b678731d30 2024-03-16T10:23:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7e17fa640 1 -- 172.21.15.77:0/3337152900 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ec09bce0 con 0x55b678731d30 2024-03-16T10:23:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.472+0000 7fc7e17fa640 1 -- 172.21.15.77:0/3337152900 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec09af40 con 0x55b678731d30 2024-03-16T10:23:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 -- 172.21.15.77:0/3337152900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678731d30 msgr2=0x55b678736830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3337152900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678731d30 0x55b678736830 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0cc710 tx=0x7fc7ec0d2cc0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc7fb7fe640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7e40a2920 0x7fc7e40a4de0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x55b678732590 tx=0x7fc7f4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 -- 172.21.15.77:0/3337152900 shutdown_connections 2024-03-16T10:23:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3337152900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678736ea0 0x55b678739330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3337152900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678731d30 0x55b678736830 secure :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0cc710 tx=0x7fc7ec0d2cc0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3337152900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b678731340 0x55b678731760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 -- 172.21.15.77:0/3337152900 >> 172.21.15.77:0/3337152900 conn(0x55b67872c8b0 msgr2=0x55b67872ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 -- 172.21.15.77:0/3337152900 shutdown_connections 2024-03-16T10:23:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 -- 172.21.15.77:0/3337152900 wait complete. 2024-03-16T10:23:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 -- start start 2024-03-16T10:23:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 Processor -- start 2024-03-16T10:23:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67873fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.474+0000 7fc803ff8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b678736ea0 0x55b678740080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc803ff8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b6787450b0 0x55b6787474a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc803ff8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b67873e260 con 0x55b6787450b0 2024-03-16T10:23:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc803ff8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55b678652d70 con 0x55b678736ea0 2024-03-16T10:23:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc803ff8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b67873e3e0 con 0x55b678731340 2024-03-16T10:23:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc803ff8740 10 client.0 _ll_register_callbacks cb 0x55b6783aaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc7faffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67873fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc7fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b6787450b0 0x55b6787474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc7faffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67873fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:45296/0 (socket says 172.21.15.77:45296) 2024-03-16T10:23:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc7faffd640 1 -- 172.21.15.77:0/2362762635 learned_addr learned my addr 172.21.15.77:0/2362762635 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc7fa7fc640 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b678736ea0 0x55b678740080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc7fb7fe640 1 -- 172.21.15.77:0/2362762635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 msgr2=0x55b67873fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc7fb7fe640 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67873fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc7fb7fe640 1 -- 172.21.15.77:0/2362762635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b678736ea0 msgr2=0x55b678740080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc7fb7fe640 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b678736ea0 0x55b678740080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc7fb7fe640 1 -- 172.21.15.77:0/2362762635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6783ab210 con 0x55b6787450b0 2024-03-16T10:23:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.475+0000 7fc7faffd640 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67873fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:23:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.476+0000 7fc7fb7fe640 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b6787450b0 0x55b6787474a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400f790 tx=0x7fc7f4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef698b55b6d2ceed server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.476+0000 7fc7deff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:23:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.476+0000 7fc7deff5640 1 -- 172.21.15.77:0/2362762635 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4004b20 con 0x55b6787450b0 2024-03-16T10:23:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.476+0000 7fc7deff5640 1 -- 172.21.15.77:0/2362762635 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fc7f4004cc0 con 0x55b6787450b0 2024-03-16T10:23:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.476+0000 7fc803ff8740 1 -- 172.21.15.77:0/2362762635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7ec09bce0 con 0x55b6787450b0 2024-03-16T10:23:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.476+0000 7fc7deff5640 1 -- 172.21.15.77:0/2362762635 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fc7f4004310 con 0x55b6787450b0 2024-03-16T10:23:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.476+0000 7fc803ff8740 10 client.17202 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.476+0000 7fc803ff8740 1 -- 172.21.15.77:0/2362762635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b678477620 con 0x55b6787450b0 2024-03-16T10:23:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.477+0000 7fc7deff5640 1 -- 172.21.15.77:0/2362762635 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ec09bce0 con 0x55b6787450b0 2024-03-16T10:23:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.477+0000 7fc7deff5640 1 -- 172.21.15.77:0/2362762635 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fc7f4026070 con 0x55b6787450b0 2024-03-16T10:23:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.477+0000 7fc803ff8740 10 client.17202 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.477+0000 7fc803ff8740 10 client.17202 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.477+0000 7fc803ff8740 20 client.17202 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.478+0000 7fc803ff8740 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b67874be00 0x55b67874e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.478+0000 7fc803ff8740 4 client.17202 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:23:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.478+0000 7fc803ff8740 1 -- 172.21.15.77:0/2362762635 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6786a3f80 con 0x55b67874be00 2024-03-16T10:23:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.478+0000 7fc7faffd640 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b67874be00 0x55b67874e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.478+0000 7fc7faffd640 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b67874be00 0x55b67874e210 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a8f24574cfa5939 server_cookie=b3513cc06794d645 in_seq=0 out_seq=0 2024-03-16T10:23:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.478+0000 7fc7deff5640 10 client.17202 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.479+0000 7fc7deff5640 1 -- 172.21.15.77:0/2362762635 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b6786a3f80 con 0x55b67874be00 2024-03-16T10:23:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.479+0000 7fc7deff5640 10 client.17202 handle_command_reply: tid=0 2024-03-16T10:23:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.479+0000 7fc803ff8740 1 client.17202 shutdown 2024-03-16T10:23:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.480+0000 7fc803ff8740 1 -- 172.21.15.77:0/2362762635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b6787450b0 msgr2=0x55b6787474a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.480+0000 7fc803ff8740 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b6787450b0 0x55b6787474a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400f790 tx=0x7fc7f4004ab0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.480+0000 7fc803ff8740 1 -- 172.21.15.77:0/2362762635 shutdown_connections 2024-03-16T10:23:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.480+0000 7fc803ff8740 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b67874be00 0x55b67874e210 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.480+0000 7fc803ff8740 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b6787450b0 0x55b6787474a0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.480+0000 7fc803ff8740 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b678736ea0 0x55b678740080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.480+0000 7fc7deff5640 0 client.17202 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.480+0000 7fc803ff8740 1 --2- 172.21.15.77:0/2362762635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67873fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.480+0000 7fc803ff8740 1 -- 172.21.15.77:0/2362762635 >> 172.21.15.77:0/2362762635 conn(0x55b67872c8b0 msgr2=0x55b67872bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.480+0000 7fc803ff8740 1 -- 172.21.15.77:0/2362762635 shutdown_connections 2024-03-16T10:23:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.480+0000 7fc803ff8740 1 -- 172.21.15.77:0/2362762635 wait complete. 2024-03-16T10:23:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.480+0000 7fc803ff8740 20 client.17202 trim_cache size 0 max 16384 2024-03-16T10:23:51.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:51 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:51.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:51 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:51.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:51 smithi077 ceph-mon[32400]: pgmap v554: 97 pgs: 97 active+clean; 790 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.2 MiB/s wr, 607 op/s 2024-03-16T10:23:51.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:51 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:51.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:51 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:51.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:51 smithi193 ceph-mon[36311]: pgmap v554: 97 pgs: 97 active+clean; 790 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.2 MiB/s wr, 607 op/s 2024-03-16T10:23:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.505+0000 7fc803ff8740 1 Processor -- start 2024-03-16T10:23:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.505+0000 7fc803ff8740 1 -- start start 2024-03-16T10:23:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.505+0000 7fc803ff8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67873e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.505+0000 7fc803ff8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 0x55b678739290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.505+0000 7fc803ff8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6787450b0 0x55b678727a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.505+0000 7fc803ff8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc1a6920 con 0x55b678736ea0 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.505+0000 7fc803ff8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc1a8460 con 0x55b6787450b0 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.505+0000 7fc803ff8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc1a6ff0 con 0x55b678731340 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.505+0000 7fc7fa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 0x55b678739290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.505+0000 7fc7fa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 0x55b678739290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37664/0 (socket says 172.21.15.77:37664) 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.505+0000 7fc7fa7fc640 1 -- 172.21.15.77:0/3769866444 learned_addr learned my addr 172.21.15.77:0/3769866444 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.505+0000 7fc7faffd640 1 --2- 172.21.15.77:0/3769866444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67873e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.506+0000 7fc7fb7fe640 1 --2- 172.21.15.77:0/3769866444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6787450b0 0x55b678727a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.506+0000 7fc7fa7fc640 1 -- 172.21.15.77:0/3769866444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 msgr2=0x55b67873e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.506+0000 7fc7fa7fc640 1 --2- 172.21.15.77:0/3769866444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67873e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.506+0000 7fc7fa7fc640 1 -- 172.21.15.77:0/3769866444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6787450b0 msgr2=0x55b678727a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.506+0000 7fc7fa7fc640 1 --2- 172.21.15.77:0/3769866444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6787450b0 0x55b678727a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.506+0000 7fc7fa7fc640 1 -- 172.21.15.77:0/3769866444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b678732110 con 0x55b678736ea0 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.506+0000 7fc7fb7fe640 1 --2- 172.21.15.77:0/3769866444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6787450b0 0x55b678727a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.506+0000 7fc7fa7fc640 1 --2- 172.21.15.77:0/3769866444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 0x55b678739290 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00057c0 tx=0x7fc7f0034ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc25b5f6ef0770b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.506+0000 7fc7e0ff9640 1 -- 172.21.15.77:0/3769866444 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0044070 con 0x55b678736ea0 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.506+0000 7fc7e0ff9640 1 -- 172.21.15.77:0/3769866444 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7f0004030 con 0x55b678736ea0 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.506+0000 7fc7e0ff9640 1 -- 172.21.15.77:0/3769866444 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0039040 con 0x55b678736ea0 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.507+0000 7fc803ff8740 1 -- 172.21.15.77:0/3769866444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 msgr2=0x55b678739290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.507+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3769866444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 0x55b678739290 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00057c0 tx=0x7fc7f0034ad0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.507+0000 7fc803ff8740 1 -- 172.21.15.77:0/3769866444 shutdown_connections 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.507+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3769866444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6787450b0 0x55b678727a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.507+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3769866444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 0x55b678739290 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.507+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3769866444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67873e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.507+0000 7fc803ff8740 1 -- 172.21.15.77:0/3769866444 >> 172.21.15.77:0/3769866444 conn(0x55b67872c8b0 msgr2=0x55b67872cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.507+0000 7fc803ff8740 1 -- 172.21.15.77:0/3769866444 shutdown_connections 2024-03-16T10:23:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.507+0000 7fc803ff8740 1 -- 172.21.15.77:0/3769866444 wait complete. 2024-03-16T10:23:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.507+0000 7fc803ff8740 1 -- start start 2024-03-16T10:23:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc803ff8740 1 Processor -- start 2024-03-16T10:23:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc803ff8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67875d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc803ff8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 0x55b67875d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc803ff8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6787450b0 0x55b67875e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc803ff8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc1a7d90 con 0x55b678736ea0 2024-03-16T10:23:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc803ff8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc1a76c0 con 0x55b6787450b0 2024-03-16T10:23:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc803ff8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc1a4900 con 0x55b678731340 2024-03-16T10:23:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc803ff8740 10 client.0 _ll_register_callbacks cb 0x7fc7f4011e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:23:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc7fa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 0x55b67875d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc7fa7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 0x55b67875d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37672/0 (socket says 172.21.15.77:37672) 2024-03-16T10:23:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc7fa7fc640 1 -- 172.21.15.77:0/3783065042 learned_addr learned my addr 172.21.15.77:0/3783065042 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:23:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc7faffd640 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67875d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc7fb7fe640 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6787450b0 0x55b67875e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc7fa7fc640 1 -- 172.21.15.77:0/3783065042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 msgr2=0x55b67875d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc7fa7fc640 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67875d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc7fa7fc640 1 -- 172.21.15.77:0/3783065042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6787450b0 msgr2=0x55b67875e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:23:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc7fa7fc640 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6787450b0 0x55b67875e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc7fa7fc640 1 -- 172.21.15.77:0/3783065042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b67875ee10 con 0x55b678736ea0 2024-03-16T10:23:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc7fa7fc640 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 0x55b67875d810 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc7f002c0b0 tx=0x7fc7f00324f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8afaeeee9503c6e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:23:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.508+0000 7fc7deff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:23:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc7deff5640 1 -- 172.21.15.77:0/3783065042 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0044070 con 0x55b678736ea0 2024-03-16T10:23:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc7deff5640 1 -- 172.21.15.77:0/3783065042 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fc7f0034700 con 0x55b678736ea0 2024-03-16T10:23:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc803ff8740 1 -- 172.21.15.77:0/3783065042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7f0004030 con 0x55b678736ea0 2024-03-16T10:23:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc7deff5640 1 -- 172.21.15.77:0/3783065042 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fc7f0039040 con 0x55b678736ea0 2024-03-16T10:23:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc803ff8740 10 client.17208 fetch_fsmap learned FSMap version 39 2024-03-16T10:23:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc803ff8740 1 -- 172.21.15.77:0/3783065042 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b6787b23f0 con 0x55b678736ea0 2024-03-16T10:23:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc7deff5640 1 -- 172.21.15.77:0/3783065042 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7f0004030 con 0x55b678736ea0 2024-03-16T10:23:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc7deff5640 1 -- 172.21.15.77:0/3783065042 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7fc7f005a050 con 0x55b678736ea0 2024-03-16T10:23:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc803ff8740 10 client.17208 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:23:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc803ff8740 10 client.17208 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:23:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc803ff8740 20 client.17208 populate_metadata read hostname 'smithi077' 2024-03-16T10:23:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b6787b2e00 0x55b6787b5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:23:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc803ff8740 4 client.17208 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:23:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc803ff8740 1 -- 172.21.15.77:0/3783065042 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b67869e2d0 con 0x55b6787b2e00 2024-03-16T10:23:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.509+0000 7fc7faffd640 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b6787b2e00 0x55b6787b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:23:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.510+0000 7fc7deff5640 10 client.17208 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.510+0000 7fc7faffd640 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b6787b2e00 0x55b6787b5220 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a443adf89d992904 server_cookie=bca646691e1b25b2 in_seq=0 out_seq=0 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc7deff5640 1 -- 172.21.15.77:0/3783065042 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b67869e2d0 con 0x55b6787b2e00 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc7deff5640 10 client.17208 handle_command_reply: tid=0 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc803ff8740 1 client.17208 shutdown 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc803ff8740 1 -- 172.21.15.77:0/3783065042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 msgr2=0x55b67875d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 0x55b67875d810 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc7f002c0b0 tx=0x7fc7f00324f0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc803ff8740 1 -- 172.21.15.77:0/3783065042 shutdown_connections 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b6787450b0 0x55b67875e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b6787b2e00 0x55b6787b5220 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b678736ea0 0x55b67875d810 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc803ff8740 1 --2- 172.21.15.77:0/3783065042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b678731340 0x55b67875d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc7deff5640 0 client.17208 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc803ff8740 1 -- 172.21.15.77:0/3783065042 >> 172.21.15.77:0/3783065042 conn(0x55b67872c8b0 msgr2=0x55b678727a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc803ff8740 1 -- 172.21.15.77:0/3783065042 shutdown_connections 2024-03-16T10:23:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.542+0000 7fc803ff8740 1 -- 172.21.15.77:0/3783065042 wait complete. 2024-03-16T10:23:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc803ff8740 20 client.17208 trim_cache size 0 max 16384 2024-03-16T10:23:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:23:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:23:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:23:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:23:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc801337640 1 -- 172.21.15.77:0/1271417242 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7e40a2920 msgr2=0x7fc7e40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc801337640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7e40a2920 0x7fc7e40a4de0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x55b678732590 tx=0x7fc7f4009500 comp rx=0 tx=0).stop 2024-03-16T10:23:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc801337640 1 -- 172.21.15.77:0/1271417242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7fc0ff890 msgr2=0x7fc7fc197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:23:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc801337640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7fc0ff890 0x7fc7fc197a60 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec00e9f0 tx=0x7fc7ec00eeb0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc801337640 1 -- 172.21.15.77:0/1271417242 shutdown_connections 2024-03-16T10:23:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc801337640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7fc102e20 0x7fc7fc19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc801337640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc7e40a2920 0x7fc7e40a4de0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc801337640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7fc100250 0x7fc7fc197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc801337640 1 --2- 172.21.15.77:0/1271417242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7fc0ff890 0x7fc7fc197a60 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:23:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc801337640 1 -- 172.21.15.77:0/1271417242 >> 172.21.15.77:0/1271417242 conn(0x7fc7fc0fb410 msgr2=0x7fc7fc105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:23:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc801337640 1 -- 172.21.15.77:0/1271417242 shutdown_connections 2024-03-16T10:23:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:23:51.543+0000 7fc801337640 1 -- 172.21.15.77:0/1271417242 wait complete. 2024-03-16T10:23:52.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:52.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:52.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:52.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:52.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:52.397 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:52.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:52.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:52.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:52.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:52.398 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:52 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:52.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:52.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:52.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:52.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:52 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:52.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:52.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:52.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:52.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:52 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:52.951 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T10:23:52.951 INFO:tasks.workunit.client.0.smithi077.stdout:Final score for writes: 173 2024-03-16T10:23:52.951 INFO:tasks.workunit.client.0.smithi077.stdout:Final score for reads : 811603 2024-03-16T10:23:52.951 INFO:tasks.workunit.client.0.smithi077.stdout: 2024-03-16T10:23:52.951 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-16T10:23:52.951 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T10:23:53.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:53 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:23:53.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:53 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:53.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:53 smithi192 ceph-mon[36354]: pgmap v555: 97 pgs: 97 active+clean; 814 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.5 MiB/s wr, 817 op/s 2024-03-16T10:23:53.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:53 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:23:53.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:53 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:53.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:53 smithi077 ceph-mon[32400]: pgmap v555: 97 pgs: 97 active+clean; 814 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.5 MiB/s wr, 817 op/s 2024-03-16T10:23:53.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:53 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:23:53.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:53 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:23:53.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:53 smithi193 ceph-mon[36311]: pgmap v555: 97 pgs: 97 active+clean; 814 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.5 MiB/s wr, 817 op/s 2024-03-16T10:23:56.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:55 smithi192 ceph-mon[36354]: pgmap v556: 97 pgs: 97 active+clean; 814 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.1 MiB/s wr, 866 op/s 2024-03-16T10:23:56.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:55 smithi077 ceph-mon[32400]: pgmap v556: 97 pgs: 97 active+clean; 814 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.1 MiB/s wr, 866 op/s 2024-03-16T10:23:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:55 smithi193 ceph-mon[36311]: pgmap v556: 97 pgs: 97 active+clean; 814 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.1 MiB/s wr, 866 op/s 2024-03-16T10:23:58.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:57 smithi192 ceph-mon[36354]: pgmap v557: 97 pgs: 97 active+clean; 801 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.0 MiB/s wr, 767 op/s 2024-03-16T10:23:58.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:57 smithi077 ceph-mon[32400]: pgmap v557: 97 pgs: 97 active+clean; 801 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.0 MiB/s wr, 767 op/s 2024-03-16T10:23:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:57 smithi193 ceph-mon[36311]: pgmap v557: 97 pgs: 97 active+clean; 801 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.0 MiB/s wr, 767 op/s 2024-03-16T10:24:00.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:23:59 smithi077 ceph-mon[32400]: pgmap v558: 97 pgs: 97 active+clean; 790 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.3 MiB/s wr, 843 op/s 2024-03-16T10:24:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:23:59 smithi193 ceph-mon[36311]: pgmap v558: 97 pgs: 97 active+clean; 790 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.3 MiB/s wr, 843 op/s 2024-03-16T10:24:00.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:23:59 smithi192 ceph-mon[36354]: pgmap v558: 97 pgs: 97 active+clean; 790 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.3 MiB/s wr, 843 op/s 2024-03-16T10:24:02.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:01 smithi077 ceph-mon[32400]: pgmap v559: 97 pgs: 97 active+clean; 777 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 679 op/s 2024-03-16T10:24:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:01 smithi193 ceph-mon[36311]: pgmap v559: 97 pgs: 97 active+clean; 777 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 679 op/s 2024-03-16T10:24:02.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:01 smithi192 ceph-mon[36354]: pgmap v559: 97 pgs: 97 active+clean; 777 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 679 op/s 2024-03-16T10:24:04.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:03 smithi077 ceph-mon[32400]: pgmap v560: 97 pgs: 97 active+clean; 767 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 699 op/s 2024-03-16T10:24:04.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:03 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:24:04.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:03 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:04.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:03 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:24:04.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:03 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:03 smithi193 ceph-mon[36311]: pgmap v560: 97 pgs: 97 active+clean; 767 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 699 op/s 2024-03-16T10:24:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:03 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:24:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:03 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:03 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:24:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:03 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:04.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:03 smithi192 ceph-mon[36354]: pgmap v560: 97 pgs: 97 active+clean; 767 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 699 op/s 2024-03-16T10:24:04.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:03 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:24:04.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:03 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:04.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:03 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:24:04.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:03 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.635+0000 7f76e7092640 1 -- 172.21.15.77:0/3841634412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76e0100700 msgr2=0x7f76e0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.635+0000 7f76e7092640 1 --2- 172.21.15.77:0/3841634412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76e0100700 0x7f76e0100b20 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f76d4009900 tx=0x7f76d402bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.635+0000 7f76e7092640 1 -- 172.21.15.77:0/3841634412 shutdown_connections 2024-03-16T10:24:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.635+0000 7f76e7092640 1 --2- 172.21.15.77:0/3841634412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76e01022b0 0x7f76e0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.638 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.635+0000 7f76e7092640 1 --2- 172.21.15.77:0/3841634412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76e01018f0 0x7f76e0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.635+0000 7f76e7092640 1 --2- 172.21.15.77:0/3841634412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76e0100700 0x7f76e0100b20 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.635+0000 7f76e7092640 1 -- 172.21.15.77:0/3841634412 >> 172.21.15.77:0/3841634412 conn(0x7f76e00fbc70 msgr2=0x7f76e00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.635+0000 7f76e7092640 1 -- 172.21.15.77:0/3841634412 shutdown_connections 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.635+0000 7f76e7092640 1 -- 172.21.15.77:0/3841634412 wait complete. 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e7092640 1 Processor -- start 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e7092640 1 -- start start 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e7092640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76e0100700 0x7f76e0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e7092640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76e01018f0 0x7f76e0196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e7092640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76e01022b0 0x7f76e019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e7092640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f76e010c370 con 0x7f76e01022b0 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e7092640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f76e010c1f0 con 0x7f76e0100700 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e7092640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76e010c4f0 con 0x7f76e01018f0 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e562f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76e01022b0 0x7f76e019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.639 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e562f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76e01022b0 0x7f76e019dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34166/0 (socket says 172.21.15.77:34166) 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e4e2e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76e0100700 0x7f76e0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e562f640 1 -- 172.21.15.77:0/1218746703 learned_addr learned my addr 172.21.15.77:0/1218746703 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76dffff640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76e01018f0 0x7f76e0196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e562f640 1 -- 172.21.15.77:0/1218746703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76e01018f0 msgr2=0x7f76e0196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e562f640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76e01018f0 0x7f76e0196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e562f640 1 -- 172.21.15.77:0/1218746703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76e0100700 msgr2=0x7f76e0196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e562f640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76e0100700 0x7f76e0196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e562f640 1 -- 172.21.15.77:0/1218746703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76e019e630 con 0x7f76e01022b0 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.636+0000 7f76e562f640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76e01022b0 0x7f76e019dfc0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f76d8009fd0 tx=0x7f76d800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.637+0000 7f76ddffb640 1 -- 172.21.15.77:0/1218746703 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d8017070 con 0x7f76e01022b0 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.637+0000 7f76ddffb640 1 -- 172.21.15.77:0/1218746703 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d8003e50 con 0x7f76e01022b0 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.637+0000 7f76e7092640 1 -- 172.21.15.77:0/1218746703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76e019e920 con 0x7f76e01022b0 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.637+0000 7f76ddffb640 1 -- 172.21.15.77:0/1218746703 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d80106c0 con 0x7f76e01022b0 2024-03-16T10:24:04.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.637+0000 7f76e7092640 1 -- 172.21.15.77:0/1218746703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76e019ee20 con 0x7f76e01022b0 2024-03-16T10:24:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.639+0000 7f76ddffb640 1 -- 172.21.15.77:0/1218746703 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f76d8003970 con 0x7f76e01022b0 2024-03-16T10:24:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.639+0000 7f76ddffb640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76d00a28a0 0x7f76d00a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.639+0000 7f76ddffb640 1 -- 172.21.15.77:0/1218746703 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f76d80ce6d0 con 0x7f76e01022b0 2024-03-16T10:24:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.639+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76d00a28a0 0x7f76d00a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76d00a28a0 0x7f76d00a4d60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f76d4000c00 tx=0x7f76d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76e9c7d740 1 Processor -- start 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76e9c7d740 1 -- start start 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76e9c7d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07a6340 0x55c3f07a6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76e9c7d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6d30 0x55c3f07ab830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76e9c7d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07abea0 0x55c3f07ae330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76e9c7d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c3f070ba40 con 0x55c3f07a6d30 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76e9c7d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c3f06b4920 con 0x55c3f07abea0 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76e9c7d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c3f0707ae0 con 0x55c3f07a6340 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76dffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6d30 0x55c3f07ab830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76e4e2e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07a6340 0x55c3f07a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76dffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6d30 0x55c3f07ab830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34172/0 (socket says 172.21.15.77:34172) 2024-03-16T10:24:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76dffff640 1 -- 172.21.15.77:0/3181066427 learned_addr learned my addr 172.21.15.77:0/3181066427 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.640+0000 7f76e562f640 1 --2- 172.21.15.77:0/3181066427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07abea0 0x55c3f07ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.641+0000 7f76e4e2e640 1 -- 172.21.15.77:0/3181066427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07abea0 msgr2=0x55c3f07ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.641+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/3181066427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07abea0 0x55c3f07ae330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.641+0000 7f76e4e2e640 1 -- 172.21.15.77:0/3181066427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6d30 msgr2=0x55c3f07ab830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.641+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/3181066427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6d30 0x55c3f07ab830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.641+0000 7f76e4e2e640 1 -- 172.21.15.77:0/3181066427 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3f06cc460 con 0x55c3f07a6340 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.641+0000 7f76dffff640 1 --2- 172.21.15.77:0/3181066427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6d30 0x55c3f07ab830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.641+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/3181066427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07a6340 0x55c3f07a6760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f76d402c170 tx=0x7f76d403e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be9f95e6c3d95a7f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.641+0000 7f76caffd640 1 -- 172.21.15.77:0/3181066427 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d404c030 con 0x55c3f07a6340 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.641+0000 7f76caffd640 1 -- 172.21.15.77:0/3181066427 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d4005530 con 0x55c3f07a6340 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.641+0000 7f76caffd640 1 -- 172.21.15.77:0/3181066427 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d4004d50 con 0x55c3f07a6340 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.642+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3181066427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07a6340 msgr2=0x55c3f07a6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.642+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/3181066427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07a6340 0x55c3f07a6760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f76d402c170 tx=0x7f76d403e000 comp rx=0 tx=0).stop 2024-03-16T10:24:04.649 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.642+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3181066427 shutdown_connections 2024-03-16T10:24:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.642+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/3181066427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07abea0 0x55c3f07ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.642+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/3181066427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6d30 0x55c3f07ab830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.642+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/3181066427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07a6340 0x55c3f07a6760 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.642+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3181066427 >> 172.21.15.77:0/3181066427 conn(0x55c3f07a18b0 msgr2=0x55c3f07a1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.642+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3181066427 shutdown_connections 2024-03-16T10:24:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.642+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3181066427 wait complete. 2024-03-16T10:24:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.642+0000 7f76e9c7d740 1 -- start start 2024-03-16T10:24:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.642+0000 7f76e9c7d740 1 Processor -- start 2024-03-16T10:24:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e9c7d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 0x55c3f07b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e9c7d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e9c7d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07abea0 0x55c3f07b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e9c7d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c3f07aedf0 con 0x55c3f07a6340 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e9c7d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c3f06c7d70 con 0x55c3f07a6d30 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e9c7d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c3f07aef70 con 0x55c3f07abea0 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e4e2e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 0x55c3f07b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76dffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e4e2e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 0x55c3f07b05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34182/0 (socket says 172.21.15.77:34182) 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e4e2e640 1 -- 172.21.15.77:0/1305711314 learned_addr learned my addr 172.21.15.77:0/1305711314 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76dffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:38804/0 (socket says 172.21.15.77:38804) 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e562f640 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07abea0 0x55c3f07b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e9c7d740 10 client.0 _ll_register_callbacks cb 0x55c3f041fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76dffff640 1 -- 172.21.15.77:0/1305711314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07abea0 msgr2=0x55c3f07b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76dffff640 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07abea0 0x55c3f07b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76dffff640 1 -- 172.21.15.77:0/1305711314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 msgr2=0x55c3f07b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76dffff640 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 0x55c3f07b05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76dffff640 1 -- 172.21.15.77:0/1305711314 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3f0420210 con 0x55c3f07a6d30 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e562f640 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07abea0 0x55c3f07b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 0x55c3f07b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1305711314 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76d4005530 con 0x55c3f07a6d30 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76dffff640 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b0b30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x55c3f07a7130 tx=0x7f76cc012b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3b24fc3d05f723c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.643+0000 7f76bffff640 10 client.26999 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76bffff640 1 -- 172.21.15.77:0/1305711314 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76cc004050 con 0x55c3f07a6d30 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76bffff640 1 -- 172.21.15.77:0/1305711314 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f76cc0041f0 con 0x55c3f07a6d30 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76bffff640 1 -- 172.21.15.77:0/1305711314 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f76cc004430 con 0x55c3f07a6d30 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76e9c7d740 10 client.26999 fetch_fsmap learned FSMap version 39 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1305711314 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({fsmap=39}) -- 0x55c3f04ec620 con 0x55c3f07a6d30 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76bffff640 1 -- 172.21.15.77:0/1305711314 <== mon.2 v2:172.21.15.192:3300/0 4 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f76cc00f8f0 con 0x55c3f07a6d30 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76e9c7d740 10 client.26999 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76e9c7d740 10 client.26999 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76e9c7d740 20 client.26999 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c3f07bc2d0 0x55c3f07be6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76e9c7d740 4 client.26999 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1305711314 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3f0718f80 con 0x55c3f07bc2d0 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.644+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c3f07bc2d0 0x55c3f07be6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.645+0000 7f76bffff640 10 client.26999 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.645+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c3f07bc2d0 0x55c3f07be6c0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d3b53462ec43a0d server_cookie=57152e1a02b80170 in_seq=0 out_seq=0 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76bffff640 1 -- 172.21.15.77:0/1305711314 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c3f0718f80 con 0x55c3f07bc2d0 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76bffff640 10 client.26999 handle_command_reply: tid=0 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76e9c7d740 1 client.26999 shutdown 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1305711314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 msgr2=0x55c3f07b0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b0b30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x55c3f07a7130 tx=0x7f76cc012b40 comp rx=0 tx=0).stop 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1305711314 shutdown_connections 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07abea0 0x55c3f07b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b0b30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c3f07bc2d0 0x55c3f07be6c0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/1305711314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 0x55c3f07b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76bffff640 0 client.26999 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1305711314 >> 172.21.15.77:0/1305711314 conn(0x55c3f07a18b0 msgr2=0x55c3f07a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1305711314 shutdown_connections 2024-03-16T10:24:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1305711314 wait complete. 2024-03-16T10:24:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.646+0000 7f76e9c7d740 20 client.26999 trim_cache size 0 max 16384 2024-03-16T10:24:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.661+0000 7f76e9c7d740 1 Processor -- start 2024-03-16T10:24:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.661+0000 7f76e9c7d740 1 -- start start 2024-03-16T10:24:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.661+0000 7f76e9c7d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 0x55c3f07ae290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.661+0000 7f76e9c7d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.661+0000 7f76e9c7d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07abea0 0x55c3f07a5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.661+0000 7f76e9c7d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f76e01a68a0 con 0x55c3f07a6340 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.661+0000 7f76e9c7d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f76e01a83e0 con 0x55c3f07a6d30 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.661+0000 7f76e9c7d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76e01a6f70 con 0x55c3f07abea0 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76e4e2e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 0x55c3f07ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76e4e2e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 0x55c3f07ae290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34194/0 (socket says 172.21.15.77:34194) 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76e4e2e640 1 -- 172.21.15.77:0/1186735936 learned_addr learned my addr 172.21.15.77:0/1186735936 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76e562f640 1 --2- 172.21.15.77:0/1186735936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07abea0 0x55c3f07a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76dffff640 1 --2- 172.21.15.77:0/1186735936 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76e4e2e640 1 -- 172.21.15.77:0/1186735936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07abea0 msgr2=0x55c3f07a5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/1186735936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07abea0 0x55c3f07a5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76e4e2e640 1 -- 172.21.15.77:0/1186735936 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 msgr2=0x55c3f07b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/1186735936 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76e4e2e640 1 -- 172.21.15.77:0/1186735936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3f07a7b40 con 0x55c3f07a6340 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/1186735936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 0x55c3f07ae290 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f76d40096c0 tx=0x7f76d402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfdd7d3bcd1eb957 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76ca7fc640 1 -- 172.21.15.77:0/1186735936 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d4041050 con 0x55c3f07a6340 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76ca7fc640 1 -- 172.21.15.77:0/1186735936 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d405e560 con 0x55c3f07a6340 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.662+0000 7f76ca7fc640 1 -- 172.21.15.77:0/1186735936 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d404c040 con 0x55c3f07a6340 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.663+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1186735936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 msgr2=0x55c3f07ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.663+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/1186735936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 0x55c3f07ae290 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f76d40096c0 tx=0x7f76d402bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.663+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1186735936 shutdown_connections 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.663+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/1186735936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07abea0 0x55c3f07a5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.663+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/1186735936 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.663+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/1186735936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07a6340 0x55c3f07ae290 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.663+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1186735936 >> 172.21.15.77:0/1186735936 conn(0x55c3f07a18b0 msgr2=0x55c3f07a4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.663+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1186735936 shutdown_connections 2024-03-16T10:24:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.663+0000 7f76e9c7d740 1 -- 172.21.15.77:0/1186735936 wait complete. 2024-03-16T10:24:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e9c7d740 1 -- start start 2024-03-16T10:24:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e9c7d740 1 Processor -- start 2024-03-16T10:24:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e9c7d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07a6340 0x55c3f07b55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e9c7d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e9c7d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07abea0 0x55c3f07b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e9c7d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f76e01a7d10 con 0x55c3f07abea0 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e9c7d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f76e01a7640 con 0x55c3f07a6d30 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e9c7d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76e01a48a0 con 0x55c3f07a6340 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e9c7d740 10 client.0 _ll_register_callbacks cb 0x7f76cc009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e562f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07abea0 0x55c3f07b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e562f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07abea0 0x55c3f07b6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:34200/0 (socket says 172.21.15.77:34200) 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e562f640 1 -- 172.21.15.77:0/3657658195 learned_addr learned my addr 172.21.15.77:0/3657658195 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76dffff640 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.664+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07a6340 0x55c3f07b55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e562f640 1 -- 172.21.15.77:0/3657658195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07a6340 msgr2=0x55c3f07b55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e562f640 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07a6340 0x55c3f07b55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e562f640 1 -- 172.21.15.77:0/3657658195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 msgr2=0x55c3f07b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e562f640 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e562f640 1 -- 172.21.15.77:0/3657658195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3f07b7130 con 0x55c3f07abea0 2024-03-16T10:24:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e562f640 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07abea0 0x55c3f07b6b40 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f76d800b760 tx=0x7f76d80bb170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e18f716b9e579b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76bffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76bffff640 1 -- 172.21.15.77:0/3657658195 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d8020040 con 0x55c3f07abea0 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76bffff640 1 -- 172.21.15.77:0/3657658195 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f76d8097820 con 0x55c3f07abea0 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3657658195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76d405e560 con 0x55c3f07abea0 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76bffff640 1 -- 172.21.15.77:0/3657658195 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f76d807e050 con 0x55c3f07abea0 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e9c7d740 10 client.17226 fetch_fsmap learned FSMap version 39 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3657658195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55c3f0822a10 con 0x55c3f07abea0 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76bffff640 1 -- 172.21.15.77:0/3657658195 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d405e560 con 0x55c3f07abea0 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76bffff640 1 -- 172.21.15.77:0/3657658195 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 39) ==== 8370+0+0 (secure 0 0 0) 0x7f76d809a070 con 0x55c3f07abea0 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e9c7d740 10 client.17226 fetch_fsmap finished waiting for FSMap version 39 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e9c7d740 10 client.17226 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e9c7d740 20 client.17226 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c3f0823400 0x55c3f08257f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e9c7d740 4 client.17226 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.665+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3657658195 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c3f07132d0 con 0x55c3f0823400 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.666+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c3f0823400 0x55c3f08257f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.666+0000 7f76bffff640 10 client.17226 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.666+0000 7f76e4e2e640 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c3f0823400 0x55c3f08257f0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b3513b4866f7a6c server_cookie=83bfbe2d33c052b1 in_seq=0 out_seq=0 2024-03-16T10:24:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76bffff640 1 -- 172.21.15.77:0/3657658195 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c3f07132d0 con 0x55c3f0823400 2024-03-16T10:24:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76bffff640 10 client.17226 handle_command_reply: tid=0 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76e9c7d740 1 client.17226 shutdown 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3657658195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07abea0 msgr2=0x55c3f07b6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07abea0 0x55c3f07b6b40 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f76d800b760 tx=0x7f76d80bb170 comp rx=0 tx=0).stop 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3657658195 shutdown_connections 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55c3f0823400 0x55c3f08257f0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3f07abea0 0x55c3f07b6b40 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76bffff640 0 client.17226 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c3f07a6d30 0x55c3f07b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76e9c7d740 1 --2- 172.21.15.77:0/3657658195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3f07a6340 0x55c3f07b55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3657658195 >> 172.21.15.77:0/3657658195 conn(0x55c3f07a18b0 msgr2=0x55c3f07a53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3657658195 shutdown_connections 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76e9c7d740 1 -- 172.21.15.77:0/3657658195 wait complete. 2024-03-16T10:24:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.674+0000 7f76e9c7d740 20 client.17226 trim_cache size 0 max 16384 2024-03-16T10:24:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:24:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:24:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:24:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:24:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.675+0000 7f76e7092640 1 -- 172.21.15.77:0/1218746703 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76d00a28a0 msgr2=0x7f76d00a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.675+0000 7f76e7092640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76d00a28a0 0x7f76d00a4d60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f76d4000c00 tx=0x7f76d4005fd0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.675+0000 7f76e7092640 1 -- 172.21.15.77:0/1218746703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76e01022b0 msgr2=0x7f76e019dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.675+0000 7f76e7092640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76e01022b0 0x7f76e019dfc0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f76d8009fd0 tx=0x7f76d800b260 comp rx=0 tx=0).stop 2024-03-16T10:24:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.675+0000 7f76e7092640 1 -- 172.21.15.77:0/1218746703 shutdown_connections 2024-03-16T10:24:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.675+0000 7f76e7092640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76e01022b0 0x7f76e019dfc0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.675+0000 7f76e7092640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76d00a28a0 0x7f76d00a4d60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.675+0000 7f76e7092640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76e01018f0 0x7f76e0196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.675+0000 7f76e7092640 1 --2- 172.21.15.77:0/1218746703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76e0100700 0x7f76e0196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.675+0000 7f76e7092640 1 -- 172.21.15.77:0/1218746703 >> 172.21.15.77:0/1218746703 conn(0x7f76e00fbc70 msgr2=0x7f76e0104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.675+0000 7f76e7092640 1 -- 172.21.15.77:0/1218746703 shutdown_connections 2024-03-16T10:24:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:04.675+0000 7f76e7092640 1 -- 172.21.15.77:0/1218746703 wait complete. 2024-03-16T10:24:06.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:05 smithi077 ceph-mon[32400]: pgmap v561: 97 pgs: 97 active+clean; 760 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 990 KiB/s wr, 609 op/s 2024-03-16T10:24:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:05 smithi193 ceph-mon[36311]: pgmap v561: 97 pgs: 97 active+clean; 760 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 990 KiB/s wr, 609 op/s 2024-03-16T10:24:06.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:05 smithi192 ceph-mon[36354]: pgmap v561: 97 pgs: 97 active+clean; 760 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 990 KiB/s wr, 609 op/s 2024-03-16T10:24:08.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:07 smithi077 ceph-mon[32400]: pgmap v562: 97 pgs: 97 active+clean; 755 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 948 KiB/s wr, 612 op/s 2024-03-16T10:24:08.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:07 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:24:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:07 smithi193 ceph-mon[36311]: pgmap v562: 97 pgs: 97 active+clean; 755 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 948 KiB/s wr, 612 op/s 2024-03-16T10:24:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:07 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:24:08.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:07 smithi192 ceph-mon[36354]: pgmap v562: 97 pgs: 97 active+clean; 755 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 948 KiB/s wr, 612 op/s 2024-03-16T10:24:08.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:07 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:24:10.218 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:09 smithi192 ceph-mon[36354]: pgmap v563: 97 pgs: 97 active+clean; 747 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 867 op/s 2024-03-16T10:24:10.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:09 smithi077 ceph-mon[32400]: pgmap v563: 97 pgs: 97 active+clean; 747 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 867 op/s 2024-03-16T10:24:10.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:09 smithi193 ceph-mon[36311]: pgmap v563: 97 pgs: 97 active+clean; 747 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 867 op/s 2024-03-16T10:24:12.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:11 smithi077 ceph-mon[32400]: pgmap v564: 97 pgs: 97 active+clean; 742 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 995 KiB/s wr, 860 op/s 2024-03-16T10:24:12.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:11 smithi193 ceph-mon[36311]: pgmap v564: 97 pgs: 97 active+clean; 742 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 995 KiB/s wr, 860 op/s 2024-03-16T10:24:12.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:11 smithi192 ceph-mon[36354]: pgmap v564: 97 pgs: 97 active+clean; 742 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 995 KiB/s wr, 860 op/s 2024-03-16T10:24:13.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:12 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:13.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:12 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:13.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:12 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:14.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:13 smithi077 ceph-mon[32400]: pgmap v565: 97 pgs: 97 active+clean; 732 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 998 op/s 2024-03-16T10:24:14.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:13 smithi193 ceph-mon[36311]: pgmap v565: 97 pgs: 97 active+clean; 732 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 998 op/s 2024-03-16T10:24:14.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:13 smithi192 ceph-mon[36354]: pgmap v565: 97 pgs: 97 active+clean; 732 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 998 op/s 2024-03-16T10:24:15.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:14 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:24:15.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:14 smithi077 ceph-mon[32400]: pgmap v566: 97 pgs: 97 active+clean; 725 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 971 op/s 2024-03-16T10:24:15.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:14 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:24:15.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:14 smithi193 ceph-mon[36311]: pgmap v566: 97 pgs: 97 active+clean; 725 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 971 op/s 2024-03-16T10:24:15.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:14 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:24:15.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:14 smithi192 ceph-mon[36354]: pgmap v566: 97 pgs: 97 active+clean; 725 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 971 op/s 2024-03-16T10:24:17.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.780+0000 7f9596337640 1 -- 172.21.15.77:0/2961391410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9590100700 msgr2=0x7f9590100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:17.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.780+0000 7f9596337640 1 --2- 172.21.15.77:0/2961391410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9590100700 0x7f9590100b20 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9580009900 tx=0x7f958002bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:17.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.780+0000 7f9596337640 1 -- 172.21.15.77:0/2961391410 shutdown_connections 2024-03-16T10:24:17.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.780+0000 7f9596337640 1 --2- 172.21.15.77:0/2961391410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95901022b0 0x7f9590108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.780+0000 7f9596337640 1 --2- 172.21.15.77:0/2961391410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95901018f0 0x7f9590101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.780+0000 7f9596337640 1 --2- 172.21.15.77:0/2961391410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9590100700 0x7f9590100b20 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.780+0000 7f9596337640 1 -- 172.21.15.77:0/2961391410 >> 172.21.15.77:0/2961391410 conn(0x7f95900fbc70 msgr2=0x7f95900fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.780+0000 7f9596337640 1 -- 172.21.15.77:0/2961391410 shutdown_connections 2024-03-16T10:24:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.780+0000 7f9596337640 1 -- 172.21.15.77:0/2961391410 wait complete. 2024-03-16T10:24:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.781+0000 7f9596337640 1 Processor -- start 2024-03-16T10:24:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.781+0000 7f9596337640 1 -- start start 2024-03-16T10:24:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.781+0000 7f9596337640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9590100700 0x7f9590197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.781+0000 7f9596337640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95901018f0 0x7f9590197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.781+0000 7f9596337640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95901022b0 0x7f959019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.781+0000 7f9596337640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f959010d4a0 con 0x7f95901018f0 2024-03-16T10:24:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.781+0000 7f9596337640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f959010d320 con 0x7f9590100700 2024-03-16T10:24:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.781+0000 7f9596337640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f959010d620 con 0x7f95901022b0 2024-03-16T10:24:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.781+0000 7f958f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95901018f0 0x7f9590197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.781+0000 7f958f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95901018f0 0x7f9590197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38640/0 (socket says 172.21.15.77:38640) 2024-03-16T10:24:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.781+0000 7f958ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9590100700 0x7f9590197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.781+0000 7f958f7fe640 1 -- 172.21.15.77:0/3122035712 learned_addr learned my addr 172.21.15.77:0/3122035712 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.782+0000 7f95948d4640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95901022b0 0x7f959019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.782+0000 7f958f7fe640 1 -- 172.21.15.77:0/3122035712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95901022b0 msgr2=0x7f959019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.782+0000 7f958f7fe640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95901022b0 0x7f959019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.782+0000 7f958f7fe640 1 -- 172.21.15.77:0/3122035712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9590100700 msgr2=0x7f9590197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.782+0000 7f958f7fe640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9590100700 0x7f9590197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.782+0000 7f958f7fe640 1 -- 172.21.15.77:0/3122035712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959019f6c0 con 0x7f95901018f0 2024-03-16T10:24:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.782+0000 7f958f7fe640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95901018f0 0x7f9590197fe0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f958400e9f0 tx=0x7f958400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.782+0000 7f958d7fa640 1 -- 172.21.15.77:0/3122035712 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958400cd80 con 0x7f95901018f0 2024-03-16T10:24:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.782+0000 7f9596337640 1 -- 172.21.15.77:0/3122035712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959019f9b0 con 0x7f95901018f0 2024-03-16T10:24:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.782+0000 7f958d7fa640 1 -- 172.21.15.77:0/3122035712 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9584010910 con 0x7f95901018f0 2024-03-16T10:24:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.782+0000 7f958d7fa640 1 -- 172.21.15.77:0/3122035712 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95840188f0 con 0x7f95901018f0 2024-03-16T10:24:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.782+0000 7f9596337640 1 -- 172.21.15.77:0/3122035712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f959019ff00 con 0x7f95901018f0 2024-03-16T10:24:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f9598ffa740 1 Processor -- start 2024-03-16T10:24:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f9598ffa740 1 -- start start 2024-03-16T10:24:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f9598ffa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 0x564369d05760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f9598ffa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d05d30 0x564369d0a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f9598ffa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d0aea0 0x564369d0d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f9598ffa740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564369c6aa40 con 0x564369d0aea0 2024-03-16T10:24:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f9598ffa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564369c13920 con 0x564369d05340 2024-03-16T10:24:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f9598ffa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564369c66ae0 con 0x564369d05d30 2024-03-16T10:24:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f95948d4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 0x564369d05760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f95948d4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 0x564369d05760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42676/0 (socket says 172.21.15.77:42676) 2024-03-16T10:24:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f95948d4640 1 -- 172.21.15.77:0/3785186486 learned_addr learned my addr 172.21.15.77:0/3785186486 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f958ffff640 1 --2- 172.21.15.77:0/3785186486 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d05d30 0x564369d0a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f958d7fa640 1 -- 172.21.15.77:0/3122035712 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f9584010ac0 con 0x7f95901018f0 2024-03-16T10:24:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f95948d4640 1 -- 172.21.15.77:0/3785186486 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d05d30 msgr2=0x564369d0a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f95948d4640 1 --2- 172.21.15.77:0/3785186486 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d05d30 0x564369d0a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f95948d4640 1 -- 172.21.15.77:0/3785186486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d0aea0 msgr2=0x564369d0d330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:24:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f95948d4640 1 --2- 172.21.15.77:0/3785186486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d0aea0 0x564369d0d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.785+0000 7f95948d4640 1 -- 172.21.15.77:0/3785186486 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564369c2b460 con 0x564369d05340 2024-03-16T10:24:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f95948d4640 1 --2- 172.21.15.77:0/3785186486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 0x564369d05760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9588009fd0 tx=0x7f95880118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ba9a192acb906a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f95767fc640 1 -- 172.21.15.77:0/3785186486 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9588004d20 con 0x564369d05340 2024-03-16T10:24:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f958d7fa640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f95780a28a0 0x7f95780a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f95767fc640 1 -- 172.21.15.77:0/3785186486 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9588004050 con 0x564369d05340 2024-03-16T10:24:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f95767fc640 1 -- 172.21.15.77:0/3785186486 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9588005280 con 0x564369d05340 2024-03-16T10:24:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f958ffff640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f95780a28a0 0x7f95780a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f958d7fa640 1 -- 172.21.15.77:0/3122035712 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f95840ced80 con 0x7f95901018f0 2024-03-16T10:24:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f9598ffa740 1 -- 172.21.15.77:0/3785186486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 msgr2=0x564369d05760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f958ffff640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f95780a28a0 0x7f95780a4d60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9580000c00 tx=0x7f9580005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f9598ffa740 1 --2- 172.21.15.77:0/3785186486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 0x564369d05760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9588009fd0 tx=0x7f95880118d0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f9598ffa740 1 -- 172.21.15.77:0/3785186486 shutdown_connections 2024-03-16T10:24:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f9598ffa740 1 --2- 172.21.15.77:0/3785186486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d0aea0 0x564369d0d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f9598ffa740 1 --2- 172.21.15.77:0/3785186486 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d05d30 0x564369d0a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f9598ffa740 1 --2- 172.21.15.77:0/3785186486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 0x564369d05760 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f9598ffa740 1 -- 172.21.15.77:0/3785186486 >> 172.21.15.77:0/3785186486 conn(0x564369d008b0 msgr2=0x564369d00cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.786+0000 7f9598ffa740 1 -- 172.21.15.77:0/3785186486 shutdown_connections 2024-03-16T10:24:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f9598ffa740 1 -- 172.21.15.77:0/3785186486 wait complete. 2024-03-16T10:24:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f9598ffa740 1 -- start start 2024-03-16T10:24:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f9598ffa740 1 Processor -- start 2024-03-16T10:24:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f9598ffa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 0x564369d0f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f9598ffa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05d30 0x564369d0fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f9598ffa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d0aea0 0x564369d16bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f9598ffa740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564369d0de50 con 0x564369d05d30 2024-03-16T10:24:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f9598ffa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564369c26d70 con 0x564369d05340 2024-03-16T10:24:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f9598ffa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564369d0dfd0 con 0x564369d0aea0 2024-03-16T10:24:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f9598ffa740 10 client.0 _ll_register_callbacks cb 0x56436997ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f958f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05d30 0x564369d0fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f958f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05d30 0x564369d0fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38656/0 (socket says 172.21.15.77:38656) 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f958ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 0x564369d0f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f958f7fe640 1 -- 172.21.15.77:0/2218518926 learned_addr learned my addr 172.21.15.77:0/2218518926 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.787+0000 7f95948d4640 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d0aea0 0x564369d16bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f958ffff640 1 -- 172.21.15.77:0/2218518926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d0aea0 msgr2=0x564369d16bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f958ffff640 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d0aea0 0x564369d16bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f958ffff640 1 -- 172.21.15.77:0/2218518926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05d30 msgr2=0x564369d0fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f958ffff640 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05d30 0x564369d0fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f958ffff640 1 -- 172.21.15.77:0/2218518926 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56436997f210 con 0x564369d05340 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f958f7fe640 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05d30 0x564369d0fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f95948d4640 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d0aea0 0x564369d16bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f958ffff640 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 0x564369d0f630 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f958002c170 tx=0x7f9580009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8023cf2b5d86f1f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f956f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:24:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f956f7fe640 1 -- 172.21.15.77:0/2218518926 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958004c070 con 0x564369d05340 2024-03-16T10:24:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f9598ffa740 1 -- 172.21.15.77:0/2218518926 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9588004050 con 0x564369d05340 2024-03-16T10:24:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f956f7fe640 1 -- 172.21.15.77:0/2218518926 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f9580005050 con 0x564369d05340 2024-03-16T10:24:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f956f7fe640 1 -- 172.21.15.77:0/2218518926 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f9580003e50 con 0x564369d05340 2024-03-16T10:24:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f9598ffa740 10 client.27017 fetch_fsmap learned FSMap version 41 2024-03-16T10:24:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.788+0000 7f9598ffa740 1 -- 172.21.15.77:0/2218518926 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x564369a4b620 con 0x564369d05340 2024-03-16T10:24:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.789+0000 7f956f7fe640 1 -- 172.21.15.77:0/2218518926 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9588004050 con 0x564369d05340 2024-03-16T10:24:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.789+0000 7f956f7fe640 1 -- 172.21.15.77:0/2218518926 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7f9580051460 con 0x564369d05340 2024-03-16T10:24:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.789+0000 7f9598ffa740 10 client.27017 fetch_fsmap finished waiting for FSMap version 41 2024-03-16T10:24:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.789+0000 7f9598ffa740 10 client.27017 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.789+0000 7f9598ffa740 20 client.27017 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.789+0000 7f9598ffa740 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564369d1b840 0x564369d1dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.789+0000 7f9598ffa740 4 client.27017 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.789+0000 7f9598ffa740 1 -- 172.21.15.77:0/2218518926 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564369c77f80 con 0x564369d1b840 2024-03-16T10:24:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.789+0000 7f958f7fe640 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564369d1b840 0x564369d1dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.790+0000 7f956f7fe640 10 client.27017 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.790+0000 7f958f7fe640 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564369d1b840 0x564369d1dcc0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=695761d4aa53b939 server_cookie=498929c9f209c46b in_seq=0 out_seq=0 2024-03-16T10:24:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.790+0000 7f956f7fe640 1 -- 172.21.15.77:0/2218518926 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564369c77f80 con 0x564369d1b840 2024-03-16T10:24:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.790+0000 7f956f7fe640 10 client.27017 handle_command_reply: tid=0 2024-03-16T10:24:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.790+0000 7f9598ffa740 1 client.27017 shutdown 2024-03-16T10:24:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.790+0000 7f9598ffa740 1 -- 172.21.15.77:0/2218518926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 msgr2=0x564369d0f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.790+0000 7f9598ffa740 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 0x564369d0f630 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f958002c170 tx=0x7f9580009260 comp rx=0 tx=0).stop 2024-03-16T10:24:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.790+0000 7f9598ffa740 1 -- 172.21.15.77:0/2218518926 shutdown_connections 2024-03-16T10:24:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.790+0000 7f9598ffa740 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d0aea0 0x564369d16bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.790+0000 7f9598ffa740 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564369d1b840 0x564369d1dcc0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.790+0000 7f9598ffa740 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05d30 0x564369d0fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.791+0000 7f9598ffa740 1 --2- 172.21.15.77:0/2218518926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05340 0x564369d0f630 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.791+0000 7f956f7fe640 0 client.27017 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.791+0000 7f9598ffa740 1 -- 172.21.15.77:0/2218518926 >> 172.21.15.77:0/2218518926 conn(0x564369d008b0 msgr2=0x564369cffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.791+0000 7f9598ffa740 1 -- 172.21.15.77:0/2218518926 shutdown_connections 2024-03-16T10:24:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.791+0000 7f9598ffa740 1 -- 172.21.15.77:0/2218518926 wait complete. 2024-03-16T10:24:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.791+0000 7f9598ffa740 20 client.27017 trim_cache size 0 max 16384 2024-03-16T10:24:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.806+0000 7f9598ffa740 1 Processor -- start 2024-03-16T10:24:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.806+0000 7f9598ffa740 1 -- start start 2024-03-16T10:24:17.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f9598ffa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d008b0 0x564369d0fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f9598ffa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 0x564369d16b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f9598ffa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05d30 0x564369cfba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f9598ffa740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f95901a8ae0 con 0x564369d05340 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f9598ffa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f95901aa650 con 0x564369d05d30 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f9598ffa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f95901a91e0 con 0x564369d008b0 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f958f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 0x564369d16b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f958f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 0x564369d16b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38672/0 (socket says 172.21.15.77:38672) 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f958f7fe640 1 -- 172.21.15.77:0/3311267841 learned_addr learned my addr 172.21.15.77:0/3311267841 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f958ffff640 1 --2- 172.21.15.77:0/3311267841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d008b0 0x564369d0fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f95948d4640 1 --2- 172.21.15.77:0/3311267841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05d30 0x564369cfba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f958f7fe640 1 -- 172.21.15.77:0/3311267841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d008b0 msgr2=0x564369d0fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f958f7fe640 1 --2- 172.21.15.77:0/3311267841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d008b0 0x564369d0fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f958f7fe640 1 -- 172.21.15.77:0/3311267841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05d30 msgr2=0x564369cfba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:17.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f958f7fe640 1 --2- 172.21.15.77:0/3311267841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05d30 0x564369cfba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f958f7fe640 1 -- 172.21.15.77:0/3311267841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564369cf2140 con 0x564369d05340 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f958f7fe640 1 --2- 172.21.15.77:0/3311267841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 0x564369d16b70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f958409b7f0 tx=0x7f958400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdcd2d1e5b687632 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f9575ffb640 1 -- 172.21.15.77:0/3311267841 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584099b60 con 0x564369d05340 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f9575ffb640 1 -- 172.21.15.77:0/3311267841 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95840860b0 con 0x564369d05340 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.807+0000 7f9575ffb640 1 -- 172.21.15.77:0/3311267841 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958409cae0 con 0x564369d05340 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.808+0000 7f9598ffa740 1 -- 172.21.15.77:0/3311267841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 msgr2=0x564369d16b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.808+0000 7f9598ffa740 1 --2- 172.21.15.77:0/3311267841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 0x564369d16b70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f958409b7f0 tx=0x7f958400cf20 comp rx=0 tx=0).stop 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.808+0000 7f9598ffa740 1 -- 172.21.15.77:0/3311267841 shutdown_connections 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.808+0000 7f9598ffa740 1 --2- 172.21.15.77:0/3311267841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05d30 0x564369cfba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.808+0000 7f9598ffa740 1 --2- 172.21.15.77:0/3311267841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 0x564369d16b70 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.808+0000 7f9598ffa740 1 --2- 172.21.15.77:0/3311267841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d008b0 0x564369d0fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.808+0000 7f9598ffa740 1 -- 172.21.15.77:0/3311267841 >> 172.21.15.77:0/3311267841 conn(0x564369d06930 msgr2=0x564369d00c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.808+0000 7f9598ffa740 1 -- 172.21.15.77:0/3311267841 shutdown_connections 2024-03-16T10:24:17.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.808+0000 7f9598ffa740 1 -- 172.21.15.77:0/3311267841 wait complete. 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.808+0000 7f9598ffa740 1 -- start start 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.808+0000 7f9598ffa740 1 Processor -- start 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f9598ffa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d008b0 0x564369d14200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f9598ffa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 0x564369d14740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f9598ffa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05d30 0x564369d15750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f9598ffa740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f95901a9f80 con 0x564369d05340 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f9598ffa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f95901a98b0 con 0x564369d05d30 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f9598ffa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f95901a6ae0 con 0x564369d008b0 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f9598ffa740 10 client.0 _ll_register_callbacks cb 0x7f9580009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f958f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 0x564369d14740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f958f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 0x564369d14740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38678/0 (socket says 172.21.15.77:38678) 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f958ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d008b0 0x564369d14200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f95948d4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05d30 0x564369d15750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f958f7fe640 1 -- 172.21.15.77:0/1301463337 learned_addr learned my addr 172.21.15.77:0/1301463337 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:17.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f958f7fe640 1 -- 172.21.15.77:0/1301463337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d008b0 msgr2=0x564369d14200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f958f7fe640 1 --2- 172.21.15.77:0/1301463337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d008b0 0x564369d14200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f958f7fe640 1 -- 172.21.15.77:0/1301463337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05d30 msgr2=0x564369d15750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f958f7fe640 1 --2- 172.21.15.77:0/1301463337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05d30 0x564369d15750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f958f7fe640 1 -- 172.21.15.77:0/1301463337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564369d15d40 con 0x564369d05340 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f958f7fe640 1 --2- 172.21.15.77:0/1301463337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 0x564369d14740 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f958401f040 tx=0x7f95840ccae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f18afda5f2068ae server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f956f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f956f7fe640 1 -- 172.21.15.77:0/1301463337 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95840b3070 con 0x564369d05340 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f956f7fe640 1 -- 172.21.15.77:0/1301463337 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f9584099b60 con 0x564369d05340 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f9598ffa740 1 -- 172.21.15.77:0/1301463337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95840860b0 con 0x564369d05340 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.809+0000 7f956f7fe640 1 -- 172.21.15.77:0/1301463337 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f9584086260 con 0x564369d05340 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f9598ffa740 10 client.17241 fetch_fsmap learned FSMap version 41 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f9598ffa740 1 -- 172.21.15.77:0/1301463337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x564369d81c00 con 0x564369d05340 2024-03-16T10:24:17.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f956f7fe640 1 -- 172.21.15.77:0/1301463337 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95840860b0 con 0x564369d05340 2024-03-16T10:24:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f956f7fe640 1 -- 172.21.15.77:0/1301463337 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7f958409cae0 con 0x564369d05340 2024-03-16T10:24:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f9598ffa740 10 client.17241 fetch_fsmap finished waiting for FSMap version 41 2024-03-16T10:24:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f9598ffa740 10 client.17241 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f9598ffa740 20 client.17241 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f9598ffa740 1 --2- 172.21.15.77:0/1301463337 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564369d0aea0 0x564369d845b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f9598ffa740 4 client.17241 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:24:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f9598ffa740 1 -- 172.21.15.77:0/1301463337 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564369c722d0 con 0x564369d0aea0 2024-03-16T10:24:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f958ffff640 1 --2- 172.21.15.77:0/1301463337 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564369d0aea0 0x564369d845b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f958ffff640 1 --2- 172.21.15.77:0/1301463337 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564369d0aea0 0x564369d845b0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40e36940209975a8 server_cookie=15bde2ab73e1a007 in_seq=0 out_seq=0 2024-03-16T10:24:17.817 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.810+0000 7f956f7fe640 10 client.17241 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.888+0000 7f956f7fe640 1 -- 172.21.15.77:0/1301463337 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564369c722d0 con 0x564369d0aea0 2024-03-16T10:24:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.888+0000 7f956f7fe640 10 client.17241 handle_command_reply: tid=0 2024-03-16T10:24:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9598ffa740 1 client.17241 shutdown 2024-03-16T10:24:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9598ffa740 1 -- 172.21.15.77:0/1301463337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 msgr2=0x564369d14740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9598ffa740 1 --2- 172.21.15.77:0/1301463337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 0x564369d14740 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f958401f040 tx=0x7f95840ccae0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9598ffa740 1 -- 172.21.15.77:0/1301463337 shutdown_connections 2024-03-16T10:24:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9598ffa740 1 --2- 172.21.15.77:0/1301463337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564369d05d30 0x564369d15750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9598ffa740 1 --2- 172.21.15.77:0/1301463337 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x564369d0aea0 0x564369d845b0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9598ffa740 1 --2- 172.21.15.77:0/1301463337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564369d05340 0x564369d14740 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9598ffa740 1 --2- 172.21.15.77:0/1301463337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564369d008b0 0x564369d14200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f956f7fe640 0 client.17241 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:17.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9598ffa740 1 -- 172.21.15.77:0/1301463337 >> 172.21.15.77:0/1301463337 conn(0x564369d06930 msgr2=0x564369cfba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:17.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9598ffa740 1 -- 172.21.15.77:0/1301463337 shutdown_connections 2024-03-16T10:24:17.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9598ffa740 1 -- 172.21.15.77:0/1301463337 wait complete. 2024-03-16T10:24:17.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9598ffa740 20 client.17241 trim_cache size 0 max 16384 2024-03-16T10:24:17.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:24:17.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:24:17.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:24:17.891 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:24:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9596337640 1 -- 172.21.15.77:0/3122035712 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f95780a28a0 msgr2=0x7f95780a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.889+0000 7f9596337640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f95780a28a0 0x7f95780a4d60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9580000c00 tx=0x7f9580005fd0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.890+0000 7f9596337640 1 -- 172.21.15.77:0/3122035712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95901018f0 msgr2=0x7f9590197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.890+0000 7f9596337640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95901018f0 0x7f9590197fe0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f958400e9f0 tx=0x7f958400eeb0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.890+0000 7f9596337640 1 -- 172.21.15.77:0/3122035712 shutdown_connections 2024-03-16T10:24:17.892 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.890+0000 7f9596337640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f95780a28a0 0x7f95780a4d60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.890+0000 7f9596337640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95901022b0 0x7f959019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.890+0000 7f9596337640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95901018f0 0x7f9590197fe0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.890+0000 7f9596337640 1 --2- 172.21.15.77:0/3122035712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9590100700 0x7f9590197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:17.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.890+0000 7f9596337640 1 -- 172.21.15.77:0/3122035712 >> 172.21.15.77:0/3122035712 conn(0x7f95900fbc70 msgr2=0x7f9590104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:17.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.890+0000 7f9596337640 1 -- 172.21.15.77:0/3122035712 shutdown_connections 2024-03-16T10:24:17.893 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:17.890+0000 7f9596337640 1 -- 172.21.15.77:0/3122035712 wait complete. 2024-03-16T10:24:18.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:17 smithi077 ceph-mon[32400]: pgmap v567: 97 pgs: 97 active+clean; 723 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 858 op/s 2024-03-16T10:24:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:17 smithi193 ceph-mon[36311]: pgmap v567: 97 pgs: 97 active+clean; 723 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 858 op/s 2024-03-16T10:24:18.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:17 smithi192 ceph-mon[36354]: pgmap v567: 97 pgs: 97 active+clean; 723 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 858 op/s 2024-03-16T10:24:19.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:24:19.895 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:24:20.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.002+0000 7f09af1e6640 1 -- 172.21.15.77:0/2927578377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a80ffac0 msgr2=0x7f09a80ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:20.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.002+0000 7f09af1e6640 1 --2- 172.21.15.77:0/2927578377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a80ffac0 0x7f09a80ffee0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f099c009930 tx=0x7f099c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:20.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.002+0000 7f09af1e6640 1 -- 172.21.15.77:0/2927578377 shutdown_connections 2024-03-16T10:24:20.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.002+0000 7f09af1e6640 1 --2- 172.21.15.77:0/2927578377 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09a8103450 0x7f09a8109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.002+0000 7f09af1e6640 1 --2- 172.21.15.77:0/2927578377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09a8102a90 0x7f09a8102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.002+0000 7f09af1e6640 1 --2- 172.21.15.77:0/2927578377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a80ffac0 0x7f09a80ffee0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.002+0000 7f09af1e6640 1 -- 172.21.15.77:0/2927578377 >> 172.21.15.77:0/2927578377 conn(0x7f09a80fb3d0 msgr2=0x7f09a80fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:20.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.002+0000 7f09af1e6640 1 -- 172.21.15.77:0/2927578377 shutdown_connections 2024-03-16T10:24:20.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.002+0000 7f09af1e6640 1 -- 172.21.15.77:0/2927578377 wait complete. 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.002+0000 7f09af1e6640 1 Processor -- start 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.002+0000 7f09af1e6640 1 -- start start 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09af1e6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09a80ffac0 0x7f09a8102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09af1e6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09a8102a90 0x7f09a8100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09af1e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a8103450 0x7f09a8100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09af1e6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f09a810d470 con 0x7f09a8103450 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09af1e6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f09a810d2f0 con 0x7f09a80ffac0 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09af1e6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f09a810d5f0 con 0x7f09a8102a90 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09ad783640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a8103450 0x7f09a8100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09ad783640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a8103450 0x7f09a8100f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38694/0 (socket says 172.21.15.77:38694) 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09ad783640 1 -- 172.21.15.77:0/3301716351 learned_addr learned my addr 172.21.15.77:0/3301716351 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09a7fff640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09a8102a90 0x7f09a8100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09acf82640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09a80ffac0 0x7f09a8102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09ad783640 1 -- 172.21.15.77:0/3301716351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09a8102a90 msgr2=0x7f09a8100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:20.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09ad783640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09a8102a90 0x7f09a8100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09ad783640 1 -- 172.21.15.77:0/3301716351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09a80ffac0 msgr2=0x7f09a8102400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:20.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09ad783640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09a80ffac0 0x7f09a8102400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09ad783640 1 -- 172.21.15.77:0/3301716351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a8101810 con 0x7f09a8103450 2024-03-16T10:24:20.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09ad783640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a8103450 0x7f09a8100f80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f09a000eb10 tx=0x7f09a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:20.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09a5ffb640 1 -- 172.21.15.77:0/3301716351 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0004b20 con 0x7f09a8103450 2024-03-16T10:24:20.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09a5ffb640 1 -- 172.21.15.77:0/3301716351 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a0004cc0 con 0x7f09a8103450 2024-03-16T10:24:20.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09af1e6640 1 -- 172.21.15.77:0/3301716351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a81a1c50 con 0x7f09a8103450 2024-03-16T10:24:20.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.003+0000 7f09a5ffb640 1 -- 172.21.15.77:0/3301716351 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0010760 con 0x7f09a8103450 2024-03-16T10:24:20.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.004+0000 7f09af1e6640 1 -- 172.21.15.77:0/3301716351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a81a2150 con 0x7f09a8103450 2024-03-16T10:24:20.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.006+0000 7f09a5ffb640 1 -- 172.21.15.77:0/3301716351 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f09a0005020 con 0x7f09a8103450 2024-03-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.006+0000 7f09b1dd0740 1 Processor -- start 2024-03-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.006+0000 7f09b1dd0740 1 -- start start 2024-03-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.006+0000 7f09b1dd0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8e170 0x563958f8e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.006+0000 7f09b1dd0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8eb60 0x563958f93660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.006+0000 7f09b1dd0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f93cd0 0x563958f96160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.006+0000 7f09b1dd0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563958ef3500 con 0x563958f93cd0 2024-03-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.006+0000 7f09b1dd0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563958e9bbe0 con 0x563958f8eb60 2024-03-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.006+0000 7f09b1dd0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563958eef5a0 con 0x563958f8e170 2024-03-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f93cd0 0x563958f96160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09acf82640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8eb60 0x563958f93660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8e170 0x563958f8e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09a5ffb640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09980a2800 0x7f09980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8e170 0x563958f8e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:55754/0 (socket says 172.21.15.77:55754) 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09acf82640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8eb60 0x563958f93660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:42728/0 (socket says 172.21.15.77:42728) 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09a7fff640 1 -- 172.21.15.77:0/1764283009 learned_addr learned my addr 172.21.15.77:0/1764283009 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09ad783640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09980a2800 0x7f09980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09a7fff640 1 -- 172.21.15.77:0/1764283009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8eb60 msgr2=0x563958f93660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09a7fff640 1 --2- 172.21.15.77:0/1764283009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8eb60 0x563958f93660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09a7fff640 1 -- 172.21.15.77:0/1764283009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f93cd0 msgr2=0x563958f96160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09a7fff640 1 --2- 172.21.15.77:0/1764283009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f93cd0 0x563958f96160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09a7fff640 1 -- 172.21.15.77:0/1764283009 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563958eaf4b0 con 0x563958f8e170 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09a5ffb640 1 -- 172.21.15.77:0/3301716351 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f09a00cefb0 con 0x7f09a8103450 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09a7fff640 1 --2- 172.21.15.77:0/1764283009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8e170 0x563958f8e590 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0994010960 tx=0x7f0994005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8fdd0fc5b537007 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f09ad783640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09980a2800 0x7f09980a4cc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f09a00983b0 tx=0x7f09a000efd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f0992ffd640 1 -- 172.21.15.77:0/1764283009 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994016070 con 0x563958f8e170 2024-03-16T10:24:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f0992ffd640 1 -- 172.21.15.77:0/1764283009 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f099400f7c0 con 0x563958f8e170 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.007+0000 7f0992ffd640 1 -- 172.21.15.77:0/1764283009 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994019040 con 0x563958f8e170 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.008+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1764283009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8e170 msgr2=0x563958f8e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.008+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/1764283009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8e170 0x563958f8e590 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0994010960 tx=0x7f0994005f90 comp rx=0 tx=0).stop 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.008+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1764283009 shutdown_connections 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.008+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/1764283009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f93cd0 0x563958f96160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.008+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/1764283009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8eb60 0x563958f93660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.008+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/1764283009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8e170 0x563958f8e590 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.008+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1764283009 >> 172.21.15.77:0/1764283009 conn(0x563958f896e0 msgr2=0x563958f89ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.008+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1764283009 shutdown_connections 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.008+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1764283009 wait complete. 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.008+0000 7f09b1dd0740 1 -- start start 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.008+0000 7f09b1dd0740 1 Processor -- start 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09b1dd0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8e170 0x563958f98620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09b1dd0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8eb60 0x563958f98b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09b1dd0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f93cd0 0x563958f9fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09b1dd0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563958f96c20 con 0x563958f93cd0 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09b1dd0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563958eaf160 con 0x563958f8e170 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09b1dd0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563958f96da0 con 0x563958f8eb60 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09b1dd0740 10 client.0 _ll_register_callbacks cb 0x563958e93150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f93cd0 0x563958f9fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f93cd0 0x563958f9fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38712/0 (socket says 172.21.15.77:38712) 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09a7fff640 1 -- 172.21.15.77:0/4085833497 learned_addr learned my addr 172.21.15.77:0/4085833497 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09acf82640 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8eb60 0x563958f98b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09a7fff640 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8e170 0x563958f98620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09a7fff640 1 -- 172.21.15.77:0/4085833497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8eb60 msgr2=0x563958f98b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09a7fff640 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8eb60 0x563958f98b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09a7fff640 1 -- 172.21.15.77:0/4085833497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8e170 msgr2=0x563958f98620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09a7fff640 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8e170 0x563958f98620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09a7fff640 1 -- 172.21.15.77:0/4085833497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563958cd17d0 con 0x563958f93cd0 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09acf82640 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8eb60 0x563958f98b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f09a7fff640 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f93cd0 0x563958f9fba0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f09940097c0 tx=0x7f0994009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd6a7bd37e33802 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.009+0000 7f0987fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f0987fff640 1 -- 172.21.15.77:0/4085833497 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994016030 con 0x563958f93cd0 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f0987fff640 1 -- 172.21.15.77:0/4085833497 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f0994004640 con 0x563958f93cd0 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f09b1dd0740 1 -- 172.21.15.77:0/4085833497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f099400f7c0 con 0x563958f93cd0 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f0987fff640 1 -- 172.21.15.77:0/4085833497 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f0994028040 con 0x563958f93cd0 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f09b1dd0740 10 client.17253 fetch_fsmap learned FSMap version 41 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f09b1dd0740 1 -- 172.21.15.77:0/4085833497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x563958c073f0 con 0x563958f93cd0 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f0987fff640 1 -- 172.21.15.77:0/4085833497 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f099400f7c0 con 0x563958f93cd0 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f0987fff640 1 -- 172.21.15.77:0/4085833497 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7f099401f550 con 0x563958f93cd0 2024-03-16T10:24:20.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f09b1dd0740 10 client.17253 fetch_fsmap finished waiting for FSMap version 41 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f09b1dd0740 10 client.17253 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f09b1dd0740 20 client.17253 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563958fa3da0 0x563958fa61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f09b1dd0740 4 client.17253 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f09b1dd0740 1 -- 172.21.15.77:0/4085833497 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563958f00a40 con 0x563958fa3da0 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.010+0000 7f09acf82640 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563958fa3da0 0x563958fa61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.011+0000 7f0987fff640 10 client.17253 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.011+0000 7f09acf82640 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563958fa3da0 0x563958fa61d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f8994a6c4d83857 server_cookie=6254816b2aa8d709 in_seq=0 out_seq=0 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f0987fff640 1 -- 172.21.15.77:0/4085833497 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563958f00a40 con 0x563958fa3da0 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f0987fff640 10 client.17253 handle_command_reply: tid=0 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f09b1dd0740 1 client.17253 shutdown 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f09b1dd0740 1 -- 172.21.15.77:0/4085833497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f93cd0 msgr2=0x563958f9fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f93cd0 0x563958f9fba0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f09940097c0 tx=0x7f0994009be0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f09b1dd0740 1 -- 172.21.15.77:0/4085833497 shutdown_connections 2024-03-16T10:24:20.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563958fa3da0 0x563958fa61d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f93cd0 0x563958f9fba0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f0987fff640 0 client.17253 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:20.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8eb60 0x563958f98b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/4085833497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8e170 0x563958f98620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f09b1dd0740 1 -- 172.21.15.77:0/4085833497 >> 172.21.15.77:0/4085833497 conn(0x563958f896e0 msgr2=0x563958f88900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:20.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f09b1dd0740 1 -- 172.21.15.77:0/4085833497 shutdown_connections 2024-03-16T10:24:20.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f09b1dd0740 1 -- 172.21.15.77:0/4085833497 wait complete. 2024-03-16T10:24:20.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.012+0000 7f09b1dd0740 20 client.17253 trim_cache size 0 max 16384 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.029+0000 7f09b1dd0740 1 Processor -- start 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.029+0000 7f09b1dd0740 1 -- start start 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09b1dd0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f896e0 0x563958f88070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09b1dd0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f8e170 0x563958f9fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09b1dd0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8eb60 0x563958f885b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09b1dd0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81aade0 con 0x563958f8e170 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09b1dd0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81ac950 con 0x563958f896e0 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09b1dd0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81ab4e0 con 0x563958f8eb60 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09acf82640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f8e170 0x563958f9fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09acf82640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f8e170 0x563958f9fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38720/0 (socket says 172.21.15.77:38720) 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09acf82640 1 -- 172.21.15.77:0/3418200507 learned_addr learned my addr 172.21.15.77:0/3418200507 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09a7fff640 1 --2- 172.21.15.77:0/3418200507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f896e0 0x563958f88070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09a7fff640 1 --2- 172.21.15.77:0/3418200507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8eb60 0x563958f885b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09acf82640 1 -- 172.21.15.77:0/3418200507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8eb60 msgr2=0x563958f885b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09acf82640 1 --2- 172.21.15.77:0/3418200507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8eb60 0x563958f885b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09acf82640 1 -- 172.21.15.77:0/3418200507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f896e0 msgr2=0x563958f88070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09acf82640 1 --2- 172.21.15.77:0/3418200507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f896e0 0x563958f88070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09acf82640 1 -- 172.21.15.77:0/3418200507 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563958bc70f0 con 0x563958f8e170 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09acf82640 1 --2- 172.21.15.77:0/3418200507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f8e170 0x563958f9fb60 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f099c009900 tx=0x7f099c03f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfa685c6dcc3fe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09927fc640 1 -- 172.21.15.77:0/3418200507 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c046d10 con 0x563958f8e170 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09927fc640 1 -- 172.21.15.77:0/3418200507 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f099c04b070 con 0x563958f8e170 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.030+0000 7f09927fc640 1 -- 172.21.15.77:0/3418200507 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c005440 con 0x563958f8e170 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.031+0000 7f09b1dd0740 1 -- 172.21.15.77:0/3418200507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f8e170 msgr2=0x563958f9fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.031+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/3418200507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f8e170 0x563958f9fb60 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f099c009900 tx=0x7f099c03f040 comp rx=0 tx=0).stop 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.031+0000 7f09b1dd0740 1 -- 172.21.15.77:0/3418200507 shutdown_connections 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.031+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/3418200507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8eb60 0x563958f885b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.031+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/3418200507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f8e170 0x563958f9fb60 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.031+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/3418200507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f896e0 0x563958f88070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.031+0000 7f09b1dd0740 1 -- 172.21.15.77:0/3418200507 >> 172.21.15.77:0/3418200507 conn(0x563958f89e30 msgr2=0x563958f98620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.031+0000 7f09b1dd0740 1 -- 172.21.15.77:0/3418200507 shutdown_connections 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.031+0000 7f09b1dd0740 1 -- 172.21.15.77:0/3418200507 wait complete. 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.031+0000 7f09b1dd0740 1 -- start start 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.031+0000 7f09b1dd0740 1 Processor -- start 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09b1dd0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f896e0 0x563958f9d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09b1dd0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8e170 0x563958f9d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09b1dd0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8eb60 0x563958f9e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09b1dd0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81ac280 con 0x563958f896e0 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09b1dd0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81abbb0 con 0x563958f8eb60 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09b1dd0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f09a81a8de0 con 0x563958f8e170 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09b1dd0740 10 client.0 _ll_register_callbacks cb 0x7f0994009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f896e0 0x563958f9d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f896e0 0x563958f9d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38728/0 (socket says 172.21.15.77:38728) 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09a7fff640 1 -- 172.21.15.77:0/1941748807 learned_addr learned my addr 172.21.15.77:0/1941748807 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:20.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09acf82640 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8e170 0x563958f9d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09a7fff640 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8eb60 0x563958f9e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09a7fff640 1 -- 172.21.15.77:0/1941748807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8e170 msgr2=0x563958f9d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09a7fff640 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8e170 0x563958f9d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09a7fff640 1 -- 172.21.15.77:0/1941748807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8eb60 msgr2=0x563958f9e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09a7fff640 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8eb60 0x563958f9e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09a7fff640 1 -- 172.21.15.77:0/1941748807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563958eba9d0 con 0x563958f896e0 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09a7fff640 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f896e0 0x563958f9d320 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f099401fb60 tx=0x7f0994010620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484738a4d693ddc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f0987fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f0987fff640 1 -- 172.21.15.77:0/1941748807 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994014030 con 0x563958f896e0 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f0987fff640 1 -- 172.21.15.77:0/1941748807 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f099401f650 con 0x563958f896e0 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1941748807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f099c04b070 con 0x563958f896e0 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f0987fff640 1 -- 172.21.15.77:0/1941748807 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f0994016070 con 0x563958f896e0 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09b1dd0740 10 client.17259 fetch_fsmap learned FSMap version 41 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.032+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1941748807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x563958f9fee0 con 0x563958f896e0 2024-03-16T10:24:20.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.033+0000 7f0987fff640 1 -- 172.21.15.77:0/1941748807 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f099c04b070 con 0x563958f896e0 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.033+0000 7f0987fff640 1 -- 172.21.15.77:0/1941748807 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7f0994049030 con 0x563958f896e0 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.033+0000 7f09b1dd0740 10 client.17259 fetch_fsmap finished waiting for FSMap version 41 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.033+0000 7f09b1dd0740 10 client.17259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.033+0000 7f09b1dd0740 20 client.17259 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.033+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563958f93cd0 0x56395900c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.033+0000 7f09b1dd0740 4 client.17259 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.033+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1941748807 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563958f8fb70 con 0x563958f93cd0 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.033+0000 7f09acf82640 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563958f93cd0 0x56395900c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.033+0000 7f0987fff640 10 client.17259 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.033+0000 7f09acf82640 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563958f93cd0 0x56395900c7a0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4eded051d1038534 server_cookie=402a427b2c3d70f8 in_seq=0 out_seq=0 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f0987fff640 1 -- 172.21.15.77:0/1941748807 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563958f8fb70 con 0x563958f93cd0 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f0987fff640 10 client.17259 handle_command_reply: tid=0 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09b1dd0740 1 client.17259 shutdown 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1941748807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f896e0 msgr2=0x563958f9d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:20.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f896e0 0x563958f9d320 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f099401fb60 tx=0x7f0994010620 comp rx=0 tx=0).stop 2024-03-16T10:24:20.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1941748807 shutdown_connections 2024-03-16T10:24:20.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563958f8eb60 0x563958f9e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563958f8e170 0x563958f9d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563958f93cd0 0x56395900c7a0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09b1dd0740 1 --2- 172.21.15.77:0/1941748807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563958f896e0 0x563958f9d320 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f0987fff640 0 client.17259 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:20.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1941748807 >> 172.21.15.77:0/1941748807 conn(0x563958f89e30 msgr2=0x563958f8c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:20.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1941748807 shutdown_connections 2024-03-16T10:24:20.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09b1dd0740 1 -- 172.21.15.77:0/1941748807 wait complete. 2024-03-16T10:24:20.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09b1dd0740 20 client.17259 trim_cache size 0 max 16384 2024-03-16T10:24:20.041 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:24:20.041 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:24:20.042 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:24:20.042 INFO:teuthology.orchestra.run.smithi077.stdout: "be0cd9e9-c385-4436-9dce-770010f9b142": { 2024-03-16T10:24:20.042 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:24:20.042 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "be0cd9e9-c385-4436-9dce-770010f9b142", 2024-03-16T10:24:20.042 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:24:20.042 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:24:20.042 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:24:20.042 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:24:20.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09af1e6640 1 -- 172.21.15.77:0/3301716351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09980a2800 msgr2=0x7f09980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:20.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09af1e6640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09980a2800 0x7f09980a4cc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f09a00983b0 tx=0x7f09a000efd0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09af1e6640 1 -- 172.21.15.77:0/3301716351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a8103450 msgr2=0x7f09a8100f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:20.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09af1e6640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a8103450 0x7f09a8100f80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f09a000eb10 tx=0x7f09a000c2d0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09af1e6640 1 -- 172.21.15.77:0/3301716351 shutdown_connections 2024-03-16T10:24:20.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09af1e6640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a8103450 0x7f09a8100f80 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09af1e6640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f09980a2800 0x7f09980a4cc0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09af1e6640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09a8102a90 0x7f09a8100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09af1e6640 1 --2- 172.21.15.77:0/3301716351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09a80ffac0 0x7f09a8102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:20.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.034+0000 7f09af1e6640 1 -- 172.21.15.77:0/3301716351 >> 172.21.15.77:0/3301716351 conn(0x7f09a80fb3d0 msgr2=0x7f09a8105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:20.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.035+0000 7f09af1e6640 1 -- 172.21.15.77:0/3301716351 shutdown_connections 2024-03-16T10:24:20.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:20.035+0000 7f09af1e6640 1 -- 172.21.15.77:0/3301716351 wait complete. 2024-03-16T10:24:20.046 INFO:tasks.cephfs.filesystem:scrub status for tag:be0cd9e9-c385-4436-9dce-770010f9b142 - {'path': '/', 'tag': 'be0cd9e9-c385-4436-9dce-770010f9b142', 'options': 'recursive,force'} 2024-03-16T10:24:20.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:19 smithi077 ceph-mon[32400]: pgmap v568: 97 pgs: 97 active+clean; 714 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 896 op/s 2024-03-16T10:24:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:19 smithi193 ceph-mon[36311]: pgmap v568: 97 pgs: 97 active+clean; 714 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 896 op/s 2024-03-16T10:24:20.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:19 smithi192 ceph-mon[36354]: pgmap v568: 97 pgs: 97 active+clean; 714 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 896 op/s 2024-03-16T10:24:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:21 smithi077 ceph-mon[32400]: pgmap v569: 97 pgs: 97 active+clean; 708 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 654 op/s 2024-03-16T10:24:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:21 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:24:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:21 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:24:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:21 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:24:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:21 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:24:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:21 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:24:22.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:21 smithi193 ceph-mon[36311]: pgmap v569: 97 pgs: 97 active+clean; 708 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 654 op/s 2024-03-16T10:24:22.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:21 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:24:22.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:21 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:24:22.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:21 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:24:22.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:21 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:24:22.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:21 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:24:22.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:21 smithi192 ceph-mon[36354]: pgmap v569: 97 pgs: 97 active+clean; 708 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 654 op/s 2024-03-16T10:24:22.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:21 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:24:22.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:21 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:24:22.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:21 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:24:22.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:21 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:24:22.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:21 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:24:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:23 smithi077 ceph-mon[32400]: pgmap v570: 97 pgs: 97 active+clean; 690 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 664 op/s 2024-03-16T10:24:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:23 smithi193 ceph-mon[36311]: pgmap v570: 97 pgs: 97 active+clean; 690 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 664 op/s 2024-03-16T10:24:24.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:23 smithi192 ceph-mon[36354]: pgmap v570: 97 pgs: 97 active+clean; 690 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 664 op/s 2024-03-16T10:24:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:25 smithi077 ceph-mon[32400]: pgmap v571: 97 pgs: 97 active+clean; 669 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 536 op/s 2024-03-16T10:24:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:25 smithi193 ceph-mon[36311]: pgmap v571: 97 pgs: 97 active+clean; 669 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 536 op/s 2024-03-16T10:24:26.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:25 smithi192 ceph-mon[36354]: pgmap v571: 97 pgs: 97 active+clean; 669 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 536 op/s 2024-03-16T10:24:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:27 smithi077 ceph-mon[32400]: pgmap v572: 97 pgs: 97 active+clean; 665 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 438 op/s 2024-03-16T10:24:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:27 smithi193 ceph-mon[36311]: pgmap v572: 97 pgs: 97 active+clean; 665 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 438 op/s 2024-03-16T10:24:28.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:27 smithi192 ceph-mon[36354]: pgmap v572: 97 pgs: 97 active+clean; 665 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 438 op/s 2024-03-16T10:24:30.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:29 smithi077 ceph-mon[32400]: pgmap v573: 97 pgs: 97 active+clean; 653 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 471 op/s 2024-03-16T10:24:30.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:29 smithi193 ceph-mon[36311]: pgmap v573: 97 pgs: 97 active+clean; 653 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 471 op/s 2024-03-16T10:24:30.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:29 smithi192 ceph-mon[36354]: pgmap v573: 97 pgs: 97 active+clean; 653 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 471 op/s 2024-03-16T10:24:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.981+0000 7eff3cec6640 1 -- 172.21.15.77:0/366589308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff381022b0 msgr2=0x7eff38108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.981+0000 7eff3cec6640 1 --2- 172.21.15.77:0/366589308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff381022b0 0x7eff38108b40 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7eff30009ca0 tx=0x7eff3002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.981+0000 7eff3cec6640 1 -- 172.21.15.77:0/366589308 shutdown_connections 2024-03-16T10:24:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.981+0000 7eff3cec6640 1 --2- 172.21.15.77:0/366589308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff381022b0 0x7eff38108b40 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.981+0000 7eff3cec6640 1 --2- 172.21.15.77:0/366589308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff381018f0 0x7eff38101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.981+0000 7eff3cec6640 1 --2- 172.21.15.77:0/366589308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff38100700 0x7eff38100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.984 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.981+0000 7eff3cec6640 1 -- 172.21.15.77:0/366589308 >> 172.21.15.77:0/366589308 conn(0x7eff380fbc70 msgr2=0x7eff380fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.981+0000 7eff3cec6640 1 -- 172.21.15.77:0/366589308 shutdown_connections 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.981+0000 7eff3cec6640 1 -- 172.21.15.77:0/366589308 wait complete. 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3cec6640 1 Processor -- start 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3cec6640 1 -- start start 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3cec6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff38100700 0x7eff38104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3cec6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff381018f0 0x7eff38102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3cec6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff381022b0 0x7eff38103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3cec6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff3810c380 con 0x7eff38100700 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3cec6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7eff3810c200 con 0x7eff381022b0 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3cec6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff3810c500 con 0x7eff381018f0 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff38100700 0x7eff38104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff38100700 0x7eff38104690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40894/0 (socket says 172.21.15.77:40894) 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3659c640 1 -- 172.21.15.77:0/4022098855 learned_addr learned my addr 172.21.15.77:0/4022098855 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:30.985 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff35d9b640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff381018f0 0x7eff38102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff36d9d640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff381022b0 0x7eff38103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3659c640 1 -- 172.21.15.77:0/4022098855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff381018f0 msgr2=0x7eff38102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3659c640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff381018f0 0x7eff38102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3659c640 1 -- 172.21.15.77:0/4022098855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff381022b0 msgr2=0x7eff38103210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3659c640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff381022b0 0x7eff38103210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3659c640 1 -- 172.21.15.77:0/4022098855 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff38103aa0 con 0x7eff38100700 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff35d9b640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff381018f0 0x7eff38102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff36d9d640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff381022b0 0x7eff38103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff3659c640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff38100700 0x7eff38104690 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7eff2800e9c0 tx=0x7eff2800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.982+0000 7eff277fe640 1 -- 172.21.15.77:0/4022098855 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2800cd80 con 0x7eff38100700 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.983+0000 7eff277fe640 1 -- 172.21.15.77:0/4022098855 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff280035f0 con 0x7eff38100700 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.983+0000 7eff277fe640 1 -- 172.21.15.77:0/4022098855 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28010690 con 0x7eff38100700 2024-03-16T10:24:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.983+0000 7eff3cec6640 1 -- 172.21.15.77:0/4022098855 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff381a1b20 con 0x7eff38100700 2024-03-16T10:24:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.983+0000 7eff3cec6640 1 -- 172.21.15.77:0/4022098855 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff381a2020 con 0x7eff38100700 2024-03-16T10:24:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.985+0000 7eff3fb92740 1 Processor -- start 2024-03-16T10:24:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.985+0000 7eff3fb92740 1 -- start start 2024-03-16T10:24:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.985+0000 7eff3fb92740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58340 0x556784d58760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.985+0000 7eff3fb92740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d58d30 0x556784d5d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.985+0000 7eff3fb92740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d5dea0 0x556784d60330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.985+0000 7eff3fb92740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556784cbda40 con 0x556784d58d30 2024-03-16T10:24:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.985+0000 7eff3fb92740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x556784c66920 con 0x556784d5dea0 2024-03-16T10:24:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.985+0000 7eff3fb92740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556784cb9ae0 con 0x556784d58340 2024-03-16T10:24:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff35d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d5dea0 0x556784d60330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff36d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58340 0x556784d58760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff35d9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d5dea0 0x556784d60330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44606/0 (socket says 172.21.15.77:44606) 2024-03-16T10:24:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff35d9b640 1 -- 172.21.15.77:0/1333287426 learned_addr learned my addr 172.21.15.77:0/1333287426 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff277fe640 1 -- 172.21.15.77:0/4022098855 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7eff280037a0 con 0x7eff38100700 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff3659c640 1 --2- 172.21.15.77:0/1333287426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d58d30 0x556784d5d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff35d9b640 1 -- 172.21.15.77:0/1333287426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58340 msgr2=0x556784d58760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff35d9b640 1 --2- 172.21.15.77:0/1333287426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58340 0x556784d58760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff35d9b640 1 -- 172.21.15.77:0/1333287426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d58d30 msgr2=0x556784d5d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff35d9b640 1 --2- 172.21.15.77:0/1333287426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d58d30 0x556784d5d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff35d9b640 1 -- 172.21.15.77:0/1333287426 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556784c7e460 con 0x556784d5dea0 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff36d9d640 1 --2- 172.21.15.77:0/1333287426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58340 0x556784d58760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff35d9b640 1 --2- 172.21.15.77:0/1333287426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d5dea0 0x556784d60330 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009cf0 tx=0x7eff2c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a264875e626130e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff277fe640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff200a28a0 0x7eff200a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff24ff9640 1 -- 172.21.15.77:0/1333287426 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c012450 con 0x556784d5dea0 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff36d9d640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff200a28a0 0x7eff200a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.986+0000 7eff24ff9640 1 -- 172.21.15.77:0/1333287426 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff2c012ad0 con 0x556784d5dea0 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.987+0000 7eff24ff9640 1 -- 172.21.15.77:0/1333287426 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c00fca0 con 0x556784d5dea0 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.987+0000 7eff277fe640 1 -- 172.21.15.77:0/4022098855 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7eff280ceee0 con 0x7eff38100700 2024-03-16T10:24:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.987+0000 7eff3fb92740 1 -- 172.21.15.77:0/1333287426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d5dea0 msgr2=0x556784d60330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.987+0000 7eff36d9d640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff200a28a0 0x7eff200a4d60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7eff30006010 tx=0x7eff30005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.987+0000 7eff3fb92740 1 --2- 172.21.15.77:0/1333287426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d5dea0 0x556784d60330 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009cf0 tx=0x7eff2c0117b0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.987+0000 7eff3fb92740 1 -- 172.21.15.77:0/1333287426 shutdown_connections 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.987+0000 7eff3fb92740 1 --2- 172.21.15.77:0/1333287426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d5dea0 0x556784d60330 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.987+0000 7eff3fb92740 1 --2- 172.21.15.77:0/1333287426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d58d30 0x556784d5d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.987+0000 7eff3fb92740 1 --2- 172.21.15.77:0/1333287426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58340 0x556784d58760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.987+0000 7eff3fb92740 1 -- 172.21.15.77:0/1333287426 >> 172.21.15.77:0/1333287426 conn(0x556784d538b0 msgr2=0x556784d53cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.987+0000 7eff3fb92740 1 -- 172.21.15.77:0/1333287426 shutdown_connections 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.987+0000 7eff3fb92740 1 -- 172.21.15.77:0/1333287426 wait complete. 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff3fb92740 1 -- start start 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff3fb92740 1 Processor -- start 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff3fb92740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58340 0x556784d625e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff3fb92740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58d30 0x556784d62b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff3fb92740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d69b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff3fb92740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556784d60e50 con 0x556784d5dea0 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff3fb92740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x556784c79d70 con 0x556784d58340 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff3fb92740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556784d60fd0 con 0x556784d58d30 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff3fb92740 10 client.0 _ll_register_callbacks cb 0x5567849d1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff36d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d69b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff35d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58d30 0x556784d62b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff36d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d69b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40910/0 (socket says 172.21.15.77:40910) 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff36d9d640 1 -- 172.21.15.77:0/2793594096 learned_addr learned my addr 172.21.15.77:0/2793594096 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff3659c640 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58340 0x556784d625e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff36d9d640 1 -- 172.21.15.77:0/2793594096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58d30 msgr2=0x556784d62b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff36d9d640 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58d30 0x556784d62b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff36d9d640 1 -- 172.21.15.77:0/2793594096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58340 msgr2=0x556784d625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff36d9d640 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58340 0x556784d625e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff36d9d640 1 -- 172.21.15.77:0/2793594096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567849d2210 con 0x556784d5dea0 2024-03-16T10:24:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.988+0000 7eff35d9b640 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58d30 0x556784d62b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff36d9d640 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d69b60 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7eff30005e30 tx=0x7eff300095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b234f9cc9fa58f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff1dffb640 1 -- 172.21.15.77:0/2793594096 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30039040 con 0x556784d5dea0 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff1dffb640 1 -- 172.21.15.77:0/2793594096 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7eff30040660 con 0x556784d5dea0 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff3fb92740 1 -- 172.21.15.77:0/2793594096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff2c012ad0 con 0x556784d5dea0 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff1dffb640 1 -- 172.21.15.77:0/2793594096 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7eff30040860 con 0x556784d5dea0 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff3fb92740 10 client.17271 fetch_fsmap learned FSMap version 41 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff3fb92740 1 -- 172.21.15.77:0/2793594096 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x556784a9e620 con 0x556784d5dea0 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff1dffb640 1 -- 172.21.15.77:0/2793594096 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff2c012ad0 con 0x556784d5dea0 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff1dffb640 1 -- 172.21.15.77:0/2793594096 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7eff3004baa0 con 0x556784d5dea0 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff3fb92740 10 client.17271 fetch_fsmap finished waiting for FSMap version 41 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff3fb92740 10 client.17271 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff3fb92740 20 client.17271 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff3fb92740 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556784d6e4d0 0x556784d70930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff3fb92740 4 client.17271 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.989+0000 7eff3fb92740 1 -- 172.21.15.77:0/2793594096 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556784ccaf80 con 0x556784d6e4d0 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.990+0000 7eff3659c640 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556784d6e4d0 0x556784d70930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.990+0000 7eff3659c640 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556784d6e4d0 0x556784d70930 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e496796f93a6e557 server_cookie=e0551cdce662e57b in_seq=0 out_seq=0 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.990+0000 7eff1dffb640 10 client.17271 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff1dffb640 1 -- 172.21.15.77:0/2793594096 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556784ccaf80 con 0x556784d6e4d0 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff1dffb640 10 client.17271 handle_command_reply: tid=0 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff3fb92740 1 client.17271 shutdown 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff3fb92740 1 -- 172.21.15.77:0/2793594096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 msgr2=0x556784d69b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff3fb92740 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d69b60 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7eff30005e30 tx=0x7eff300095c0 comp rx=0 tx=0).stop 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff3fb92740 1 -- 172.21.15.77:0/2793594096 shutdown_connections 2024-03-16T10:24:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff3fb92740 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556784d6e4d0 0x556784d70930 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff3fb92740 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d69b60 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff3fb92740 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58d30 0x556784d62b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff3fb92740 1 --2- 172.21.15.77:0/2793594096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58340 0x556784d625e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff3fb92740 1 -- 172.21.15.77:0/2793594096 >> 172.21.15.77:0/2793594096 conn(0x556784d538b0 msgr2=0x556784d52bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff1dffb640 0 client.17271 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff3fb92740 1 -- 172.21.15.77:0/2793594096 shutdown_connections 2024-03-16T10:24:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff3fb92740 1 -- 172.21.15.77:0/2793594096 wait complete. 2024-03-16T10:24:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:30.991+0000 7eff3fb92740 20 client.17271 trim_cache size 0 max 16384 2024-03-16T10:24:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.006+0000 7eff3fb92740 1 Processor -- start 2024-03-16T10:24:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.006+0000 7eff3fb92740 1 -- start start 2024-03-16T10:24:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.006+0000 7eff3fb92740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58340 0x556784d613d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.006+0000 7eff3fb92740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58d30 0x556784d69b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.006+0000 7eff3fb92740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d56b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.006+0000 7eff3fb92740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff381aacd0 con 0x556784d5dea0 2024-03-16T10:24:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.006+0000 7eff3fb92740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7eff381ac840 con 0x556784d58d30 2024-03-16T10:24:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.006+0000 7eff3fb92740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff381ab3d0 con 0x556784d58340 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff36d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d56b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff36d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d56b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40912/0 (socket says 172.21.15.77:40912) 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff36d9d640 1 -- 172.21.15.77:0/3963819532 learned_addr learned my addr 172.21.15.77:0/3963819532 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff3659c640 1 --2- 172.21.15.77:0/3963819532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58340 0x556784d613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff35d9b640 1 --2- 172.21.15.77:0/3963819532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58d30 0x556784d69b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff36d9d640 1 -- 172.21.15.77:0/3963819532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58340 msgr2=0x556784d613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff36d9d640 1 --2- 172.21.15.77:0/3963819532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58340 0x556784d613d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff36d9d640 1 -- 172.21.15.77:0/3963819532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58d30 msgr2=0x556784d69b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff36d9d640 1 --2- 172.21.15.77:0/3963819532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58d30 0x556784d69b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff36d9d640 1 -- 172.21.15.77:0/3963819532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556784d45140 con 0x556784d5dea0 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff36d9d640 1 --2- 172.21.15.77:0/3963819532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d56b10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7eff3004beb0 tx=0x7eff300093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183ea75d91c1940c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff1ffff640 1 -- 172.21.15.77:0/3963819532 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30063070 con 0x556784d5dea0 2024-03-16T10:24:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff1ffff640 1 -- 172.21.15.77:0/3963819532 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff30050070 con 0x556784d5dea0 2024-03-16T10:24:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.007+0000 7eff1ffff640 1 -- 172.21.15.77:0/3963819532 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3004bb00 con 0x556784d5dea0 2024-03-16T10:24:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 -- 172.21.15.77:0/3963819532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 msgr2=0x556784d56b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 --2- 172.21.15.77:0/3963819532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d56b10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7eff3004beb0 tx=0x7eff300093e0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 -- 172.21.15.77:0/3963819532 shutdown_connections 2024-03-16T10:24:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 --2- 172.21.15.77:0/3963819532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d56b10 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 --2- 172.21.15.77:0/3963819532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58d30 0x556784d69b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 --2- 172.21.15.77:0/3963819532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58340 0x556784d613d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 -- 172.21.15.77:0/3963819532 >> 172.21.15.77:0/3963819532 conn(0x556784d538b0 msgr2=0x556784d59f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 -- 172.21.15.77:0/3963819532 shutdown_connections 2024-03-16T10:24:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 -- 172.21.15.77:0/3963819532 wait complete. 2024-03-16T10:24:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 -- start start 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 Processor -- start 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58340 0x556784d66f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58d30 0x556784d67440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d68450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff381ac170 con 0x556784d5dea0 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7eff381abaa0 con 0x556784d58340 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff381a8cb0 con 0x556784d58d30 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff3fb92740 10 client.0 _ll_register_callbacks cb 0x7eff2c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.008+0000 7eff36d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d68450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff3659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58340 0x556784d66f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff3659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58340 0x556784d66f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:44634/0 (socket says 172.21.15.77:44634) 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff3659c640 1 -- 172.21.15.77:0/666983868 learned_addr learned my addr 172.21.15.77:0/666983868 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff36d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d68450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40914/0 (socket says 172.21.15.77:40914) 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff35d9b640 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58d30 0x556784d67440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff36d9d640 1 -- 172.21.15.77:0/666983868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58d30 msgr2=0x556784d67440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff36d9d640 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58d30 0x556784d67440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff36d9d640 1 -- 172.21.15.77:0/666983868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58340 msgr2=0x556784d66f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff36d9d640 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58340 0x556784d66f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff36d9d640 1 -- 172.21.15.77:0/666983868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556784d68a40 con 0x556784d5dea0 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff3659c640 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58340 0x556784d66f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff35d9b640 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58d30 0x556784d67440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff36d9d640 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d68450 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7eff3002c630 tx=0x7eff30005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4fbce66cb6b77ac server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff1dffb640 1 -- 172.21.15.77:0/666983868 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30063070 con 0x556784d5dea0 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff1dffb640 1 -- 172.21.15.77:0/666983868 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7eff3004ba80 con 0x556784d5dea0 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff3fb92740 1 -- 172.21.15.77:0/666983868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff30050070 con 0x556784d5dea0 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff1dffb640 1 -- 172.21.15.77:0/666983868 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7eff3004b450 con 0x556784d5dea0 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff3fb92740 10 client.17277 fetch_fsmap learned FSMap version 41 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.009+0000 7eff3fb92740 1 -- 172.21.15.77:0/666983868 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x556784d69d50 con 0x556784d5dea0 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.010+0000 7eff1dffb640 1 -- 172.21.15.77:0/666983868 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff30050070 con 0x556784d5dea0 2024-03-16T10:24:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.010+0000 7eff1dffb640 1 -- 172.21.15.77:0/666983868 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7eff30066030 con 0x556784d5dea0 2024-03-16T10:24:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.010+0000 7eff3fb92740 10 client.17277 fetch_fsmap finished waiting for FSMap version 41 2024-03-16T10:24:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.010+0000 7eff3fb92740 10 client.17277 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.010+0000 7eff3fb92740 20 client.17277 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.010+0000 7eff3fb92740 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556784dd4eb0 0x556784dd72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.010+0000 7eff3fb92740 4 client.17277 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:24:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.010+0000 7eff3fb92740 1 -- 172.21.15.77:0/666983868 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556784cc52d0 con 0x556784dd4eb0 2024-03-16T10:24:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.010+0000 7eff3659c640 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556784dd4eb0 0x556784dd72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.010+0000 7eff1dffb640 10 client.17277 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.010+0000 7eff3659c640 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556784dd4eb0 0x556784dd72d0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6487900a6e0f366 server_cookie=a69d11d5898a204f in_seq=0 out_seq=0 2024-03-16T10:24:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff1dffb640 1 -- 172.21.15.77:0/666983868 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556784cc52d0 con 0x556784dd4eb0 2024-03-16T10:24:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff1dffb640 10 client.17277 handle_command_reply: tid=0 2024-03-16T10:24:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff3fb92740 1 client.17277 shutdown 2024-03-16T10:24:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff3fb92740 1 -- 172.21.15.77:0/666983868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 msgr2=0x556784d68450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff3fb92740 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d68450 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7eff3002c630 tx=0x7eff30005ce0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff3fb92740 1 -- 172.21.15.77:0/666983868 shutdown_connections 2024-03-16T10:24:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff3fb92740 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556784dd4eb0 0x556784dd72d0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff3fb92740 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556784d5dea0 0x556784d68450 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff3fb92740 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556784d58d30 0x556784d67440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff1dffb640 0 client.17277 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff3fb92740 1 --2- 172.21.15.77:0/666983868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556784d58340 0x556784d66f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff3fb92740 1 -- 172.21.15.77:0/666983868 >> 172.21.15.77:0/666983868 conn(0x556784d538b0 msgr2=0x556784d56b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff3fb92740 1 -- 172.21.15.77:0/666983868 shutdown_connections 2024-03-16T10:24:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff3fb92740 1 -- 172.21.15.77:0/666983868 wait complete. 2024-03-16T10:24:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.137+0000 7eff3fb92740 20 client.17277 trim_cache size 0 max 16384 2024-03-16T10:24:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:24:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:24:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:24:31.140 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:24:31.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.138+0000 7eff3cec6640 1 -- 172.21.15.77:0/4022098855 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff200a28a0 msgr2=0x7eff200a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:31.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.138+0000 7eff3cec6640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff200a28a0 0x7eff200a4d60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7eff30006010 tx=0x7eff30005d80 comp rx=0 tx=0).stop 2024-03-16T10:24:31.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.138+0000 7eff3cec6640 1 -- 172.21.15.77:0/4022098855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff38100700 msgr2=0x7eff38104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:31.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.138+0000 7eff3cec6640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff38100700 0x7eff38104690 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7eff2800e9c0 tx=0x7eff2800ee80 comp rx=0 tx=0).stop 2024-03-16T10:24:31.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.138+0000 7eff3cec6640 1 -- 172.21.15.77:0/4022098855 shutdown_connections 2024-03-16T10:24:31.141 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.138+0000 7eff3cec6640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff381022b0 0x7eff38103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.138+0000 7eff3cec6640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7eff200a28a0 0x7eff200a4d60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.138+0000 7eff3cec6640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff381018f0 0x7eff38102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.138+0000 7eff3cec6640 1 --2- 172.21.15.77:0/4022098855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff38100700 0x7eff38104690 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:31.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.138+0000 7eff3cec6640 1 -- 172.21.15.77:0/4022098855 >> 172.21.15.77:0/4022098855 conn(0x7eff380fbc70 msgr2=0x7eff380fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:31.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.138+0000 7eff3cec6640 1 -- 172.21.15.77:0/4022098855 shutdown_connections 2024-03-16T10:24:31.142 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:31.138+0000 7eff3cec6640 1 -- 172.21.15.77:0/4022098855 wait complete. 2024-03-16T10:24:32.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:31 smithi077 ceph-mon[32400]: pgmap v574: 97 pgs: 97 active+clean; 646 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 376 op/s 2024-03-16T10:24:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:31 smithi193 ceph-mon[36311]: pgmap v574: 97 pgs: 97 active+clean; 646 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 376 op/s 2024-03-16T10:24:32.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:31 smithi192 ceph-mon[36354]: pgmap v574: 97 pgs: 97 active+clean; 646 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.3 MiB/s wr, 376 op/s 2024-03-16T10:24:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: pgmap v575: 97 pgs: 97 active+clean; 639 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 408 op/s 2024-03-16T10:24:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:24:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:24:34.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:24:34.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:33 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:24:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: pgmap v575: 97 pgs: 97 active+clean; 639 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 408 op/s 2024-03-16T10:24:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:24:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:24:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:24:34.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:33 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:24:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: pgmap v575: 97 pgs: 97 active+clean; 639 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 408 op/s 2024-03-16T10:24:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:24:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T10:24:34.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:24:34.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:24:34.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:33 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:24:35.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:34 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-16T10:24:35.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:34 smithi077 ceph-mon[32400]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-16T10:24:35.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:34 smithi077 ceph-mon[32400]: Adjusting osd_memory_target on smithi192 to 174.1M 2024-03-16T10:24:35.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:34 smithi077 ceph-mon[32400]: Unable to set osd_memory_target on smithi192 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-16T10:24:35.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:34 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-16T10:24:35.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:34 smithi193 ceph-mon[36311]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-16T10:24:35.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:34 smithi193 ceph-mon[36311]: Adjusting osd_memory_target on smithi192 to 174.1M 2024-03-16T10:24:35.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:34 smithi193 ceph-mon[36311]: Unable to set osd_memory_target on smithi192 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-16T10:24:35.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:34 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-16T10:24:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:34 smithi192 ceph-mon[36354]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-16T10:24:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:34 smithi192 ceph-mon[36354]: Adjusting osd_memory_target on smithi192 to 174.1M 2024-03-16T10:24:35.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:34 smithi192 ceph-mon[36354]: Unable to set osd_memory_target on smithi192 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-16T10:24:36.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:35 smithi077 ceph-mon[32400]: pgmap v576: 97 pgs: 97 active+clean; 630 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 359 op/s 2024-03-16T10:24:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:35 smithi193 ceph-mon[36311]: pgmap v576: 97 pgs: 97 active+clean; 630 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 359 op/s 2024-03-16T10:24:36.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:35 smithi192 ceph-mon[36354]: pgmap v576: 97 pgs: 97 active+clean; 630 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 359 op/s 2024-03-16T10:24:37.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:36 smithi077 ceph-mon[32400]: pgmap v577: 97 pgs: 97 active+clean; 627 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 344 op/s 2024-03-16T10:24:37.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:36 smithi193 ceph-mon[36311]: pgmap v577: 97 pgs: 97 active+clean; 627 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 344 op/s 2024-03-16T10:24:37.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:36 smithi192 ceph-mon[36354]: pgmap v577: 97 pgs: 97 active+clean; 627 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 344 op/s 2024-03-16T10:24:40.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:39 smithi077 ceph-mon[32400]: pgmap v578: 97 pgs: 97 active+clean; 618 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 422 op/s 2024-03-16T10:24:40.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:39 smithi193 ceph-mon[36311]: pgmap v578: 97 pgs: 97 active+clean; 618 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 422 op/s 2024-03-16T10:24:40.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:39 smithi192 ceph-mon[36354]: pgmap v578: 97 pgs: 97 active+clean; 618 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 422 op/s 2024-03-16T10:24:42.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:41 smithi077 ceph-mon[32400]: pgmap v579: 97 pgs: 97 active+clean; 608 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 407 op/s 2024-03-16T10:24:42.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:41 smithi193 ceph-mon[36311]: pgmap v579: 97 pgs: 97 active+clean; 608 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 407 op/s 2024-03-16T10:24:42.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:41 smithi192 ceph-mon[36354]: pgmap v579: 97 pgs: 97 active+clean; 608 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 407 op/s 2024-03-16T10:24:43.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:42 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:24:43.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:42 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:24:43.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:42 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:24:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.226+0000 7f25f45d3640 1 -- 172.21.15.77:0/697578296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec100710 msgr2=0x7f25ec100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.226+0000 7f25f45d3640 1 --2- 172.21.15.77:0/697578296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec100710 0x7f25ec100b30 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f25e8009900 tx=0x7f25e802bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.226+0000 7f25f45d3640 1 -- 172.21.15.77:0/697578296 shutdown_connections 2024-03-16T10:24:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.226+0000 7f25f45d3640 1 --2- 172.21.15.77:0/697578296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25ec1022c0 0x7f25ec108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.226+0000 7f25f45d3640 1 --2- 172.21.15.77:0/697578296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25ec101900 0x7f25ec101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.226+0000 7f25f45d3640 1 --2- 172.21.15.77:0/697578296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec100710 0x7f25ec100b30 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.226+0000 7f25f45d3640 1 -- 172.21.15.77:0/697578296 >> 172.21.15.77:0/697578296 conn(0x7f25ec0fbca0 msgr2=0x7f25ec0fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.226+0000 7f25f45d3640 1 -- 172.21.15.77:0/697578296 shutdown_connections 2024-03-16T10:24:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.226+0000 7f25f45d3640 1 -- 172.21.15.77:0/697578296 wait complete. 2024-03-16T10:24:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f45d3640 1 Processor -- start 2024-03-16T10:24:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f45d3640 1 -- start start 2024-03-16T10:24:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f45d3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25ec100710 0x7f25ec197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f45d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25ec101900 0x7f25ec197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f45d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec1022c0 0x7f25ec19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f45d3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec10c390 con 0x7f25ec1022c0 2024-03-16T10:24:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f45d3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec10c210 con 0x7f25ec101900 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f45d3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec10c510 con 0x7f25ec100710 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f2b70640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec1022c0 0x7f25ec19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f236f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25ec100710 0x7f25ec197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f2b70640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec1022c0 0x7f25ec19f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48950/0 (socket says 172.21.15.77:48950) 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f2b70640 1 -- 172.21.15.77:0/606606312 learned_addr learned my addr 172.21.15.77:0/606606312 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25ec101900 0x7f25ec197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f2b70640 1 -- 172.21.15.77:0/606606312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25ec100710 msgr2=0x7f25ec197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f2b70640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25ec100710 0x7f25ec197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f2b70640 1 -- 172.21.15.77:0/606606312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25ec101900 msgr2=0x7f25ec197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f2b70640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25ec101900 0x7f25ec197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.227+0000 7f25f2b70640 1 -- 172.21.15.77:0/606606312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25ec19f670 con 0x7f25ec1022c0 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.228+0000 7f25f2b70640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec1022c0 0x7f25ec19f000 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f25e400eb10 tx=0x7f25e400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.228+0000 7f25e37fe640 1 -- 172.21.15.77:0/606606312 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e4004b20 con 0x7f25ec1022c0 2024-03-16T10:24:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.228+0000 7f25e37fe640 1 -- 172.21.15.77:0/606606312 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25e4004cc0 con 0x7f25ec1022c0 2024-03-16T10:24:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.228+0000 7f25f45d3640 1 -- 172.21.15.77:0/606606312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25ec19f960 con 0x7f25ec1022c0 2024-03-16T10:24:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.228+0000 7f25e37fe640 1 -- 172.21.15.77:0/606606312 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e4010760 con 0x7f25ec1022c0 2024-03-16T10:24:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.228+0000 7f25f45d3640 1 -- 172.21.15.77:0/606606312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25ec19ff00 con 0x7f25ec1022c0 2024-03-16T10:24:44.238 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.230+0000 7f25e37fe640 1 -- 172.21.15.77:0/606606312 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f25e4005020 con 0x7f25ec1022c0 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.230+0000 7f25f71b7740 1 Processor -- start 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f71b7740 1 -- start start 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f71b7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f71b7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563788137140 0x56378813bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f71b7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56378813c2b0 0x56378813e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f71b7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563788097a40 con 0x563788137140 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f71b7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563788040920 con 0x563788136750 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f71b7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563788093ae0 con 0x56378813c2b0 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25e37fe640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f25d80a2920 0x7f25d80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f1b6e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f236f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563788137140 0x56378813bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f1b6e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788136b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:60988/0 (socket says 172.21.15.77:60988) 2024-03-16T10:24:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f1b6e640 1 -- 172.21.15.77:0/380715527 learned_addr learned my addr 172.21.15.77:0/380715527 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/380715527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56378813c2b0 0x56378813e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f2b70640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f25d80a2920 0x7f25d80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25e37fe640 1 -- 172.21.15.77:0/606606312 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f25e40cf040 con 0x7f25ec1022c0 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f1b6e640 1 -- 172.21.15.77:0/380715527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56378813c2b0 msgr2=0x56378813e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/380715527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56378813c2b0 0x56378813e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f1b6e640 1 -- 172.21.15.77:0/380715527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563788137140 msgr2=0x56378813bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/380715527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563788137140 0x56378813bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.231+0000 7f25f1b6e640 1 -- 172.21.15.77:0/380715527 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563788058460 con 0x563788136750 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.232+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/380715527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788136b70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0058c0 tx=0x7f25dc005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cab9bf74a6ad559c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.232+0000 7f25f2b70640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f25d80a2920 0x7f25d80a4de0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f25e4098420 tx=0x7f25e401d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.232+0000 7f25e0bfe640 1 -- 172.21.15.77:0/380715527 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc004030 con 0x563788136750 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.232+0000 7f25e0bfe640 1 -- 172.21.15.77:0/380715527 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25dc005500 con 0x563788136750 2024-03-16T10:24:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.232+0000 7f25e0bfe640 1 -- 172.21.15.77:0/380715527 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc004c10 con 0x563788136750 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.232+0000 7f25f71b7740 1 -- 172.21.15.77:0/380715527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 msgr2=0x563788136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.232+0000 7f25f71b7740 1 --2- 172.21.15.77:0/380715527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788136b70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0058c0 tx=0x7f25dc005850 comp rx=0 tx=0).stop 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 -- 172.21.15.77:0/380715527 shutdown_connections 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 --2- 172.21.15.77:0/380715527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56378813c2b0 0x56378813e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 --2- 172.21.15.77:0/380715527 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563788137140 0x56378813bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 --2- 172.21.15.77:0/380715527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788136b70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 -- 172.21.15.77:0/380715527 >> 172.21.15.77:0/380715527 conn(0x56378812d8b0 msgr2=0x56378812dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 -- 172.21.15.77:0/380715527 shutdown_connections 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 -- 172.21.15.77:0/380715527 wait complete. 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 -- start start 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 Processor -- start 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563788136750 0x563788140a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788140f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56378813c2b0 0x563788147f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56378813f260 con 0x563788136750 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563788053d70 con 0x56378813c2b0 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56378813f3e0 con 0x563788137140 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f71b7740 10 client.0 _ll_register_callbacks cb 0x563787daba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f1b6e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563788136750 0x563788140a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f1b6e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563788136750 0x563788140a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48964/0 (socket says 172.21.15.77:48964) 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f1b6e640 1 -- 172.21.15.77:0/1380915687 learned_addr learned my addr 172.21.15.77:0/1380915687 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f236f640 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.233+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56378813c2b0 0x563788147f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25f1b6e640 1 -- 172.21.15.77:0/1380915687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 msgr2=0x563788140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788140f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25f1b6e640 1 -- 172.21.15.77:0/1380915687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56378813c2b0 msgr2=0x563788147f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56378813c2b0 0x563788147f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25f1b6e640 1 -- 172.21.15.77:0/1380915687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563787dac210 con 0x563788136750 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563788136750 0x563788140a00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f25dc009800 tx=0x7f25dc009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43cedf6c76226704 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25d5ffb640 1 -- 172.21.15.77:0/1380915687 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc023070 con 0x563788136750 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25d5ffb640 1 -- 172.21.15.77:0/1380915687 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f25dc004af0 con 0x563788136750 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25f71b7740 1 -- 172.21.15.77:0/1380915687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25dc005500 con 0x563788136750 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25d5ffb640 1 -- 172.21.15.77:0/1380915687 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f25dc036050 con 0x563788136750 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25f71b7740 10 client.17289 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25f71b7740 1 -- 172.21.15.77:0/1380915687 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x563787e78620 con 0x563788136750 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.234+0000 7f25d5ffb640 1 -- 172.21.15.77:0/1380915687 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25dc005500 con 0x563788136750 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.235+0000 7f25d5ffb640 1 -- 172.21.15.77:0/1380915687 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f25dc01f600 con 0x563788136750 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.235+0000 7f25f71b7740 10 client.17289 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.235+0000 7f25f71b7740 10 client.17289 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.235+0000 7f25f71b7740 20 client.17289 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.235+0000 7f25f71b7740 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56378814c590 0x56378814e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.235+0000 7f25f71b7740 4 client.17289 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.235+0000 7f25f71b7740 1 -- 172.21.15.77:0/1380915687 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637880a4f80 con 0x56378814c590 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.235+0000 7f25f236f640 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56378814c590 0x56378814e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.235+0000 7f25d5ffb640 10 client.17289 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.235+0000 7f25f236f640 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56378814c590 0x56378814e980 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49f57dd08059623b server_cookie=1381a64dfb945235 in_seq=0 out_seq=0 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.236+0000 7f25d5ffb640 1 -- 172.21.15.77:0/1380915687 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5637880a4f80 con 0x56378814c590 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.236+0000 7f25d5ffb640 10 client.17289 handle_command_reply: tid=0 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.236+0000 7f25f71b7740 1 client.17289 shutdown 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.236+0000 7f25f71b7740 1 -- 172.21.15.77:0/1380915687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563788136750 msgr2=0x563788140a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.236+0000 7f25f71b7740 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563788136750 0x563788140a00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f25dc009800 tx=0x7f25dc009fd0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.236+0000 7f25f71b7740 1 -- 172.21.15.77:0/1380915687 shutdown_connections 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.236+0000 7f25f71b7740 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56378813c2b0 0x563788147f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.236+0000 7f25f71b7740 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.236+0000 7f25f71b7740 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x56378814c590 0x56378814e980 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.236+0000 7f25f71b7740 1 --2- 172.21.15.77:0/1380915687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563788136750 0x563788140a00 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.236+0000 7f25d5ffb640 0 client.17289 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.236+0000 7f25f71b7740 1 -- 172.21.15.77:0/1380915687 >> 172.21.15.77:0/1380915687 conn(0x56378812d8b0 msgr2=0x56378812cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.237+0000 7f25f71b7740 1 -- 172.21.15.77:0/1380915687 shutdown_connections 2024-03-16T10:24:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.237+0000 7f25f71b7740 1 -- 172.21.15.77:0/1380915687 wait complete. 2024-03-16T10:24:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.237+0000 7f25f71b7740 20 client.17289 trim_cache size 0 max 16384 2024-03-16T10:24:44.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:43 smithi077 ceph-mon[32400]: pgmap v580: 97 pgs: 97 active+clean; 602 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 435 op/s 2024-03-16T10:24:44.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:43 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:24:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.251+0000 7f25f71b7740 1 Processor -- start 2024-03-16T10:24:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.251+0000 7f25f71b7740 1 -- start start 2024-03-16T10:24:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.251+0000 7f25f71b7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.251+0000 7f25f71b7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788147f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.251+0000 7f25f71b7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 0x563788128a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.251+0000 7f25f71b7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec1a6970 con 0x56378813c2b0 2024-03-16T10:24:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.251+0000 7f25f71b7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec1a8540 con 0x563788136750 2024-03-16T10:24:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.251+0000 7f25f71b7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec1a7070 con 0x563788137140 2024-03-16T10:24:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.251+0000 7f25f1b6e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 0x563788128a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25f1b6e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 0x563788128a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:48968/0 (socket says 172.21.15.77:48968) 2024-03-16T10:24:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25f1b6e640 1 -- 172.21.15.77:0/2241452034 learned_addr learned my addr 172.21.15.77:0/2241452034 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25f236f640 1 --2- 172.21.15.77:0/2241452034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/2241452034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25f1b6e640 1 -- 172.21.15.77:0/2241452034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 msgr2=0x563788147f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/2241452034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788147f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25f1b6e640 1 -- 172.21.15.77:0/2241452034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 msgr2=0x563788140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/2241452034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25f1b6e640 1 -- 172.21.15.77:0/2241452034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563788136b30 con 0x56378813c2b0 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25f236f640 1 --2- 172.21.15.77:0/2241452034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788147f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/2241452034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 0x563788128a40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f25dc006010 tx=0x7f25dc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7e6f84859f646e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25d7fff640 1 -- 172.21.15.77:0/2241452034 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc023070 con 0x56378813c2b0 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25d7fff640 1 -- 172.21.15.77:0/2241452034 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25dc01fbf0 con 0x56378813c2b0 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.252+0000 7f25d7fff640 1 -- 172.21.15.77:0/2241452034 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc029040 con 0x56378813c2b0 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 -- 172.21.15.77:0/2241452034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 msgr2=0x563788128a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 --2- 172.21.15.77:0/2241452034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 0x563788128a40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f25dc006010 tx=0x7f25dc000bc0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 -- 172.21.15.77:0/2241452034 shutdown_connections 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 --2- 172.21.15.77:0/2241452034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 0x563788128a40 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 --2- 172.21.15.77:0/2241452034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788147f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 --2- 172.21.15.77:0/2241452034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 -- 172.21.15.77:0/2241452034 >> 172.21.15.77:0/2241452034 conn(0x56378812d8b0 msgr2=0x56378812dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 -- 172.21.15.77:0/2241452034 shutdown_connections 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 -- 172.21.15.77:0/2241452034 wait complete. 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 -- start start 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 Processor -- start 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788145010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788145550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 0x5637881b0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec1a7e70 con 0x56378813c2b0 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec1a7770 con 0x563788136750 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec1a4950 con 0x563788137140 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f71b7740 10 client.0 _ll_register_callbacks cb 0x7f25e40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f236f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f1b6e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 0x5637881b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f1b6e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f236f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788145550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:60740/0 (socket says 172.21.15.77:60740) 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.253+0000 7f25f236f640 1 -- 172.21.15.77:0/50732218 learned_addr learned my addr 172.21.15.77:0/50732218 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25f236f640 1 -- 172.21.15.77:0/50732218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 msgr2=0x563788145010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25f236f640 1 --2- 172.21.15.77:0/50732218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788145010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25f236f640 1 -- 172.21.15.77:0/50732218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 msgr2=0x5637881b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25f236f640 1 --2- 172.21.15.77:0/50732218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 0x5637881b0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25f236f640 1 -- 172.21.15.77:0/50732218 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637881b0dd0 con 0x563788137140 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/50732218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 0x5637881b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25f236f640 1 --2- 172.21.15.77:0/50732218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788145550 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f25e802c7b0 tx=0x7f25e802cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=636bb980dd120ede server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25d5ffb640 1 -- 172.21.15.77:0/50732218 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e8046d10 con 0x563788137140 2024-03-16T10:24:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25f71b7740 1 -- 172.21.15.77:0/50732218 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25dc01fbf0 con 0x563788137140 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25d5ffb640 1 -- 172.21.15.77:0/50732218 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f25e80463a0 con 0x563788137140 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25d5ffb640 1 -- 172.21.15.77:0/50732218 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f25e80465a0 con 0x563788137140 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25f71b7740 10 client.27049 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25f71b7740 1 -- 172.21.15.77:0/50732218 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5637881b2130 con 0x563788137140 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.254+0000 7f25d5ffb640 1 -- 172.21.15.77:0/50732218 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25dc01fbf0 con 0x563788137140 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.255+0000 7f25d5ffb640 1 -- 172.21.15.77:0/50732218 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f25e8003a60 con 0x563788137140 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.255+0000 7f25f71b7740 10 client.27049 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.255+0000 7f25f71b7740 10 client.27049 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.255+0000 7f25f71b7740 20 client.27049 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.255+0000 7f25f71b7740 1 --2- 172.21.15.77:0/50732218 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5637881b2ae0 0x5637881b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.255+0000 7f25f71b7740 4 client.27049 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.255+0000 7f25f71b7740 1 -- 172.21.15.77:0/50732218 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56378809f2d0 con 0x5637881b2ae0 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.255+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/50732218 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5637881b2ae0 0x5637881b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.255+0000 7f25d5ffb640 10 client.27049 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.255+0000 7f25f1b6e640 1 --2- 172.21.15.77:0/50732218 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5637881b2ae0 0x5637881b4f00 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=754eb0cd2d3a047a server_cookie=c57d70f24711d704 in_seq=0 out_seq=0 2024-03-16T10:24:44.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:43 smithi193 ceph-mon[36311]: pgmap v580: 97 pgs: 97 active+clean; 602 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 435 op/s 2024-03-16T10:24:44.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:43 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:24:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25d5ffb640 1 -- 172.21.15.77:0/50732218 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56378809f2d0 con 0x5637881b2ae0 2024-03-16T10:24:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25d5ffb640 10 client.27049 handle_command_reply: tid=0 2024-03-16T10:24:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25f71b7740 1 client.27049 shutdown 2024-03-16T10:24:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25f71b7740 1 -- 172.21.15.77:0/50732218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 msgr2=0x563788145550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25f71b7740 1 --2- 172.21.15.77:0/50732218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788145550 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f25e802c7b0 tx=0x7f25e802cb70 comp rx=0 tx=0).stop 2024-03-16T10:24:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25f71b7740 1 -- 172.21.15.77:0/50732218 shutdown_connections 2024-03-16T10:24:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25f71b7740 1 --2- 172.21.15.77:0/50732218 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5637881b2ae0 0x5637881b4f00 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25f71b7740 1 --2- 172.21.15.77:0/50732218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56378813c2b0 0x5637881b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25d5ffb640 0 client.27049 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25f71b7740 1 --2- 172.21.15.77:0/50732218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563788137140 0x563788145550 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25f71b7740 1 --2- 172.21.15.77:0/50732218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563788136750 0x563788145010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25f71b7740 1 -- 172.21.15.77:0/50732218 >> 172.21.15.77:0/50732218 conn(0x56378812d8b0 msgr2=0x563788128a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25f71b7740 1 -- 172.21.15.77:0/50732218 shutdown_connections 2024-03-16T10:24:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25f71b7740 1 -- 172.21.15.77:0/50732218 wait complete. 2024-03-16T10:24:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.281+0000 7f25f71b7740 20 client.27049 trim_cache size 0 max 16384 2024-03-16T10:24:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:24:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:24:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:24:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:24:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.282+0000 7f25f45d3640 1 -- 172.21.15.77:0/606606312 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f25d80a2920 msgr2=0x7f25d80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.282+0000 7f25f45d3640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f25d80a2920 0x7f25d80a4de0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f25e4098420 tx=0x7f25e401d000 comp rx=0 tx=0).stop 2024-03-16T10:24:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.282+0000 7f25f45d3640 1 -- 172.21.15.77:0/606606312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec1022c0 msgr2=0x7f25ec19f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.282+0000 7f25f45d3640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec1022c0 0x7f25ec19f000 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f25e400eb10 tx=0x7f25e400c2d0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.282+0000 7f25f45d3640 1 -- 172.21.15.77:0/606606312 shutdown_connections 2024-03-16T10:24:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.282+0000 7f25f45d3640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25ec1022c0 0x7f25ec19f000 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.282+0000 7f25f45d3640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25ec101900 0x7f25ec197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.282+0000 7f25f45d3640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f25d80a2920 0x7f25d80a4de0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.282+0000 7f25f45d3640 1 --2- 172.21.15.77:0/606606312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25ec100710 0x7f25ec197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.282+0000 7f25f45d3640 1 -- 172.21.15.77:0/606606312 >> 172.21.15.77:0/606606312 conn(0x7f25ec0fbca0 msgr2=0x7f25ec0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.282+0000 7f25f45d3640 1 -- 172.21.15.77:0/606606312 shutdown_connections 2024-03-16T10:24:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:44.282+0000 7f25f45d3640 1 -- 172.21.15.77:0/606606312 wait complete. 2024-03-16T10:24:44.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:43 smithi192 ceph-mon[36354]: pgmap v580: 97 pgs: 97 active+clean; 602 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 435 op/s 2024-03-16T10:24:44.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:43 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:24:46.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:45 smithi077 ceph-mon[32400]: pgmap v581: 97 pgs: 97 active+clean; 593 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 1.0 MiB/s wr, 421 op/s 2024-03-16T10:24:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:45 smithi193 ceph-mon[36311]: pgmap v581: 97 pgs: 97 active+clean; 593 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 1.0 MiB/s wr, 421 op/s 2024-03-16T10:24:46.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:45 smithi192 ceph-mon[36354]: pgmap v581: 97 pgs: 97 active+clean; 593 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 1.0 MiB/s wr, 421 op/s 2024-03-16T10:24:48.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:47 smithi077 ceph-mon[32400]: pgmap v582: 97 pgs: 97 active+clean; 583 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 411 op/s 2024-03-16T10:24:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:47 smithi193 ceph-mon[36311]: pgmap v582: 97 pgs: 97 active+clean; 583 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 411 op/s 2024-03-16T10:24:48.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:47 smithi192 ceph-mon[36354]: pgmap v582: 97 pgs: 97 active+clean; 583 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 411 op/s 2024-03-16T10:24:49.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:48 smithi077 ceph-mon[32400]: pgmap v583: 97 pgs: 97 active+clean; 554 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 457 op/s 2024-03-16T10:24:49.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:48 smithi193 ceph-mon[36311]: pgmap v583: 97 pgs: 97 active+clean; 554 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 457 op/s 2024-03-16T10:24:49.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:48 smithi192 ceph-mon[36354]: pgmap v583: 97 pgs: 97 active+clean; 554 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 457 op/s 2024-03-16T10:24:50.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:24:50.047 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:24:50.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.158+0000 7f0a698c6640 1 -- 172.21.15.77:0/2302118243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a641018f0 msgr2=0x7f0a64101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.158+0000 7f0a698c6640 1 --2- 172.21.15.77:0/2302118243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a641018f0 0x7f0a64101d70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009900 tx=0x7f0a5402bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:50.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.158+0000 7f0a698c6640 1 -- 172.21.15.77:0/2302118243 shutdown_connections 2024-03-16T10:24:50.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.158+0000 7f0a698c6640 1 --2- 172.21.15.77:0/2302118243 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a641022b0 0x7f0a64108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.158+0000 7f0a698c6640 1 --2- 172.21.15.77:0/2302118243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a641018f0 0x7f0a64101d70 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.158+0000 7f0a698c6640 1 --2- 172.21.15.77:0/2302118243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a64100700 0x7f0a64100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.158+0000 7f0a698c6640 1 -- 172.21.15.77:0/2302118243 >> 172.21.15.77:0/2302118243 conn(0x7f0a640fbc90 msgr2=0x7f0a640fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.158+0000 7f0a698c6640 1 -- 172.21.15.77:0/2302118243 shutdown_connections 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.158+0000 7f0a698c6640 1 -- 172.21.15.77:0/2302118243 wait complete. 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a698c6640 1 Processor -- start 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a698c6640 1 -- start start 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a698c6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a64100700 0x7f0a64197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a698c6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a641018f0 0x7f0a64198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a698c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a641022b0 0x7f0a6419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a698c6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6410d4a0 con 0x7f0a641022b0 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a698c6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6410d320 con 0x7f0a641018f0 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a698c6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6410d620 con 0x7f0a64100700 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a637fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a641022b0 0x7f0a6419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a637fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a641022b0 0x7f0a6419f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49006/0 (socket says 172.21.15.77:49006) 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a637fe640 1 -- 172.21.15.77:0/2321791175 learned_addr learned my addr 172.21.15.77:0/2321791175 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a62ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a64100700 0x7f0a64197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.159+0000 7f0a627fc640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a641018f0 0x7f0a64198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.160+0000 7f0a637fe640 1 -- 172.21.15.77:0/2321791175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a64100700 msgr2=0x7f0a64197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.160+0000 7f0a637fe640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a64100700 0x7f0a64197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.160+0000 7f0a637fe640 1 -- 172.21.15.77:0/2321791175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a641018f0 msgr2=0x7f0a64198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.160+0000 7f0a637fe640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a641018f0 0x7f0a64198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.160+0000 7f0a637fe640 1 -- 172.21.15.77:0/2321791175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6419f740 con 0x7f0a641022b0 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.160+0000 7f0a62ffd640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a64100700 0x7f0a64197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.160+0000 7f0a637fe640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a641022b0 0x7f0a6419f0d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00eb10 tx=0x7f0a5c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.160+0000 7f0a688c4640 1 -- 172.21.15.77:0/2321791175 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c004b20 con 0x7f0a641022b0 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.160+0000 7f0a698c6640 1 -- 172.21.15.77:0/2321791175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a6419fa30 con 0x7f0a641022b0 2024-03-16T10:24:50.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.160+0000 7f0a688c4640 1 -- 172.21.15.77:0/2321791175 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a5c004cc0 con 0x7f0a641022b0 2024-03-16T10:24:50.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.160+0000 7f0a688c4640 1 -- 172.21.15.77:0/2321791175 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c010760 con 0x7f0a641022b0 2024-03-16T10:24:50.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.160+0000 7f0a698c6640 1 -- 172.21.15.77:0/2321791175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a6419ff00 con 0x7f0a641022b0 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.162+0000 7f0a688c4640 1 -- 172.21.15.77:0/2321791175 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f0a5c005020 con 0x7f0a641022b0 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.163+0000 7f0a688c4640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0a4c0a2800 0x7f0a4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.163+0000 7f0a688c4640 1 -- 172.21.15.77:0/2321791175 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f0a5c0cef40 con 0x7f0a641022b0 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.163+0000 7f0a62ffd640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0a4c0a2800 0x7f0a4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.163+0000 7f0a62ffd640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0a4c0a2800 0x7f0a4c0a4cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0a64101750 tx=0x7f0a58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.163+0000 7f0a6c592740 1 Processor -- start 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.163+0000 7f0a6c592740 1 -- start start 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a6c592740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002170 0x5575bc002590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a6c592740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002b60 0x5575bc007660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a6c592740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc007cd0 0x5575bc00a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a6c592740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5575bbf67500 con 0x5575bc002b60 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a6c592740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5575bbf0fbe0 con 0x5575bc007cd0 2024-03-16T10:24:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a6c592740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5575bbf635a0 con 0x5575bc002170 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a627fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002b60 0x5575bc007660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a627fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002b60 0x5575bc007660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49016/0 (socket says 172.21.15.77:49016) 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a627fc640 1 -- 172.21.15.77:0/2250767363 learned_addr learned my addr 172.21.15.77:0/2250767363 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a637fe640 1 --2- 172.21.15.77:0/2250767363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc007cd0 0x5575bc00a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a62ffd640 1 --2- 172.21.15.77:0/2250767363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002170 0x5575bc002590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a627fc640 1 -- 172.21.15.77:0/2250767363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002170 msgr2=0x5575bc002590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a627fc640 1 --2- 172.21.15.77:0/2250767363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002170 0x5575bc002590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a627fc640 1 -- 172.21.15.77:0/2250767363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc007cd0 msgr2=0x5575bc00a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a627fc640 1 --2- 172.21.15.77:0/2250767363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc007cd0 0x5575bc00a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a627fc640 1 -- 172.21.15.77:0/2250767363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575bbf234b0 con 0x5575bc002b60 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a627fc640 1 --2- 172.21.15.77:0/2250767363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002b60 0x5575bc007660 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009900 tx=0x7f0a54009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8925175f9fb0f853 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a49ffb640 1 -- 172.21.15.77:0/2250767363 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5402cb80 con 0x5575bc002b60 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a49ffb640 1 -- 172.21.15.77:0/2250767363 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a54005040 con 0x5575bc002b60 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.164+0000 7f0a49ffb640 1 -- 172.21.15.77:0/2250767363 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54004900 con 0x5575bc002b60 2024-03-16T10:24:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.165+0000 7f0a6c592740 1 -- 172.21.15.77:0/2250767363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002b60 msgr2=0x5575bc007660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.165+0000 7f0a6c592740 1 --2- 172.21.15.77:0/2250767363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002b60 0x5575bc007660 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009900 tx=0x7f0a54009560 comp rx=0 tx=0).stop 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.165+0000 7f0a6c592740 1 -- 172.21.15.77:0/2250767363 shutdown_connections 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.165+0000 7f0a6c592740 1 --2- 172.21.15.77:0/2250767363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc007cd0 0x5575bc00a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.165+0000 7f0a6c592740 1 --2- 172.21.15.77:0/2250767363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002b60 0x5575bc007660 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.165+0000 7f0a6c592740 1 --2- 172.21.15.77:0/2250767363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002170 0x5575bc002590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.165+0000 7f0a6c592740 1 -- 172.21.15.77:0/2250767363 >> 172.21.15.77:0/2250767363 conn(0x5575bbffd6e0 msgr2=0x5575bbffdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.165+0000 7f0a6c592740 1 -- 172.21.15.77:0/2250767363 shutdown_connections 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.165+0000 7f0a6c592740 1 -- 172.21.15.77:0/2250767363 wait complete. 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.165+0000 7f0a6c592740 1 -- start start 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a6c592740 1 Processor -- start 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a6c592740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002170 0x5575bc00c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a6c592740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002b60 0x5575bc00cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a6c592740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 0x5575bc013bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a6c592740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5575bc00ac20 con 0x5575bc007cd0 2024-03-16T10:24:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a6c592740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5575bbf23160 con 0x5575bc002170 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a6c592740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5575bc00ada0 con 0x5575bc002b60 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a6c592740 10 client.0 _ll_register_callbacks cb 0x5575bbf07150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a637fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 0x5575bc013bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a62ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002170 0x5575bc00c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a637fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 0x5575bc013bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49028/0 (socket says 172.21.15.77:49028) 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a637fe640 1 -- 172.21.15.77:0/368509217 learned_addr learned my addr 172.21.15.77:0/368509217 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a627fc640 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002b60 0x5575bc00cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a637fe640 1 -- 172.21.15.77:0/368509217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002b60 msgr2=0x5575bc00cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a637fe640 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002b60 0x5575bc00cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a637fe640 1 -- 172.21.15.77:0/368509217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002170 msgr2=0x5575bc00c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a637fe640 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002170 0x5575bc00c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a637fe640 1 -- 172.21.15.77:0/368509217 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575bbd457d0 con 0x5575bc007cd0 2024-03-16T10:24:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.166+0000 7f0a627fc640 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002b60 0x5575bc00cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a637fe640 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 0x5575bc013bf0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c09b900 tx=0x7f0a5c09c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6d9d1ec31652a8e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a42ffd640 1 -- 172.21.15.77:0/368509217 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c0b3070 con 0x5575bc007cd0 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a42ffd640 1 -- 172.21.15.77:0/368509217 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f0a5c0a3e80 con 0x5575bc007cd0 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a6c592740 1 -- 172.21.15.77:0/368509217 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a54005040 con 0x5575bc007cd0 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a42ffd640 1 -- 172.21.15.77:0/368509217 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0a5c0a4110 con 0x5575bc007cd0 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a6c592740 10 client.17307 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a6c592740 1 -- 172.21.15.77:0/368509217 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5575bbc7b3f0 con 0x5575bc007cd0 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a42ffd640 1 -- 172.21.15.77:0/368509217 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a54005040 con 0x5575bc007cd0 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a42ffd640 1 -- 172.21.15.77:0/368509217 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f0a5c086140 con 0x5575bc007cd0 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a6c592740 10 client.17307 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a6c592740 10 client.17307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a6c592740 20 client.17307 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a6c592740 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5575bc0182f0 0x5575bc01a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a6c592740 4 client.17307 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.167+0000 7f0a6c592740 1 -- 172.21.15.77:0/368509217 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575bbf74a40 con 0x5575bc0182f0 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.168+0000 7f0a62ffd640 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5575bc0182f0 0x5575bc01a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.168+0000 7f0a42ffd640 10 client.17307 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.168+0000 7f0a62ffd640 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5575bc0182f0 0x5575bc01a6e0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5783b790b303a348 server_cookie=65052e1d5a19db3f in_seq=0 out_seq=0 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a42ffd640 1 -- 172.21.15.77:0/368509217 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5575bbf74a40 con 0x5575bc0182f0 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a42ffd640 10 client.17307 handle_command_reply: tid=0 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a6c592740 1 client.17307 shutdown 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a6c592740 1 -- 172.21.15.77:0/368509217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 msgr2=0x5575bc013bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a6c592740 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 0x5575bc013bf0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c09b900 tx=0x7f0a5c09c070 comp rx=0 tx=0).stop 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a6c592740 1 -- 172.21.15.77:0/368509217 shutdown_connections 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a6c592740 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5575bc0182f0 0x5575bc01a6e0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a6c592740 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 0x5575bc013bf0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a42ffd640 0 client.17307 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a6c592740 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002b60 0x5575bc00cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a6c592740 1 --2- 172.21.15.77:0/368509217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002170 0x5575bc00c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a6c592740 1 -- 172.21.15.77:0/368509217 >> 172.21.15.77:0/368509217 conn(0x5575bbffd6e0 msgr2=0x5575bbffc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a6c592740 1 -- 172.21.15.77:0/368509217 shutdown_connections 2024-03-16T10:24:50.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.169+0000 7f0a6c592740 1 -- 172.21.15.77:0/368509217 wait complete. 2024-03-16T10:24:50.177 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.170+0000 7f0a6c592740 20 client.17307 trim_cache size 0 max 16384 2024-03-16T10:24:50.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a6c592740 1 Processor -- start 2024-03-16T10:24:50.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a6c592740 1 -- start start 2024-03-16T10:24:50.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a6c592740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002170 0x5575bc00b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a6c592740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002b60 0x5575bc013bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a6c592740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 0x5575bc00b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a6c592740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641a8ac0 con 0x5575bc007cd0 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a6c592740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641aa690 con 0x5575bc002b60 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a6c592740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641a91c0 con 0x5575bc002170 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a637fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 0x5575bc00b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a627fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002b60 0x5575bc013bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a637fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 0x5575bc00b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49032/0 (socket says 172.21.15.77:49032) 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a637fe640 1 -- 172.21.15.77:0/937327467 learned_addr learned my addr 172.21.15.77:0/937327467 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a62ffd640 1 --2- 172.21.15.77:0/937327467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002170 0x5575bc00b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a637fe640 1 -- 172.21.15.77:0/937327467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002170 msgr2=0x5575bc00b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a637fe640 1 --2- 172.21.15.77:0/937327467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002170 0x5575bc00b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a637fe640 1 -- 172.21.15.77:0/937327467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002b60 msgr2=0x5575bc013bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a637fe640 1 --2- 172.21.15.77:0/937327467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002b60 0x5575bc013bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a637fe640 1 -- 172.21.15.77:0/937327467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575bbc3b0f0 con 0x5575bc007cd0 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a627fc640 1 --2- 172.21.15.77:0/937327467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002b60 0x5575bc013bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.187+0000 7f0a637fe640 1 --2- 172.21.15.77:0/937327467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 0x5575bc00b6b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c097dd0 tx=0x7f0a5c099e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9174a3e13b96246e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.188+0000 7f0a497fa640 1 -- 172.21.15.77:0/937327467 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c0b3070 con 0x5575bc007cd0 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.188+0000 7f0a497fa640 1 -- 172.21.15.77:0/937327467 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a5c0862b0 con 0x5575bc007cd0 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.188+0000 7f0a497fa640 1 -- 172.21.15.77:0/937327467 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c0a3c80 con 0x5575bc007cd0 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.188+0000 7f0a6c592740 1 -- 172.21.15.77:0/937327467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 msgr2=0x5575bc00b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.188+0000 7f0a6c592740 1 --2- 172.21.15.77:0/937327467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 0x5575bc00b6b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c097dd0 tx=0x7f0a5c099e80 comp rx=0 tx=0).stop 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.188+0000 7f0a6c592740 1 -- 172.21.15.77:0/937327467 shutdown_connections 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.188+0000 7f0a6c592740 1 --2- 172.21.15.77:0/937327467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc007cd0 0x5575bc00b6b0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.188+0000 7f0a6c592740 1 --2- 172.21.15.77:0/937327467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002b60 0x5575bc013bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.188+0000 7f0a6c592740 1 --2- 172.21.15.77:0/937327467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc002170 0x5575bc00b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.188+0000 7f0a6c592740 1 -- 172.21.15.77:0/937327467 >> 172.21.15.77:0/937327467 conn(0x5575bbffd6e0 msgr2=0x5575bc00c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.188+0000 7f0a6c592740 1 -- 172.21.15.77:0/937327467 shutdown_connections 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.188+0000 7f0a6c592740 1 -- 172.21.15.77:0/937327467 wait complete. 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a6c592740 1 -- start start 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a6c592740 1 Processor -- start 2024-03-16T10:24:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a6c592740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002170 0x5575bc010c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a6c592740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002b60 0x5575bc011140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a6c592740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc007cd0 0x5575bc012150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a6c592740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641a9fc0 con 0x5575bc002170 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a6c592740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641a98c0 con 0x5575bc002b60 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a6c592740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641a6a80 con 0x5575bc007cd0 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a6c592740 10 client.0 _ll_register_callbacks cb 0x7f0a5c097c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a62ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002170 0x5575bc010c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a62ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002170 0x5575bc010c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49038/0 (socket says 172.21.15.77:49038) 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a62ffd640 1 -- 172.21.15.77:0/2897451682 learned_addr learned my addr 172.21.15.77:0/2897451682 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a627fc640 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002b60 0x5575bc011140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a637fe640 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc007cd0 0x5575bc012150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a62ffd640 1 -- 172.21.15.77:0/2897451682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc007cd0 msgr2=0x5575bc012150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a62ffd640 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc007cd0 0x5575bc012150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a62ffd640 1 -- 172.21.15.77:0/2897451682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002b60 msgr2=0x5575bc011140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a62ffd640 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002b60 0x5575bc011140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.189+0000 7f0a62ffd640 1 -- 172.21.15.77:0/2897451682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575bbf2e9d0 con 0x5575bc002170 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a62ffd640 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002170 0x5575bc010c00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009e80 tx=0x7f0a58013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6662680ff3dec93 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a42ffd640 1 -- 172.21.15.77:0/2897451682 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a58030030 con 0x5575bc002170 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a42ffd640 1 -- 172.21.15.77:0/2897451682 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f0a5801ba50 con 0x5575bc002170 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a6c592740 1 -- 172.21.15.77:0/2897451682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a5c0862b0 con 0x5575bc002170 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a42ffd640 1 -- 172.21.15.77:0/2897451682 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0a5803c050 con 0x5575bc002170 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a6c592740 10 client.17313 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a6c592740 1 -- 172.21.15.77:0/2897451682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5575bc013950 con 0x5575bc002170 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a42ffd640 1 -- 172.21.15.77:0/2897451682 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a5c0862b0 con 0x5575bc002170 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a42ffd640 1 -- 172.21.15.77:0/2897451682 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f0a5802a520 con 0x5575bc002170 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a6c592740 10 client.17313 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a6c592740 10 client.17313 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a6c592740 20 client.17313 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a6c592740 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5575bc0182f0 0x5575bc0807a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a6c592740 4 client.17313 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.190+0000 7f0a6c592740 1 -- 172.21.15.77:0/2897451682 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5575bbf6ed90 con 0x5575bc0182f0 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.191+0000 7f0a627fc640 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5575bc0182f0 0x5575bc0807a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.191+0000 7f0a42ffd640 10 client.17313 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.191+0000 7f0a627fc640 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5575bc0182f0 0x5575bc0807a0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c422202c2861efa server_cookie=b13a9b18b8512190 in_seq=0 out_seq=0 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.191+0000 7f0a42ffd640 1 -- 172.21.15.77:0/2897451682 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5575bbf6ed90 con 0x5575bc0182f0 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.191+0000 7f0a42ffd640 10 client.17313 handle_command_reply: tid=0 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.191+0000 7f0a6c592740 1 client.17313 shutdown 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.191+0000 7f0a6c592740 1 -- 172.21.15.77:0/2897451682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002170 msgr2=0x5575bc010c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.191+0000 7f0a6c592740 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002170 0x5575bc010c00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009e80 tx=0x7f0a58013040 comp rx=0 tx=0).stop 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a6c592740 1 -- 172.21.15.77:0/2897451682 shutdown_connections 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a6c592740 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575bc007cd0 0x5575bc012150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a6c592740 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5575bc002b60 0x5575bc011140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a6c592740 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5575bc0182f0 0x5575bc0807a0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a6c592740 1 --2- 172.21.15.77:0/2897451682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575bc002170 0x5575bc010c00 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a42ffd640 0 client.17313 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:50.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a6c592740 1 -- 172.21.15.77:0/2897451682 >> 172.21.15.77:0/2897451682 conn(0x5575bbffd6e0 msgr2=0x5575bc00c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a6c592740 1 -- 172.21.15.77:0/2897451682 shutdown_connections 2024-03-16T10:24:50.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a6c592740 1 -- 172.21.15.77:0/2897451682 wait complete. 2024-03-16T10:24:50.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a6c592740 20 client.17313 trim_cache size 0 max 16384 2024-03-16T10:24:50.199 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:24:50.199 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T10:24:50.199 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T10:24:50.199 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:24:50.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a698c6640 1 -- 172.21.15.77:0/2321791175 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0a4c0a2800 msgr2=0x7f0a4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a698c6640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0a4c0a2800 0x7f0a4c0a4cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0a64101750 tx=0x7f0a58005f90 comp rx=0 tx=0).stop 2024-03-16T10:24:50.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a698c6640 1 -- 172.21.15.77:0/2321791175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a641022b0 msgr2=0x7f0a6419f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a698c6640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a641022b0 0x7f0a6419f0d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00eb10 tx=0x7f0a5c00c2d0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a698c6640 1 -- 172.21.15.77:0/2321791175 shutdown_connections 2024-03-16T10:24:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a698c6640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0a641022b0 0x7f0a6419f0d0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a698c6640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a641018f0 0x7f0a64198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a698c6640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0a4c0a2800 0x7f0a4c0a4cc0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a698c6640 1 --2- 172.21.15.77:0/2321791175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a64100700 0x7f0a64197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a698c6640 1 -- 172.21.15.77:0/2321791175 >> 172.21.15.77:0/2321791175 conn(0x7f0a640fbc90 msgr2=0x7f0a64104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a698c6640 1 -- 172.21.15.77:0/2321791175 shutdown_connections 2024-03-16T10:24:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.192+0000 7f0a698c6640 1 -- 172.21.15.77:0/2321791175 wait complete. 2024-03-16T10:24:50.204 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T10:24:50.204 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:24:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.308+0000 7fb6e073b640 1 -- 172.21.15.77:0/604759429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6d81018f0 msgr2=0x7fb6d8101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.308+0000 7fb6e073b640 1 --2- 172.21.15.77:0/604759429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6d81018f0 0x7fb6d8101d70 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009900 tx=0x7fb6c802bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.308+0000 7fb6e073b640 1 -- 172.21.15.77:0/604759429 shutdown_connections 2024-03-16T10:24:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.308+0000 7fb6e073b640 1 --2- 172.21.15.77:0/604759429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6d81022b0 0x7fb6d8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.308+0000 7fb6e073b640 1 --2- 172.21.15.77:0/604759429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6d81018f0 0x7fb6d8101d70 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.308+0000 7fb6e073b640 1 --2- 172.21.15.77:0/604759429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6d8100700 0x7fb6d8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.308+0000 7fb6e073b640 1 -- 172.21.15.77:0/604759429 >> 172.21.15.77:0/604759429 conn(0x7fb6d80fbc70 msgr2=0x7fb6d80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.308+0000 7fb6e073b640 1 -- 172.21.15.77:0/604759429 shutdown_connections 2024-03-16T10:24:50.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.308+0000 7fb6e073b640 1 -- 172.21.15.77:0/604759429 wait complete. 2024-03-16T10:24:50.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.308+0000 7fb6e073b640 1 Processor -- start 2024-03-16T10:24:50.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6e073b640 1 -- start start 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6e073b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6d8100700 0x7fb6d8196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6e073b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6d81018f0 0x7fb6d8196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6e073b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6d81022b0 0x7fb6d819dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6e073b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d810c380 con 0x7fb6d8100700 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6e073b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d810c200 con 0x7fb6d81018f0 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6e073b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d810c500 con 0x7fb6d81022b0 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6de4d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6d8100700 0x7fb6d8196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6de4d7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6d8100700 0x7fb6d8196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49050/0 (socket says 172.21.15.77:49050) 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6de4d7640 1 -- 172.21.15.77:0/3626710502 learned_addr learned my addr 172.21.15.77:0/3626710502 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6decd8640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6d81022b0 0x7fb6d819dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6ddcd6640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6d81018f0 0x7fb6d8196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6de4d7640 1 -- 172.21.15.77:0/3626710502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6d81022b0 msgr2=0x7fb6d819dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6de4d7640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6d81022b0 0x7fb6d819dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6de4d7640 1 -- 172.21.15.77:0/3626710502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6d81018f0 msgr2=0x7fb6d8196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6de4d7640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6d81018f0 0x7fb6d8196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6de4d7640 1 -- 172.21.15.77:0/3626710502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d819e640 con 0x7fb6d8100700 2024-03-16T10:24:50.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6ddcd6640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6d81018f0 0x7fb6d8196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:24:50.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6de4d7640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6d8100700 0x7fb6d8196a20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400e9c0 tx=0x7fb6d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6cf7fe640 1 -- 172.21.15.77:0/3626710502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d400cd80 con 0x7fb6d8100700 2024-03-16T10:24:50.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6cf7fe640 1 -- 172.21.15.77:0/3626710502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6d40035f0 con 0x7fb6d8100700 2024-03-16T10:24:50.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6cf7fe640 1 -- 172.21.15.77:0/3626710502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4010690 con 0x7fb6d8100700 2024-03-16T10:24:50.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.309+0000 7fb6e073b640 1 -- 172.21.15.77:0/3626710502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d819e930 con 0x7fb6d8100700 2024-03-16T10:24:50.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.310+0000 7fb6e073b640 1 -- 172.21.15.77:0/3626710502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d819eea0 con 0x7fb6d8100700 2024-03-16T10:24:50.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.310+0000 7fb6e073b640 1 -- 172.21.15.77:0/3626710502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d8107a20 con 0x7fb6d8100700 2024-03-16T10:24:50.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.311+0000 7fb6cf7fe640 1 -- 172.21.15.77:0/3626710502 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fb6d40037a0 con 0x7fb6d8100700 2024-03-16T10:24:50.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.312+0000 7fb6cf7fe640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb6c40a2880 0x7fb6c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.312+0000 7fb6cf7fe640 1 -- 172.21.15.77:0/3626710502 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fb6d40cedb0 con 0x7fb6d8100700 2024-03-16T10:24:50.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.312+0000 7fb6ddcd6640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb6c40a2880 0x7fb6c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.313+0000 7fb6ddcd6640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb6c40a2880 0x7fb6c40a4d40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8006010 tx=0x7fb6c802c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.317+0000 7fb6cf7fe640 1 -- 172.21.15.77:0/3626710502 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb6d409bee0 con 0x7fb6d8100700 2024-03-16T10:24:50.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.561+0000 7fb6e073b640 1 -- 172.21.15.77:0/3626710502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6d802d1b0 con 0x7fb6d8100700 2024-03-16T10:24:50.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.562+0000 7fb6cf7fe640 1 -- 172.21.15.77:0/3626710502 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+11946 (secure 0 0 0) 0x7fb6d409b7e0 con 0x7fb6d8100700 2024-03-16T10:24:50.563 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:24:50.563 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":43,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:24:42.466172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26800":{"gid":26800,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/946505511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":946505511},{"type":"v1","addr":"172.21.15.193:6835","nonce":946505511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24475,24454,24482,14658,24455]},"id":1}]} 2024-03-16T10:24:50.563 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 43 2024-03-16T10:24:50.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.565+0000 7fb6e073b640 1 -- 172.21.15.77:0/3626710502 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb6c40a2880 msgr2=0x7fb6c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.565+0000 7fb6e073b640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb6c40a2880 0x7fb6c40a4d40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8006010 tx=0x7fb6c802c8a0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.565+0000 7fb6e073b640 1 -- 172.21.15.77:0/3626710502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6d8100700 msgr2=0x7fb6d8196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.565+0000 7fb6e073b640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6d8100700 0x7fb6d8196a20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400e9c0 tx=0x7fb6d400ee80 comp rx=0 tx=0).stop 2024-03-16T10:24:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.565+0000 7fb6e073b640 1 -- 172.21.15.77:0/3626710502 shutdown_connections 2024-03-16T10:24:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.565+0000 7fb6e073b640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb6c40a2880 0x7fb6c40a4d40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.565+0000 7fb6e073b640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6d81022b0 0x7fb6d819dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.565+0000 7fb6e073b640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6d81018f0 0x7fb6d8196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.565+0000 7fb6e073b640 1 --2- 172.21.15.77:0/3626710502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6d8100700 0x7fb6d8196a20 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.565+0000 7fb6e073b640 1 -- 172.21.15.77:0/3626710502 >> 172.21.15.77:0/3626710502 conn(0x7fb6d80fbc70 msgr2=0x7fb6d8104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.565+0000 7fb6e073b640 1 -- 172.21.15.77:0/3626710502 shutdown_connections 2024-03-16T10:24:50.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.565+0000 7fb6e073b640 1 -- 172.21.15.77:0/3626710502 wait complete. 2024-03-16T10:24:50.583 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-16T10:24:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.728+0000 7f86eded1640 1 -- 172.21.15.77:0/3996438381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e8102a00 msgr2=0x7f86e8102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.728+0000 7f86eded1640 1 --2- 172.21.15.77:0/3996438381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e8102a00 0x7f86e8102e80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009900 tx=0x7f86dc02bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:50.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.728+0000 7f86eded1640 1 -- 172.21.15.77:0/3996438381 shutdown_connections 2024-03-16T10:24:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.728+0000 7f86eded1640 1 --2- 172.21.15.77:0/3996438381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86e81033c0 0x7f86e8109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.728+0000 7f86eded1640 1 --2- 172.21.15.77:0/3996438381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e8102a00 0x7f86e8102e80 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.728+0000 7f86eded1640 1 --2- 172.21.15.77:0/3996438381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86e8101810 0x7f86e8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.728+0000 7f86eded1640 1 -- 172.21.15.77:0/3996438381 >> 172.21.15.77:0/3996438381 conn(0x7f86e80fcd80 msgr2=0x7f86e80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.728+0000 7f86eded1640 1 -- 172.21.15.77:0/3996438381 shutdown_connections 2024-03-16T10:24:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.728+0000 7f86eded1640 1 -- 172.21.15.77:0/3996438381 wait complete. 2024-03-16T10:24:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.729+0000 7f86eded1640 1 Processor -- start 2024-03-16T10:24:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.729+0000 7f86eded1640 1 -- start start 2024-03-16T10:24:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.729+0000 7f86eded1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86e8101810 0x7f86e8197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.729+0000 7f86eded1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e8102a00 0x7f86e8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.729+0000 7f86eded1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86e81033c0 0x7f86e819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.729+0000 7f86eded1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86e810d490 con 0x7f86e8102a00 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.729+0000 7f86eded1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f86e810d310 con 0x7f86e81033c0 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.729+0000 7f86eded1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f86e810d610 con 0x7f86e8101810 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.729+0000 7f86e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e8102a00 0x7f86e8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86e77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86e8101810 0x7f86e8197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.729+0000 7f86e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e8102a00 0x7f86e8198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49074/0 (socket says 172.21.15.77:49074) 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86e6ffd640 1 -- 172.21.15.77:0/1042991295 learned_addr learned my addr 172.21.15.77:0/1042991295 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86e7fff640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86e81033c0 0x7f86e819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86e6ffd640 1 -- 172.21.15.77:0/1042991295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86e8101810 msgr2=0x7f86e8197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86e6ffd640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86e8101810 0x7f86e8197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86e6ffd640 1 -- 172.21.15.77:0/1042991295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86e81033c0 msgr2=0x7f86e819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86e6ffd640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86e81033c0 0x7f86e819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86e6ffd640 1 -- 172.21.15.77:0/1042991295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e819f720 con 0x7f86e8102a00 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86e6ffd640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e8102a00 0x7f86e8198040 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009230 tx=0x7f86dc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86e4ff9640 1 -- 172.21.15.77:0/1042991295 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc041070 con 0x7f86e8102a00 2024-03-16T10:24:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86e4ff9640 1 -- 172.21.15.77:0/1042991295 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86dc004450 con 0x7f86e8102a00 2024-03-16T10:24:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86eded1640 1 -- 172.21.15.77:0/1042991295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e819f9b0 con 0x7f86e8102a00 2024-03-16T10:24:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86e4ff9640 1 -- 172.21.15.77:0/1042991295 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc003e20 con 0x7f86e8102a00 2024-03-16T10:24:50.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.730+0000 7f86eded1640 1 -- 172.21.15.77:0/1042991295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e819fef0 con 0x7f86e8102a00 2024-03-16T10:24:50.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86f0b97740 1 Processor -- start 2024-03-16T10:24:50.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86f0b97740 1 -- start start 2024-03-16T10:24:50.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86e4ff9640 1 -- 172.21.15.77:0/1042991295 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f86dc044070 con 0x7f86e8102a00 2024-03-16T10:24:50.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86f0b97740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efbff850 0x55b5efbffc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86f0b97740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efc00240 0x55b5efc04d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86f0b97740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc053b0 0x55b5efc07840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86f0b97740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b5efb64490 con 0x55b5efc053b0 2024-03-16T10:24:50.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86f0b97740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55b5efb0cc90 con 0x55b5efc00240 2024-03-16T10:24:50.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86f0b97740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b5efb60530 con 0x55b5efbff850 2024-03-16T10:24:50.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc053b0 0x55b5efc07840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc053b0 0x55b5efc07840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49088/0 (socket says 172.21.15.77:49088) 2024-03-16T10:24:50.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86e6ffd640 1 -- 172.21.15.77:0/1561044280 learned_addr learned my addr 172.21.15.77:0/1561044280 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86e77fe640 1 --2- 172.21.15.77:0/1561044280 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efc00240 0x55b5efc04d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.733+0000 7f86e7fff640 1 --2- 172.21.15.77:0/1561044280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efbff850 0x55b5efbffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.734+0000 7f86e7fff640 1 -- 172.21.15.77:0/1561044280 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efc00240 msgr2=0x55b5efc04d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.734+0000 7f86e7fff640 1 --2- 172.21.15.77:0/1561044280 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efc00240 0x55b5efc04d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.734+0000 7f86e7fff640 1 -- 172.21.15.77:0/1561044280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc053b0 msgr2=0x55b5efc07840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.734+0000 7f86e7fff640 1 --2- 172.21.15.77:0/1561044280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc053b0 0x55b5efc07840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.734+0000 7f86e7fff640 1 -- 172.21.15.77:0/1561044280 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5efbc7500 con 0x55b5efbff850 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.734+0000 7f86e4ff9640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f86d00a2880 0x7f86d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.734+0000 7f86e77fe640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f86d00a2880 0x7f86d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.734+0000 7f86e4ff9640 1 -- 172.21.15.77:0/1042991295 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f86dc0f04c0 con 0x7f86e8102a00 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.734+0000 7f86e7fff640 1 --2- 172.21.15.77:0/1561044280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efbff850 0x55b5efbffc70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f86e0009f00 tx=0x7f86e0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4359e5337846e712 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.734+0000 7f86cdffb640 1 -- 172.21.15.77:0/1561044280 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e0004d20 con 0x55b5efbff850 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.734+0000 7f86cdffb640 1 -- 172.21.15.77:0/1561044280 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86e0004030 con 0x55b5efbff850 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.734+0000 7f86cdffb640 1 -- 172.21.15.77:0/1561044280 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e0005260 con 0x55b5efbff850 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.735+0000 7f86f0b97740 1 -- 172.21.15.77:0/1561044280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efbff850 msgr2=0x55b5efbffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.735+0000 7f86f0b97740 1 --2- 172.21.15.77:0/1561044280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efbff850 0x55b5efbffc70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f86e0009f00 tx=0x7f86e0011810 comp rx=0 tx=0).stop 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.735+0000 7f86e77fe640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f86d00a2880 0x7f86d00a4d40 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x55b5efc00640 tx=0x7f86d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.735+0000 7f86f0b97740 1 -- 172.21.15.77:0/1561044280 shutdown_connections 2024-03-16T10:24:50.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.735+0000 7f86f0b97740 1 --2- 172.21.15.77:0/1561044280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc053b0 0x55b5efc07840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.735+0000 7f86f0b97740 1 --2- 172.21.15.77:0/1561044280 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efc00240 0x55b5efc04d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.735+0000 7f86f0b97740 1 --2- 172.21.15.77:0/1561044280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efbff850 0x55b5efbffc70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.735+0000 7f86f0b97740 1 -- 172.21.15.77:0/1561044280 >> 172.21.15.77:0/1561044280 conn(0x55b5efbfadc0 msgr2=0x55b5efbfb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.735+0000 7f86f0b97740 1 -- 172.21.15.77:0/1561044280 shutdown_connections 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.735+0000 7f86f0b97740 1 -- 172.21.15.77:0/1561044280 wait complete. 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.735+0000 7f86f0b97740 1 -- start start 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.735+0000 7f86f0b97740 1 Processor -- start 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86f0b97740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efbff850 0x55b5efc09df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86f0b97740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc00240 0x55b5efc0a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86f0b97740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efc053b0 0x55b5efc11330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86f0b97740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b5efc08360 con 0x55b5efbff850 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86f0b97740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55b5efb20230 con 0x55b5efc053b0 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86f0b97740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b5efc084e0 con 0x55b5efc00240 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86f0b97740 10 client.0 _ll_register_callbacks cb 0x55b5efb5c010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86e77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efbff850 0x55b5efc09df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86e77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efbff850 0x55b5efc09df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49102/0 (socket says 172.21.15.77:49102) 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86e77fe640 1 -- 172.21.15.77:0/1372033799 learned_addr learned my addr 172.21.15.77:0/1372033799 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86e6ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc00240 0x55b5efc0a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86e7fff640 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efc053b0 0x55b5efc11330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86e6ffd640 1 -- 172.21.15.77:0/1372033799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efc053b0 msgr2=0x55b5efc11330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86e6ffd640 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efc053b0 0x55b5efc11330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86e6ffd640 1 -- 172.21.15.77:0/1372033799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efbff850 msgr2=0x55b5efc09df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86e6ffd640 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efbff850 0x55b5efc09df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86e6ffd640 1 -- 172.21.15.77:0/1372033799 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5efb2f100 con 0x55b5efc00240 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86e77fe640 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efbff850 0x55b5efc09df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86e6ffd640 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc00240 0x55b5efc0a330 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009430 tx=0x7f86dc005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=993068ef0db30ac3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86c6ffd640 1 -- 172.21.15.77:0/1372033799 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc04b070 con 0x55b5efc00240 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86c6ffd640 1 -- 172.21.15.77:0/1372033799 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f86dc0f00c0 con 0x55b5efc00240 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.736+0000 7f86f0b97740 1 -- 172.21.15.77:0/1372033799 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86e0004030 con 0x55b5efc00240 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.737+0000 7f86c6ffd640 1 -- 172.21.15.77:0/1372033799 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f86dc0bcae0 con 0x55b5efc00240 2024-03-16T10:24:50.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.737+0000 7f86f0b97740 10 client.27085 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.737+0000 7f86f0b97740 1 -- 172.21.15.77:0/1372033799 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55b5ef8783f0 con 0x55b5efc00240 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.737+0000 7f86c6ffd640 1 -- 172.21.15.77:0/1372033799 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86e0004030 con 0x55b5efc00240 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.737+0000 7f86c6ffd640 1 -- 172.21.15.77:0/1372033799 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f86dc0c5200 con 0x55b5efc00240 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.737+0000 7f86f0b97740 10 client.27085 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.737+0000 7f86f0b97740 10 client.27085 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.737+0000 7f86f0b97740 20 client.27085 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.737+0000 7f86f0b97740 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b5efc15c60 0x55b5efc180e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.737+0000 7f86f0b97740 4 client.27085 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.737+0000 7f86f0b97740 1 -- 172.21.15.77:0/1372033799 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5efb719d0 con 0x55b5efc15c60 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.737+0000 7f86e77fe640 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b5efc15c60 0x55b5efc180e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86c6ffd640 10 client.27085 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86e77fe640 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b5efc15c60 0x55b5efc180e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9647989c93cc986e server_cookie=ee5fe021f76bf67a in_seq=0 out_seq=0 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86c6ffd640 1 -- 172.21.15.77:0/1372033799 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b5efb719d0 con 0x55b5efc15c60 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86c6ffd640 10 client.27085 handle_command_reply: tid=0 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86f0b97740 1 client.27085 shutdown 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86f0b97740 1 -- 172.21.15.77:0/1372033799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc00240 msgr2=0x55b5efc0a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86f0b97740 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc00240 0x55b5efc0a330 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009430 tx=0x7f86dc005bd0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86f0b97740 1 -- 172.21.15.77:0/1372033799 shutdown_connections 2024-03-16T10:24:50.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86f0b97740 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efc053b0 0x55b5efc11330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86f0b97740 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b5efc15c60 0x55b5efc180e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86f0b97740 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc00240 0x55b5efc0a330 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86c6ffd640 0 client.27085 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:24:50.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86f0b97740 1 --2- 172.21.15.77:0/1372033799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efbff850 0x55b5efc09df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86f0b97740 1 -- 172.21.15.77:0/1372033799 >> 172.21.15.77:0/1372033799 conn(0x55b5efbfadc0 msgr2=0x55b5efbf9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86f0b97740 1 -- 172.21.15.77:0/1372033799 shutdown_connections 2024-03-16T10:24:50.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.738+0000 7f86f0b97740 1 -- 172.21.15.77:0/1372033799 wait complete. 2024-03-16T10:24:50.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.739+0000 7f86f0b97740 20 client.27085 trim_cache size 0 max 16384 2024-03-16T10:24:50.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86f0b97740 1 Processor -- start 2024-03-16T10:24:50.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86f0b97740 1 -- start start 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86f0b97740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efbff850 0x55b5efc017d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86f0b97740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 0x55b5efc01d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86f0b97740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc053b0 0x55b5efc112f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86f0b97740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81a8b10 con 0x55b5efc00240 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86f0b97740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81aa6e0 con 0x55b5efbff850 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86f0b97740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81a9210 con 0x55b5efc053b0 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 0x55b5efc01d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 0x55b5efc01d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49116/0 (socket says 172.21.15.77:49116) 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86e6ffd640 1 -- 172.21.15.77:0/4151088631 learned_addr learned my addr 172.21.15.77:0/4151088631 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86e77fe640 1 --2- 172.21.15.77:0/4151088631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efbff850 0x55b5efc017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86e7fff640 1 --2- 172.21.15.77:0/4151088631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc053b0 0x55b5efc112f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86e6ffd640 1 -- 172.21.15.77:0/4151088631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc053b0 msgr2=0x55b5efc112f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86e6ffd640 1 --2- 172.21.15.77:0/4151088631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc053b0 0x55b5efc112f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86e6ffd640 1 -- 172.21.15.77:0/4151088631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efbff850 msgr2=0x55b5efc017d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86e6ffd640 1 --2- 172.21.15.77:0/4151088631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efbff850 0x55b5efc017d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.752+0000 7f86e6ffd640 1 -- 172.21.15.77:0/4151088631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ef9427d0 con 0x55b5efc00240 2024-03-16T10:24:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86e6ffd640 1 --2- 172.21.15.77:0/4151088631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 0x55b5efc01d10 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0bcca0 tx=0x7f86dc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a3ca2aa9cef330 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86cd7fa640 1 -- 172.21.15.77:0/4151088631 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc04b070 con 0x55b5efc00240 2024-03-16T10:24:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86cd7fa640 1 -- 172.21.15.77:0/4151088631 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86dc004c60 con 0x55b5efc00240 2024-03-16T10:24:50.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86cd7fa640 1 -- 172.21.15.77:0/4151088631 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc0bc900 con 0x55b5efc00240 2024-03-16T10:24:50.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86f0b97740 1 -- 172.21.15.77:0/4151088631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 msgr2=0x55b5efc01d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86f0b97740 1 --2- 172.21.15.77:0/4151088631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 0x55b5efc01d10 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0bcca0 tx=0x7f86dc02bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:50.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86f0b97740 1 -- 172.21.15.77:0/4151088631 shutdown_connections 2024-03-16T10:24:50.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86f0b97740 1 --2- 172.21.15.77:0/4151088631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc053b0 0x55b5efc112f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86f0b97740 1 --2- 172.21.15.77:0/4151088631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 0x55b5efc01d10 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86f0b97740 1 --2- 172.21.15.77:0/4151088631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efbff850 0x55b5efc017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86f0b97740 1 -- 172.21.15.77:0/4151088631 >> 172.21.15.77:0/4151088631 conn(0x55b5efbfadc0 msgr2=0x55b5efc09df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86f0b97740 1 -- 172.21.15.77:0/4151088631 shutdown_connections 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.753+0000 7f86f0b97740 1 -- 172.21.15.77:0/4151088631 wait complete. 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86f0b97740 1 -- start start 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86f0b97740 1 Processor -- start 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86f0b97740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efbff850 0x55b5efc0e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86f0b97740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 0x55b5efc0ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86f0b97740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc053b0 0x55b5efc0fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86f0b97740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81aa010 con 0x55b5efc00240 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86f0b97740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81a9910 con 0x55b5efbff850 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86f0b97740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81a6af0 con 0x55b5efc053b0 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86f0b97740 10 client.0 _ll_register_callbacks cb 0x7f86dc009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:50.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 0x55b5efc0ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86e6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 0x55b5efc0ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49130/0 (socket says 172.21.15.77:49130) 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86e6ffd640 1 -- 172.21.15.77:0/853000351 learned_addr learned my addr 172.21.15.77:0/853000351 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86e77fe640 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efbff850 0x55b5efc0e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86e7fff640 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc053b0 0x55b5efc0fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86e6ffd640 1 -- 172.21.15.77:0/853000351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc053b0 msgr2=0x55b5efc0fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.754+0000 7f86e6ffd640 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc053b0 0x55b5efc0fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86e6ffd640 1 -- 172.21.15.77:0/853000351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efbff850 msgr2=0x55b5efc0e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86e6ffd640 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efbff850 0x55b5efc0e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86e6ffd640 1 -- 172.21.15.77:0/853000351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5efc10220 con 0x55b5efc00240 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86e77fe640 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efbff850 0x55b5efc0e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86e6ffd640 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 0x55b5efc0ec20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0b9400 tx=0x7f86dc0f03f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75cea55d4c5c669d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86c6ffd640 1 -- 172.21.15.77:0/853000351 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc04b040 con 0x55b5efc00240 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86c6ffd640 1 -- 172.21.15.77:0/853000351 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f86dc0b7b10 con 0x55b5efc00240 2024-03-16T10:24:50.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86f0b97740 1 -- 172.21.15.77:0/853000351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86dc004c60 con 0x55b5efc00240 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86c6ffd640 1 -- 172.21.15.77:0/853000351 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f86dc0bc900 con 0x55b5efc00240 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86f0b97740 10 client.17337 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86f0b97740 1 -- 172.21.15.77:0/853000351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55b5efc10220 con 0x55b5efc00240 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86c6ffd640 1 -- 172.21.15.77:0/853000351 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86dc004c60 con 0x55b5efc00240 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86c6ffd640 1 -- 172.21.15.77:0/853000351 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f86dc0c1f60 con 0x55b5efc00240 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86f0b97740 10 client.17337 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86f0b97740 10 client.17337 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86f0b97740 20 client.17337 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86f0b97740 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b5efc7c230 0x55b5efc7e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86f0b97740 4 client.17337 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.755+0000 7f86f0b97740 1 -- 172.21.15.77:0/853000351 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5efb6bd20 con 0x55b5efc7c230 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86e77fe640 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b5efc7c230 0x55b5efc7e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86c6ffd640 10 client.17337 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:24:50.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86e77fe640 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b5efc7c230 0x55b5efc7e620 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3fb7efd30ae5524c server_cookie=7a3a028e84156e4f in_seq=0 out_seq=0 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86c6ffd640 1 -- 172.21.15.77:0/853000351 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5efb6bd20 con 0x55b5efc7c230 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86c6ffd640 10 client.17337 handle_command_reply: tid=0 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86f0b97740 1 client.17337 shutdown 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86f0b97740 1 -- 172.21.15.77:0/853000351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 msgr2=0x55b5efc0ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86f0b97740 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 0x55b5efc0ec20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0b9400 tx=0x7f86dc0f03f0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86f0b97740 1 -- 172.21.15.77:0/853000351 shutdown_connections 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86f0b97740 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55b5efc7c230 0x55b5efc7e620 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86f0b97740 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5efc053b0 0x55b5efc0fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86c6ffd640 0 client.17337 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86f0b97740 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5efc00240 0x55b5efc0ec20 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86f0b97740 1 --2- 172.21.15.77:0/853000351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5efbff850 0x55b5efc0e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86f0b97740 1 -- 172.21.15.77:0/853000351 >> 172.21.15.77:0/853000351 conn(0x55b5efbfadc0 msgr2=0x55b5efc09db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86f0b97740 1 -- 172.21.15.77:0/853000351 shutdown_connections 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86f0b97740 1 -- 172.21.15.77:0/853000351 wait complete. 2024-03-16T10:24:50.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.756+0000 7f86f0b97740 20 client.17337 trim_cache size 0 max 16384 2024-03-16T10:24:50.764 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:24:50.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.757+0000 7f86eded1640 1 -- 172.21.15.77:0/1042991295 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f86d00a2880 msgr2=0x7f86d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.757+0000 7f86eded1640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f86d00a2880 0x7f86d00a4d40 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x55b5efc00640 tx=0x7f86d8005f90 comp rx=0 tx=0).stop 2024-03-16T10:24:50.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.757+0000 7f86eded1640 1 -- 172.21.15.77:0/1042991295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e8102a00 msgr2=0x7f86e8198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.757+0000 7f86eded1640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e8102a00 0x7f86e8198040 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009230 tx=0x7f86dc02c650 comp rx=0 tx=0).stop 2024-03-16T10:24:50.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.757+0000 7f86eded1640 1 -- 172.21.15.77:0/1042991295 shutdown_connections 2024-03-16T10:24:50.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.757+0000 7f86eded1640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86e81033c0 0x7f86e819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.757+0000 7f86eded1640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86e8102a00 0x7f86e8198040 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.757+0000 7f86eded1640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f86d00a2880 0x7f86d00a4d40 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.757+0000 7f86eded1640 1 --2- 172.21.15.77:0/1042991295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86e8101810 0x7f86e8197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.757+0000 7f86eded1640 1 -- 172.21.15.77:0/1042991295 >> 172.21.15.77:0/1042991295 conn(0x7f86e80fcd80 msgr2=0x7f86e8105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.757+0000 7f86eded1640 1 -- 172.21.15.77:0/1042991295 shutdown_connections 2024-03-16T10:24:50.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.757+0000 7f86eded1640 1 -- 172.21.15.77:0/1042991295 wait complete. 2024-03-16T10:24:50.769 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-16T10:24:50.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.879+0000 7fcc6410c640 1 -- 172.21.15.77:0/1307256179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc5c1033c0 msgr2=0x7fcc5c109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.879+0000 7fcc6410c640 1 --2- 172.21.15.77:0/1307256179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc5c1033c0 0x7fcc5c109c50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fcc54009be0 tx=0x7fcc5402bef0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.879+0000 7fcc6410c640 1 -- 172.21.15.77:0/1307256179 shutdown_connections 2024-03-16T10:24:50.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.879+0000 7fcc6410c640 1 --2- 172.21.15.77:0/1307256179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc5c1033c0 0x7fcc5c109c50 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.879+0000 7fcc6410c640 1 --2- 172.21.15.77:0/1307256179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5c102a00 0x7fcc5c102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.879+0000 7fcc6410c640 1 --2- 172.21.15.77:0/1307256179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc5c101810 0x7fcc5c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.879+0000 7fcc6410c640 1 -- 172.21.15.77:0/1307256179 >> 172.21.15.77:0/1307256179 conn(0x7fcc5c0fcd80 msgr2=0x7fcc5c0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.879+0000 7fcc6410c640 1 -- 172.21.15.77:0/1307256179 shutdown_connections 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.879+0000 7fcc6410c640 1 -- 172.21.15.77:0/1307256179 wait complete. 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc6410c640 1 Processor -- start 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc6410c640 1 -- start start 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc6410c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5c101810 0x7fcc5c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc6410c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc5c102a00 0x7fcc5c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc6410c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc5c1033c0 0x7fcc5c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc6410c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c10d470 con 0x7fcc5c102a00 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc6410c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c10d2f0 con 0x7fcc5c1033c0 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc6410c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c10d5f0 con 0x7fcc5c101810 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc616a7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc5c102a00 0x7fcc5c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc616a7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc5c102a00 0x7fcc5c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49142/0 (socket says 172.21.15.77:49142) 2024-03-16T10:24:50.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc616a7640 1 -- 172.21.15.77:0/2271881069 learned_addr learned my addr 172.21.15.77:0/2271881069 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc61ea8640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5c101810 0x7fcc5c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc626a9640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc5c1033c0 0x7fcc5c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc616a7640 1 -- 172.21.15.77:0/2271881069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5c101810 msgr2=0x7fcc5c197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc616a7640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5c101810 0x7fcc5c197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc616a7640 1 -- 172.21.15.77:0/2271881069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc5c1033c0 msgr2=0x7fcc5c19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc616a7640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc5c1033c0 0x7fcc5c19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc616a7640 1 -- 172.21.15.77:0/2271881069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5c19f6d0 con 0x7fcc5c102a00 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.880+0000 7fcc616a7640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc5c102a00 0x7fcc5c197ff0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800ea60 tx=0x7fcc5800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.881+0000 7fcc52ffd640 1 -- 172.21.15.77:0/2271881069 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5800cde0 con 0x7fcc5c102a00 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.881+0000 7fcc6410c640 1 -- 172.21.15.77:0/2271881069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc5c19f9c0 con 0x7fcc5c102a00 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.881+0000 7fcc52ffd640 1 -- 172.21.15.77:0/2271881069 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc580035f0 con 0x7fcc5c102a00 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.881+0000 7fcc52ffd640 1 -- 172.21.15.77:0/2271881069 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc58010670 con 0x7fcc5c102a00 2024-03-16T10:24:50.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.881+0000 7fcc6410c640 1 -- 172.21.15.77:0/2271881069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc5c19ff60 con 0x7fcc5c102a00 2024-03-16T10:24:50.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.883+0000 7fcc52ffd640 1 -- 172.21.15.77:0/2271881069 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fcc580037a0 con 0x7fcc5c102a00 2024-03-16T10:24:50.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.883+0000 7fcc66dcc740 1 Processor -- start 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc66dcc740 1 -- start start 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc66dcc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 0x55c866563c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc66dcc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c866568d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc66dcc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 0x55c86656b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc66dcc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c8664c8490 con 0x55c866563850 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc66dcc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c866470c90 con 0x55c866564240 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc66dcc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c8664c4530 con 0x55c8665693b0 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc626a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 0x55c866563c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc61ea8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c866568d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc626a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 0x55c866563c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49148/0 (socket says 172.21.15.77:49148) 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc626a9640 1 -- 172.21.15.77:0/2794951660 learned_addr learned my addr 172.21.15.77:0/2794951660 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc52ffd640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcc480a2800 0x7fcc480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc616a7640 1 --2- 172.21.15.77:0/2794951660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 0x55c86656b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc626a9640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcc480a2800 0x7fcc480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc61ea8640 1 -- 172.21.15.77:0/2794951660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 msgr2=0x55c86656b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc61ea8640 1 --2- 172.21.15.77:0/2794951660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 0x55c86656b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc61ea8640 1 -- 172.21.15.77:0/2794951660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 msgr2=0x55c866563c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc61ea8640 1 --2- 172.21.15.77:0/2794951660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 0x55c866563c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc61ea8640 1 -- 172.21.15.77:0/2794951660 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c86652b500 con 0x55c866564240 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc52ffd640 1 -- 172.21.15.77:0/2271881069 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fcc580cecc0 con 0x7fcc5c102a00 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.884+0000 7fcc61ea8640 1 --2- 172.21.15.77:0/2794951660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c866568d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c102860 tx=0x7fcc4c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97ef225449b39151 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.885+0000 7fcc626a9640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcc480a2800 0x7fcc480a4cc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x55c866564aa0 tx=0x7fcc5402bef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.885+0000 7fcc47fff640 1 -- 172.21.15.77:0/2794951660 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c01e070 con 0x55c866564240 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.885+0000 7fcc47fff640 1 -- 172.21.15.77:0/2794951660 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc4c019910 con 0x55c866564240 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.885+0000 7fcc47fff640 1 -- 172.21.15.77:0/2794951660 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c00ea10 con 0x55c866564240 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.885+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2794951660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 msgr2=0x55c866568d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.885+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2794951660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c866568d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c102860 tx=0x7fcc4c011810 comp rx=0 tx=0).stop 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.885+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2794951660 shutdown_connections 2024-03-16T10:24:50.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.885+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2794951660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 0x55c86656b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.885+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2794951660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c866568d40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.885+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2794951660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 0x55c866563c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.885+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2794951660 >> 172.21.15.77:0/2794951660 conn(0x55c86655edc0 msgr2=0x55c86655f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2794951660 shutdown_connections 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2794951660 wait complete. 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc66dcc740 1 -- start start 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc66dcc740 1 Processor -- start 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc66dcc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866563850 0x55c86656dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc66dcc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866564240 0x55c86656e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc66dcc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 0x55c866575300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc66dcc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c86656c360 con 0x55c866564240 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc66dcc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c866484230 con 0x55c866563850 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc66dcc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c86656c4e0 con 0x55c8665693b0 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc66dcc740 10 client.0 _ll_register_callbacks cb 0x55c8664c0010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:50.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc616a7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866564240 0x55c86656e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.886+0000 7fcc61ea8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866563850 0x55c86656dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc616a7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866564240 0x55c86656e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49158/0 (socket says 172.21.15.77:49158) 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc626a9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 0x55c866575300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc616a7640 1 -- 172.21.15.77:0/2185686825 learned_addr learned my addr 172.21.15.77:0/2185686825 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc616a7640 1 -- 172.21.15.77:0/2185686825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 msgr2=0x55c866575300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc616a7640 1 --2- 172.21.15.77:0/2185686825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 0x55c866575300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc616a7640 1 -- 172.21.15.77:0/2185686825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866563850 msgr2=0x55c86656dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc616a7640 1 --2- 172.21.15.77:0/2185686825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866563850 0x55c86656dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc616a7640 1 -- 172.21.15.77:0/2185686825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c866493100 con 0x55c866564240 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc616a7640 1 --2- 172.21.15.77:0/2185686825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866564240 0x55c86656e2c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800c760 tx=0x7fcc580bb7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3ea6613943f27 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc457fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc457fa640 1 -- 172.21.15.77:0/2185686825 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc580b9070 con 0x55c866564240 2024-03-16T10:24:50.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc457fa640 1 -- 172.21.15.77:0/2185686825 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fcc58003b30 con 0x55c866564240 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2185686825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc4c019910 con 0x55c866564240 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc457fa640 1 -- 172.21.15.77:0/2185686825 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fcc580d0070 con 0x55c866564240 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc66dcc740 10 client.17349 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2185686825 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55c8661dc3f0 con 0x55c866564240 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.887+0000 7fcc457fa640 1 -- 172.21.15.77:0/2185686825 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc4c019910 con 0x55c866564240 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.888+0000 7fcc457fa640 1 -- 172.21.15.77:0/2185686825 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fcc5809ab40 con 0x55c866564240 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.888+0000 7fcc66dcc740 10 client.17349 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.888+0000 7fcc66dcc740 10 client.17349 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.888+0000 7fcc66dcc740 20 client.17349 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.888+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2185686825 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55c866579ba0 0x55c86657bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.888+0000 7fcc66dcc740 4 client.17349 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.888+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2185686825 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8664d59d0 con 0x55c866579ba0 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.888+0000 7fcc61ea8640 1 --2- 172.21.15.77:0/2185686825 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55c866579ba0 0x55c86657bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.888+0000 7fcc457fa640 10 client.17349 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:24:50.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.888+0000 7fcc61ea8640 1 --2- 172.21.15.77:0/2185686825 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55c866579ba0 0x55c86657bff0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9755b40f19ec201c server_cookie=d39ba8fbd27b3664 in_seq=0 out_seq=0 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.889+0000 7fcc457fa640 1 -- 172.21.15.77:0/2185686825 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c8664d59d0 con 0x55c866579ba0 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.889+0000 7fcc457fa640 10 client.17349 handle_command_reply: tid=0 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.889+0000 7fcc66dcc740 1 client.17349 shutdown 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.889+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2185686825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866564240 msgr2=0x55c86656e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.889+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2185686825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866564240 0x55c86656e2c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800c760 tx=0x7fcc580bb7e0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.889+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2185686825 shutdown_connections 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.889+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2185686825 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55c866579ba0 0x55c86657bff0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.889+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2185686825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 0x55c866575300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.890+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2185686825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866564240 0x55c86656e2c0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.890+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2185686825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866563850 0x55c86656dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.890+0000 7fcc457fa640 0 client.17349 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.890+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2185686825 >> 172.21.15.77:0/2185686825 conn(0x55c86655edc0 msgr2=0x55c86655dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.890+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2185686825 shutdown_connections 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.890+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2185686825 wait complete. 2024-03-16T10:24:50.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.890+0000 7fcc66dcc740 20 client.17349 trim_cache size 0 max 16384 2024-03-16T10:24:50.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.903+0000 7fcc66dcc740 1 Processor -- start 2024-03-16T10:24:50.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.903+0000 7fcc66dcc740 1 -- start start 2024-03-16T10:24:50.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.903+0000 7fcc66dcc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c866563850 0x55c866565740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.903+0000 7fcc66dcc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c8665752c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.903+0000 7fcc66dcc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c8665693b0 0x55c866565c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.903+0000 7fcc66dcc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c1a8be0 con 0x55c8665693b0 2024-03-16T10:24:50.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.903+0000 7fcc66dcc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c1aa7b0 con 0x55c866564240 2024-03-16T10:24:50.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.903+0000 7fcc66dcc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c1a92e0 con 0x55c866563850 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc626a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c8665693b0 0x55c866565c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc626a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c8665693b0 0x55c866565c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49164/0 (socket says 172.21.15.77:49164) 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc626a9640 1 -- 172.21.15.77:0/2384466124 learned_addr learned my addr 172.21.15.77:0/2384466124 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc61ea8640 1 --2- 172.21.15.77:0/2384466124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c866563850 0x55c866565740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc616a7640 1 --2- 172.21.15.77:0/2384466124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c8665752c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc626a9640 1 -- 172.21.15.77:0/2384466124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c866563850 msgr2=0x55c866565740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc626a9640 1 --2- 172.21.15.77:0/2384466124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c866563850 0x55c866565740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc626a9640 1 -- 172.21.15.77:0/2384466124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 msgr2=0x55c8665752c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc626a9640 1 --2- 172.21.15.77:0/2384466124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c8665752c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc626a9640 1 -- 172.21.15.77:0/2384466124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8662a67d0 con 0x55c8665693b0 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc626a9640 1 --2- 172.21.15.77:0/2384466124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c8665693b0 0x55c866565c80 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fcc54005f00 tx=0x7fcc54005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d67d5b562616fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc477fe640 1 -- 172.21.15.77:0/2384466124 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc54049070 con 0x55c8665693b0 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc477fe640 1 -- 172.21.15.77:0/2384466124 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc54054070 con 0x55c8665693b0 2024-03-16T10:24:50.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.904+0000 7fcc477fe640 1 -- 172.21.15.77:0/2384466124 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc54034690 con 0x55c8665693b0 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2384466124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c8665693b0 msgr2=0x55c866565c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2384466124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c8665693b0 0x55c866565c80 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fcc54005f00 tx=0x7fcc54005e30 comp rx=0 tx=0).stop 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2384466124 shutdown_connections 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2384466124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c8665693b0 0x55c866565c80 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2384466124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c8665752c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/2384466124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c866563850 0x55c866565740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2384466124 >> 172.21.15.77:0/2384466124 conn(0x55c86655edc0 msgr2=0x55c86656dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2384466124 shutdown_connections 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 -- 172.21.15.77:0/2384466124 wait complete. 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 -- start start 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 Processor -- start 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 0x55c866572380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c8665728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 0x55c8665738d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c1aa0e0 con 0x55c866563850 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c1a99e0 con 0x55c866564240 2024-03-16T10:24:50.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.905+0000 7fcc66dcc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c1a6be0 con 0x55c8665693b0 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc66dcc740 10 client.0 _ll_register_callbacks cb 0x7fcc58003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc61ea8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 0x55c866572380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc61ea8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 0x55c866572380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49170/0 (socket says 172.21.15.77:49170) 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc61ea8640 1 -- 172.21.15.77:0/713198285 learned_addr learned my addr 172.21.15.77:0/713198285 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc616a7640 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c8665728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc626a9640 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 0x55c8665738d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc61ea8640 1 -- 172.21.15.77:0/713198285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 msgr2=0x55c8665738d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc61ea8640 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 0x55c8665738d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc61ea8640 1 -- 172.21.15.77:0/713198285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 msgr2=0x55c8665728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc61ea8640 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c8665728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc61ea8640 1 -- 172.21.15.77:0/713198285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c866573ec0 con 0x55c866563850 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc61ea8640 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 0x55c866572380 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c01c040 tx=0x7fcc4c0092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2da1fe431619824 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc457fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc457fa640 1 -- 172.21.15.77:0/713198285 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c01e040 con 0x55c866563850 2024-03-16T10:24:50.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc457fa640 1 -- 172.21.15.77:0/713198285 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fcc4c005240 con 0x55c866563850 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc66dcc740 1 -- 172.21.15.77:0/713198285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc54054070 con 0x55c866563850 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc457fa640 1 -- 172.21.15.77:0/713198285 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fcc4c005d10 con 0x55c866563850 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc66dcc740 10 client.17355 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.906+0000 7fcc66dcc740 1 -- 172.21.15.77:0/713198285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55c866575230 con 0x55c866563850 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.907+0000 7fcc457fa640 1 -- 172.21.15.77:0/713198285 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc54054070 con 0x55c866563850 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.907+0000 7fcc457fa640 1 -- 172.21.15.77:0/713198285 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fcc4c031420 con 0x55c866563850 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.907+0000 7fcc66dcc740 10 client.17355 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.907+0000 7fcc66dcc740 10 client.17355 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.907+0000 7fcc66dcc740 20 client.17355 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.907+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55c8665e00d0 0x55c8665e24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.907+0000 7fcc66dcc740 4 client.17355 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.907+0000 7fcc66dcc740 1 -- 172.21.15.77:0/713198285 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c8664cfd20 con 0x55c8665e00d0 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.907+0000 7fcc616a7640 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55c8665e00d0 0x55c8665e24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.907+0000 7fcc457fa640 10 client.17355 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.907+0000 7fcc616a7640 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55c8665e00d0 0x55c8665e24c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=668dc1e4af33aafd server_cookie=c41a52e02807ee64 in_seq=0 out_seq=0 2024-03-16T10:24:50.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc457fa640 1 -- 172.21.15.77:0/713198285 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c8664cfd20 con 0x55c8665e00d0 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc457fa640 10 client.17355 handle_command_reply: tid=0 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc66dcc740 1 client.17355 shutdown 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc66dcc740 1 -- 172.21.15.77:0/713198285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 msgr2=0x55c866572380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 0x55c866572380 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c01c040 tx=0x7fcc4c0092f0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc66dcc740 1 -- 172.21.15.77:0/713198285 shutdown_connections 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55c8665e00d0 0x55c8665e24c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8665693b0 0x55c8665738d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc457fa640 0 client.17355 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c866564240 0x55c8665728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc66dcc740 1 --2- 172.21.15.77:0/713198285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c866563850 0x55c866572380 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc66dcc740 1 -- 172.21.15.77:0/713198285 >> 172.21.15.77:0/713198285 conn(0x55c86655edc0 msgr2=0x55c86656dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc66dcc740 1 -- 172.21.15.77:0/713198285 shutdown_connections 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc66dcc740 1 -- 172.21.15.77:0/713198285 wait complete. 2024-03-16T10:24:50.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc66dcc740 20 client.17355 trim_cache size 0 max 16384 2024-03-16T10:24:50.915 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:24:50.915 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc6410c640 1 -- 172.21.15.77:0/2271881069 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcc480a2800 msgr2=0x7fcc480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc6410c640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcc480a2800 0x7fcc480a4cc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x55c866564aa0 tx=0x7fcc5402bef0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc6410c640 1 -- 172.21.15.77:0/2271881069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc5c102a00 msgr2=0x7fcc5c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:50.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc6410c640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc5c102a00 0x7fcc5c197ff0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800ea60 tx=0x7fcc5800ef20 comp rx=0 tx=0).stop 2024-03-16T10:24:50.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc6410c640 1 -- 172.21.15.77:0/2271881069 shutdown_connections 2024-03-16T10:24:50.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc6410c640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc5c1033c0 0x7fcc5c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc6410c640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc5c102a00 0x7fcc5c197ff0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc6410c640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fcc480a2800 0x7fcc480a4cc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc6410c640 1 --2- 172.21.15.77:0/2271881069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5c101810 0x7fcc5c197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:50.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.908+0000 7fcc6410c640 1 -- 172.21.15.77:0/2271881069 >> 172.21.15.77:0/2271881069 conn(0x7fcc5c0fcd80 msgr2=0x7fcc5c105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:50.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.909+0000 7fcc6410c640 1 -- 172.21.15.77:0/2271881069 shutdown_connections 2024-03-16T10:24:50.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:50.909+0000 7fcc6410c640 1 -- 172.21.15.77:0/2271881069 wait complete. 2024-03-16T10:24:50.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:50 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3626710502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:24:50.919 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:50 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3626710502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:24:50.920 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T10:24:51.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:50 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3626710502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:24:51.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.030+0000 7fba581f4640 1 -- 172.21.15.77:0/551192163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba501006e0 msgr2=0x7fba50100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.030+0000 7fba581f4640 1 --2- 172.21.15.77:0/551192163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba501006e0 0x7fba50100b00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009860 tx=0x7fba4c02bb70 comp rx=0 tx=0).stop 2024-03-16T10:24:51.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.030+0000 7fba581f4640 1 -- 172.21.15.77:0/551192163 shutdown_connections 2024-03-16T10:24:51.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.030+0000 7fba581f4640 1 --2- 172.21.15.77:0/551192163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba50102290 0x7fba50108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.033 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.030+0000 7fba581f4640 1 --2- 172.21.15.77:0/551192163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba501018d0 0x7fba50101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.030+0000 7fba581f4640 1 --2- 172.21.15.77:0/551192163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba501006e0 0x7fba50100b00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.030+0000 7fba581f4640 1 -- 172.21.15.77:0/551192163 >> 172.21.15.77:0/551192163 conn(0x7fba500fbc70 msgr2=0x7fba500fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.030+0000 7fba581f4640 1 -- 172.21.15.77:0/551192163 shutdown_connections 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.030+0000 7fba581f4640 1 -- 172.21.15.77:0/551192163 wait complete. 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba581f4640 1 Processor -- start 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba581f4640 1 -- start start 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba581f4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba501006e0 0x7fba50196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba581f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba501018d0 0x7fba50196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba581f4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba50102290 0x7fba5019e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba581f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fba5010c350 con 0x7fba501018d0 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba581f4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fba5010c1d0 con 0x7fba501006e0 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba581f4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba5010c4d0 con 0x7fba50102290 2024-03-16T10:24:51.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba5578f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba501018d0 0x7fba50196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba5578f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba501018d0 0x7fba50196fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49188/0 (socket says 172.21.15.77:49188) 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba5578f640 1 -- 172.21.15.77:0/3014598097 learned_addr learned my addr 172.21.15.77:0/3014598097 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba55f90640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba501006e0 0x7fba50196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba56791640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba50102290 0x7fba5019e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba5578f640 1 -- 172.21.15.77:0/3014598097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba50102290 msgr2=0x7fba5019e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.031+0000 7fba5578f640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba50102290 0x7fba5019e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.032+0000 7fba5578f640 1 -- 172.21.15.77:0/3014598097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba501006e0 msgr2=0x7fba50196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.032+0000 7fba5578f640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba501006e0 0x7fba50196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.032+0000 7fba5578f640 1 -- 172.21.15.77:0/3014598097 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5019e6a0 con 0x7fba501018d0 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.032+0000 7fba5578f640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba501018d0 0x7fba50196fc0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fba4000e9f0 tx=0x7fba4000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.032+0000 7fba46ffd640 1 -- 172.21.15.77:0/3014598097 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4000cd80 con 0x7fba501018d0 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.032+0000 7fba581f4640 1 -- 172.21.15.77:0/3014598097 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba5019e990 con 0x7fba501018d0 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.032+0000 7fba46ffd640 1 -- 172.21.15.77:0/3014598097 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba40005500 con 0x7fba501018d0 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.032+0000 7fba46ffd640 1 -- 172.21.15.77:0/3014598097 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba40010690 con 0x7fba501018d0 2024-03-16T10:24:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.032+0000 7fba581f4640 1 -- 172.21.15.77:0/3014598097 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba5019ee90 con 0x7fba501018d0 2024-03-16T10:24:51.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.034+0000 7fba46ffd640 1 -- 172.21.15.77:0/3014598097 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fba400046c0 con 0x7fba501018d0 2024-03-16T10:24:51.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.035+0000 7fba46ffd640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fba3c0a2800 0x7fba3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.035+0000 7fba5ade8740 1 Processor -- start 2024-03-16T10:24:51.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.035+0000 7fba55f90640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fba3c0a2800 0x7fba3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.035+0000 7fba46ffd640 1 -- 172.21.15.77:0/3014598097 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fba400cee60 con 0x7fba501018d0 2024-03-16T10:24:51.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.035+0000 7fba5ade8740 1 -- start start 2024-03-16T10:24:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba5ade8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d12dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba5ade8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 0x563c9d132d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba5ade8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d1333b0 0x563c9d135840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba5ade8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563c9d092490 con 0x563c9d12e240 2024-03-16T10:24:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba5ade8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563c9d03ac90 con 0x563c9d12d850 2024-03-16T10:24:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba5ade8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c9d08e530 con 0x563c9d1333b0 2024-03-16T10:24:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba55f90640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fba3c0a2800 0x7fba3c0a4cc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fba4c0349e0 tx=0x7fba4c034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba5578f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 0x563c9d132d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba55f90640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d12dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba56791640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d1333b0 0x563c9d135840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba5578f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 0x563c9d132d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49192/0 (socket says 172.21.15.77:49192) 2024-03-16T10:24:51.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba5578f640 1 -- 172.21.15.77:0/3282414489 learned_addr learned my addr 172.21.15.77:0/3282414489 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba55f90640 1 -- 172.21.15.77:0/3282414489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d1333b0 msgr2=0x563c9d135840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba55f90640 1 --2- 172.21.15.77:0/3282414489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d1333b0 0x563c9d135840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba55f90640 1 -- 172.21.15.77:0/3282414489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 msgr2=0x563c9d132d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba55f90640 1 --2- 172.21.15.77:0/3282414489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 0x563c9d132d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba55f90640 1 -- 172.21.15.77:0/3282414489 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c9d0f5500 con 0x563c9d12d850 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba5578f640 1 --2- 172.21.15.77:0/3282414489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 0x563c9d132d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.036+0000 7fba55f90640 1 --2- 172.21.15.77:0/3282414489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d12dc70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009780 tx=0x7fba4c03e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c20f6047f6bc94e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.037+0000 7fba3bfff640 1 -- 172.21.15.77:0/3282414489 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c039040 con 0x563c9d12d850 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.037+0000 7fba3bfff640 1 -- 172.21.15.77:0/3282414489 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba4c04f070 con 0x563c9d12d850 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.037+0000 7fba3bfff640 1 -- 172.21.15.77:0/3282414489 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c0328a0 con 0x563c9d12d850 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.037+0000 7fba5ade8740 1 -- 172.21.15.77:0/3282414489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 msgr2=0x563c9d12dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.037+0000 7fba5ade8740 1 --2- 172.21.15.77:0/3282414489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d12dc70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009780 tx=0x7fba4c03e000 comp rx=0 tx=0).stop 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.037+0000 7fba5ade8740 1 -- 172.21.15.77:0/3282414489 shutdown_connections 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.037+0000 7fba5ade8740 1 --2- 172.21.15.77:0/3282414489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d1333b0 0x563c9d135840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.037+0000 7fba5ade8740 1 --2- 172.21.15.77:0/3282414489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 0x563c9d132d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.037+0000 7fba5ade8740 1 --2- 172.21.15.77:0/3282414489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d12dc70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.037+0000 7fba5ade8740 1 -- 172.21.15.77:0/3282414489 >> 172.21.15.77:0/3282414489 conn(0x563c9d128dc0 msgr2=0x563c9d1291c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.037+0000 7fba5ade8740 1 -- 172.21.15.77:0/3282414489 shutdown_connections 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.037+0000 7fba5ade8740 1 -- 172.21.15.77:0/3282414489 wait complete. 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba5ade8740 1 -- start start 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba5ade8740 1 Processor -- start 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba5ade8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d137d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba5ade8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12e240 0x563c9d1382a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba5ade8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 0x563c9d13f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba5ade8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563c9d136300 con 0x563c9d1333b0 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba5ade8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563c9d04e230 con 0x563c9d12d850 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba5ade8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c9d136480 con 0x563c9d12e240 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba5ade8740 10 client.0 _ll_register_callbacks cb 0x563c9d08a010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba56791640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 0x563c9d13f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba5578f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12e240 0x563c9d1382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba55f90640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba56791640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 0x563c9d13f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49196/0 (socket says 172.21.15.77:49196) 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.038+0000 7fba56791640 1 -- 172.21.15.77:0/321271491 learned_addr learned my addr 172.21.15.77:0/321271491 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba56791640 1 -- 172.21.15.77:0/321271491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12e240 msgr2=0x563c9d1382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba56791640 1 --2- 172.21.15.77:0/321271491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12e240 0x563c9d1382a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba56791640 1 -- 172.21.15.77:0/321271491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 msgr2=0x563c9d137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba56791640 1 --2- 172.21.15.77:0/321271491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d137d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba56791640 1 -- 172.21.15.77:0/321271491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c9d05d100 con 0x563c9d1333b0 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba56791640 1 --2- 172.21.15.77:0/321271491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 0x563c9d13f2a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fba4800eec0 tx=0x7fba4800d7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b6b028e8f40033e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba397fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba397fa640 1 -- 172.21.15.77:0/321271491 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4801c030 con 0x563c9d1333b0 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba5ade8740 1 -- 172.21.15.77:0/321271491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba4c04f070 con 0x563c9d1333b0 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba397fa640 1 -- 172.21.15.77:0/321271491 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fba48018460 con 0x563c9d1333b0 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba397fa640 1 -- 172.21.15.77:0/321271491 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fba48018660 con 0x563c9d1333b0 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba5ade8740 10 client.17367 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba5ade8740 1 -- 172.21.15.77:0/321271491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x563c9cda63f0 con 0x563c9d1333b0 2024-03-16T10:24:51.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba397fa640 1 -- 172.21.15.77:0/321271491 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba4c04f070 con 0x563c9d1333b0 2024-03-16T10:24:51.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba397fa640 1 -- 172.21.15.77:0/321271491 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fba4800b040 con 0x563c9d1333b0 2024-03-16T10:24:51.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba5ade8740 10 client.17367 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:51.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba5ade8740 10 client.17367 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:24:51.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba5ade8740 20 client.17367 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:51.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba5ade8740 1 --2- 172.21.15.77:0/321271491 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x563c9d143b80 0x563c9d146020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba5ade8740 4 client.17367 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:51.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.039+0000 7fba5ade8740 1 -- 172.21.15.77:0/321271491 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c9d09f9d0 con 0x563c9d143b80 2024-03-16T10:24:51.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.040+0000 7fba55f90640 1 --2- 172.21.15.77:0/321271491 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x563c9d143b80 0x563c9d146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.040+0000 7fba397fa640 10 client.17367 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:24:51.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.040+0000 7fba55f90640 1 --2- 172.21.15.77:0/321271491 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x563c9d143b80 0x563c9d146020 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=61872d55dd47877f server_cookie=e0564a2180e44a6f in_seq=0 out_seq=0 2024-03-16T10:24:51.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba397fa640 1 -- 172.21.15.77:0/321271491 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563c9d09f9d0 con 0x563c9d143b80 2024-03-16T10:24:51.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba397fa640 10 client.17367 handle_command_reply: tid=0 2024-03-16T10:24:51.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba5ade8740 1 client.17367 shutdown 2024-03-16T10:24:51.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba5ade8740 1 -- 172.21.15.77:0/321271491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 msgr2=0x563c9d13f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba5ade8740 1 --2- 172.21.15.77:0/321271491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 0x563c9d13f2a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fba4800eec0 tx=0x7fba4800d7b0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba5ade8740 1 -- 172.21.15.77:0/321271491 shutdown_connections 2024-03-16T10:24:51.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba5ade8740 1 --2- 172.21.15.77:0/321271491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 0x563c9d13f2a0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba5ade8740 1 --2- 172.21.15.77:0/321271491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12e240 0x563c9d1382a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba5ade8740 1 --2- 172.21.15.77:0/321271491 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x563c9d143b80 0x563c9d146020 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba5ade8740 1 --2- 172.21.15.77:0/321271491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d137d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba5ade8740 1 -- 172.21.15.77:0/321271491 >> 172.21.15.77:0/321271491 conn(0x563c9d128dc0 msgr2=0x563c9d127fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba397fa640 0 client.17367 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:24:51.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba5ade8740 1 -- 172.21.15.77:0/321271491 shutdown_connections 2024-03-16T10:24:51.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.041+0000 7fba5ade8740 1 -- 172.21.15.77:0/321271491 wait complete. 2024-03-16T10:24:51.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.042+0000 7fba5ade8740 20 client.17367 trim_cache size 0 max 16384 2024-03-16T10:24:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.055+0000 7fba5ade8740 1 Processor -- start 2024-03-16T10:24:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.055+0000 7fba5ade8740 1 -- start start 2024-03-16T10:24:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.055+0000 7fba5ade8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12d850 0x563c9d137d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.055+0000 7fba5ade8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 0x563c9d13f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.055+0000 7fba5ade8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d1333b0 0x563c9d1382a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.055+0000 7fba5ade8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fba501a7a90 con 0x563c9d12e240 2024-03-16T10:24:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.055+0000 7fba5ade8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fba501a9660 con 0x563c9d1333b0 2024-03-16T10:24:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.055+0000 7fba5ade8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba501a8190 con 0x563c9d12d850 2024-03-16T10:24:51.057 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba5578f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 0x563c9d13f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba5578f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 0x563c9d13f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49208/0 (socket says 172.21.15.77:49208) 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba5578f640 1 -- 172.21.15.77:0/1387842521 learned_addr learned my addr 172.21.15.77:0/1387842521 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba55f90640 1 --2- 172.21.15.77:0/1387842521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12d850 0x563c9d137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba56791640 1 --2- 172.21.15.77:0/1387842521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d1333b0 0x563c9d1382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba5578f640 1 -- 172.21.15.77:0/1387842521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12d850 msgr2=0x563c9d137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba5578f640 1 --2- 172.21.15.77:0/1387842521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12d850 0x563c9d137d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba5578f640 1 -- 172.21.15.77:0/1387842521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d1333b0 msgr2=0x563c9d1382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba5578f640 1 --2- 172.21.15.77:0/1387842521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d1333b0 0x563c9d1382a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba5578f640 1 -- 172.21.15.77:0/1387842521 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c9ce707d0 con 0x563c9d12e240 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba5578f640 1 --2- 172.21.15.77:0/1387842521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 0x563c9d13f260 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fba4000c6a0 tx=0x7fba40097450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b64873f7c80c52 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba3b7fe640 1 -- 172.21.15.77:0/1387842521 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba400b3070 con 0x563c9d12e240 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba3b7fe640 1 -- 172.21.15.77:0/1387842521 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba4009b7f0 con 0x563c9d12e240 2024-03-16T10:24:51.058 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.056+0000 7fba3b7fe640 1 -- 172.21.15.77:0/1387842521 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba40097e60 con 0x563c9d12e240 2024-03-16T10:24:51.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.057+0000 7fba5ade8740 1 -- 172.21.15.77:0/1387842521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 msgr2=0x563c9d13f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.057+0000 7fba5ade8740 1 --2- 172.21.15.77:0/1387842521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 0x563c9d13f260 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fba4000c6a0 tx=0x7fba40097450 comp rx=0 tx=0).stop 2024-03-16T10:24:51.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.057+0000 7fba5ade8740 1 -- 172.21.15.77:0/1387842521 shutdown_connections 2024-03-16T10:24:51.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.057+0000 7fba5ade8740 1 --2- 172.21.15.77:0/1387842521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d1333b0 0x563c9d1382a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.057+0000 7fba5ade8740 1 --2- 172.21.15.77:0/1387842521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d12e240 0x563c9d13f260 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.057+0000 7fba5ade8740 1 --2- 172.21.15.77:0/1387842521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12d850 0x563c9d137d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.057+0000 7fba5ade8740 1 -- 172.21.15.77:0/1387842521 >> 172.21.15.77:0/1387842521 conn(0x563c9d128dc0 msgr2=0x563c9d1291a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.057+0000 7fba5ade8740 1 -- 172.21.15.77:0/1387842521 shutdown_connections 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.057+0000 7fba5ade8740 1 -- 172.21.15.77:0/1387842521 wait complete. 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.057+0000 7fba5ade8740 1 -- start start 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.057+0000 7fba5ade8740 1 Processor -- start 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba5ade8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d13c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba5ade8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12e240 0x563c9d13cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba5ade8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 0x563c9d13dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba5ade8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fba501a8f90 con 0x563c9d1333b0 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba5ade8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fba501a8890 con 0x563c9d12d850 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba5ade8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba501a5a70 con 0x563c9d12e240 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba5ade8740 10 client.0 _ll_register_callbacks cb 0x7fba48009820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba56791640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 0x563c9d13dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba56791640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 0x563c9d13dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49210/0 (socket says 172.21.15.77:49210) 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba56791640 1 -- 172.21.15.77:0/887561392 learned_addr learned my addr 172.21.15.77:0/887561392 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba5578f640 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12e240 0x563c9d13cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba55f90640 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d13c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba56791640 1 -- 172.21.15.77:0/887561392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12e240 msgr2=0x563c9d13cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba56791640 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12e240 0x563c9d13cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba56791640 1 -- 172.21.15.77:0/887561392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 msgr2=0x563c9d13c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba56791640 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d13c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba56791640 1 -- 172.21.15.77:0/887561392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c9d13e350 con 0x563c9d1333b0 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba56791640 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 0x563c9d13dd60 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fba48021ae0 tx=0x7fba48021700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc5408e4b0f5df2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba397fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba397fa640 1 -- 172.21.15.77:0/887561392 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba48031060 con 0x563c9d1333b0 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba5ade8740 1 -- 172.21.15.77:0/887561392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba4009b7f0 con 0x563c9d1333b0 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.058+0000 7fba397fa640 1 -- 172.21.15.77:0/887561392 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fba48003970 con 0x563c9d1333b0 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.059+0000 7fba397fa640 1 -- 172.21.15.77:0/887561392 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fba4801c040 con 0x563c9d1333b0 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.059+0000 7fba5ade8740 10 client.17373 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.059+0000 7fba5ade8740 1 -- 172.21.15.77:0/887561392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x563c9d1ae370 con 0x563c9d1333b0 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.059+0000 7fba397fa640 1 -- 172.21.15.77:0/887561392 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba4009b7f0 con 0x563c9d1333b0 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.059+0000 7fba397fa640 1 -- 172.21.15.77:0/887561392 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fba4800b040 con 0x563c9d1333b0 2024-03-16T10:24:51.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.059+0000 7fba5ade8740 10 client.17373 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.059+0000 7fba5ade8740 10 client.17373 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.059+0000 7fba5ade8740 20 client.17373 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.059+0000 7fba5ade8740 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x563c9d1aed70 0x563c9d1b1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.059+0000 7fba5ade8740 4 client.17373 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.059+0000 7fba5ade8740 1 -- 172.21.15.77:0/887561392 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563c9d099d20 con 0x563c9d1aed70 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.059+0000 7fba55f90640 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x563c9d1aed70 0x563c9d1b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba397fa640 10 client.17373 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba55f90640 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x563c9d1aed70 0x563c9d1b1190 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f47c231ab1dcc3b9 server_cookie=2e5ab0fa4ec80ac2 in_seq=0 out_seq=0 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba397fa640 1 -- 172.21.15.77:0/887561392 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563c9d099d20 con 0x563c9d1aed70 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba397fa640 10 client.17373 handle_command_reply: tid=0 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba5ade8740 1 client.17373 shutdown 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba5ade8740 1 -- 172.21.15.77:0/887561392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 msgr2=0x563c9d13dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba5ade8740 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 0x563c9d13dd60 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fba48021ae0 tx=0x7fba48021700 comp rx=0 tx=0).stop 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba5ade8740 1 -- 172.21.15.77:0/887561392 shutdown_connections 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba5ade8740 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563c9d1333b0 0x563c9d13dd60 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.065 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba5ade8740 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c9d12e240 0x563c9d13cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba5ade8740 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x563c9d1aed70 0x563c9d1b1190 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba5ade8740 1 --2- 172.21.15.77:0/887561392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c9d12d850 0x563c9d13c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba5ade8740 1 -- 172.21.15.77:0/887561392 >> 172.21.15.77:0/887561392 conn(0x563c9d128dc0 msgr2=0x563c9d12f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba397fa640 0 client.17373 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:24:51.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba5ade8740 1 -- 172.21.15.77:0/887561392 shutdown_connections 2024-03-16T10:24:51.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba5ade8740 1 -- 172.21.15.77:0/887561392 wait complete. 2024-03-16T10:24:51.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.060+0000 7fba5ade8740 20 client.17373 trim_cache size 0 max 16384 2024-03-16T10:24:51.066 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:24:51.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.061+0000 7fba581f4640 1 -- 172.21.15.77:0/3014598097 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fba3c0a2800 msgr2=0x7fba3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.061+0000 7fba581f4640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fba3c0a2800 0x7fba3c0a4cc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fba4c0349e0 tx=0x7fba4c034ab0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.061+0000 7fba581f4640 1 -- 172.21.15.77:0/3014598097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba501018d0 msgr2=0x7fba50196fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.061+0000 7fba581f4640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba501018d0 0x7fba50196fc0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fba4000e9f0 tx=0x7fba4000edf0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.061+0000 7fba581f4640 1 -- 172.21.15.77:0/3014598097 shutdown_connections 2024-03-16T10:24:51.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.061+0000 7fba581f4640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fba3c0a2800 0x7fba3c0a4cc0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.061+0000 7fba581f4640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba50102290 0x7fba5019e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.061+0000 7fba581f4640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba501018d0 0x7fba50196fc0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.061+0000 7fba581f4640 1 --2- 172.21.15.77:0/3014598097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba501006e0 0x7fba50196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.061+0000 7fba581f4640 1 -- 172.21.15.77:0/3014598097 >> 172.21.15.77:0/3014598097 conn(0x7fba500fbc70 msgr2=0x7fba50104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.061+0000 7fba581f4640 1 -- 172.21.15.77:0/3014598097 shutdown_connections 2024-03-16T10:24:51.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.061+0000 7fba581f4640 1 -- 172.21.15.77:0/3014598097 wait complete. 2024-03-16T10:24:51.075 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T10:24:51.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.181+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4204590273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f0101810 msgr2=0x7fb6f0101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.181+0000 7fb6f7bd4640 1 --2- 172.21.15.77:0/4204590273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f0101810 0x7fb6f0101c30 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0009900 tx=0x7fb6e002bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4204590273 shutdown_connections 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 --2- 172.21.15.77:0/4204590273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6f01033c0 0x7fb6f0109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 --2- 172.21.15.77:0/4204590273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6f0102a00 0x7fb6f0102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 --2- 172.21.15.77:0/4204590273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f0101810 0x7fb6f0101c30 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4204590273 >> 172.21.15.77:0/4204590273 conn(0x7fb6f00fcd80 msgr2=0x7fb6f00ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4204590273 shutdown_connections 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4204590273 wait complete. 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 Processor -- start 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 -- start start 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6f0101810 0x7fb6f0197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6f0102a00 0x7fb6f01980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f01033c0 0x7fb6f019f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f010d460 con 0x7fb6f01033c0 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f010d2e0 con 0x7fb6f0102a00 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f7bd4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f010d5e0 con 0x7fb6f0101810 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.182+0000 7fb6f6171640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f01033c0 0x7fb6f019f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6f6171640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f01033c0 0x7fb6f019f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49232/0 (socket says 172.21.15.77:49232) 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6f6171640 1 -- 172.21.15.77:0/4092774043 learned_addr learned my addr 172.21.15.77:0/4092774043 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6f516f640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6f0102a00 0x7fb6f01980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6f5970640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6f0101810 0x7fb6f0197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6f6171640 1 -- 172.21.15.77:0/4092774043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6f0101810 msgr2=0x7fb6f0197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6f6171640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6f0101810 0x7fb6f0197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6f6171640 1 -- 172.21.15.77:0/4092774043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6f0102a00 msgr2=0x7fb6f01980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6f6171640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6f0102a00 0x7fb6f01980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6f6171640 1 -- 172.21.15.77:0/4092774043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f019f7a0 con 0x7fb6f01033c0 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6f6171640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f01033c0 0x7fb6f019f130 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009fd0 tx=0x7fb6e800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6e6ffd640 1 -- 172.21.15.77:0/4092774043 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8017070 con 0x7fb6f01033c0 2024-03-16T10:24:51.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4092774043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6f019fa90 con 0x7fb6f01033c0 2024-03-16T10:24:51.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6e6ffd640 1 -- 172.21.15.77:0/4092774043 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6e8003e50 con 0x7fb6f01033c0 2024-03-16T10:24:51.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6e6ffd640 1 -- 172.21.15.77:0/4092774043 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e80106c0 con 0x7fb6f01033c0 2024-03-16T10:24:51.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.183+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4092774043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6f019ff90 con 0x7fb6f01033c0 2024-03-16T10:24:51.193 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.185+0000 7fb6e6ffd640 1 -- 172.21.15.77:0/4092774043 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fb6e8003970 con 0x7fb6f01033c0 2024-03-16T10:24:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6fa7c3740 1 Processor -- start 2024-03-16T10:24:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6fa7c3740 1 -- start start 2024-03-16T10:24:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6fa7c3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e808c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6fa7c3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e809240 0x55f36e80dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6fa7c3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e80e3b0 0x55f36e810840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6fa7c3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f36e76d490 con 0x55f36e808850 2024-03-16T10:24:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6fa7c3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55f36e715c90 con 0x55f36e80e3b0 2024-03-16T10:24:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6fa7c3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f36e769530 con 0x55f36e809240 2024-03-16T10:24:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6e6ffd640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb6dc0a2800 0x7fb6dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6f516f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e808c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6f516f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e808c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49234/0 (socket says 172.21.15.77:49234) 2024-03-16T10:24:51.194 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6f516f640 1 -- 172.21.15.77:0/41037336 learned_addr learned my addr 172.21.15.77:0/41037336 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.186+0000 7fb6f5970640 1 --2- 172.21.15.77:0/41037336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e809240 0x55f36e80dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6f516f640 1 --2- 172.21.15.77:0/41037336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e80e3b0 0x55f36e810840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6f6171640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb6dc0a2800 0x7fb6dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6e6ffd640 1 -- 172.21.15.77:0/4092774043 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fb6e80ce770 con 0x7fb6f01033c0 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6f5970640 1 -- 172.21.15.77:0/41037336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e80e3b0 msgr2=0x55f36e810840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6f5970640 1 --2- 172.21.15.77:0/41037336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e80e3b0 0x55f36e810840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6f5970640 1 -- 172.21.15.77:0/41037336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 msgr2=0x55f36e808c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6f5970640 1 --2- 172.21.15.77:0/41037336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e808c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6f5970640 1 -- 172.21.15.77:0/41037336 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f36e7d0500 con 0x55f36e809240 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6f516f640 1 --2- 172.21.15.77:0/41037336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e808c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6f6171640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb6dc0a2800 0x7fb6dc0a4cc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8097610 tx=0x7fb6e8095be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6f5970640 1 --2- 172.21.15.77:0/41037336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e809240 0x55f36e80dd40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb6e002c170 tx=0x7fb6e0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8403ccc41ae3cf9d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6dbfff640 1 -- 172.21.15.77:0/41037336 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e002cb80 con 0x55f36e809240 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6dbfff640 1 -- 172.21.15.77:0/41037336 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6e0005040 con 0x55f36e809240 2024-03-16T10:24:51.195 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.187+0000 7fb6dbfff640 1 -- 172.21.15.77:0/41037336 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0004920 con 0x55f36e809240 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.188+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/41037336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e809240 msgr2=0x55f36e80dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.188+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/41037336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e809240 0x55f36e80dd40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb6e002c170 tx=0x7fb6e0009560 comp rx=0 tx=0).stop 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.188+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/41037336 shutdown_connections 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.188+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/41037336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e80e3b0 0x55f36e810840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.188+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/41037336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e809240 0x55f36e80dd40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.188+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/41037336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e808c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.188+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/41037336 >> 172.21.15.77:0/41037336 conn(0x55f36e803dc0 msgr2=0x55f36e8041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.188+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/41037336 shutdown_connections 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.188+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/41037336 wait complete. 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.189+0000 7fb6fa7c3740 1 -- start start 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.189+0000 7fb6fa7c3740 1 Processor -- start 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.189+0000 7fb6fa7c3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e812d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.189+0000 7fb6fa7c3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e809240 0x55f36e8132d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.189+0000 7fb6fa7c3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 0x55f36e81a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.189+0000 7fb6fa7c3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f36e811300 con 0x55f36e808850 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.189+0000 7fb6fa7c3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55f36e729230 con 0x55f36e809240 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.189+0000 7fb6fa7c3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f36e811480 con 0x55f36e80e3b0 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.189+0000 7fb6fa7c3740 10 client.0 _ll_register_callbacks cb 0x55f36e765010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.189+0000 7fb6f516f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e812d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6f5970640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e809240 0x55f36e8132d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6f516f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e812d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49240/0 (socket says 172.21.15.77:49240) 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6f516f640 1 -- 172.21.15.77:0/1481458646 learned_addr learned my addr 172.21.15.77:0/1481458646 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6f516f640 1 --2- 172.21.15.77:0/1481458646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 0x55f36e81a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6f516f640 1 -- 172.21.15.77:0/1481458646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 msgr2=0x55f36e81a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6f516f640 1 --2- 172.21.15.77:0/1481458646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 0x55f36e81a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6f516f640 1 -- 172.21.15.77:0/1481458646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e809240 msgr2=0x55f36e8132d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6f516f640 1 --2- 172.21.15.77:0/1481458646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e809240 0x55f36e8132d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6f516f640 1 -- 172.21.15.77:0/1481458646 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f36e738100 con 0x55f36e808850 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6f516f640 1 --2- 172.21.15.77:0/1481458646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e812d90 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec013e50 tx=0x7fb6ec009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15cbf101a9480897 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.197 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6d97fa640 1 -- 172.21.15.77:0/1481458646 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec005220 con 0x55f36e808850 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6d97fa640 1 -- 172.21.15.77:0/1481458646 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fb6ec0053c0 con 0x55f36e808850 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1481458646 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6e0005040 con 0x55f36e808850 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6d97fa640 1 -- 172.21.15.77:0/1481458646 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fb6ec00fe60 con 0x55f36e808850 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6fa7c3740 10 client.17385 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1481458646 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55f36e4813f0 con 0x55f36e808850 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.190+0000 7fb6d97fa640 1 -- 172.21.15.77:0/1481458646 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6e0005040 con 0x55f36e808850 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.191+0000 7fb6d97fa640 1 -- 172.21.15.77:0/1481458646 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fb6ec026020 con 0x55f36e808850 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.191+0000 7fb6fa7c3740 10 client.17385 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.191+0000 7fb6fa7c3740 10 client.17385 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.191+0000 7fb6fa7c3740 20 client.17385 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.191+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/1481458646 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55f36e81ec80 0x55f36e8210f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.191+0000 7fb6fa7c3740 4 client.17385 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.191+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1481458646 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f36e77a9d0 con 0x55f36e81ec80 2024-03-16T10:24:51.198 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.191+0000 7fb6f5970640 1 --2- 172.21.15.77:0/1481458646 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55f36e81ec80 0x55f36e8210f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6f5970640 1 --2- 172.21.15.77:0/1481458646 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55f36e81ec80 0x55f36e8210f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eb0c781b53144c0a server_cookie=cbc65dbfe953bb03 in_seq=0 out_seq=0 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6d97fa640 10 client.17385 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6d97fa640 1 -- 172.21.15.77:0/1481458646 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f36e77a9d0 con 0x55f36e81ec80 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6d97fa640 10 client.17385 handle_command_reply: tid=0 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6fa7c3740 1 client.17385 shutdown 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1481458646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 msgr2=0x55f36e812d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/1481458646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e812d90 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec013e50 tx=0x7fb6ec009950 comp rx=0 tx=0).stop 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1481458646 shutdown_connections 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/1481458646 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55f36e81ec80 0x55f36e8210f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/1481458646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 0x55f36e81a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/1481458646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e809240 0x55f36e8132d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/1481458646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e812d90 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6d97fa640 0 client.17385 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1481458646 >> 172.21.15.77:0/1481458646 conn(0x55f36e803dc0 msgr2=0x55f36e802fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.192+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1481458646 shutdown_connections 2024-03-16T10:24:51.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.193+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1481458646 wait complete. 2024-03-16T10:24:51.200 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.193+0000 7fb6fa7c3740 20 client.17385 trim_cache size 0 max 16384 2024-03-16T10:24:51.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6fa7c3740 1 Processor -- start 2024-03-16T10:24:51.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6fa7c3740 1 -- start start 2024-03-16T10:24:51.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6fa7c3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e8130b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6fa7c3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e809240 0x55f36e81a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6fa7c3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 0x55f36e802750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6fa7c3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f01a8ab0 con 0x55f36e808850 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6fa7c3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f01aa680 con 0x55f36e809240 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6fa7c3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f01a91b0 con 0x55f36e80e3b0 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6f516f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e8130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6f516f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e8130b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49250/0 (socket says 172.21.15.77:49250) 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6f516f640 1 -- 172.21.15.77:0/1565515094 learned_addr learned my addr 172.21.15.77:0/1565515094 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6f5970640 1 --2- 172.21.15.77:0/1565515094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e809240 0x55f36e81a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6f516f640 1 --2- 172.21.15.77:0/1565515094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 0x55f36e802750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6f516f640 1 -- 172.21.15.77:0/1565515094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 msgr2=0x55f36e802750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6f516f640 1 --2- 172.21.15.77:0/1565515094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 0x55f36e802750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6f516f640 1 -- 172.21.15.77:0/1565515094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e809240 msgr2=0x55f36e81a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6f516f640 1 --2- 172.21.15.77:0/1565515094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e809240 0x55f36e81a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.206+0000 7fb6f516f640 1 -- 172.21.15.77:0/1565515094 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f36e54b7d0 con 0x55f36e808850 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6f516f640 1 --2- 172.21.15.77:0/1565515094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e8130b0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec006010 tx=0x7fb6ec009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36532760416b37f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6db7fe640 1 -- 172.21.15.77:0/1565515094 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec010a60 con 0x55f36e808850 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6db7fe640 1 -- 172.21.15.77:0/1565515094 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6ec010430 con 0x55f36e808850 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6db7fe640 1 -- 172.21.15.77:0/1565515094 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec028440 con 0x55f36e808850 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1565515094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 msgr2=0x55f36e8130b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/1565515094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e8130b0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec006010 tx=0x7fb6ec009260 comp rx=0 tx=0).stop 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1565515094 shutdown_connections 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/1565515094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 0x55f36e802750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/1565515094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e809240 0x55f36e81a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/1565515094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e808850 0x55f36e8130b0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1565515094 >> 172.21.15.77:0/1565515094 conn(0x55f36e803dc0 msgr2=0x55f36e8129a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1565515094 shutdown_connections 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.207+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/1565515094 wait complete. 2024-03-16T10:24:51.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6fa7c3740 1 -- start start 2024-03-16T10:24:51.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6fa7c3740 1 Processor -- start 2024-03-16T10:24:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6fa7c3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e808850 0x55f36e817210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6fa7c3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e809240 0x55f36e817750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6fa7c3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 0x55f36e882fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6fa7c3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f01a9fb0 con 0x55f36e809240 2024-03-16T10:24:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6fa7c3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f01a98b0 con 0x55f36e808850 2024-03-16T10:24:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6fa7c3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f01a6ab0 con 0x55f36e80e3b0 2024-03-16T10:24:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6fa7c3740 10 client.0 _ll_register_callbacks cb 0x7fb6ec013050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6f516f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e808850 0x55f36e817210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6f5970640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e809240 0x55f36e817750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6f516f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e808850 0x55f36e817210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:33056/0 (socket says 172.21.15.77:33056) 2024-03-16T10:24:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6f516f640 1 -- 172.21.15.77:0/3728550798 learned_addr learned my addr 172.21.15.77:0/3728550798 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.208+0000 7fb6f516f640 1 --2- 172.21.15.77:0/3728550798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 0x55f36e882fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6f5970640 1 -- 172.21.15.77:0/3728550798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 msgr2=0x55f36e882fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6f5970640 1 --2- 172.21.15.77:0/3728550798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 0x55f36e882fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6f5970640 1 -- 172.21.15.77:0/3728550798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e808850 msgr2=0x55f36e817210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6f5970640 1 --2- 172.21.15.77:0/3728550798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e808850 0x55f36e817210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6f5970640 1 -- 172.21.15.77:0/3728550798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f36e883520 con 0x55f36e809240 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6f5970640 1 --2- 172.21.15.77:0/3728550798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e809240 0x55f36e817750 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x55f36e804d30 tx=0x7fb6e002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd5d92921f371fe server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6d97fa640 1 -- 172.21.15.77:0/3728550798 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0055060 con 0x55f36e809240 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6d97fa640 1 -- 172.21.15.77:0/3728550798 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fb6e0050d90 con 0x55f36e809240 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/3728550798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6ec010430 con 0x55f36e809240 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6d97fa640 1 -- 172.21.15.77:0/3728550798 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fb6e0005d50 con 0x55f36e809240 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6fa7c3740 10 client.17391 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/3728550798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55f36e884740 con 0x55f36e809240 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6d97fa640 1 -- 172.21.15.77:0/3728550798 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6ec010430 con 0x55f36e809240 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6d97fa640 1 -- 172.21.15.77:0/3728550798 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fb6e00054d0 con 0x55f36e809240 2024-03-16T10:24:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6fa7c3740 10 client.17391 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6fa7c3740 10 client.17391 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.209+0000 7fb6fa7c3740 20 client.17391 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/3728550798 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55f36e884dc0 0x55f36e8871b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 4 client.17391 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/3728550798 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f36e774d20 con 0x55f36e884dc0 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6f516f640 1 --2- 172.21.15.77:0/3728550798 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55f36e884dc0 0x55f36e8871b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6f516f640 1 --2- 172.21.15.77:0/3728550798 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55f36e884dc0 0x55f36e8871b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=da7366cbb2f98b89 server_cookie=43f207157d975902 in_seq=0 out_seq=0 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6d97fa640 10 client.17391 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6d97fa640 1 -- 172.21.15.77:0/3728550798 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f36e774d20 con 0x55f36e884dc0 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6d97fa640 10 client.17391 handle_command_reply: tid=0 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 client.17391 shutdown 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/3728550798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e809240 msgr2=0x55f36e817750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/3728550798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e809240 0x55f36e817750 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x55f36e804d30 tx=0x7fb6e002bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/3728550798 shutdown_connections 2024-03-16T10:24:51.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/3728550798 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55f36e884dc0 0x55f36e8871b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/3728550798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f36e80e3b0 0x55f36e882fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6d97fa640 0 client.17391 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:24:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/3728550798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f36e809240 0x55f36e817750 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 --2- 172.21.15.77:0/3728550798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f36e808850 0x55f36e817210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/3728550798 >> 172.21.15.77:0/3728550798 conn(0x55f36e803dc0 msgr2=0x55f36e807020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/3728550798 shutdown_connections 2024-03-16T10:24:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.210+0000 7fb6fa7c3740 1 -- 172.21.15.77:0/3728550798 wait complete. 2024-03-16T10:24:51.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6fa7c3740 20 client.17391 trim_cache size 0 max 16384 2024-03-16T10:24:51.217 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:24:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4092774043 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb6dc0a2800 msgr2=0x7fb6dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6f7bd4640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb6dc0a2800 0x7fb6dc0a4cc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8097610 tx=0x7fb6e8095be0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4092774043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f01033c0 msgr2=0x7fb6f019f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6f7bd4640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f01033c0 0x7fb6f019f130 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009fd0 tx=0x7fb6e800b260 comp rx=0 tx=0).stop 2024-03-16T10:24:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4092774043 shutdown_connections 2024-03-16T10:24:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6f7bd4640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6f01033c0 0x7fb6f019f130 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6f7bd4640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6f0102a00 0x7fb6f01980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6f7bd4640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb6dc0a2800 0x7fb6dc0a4cc0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6f7bd4640 1 --2- 172.21.15.77:0/4092774043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6f0101810 0x7fb6f0197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4092774043 >> 172.21.15.77:0/4092774043 conn(0x7fb6f00fcd80 msgr2=0x7fb6f0105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4092774043 shutdown_connections 2024-03-16T10:24:51.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.211+0000 7fb6f7bd4640 1 -- 172.21.15.77:0/4092774043 wait complete. 2024-03-16T10:24:51.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T10:24:51.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.331+0000 7f89e39e8640 1 -- 172.21.15.77:0/2819365636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc1018d0 msgr2=0x7f89dc101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.331+0000 7f89e39e8640 1 --2- 172.21.15.77:0/2819365636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc1018d0 0x7f89dc101d50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009900 tx=0x7f89cc02bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:51.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.331+0000 7f89e39e8640 1 -- 172.21.15.77:0/2819365636 shutdown_connections 2024-03-16T10:24:51.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.331+0000 7f89e39e8640 1 --2- 172.21.15.77:0/2819365636 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89dc102290 0x7f89dc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.331+0000 7f89e39e8640 1 --2- 172.21.15.77:0/2819365636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc1018d0 0x7f89dc101d50 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.331+0000 7f89e39e8640 1 --2- 172.21.15.77:0/2819365636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89dc1006e0 0x7f89dc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.331+0000 7f89e39e8640 1 -- 172.21.15.77:0/2819365636 >> 172.21.15.77:0/2819365636 conn(0x7f89dc0fbc70 msgr2=0x7f89dc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.331+0000 7f89e39e8640 1 -- 172.21.15.77:0/2819365636 shutdown_connections 2024-03-16T10:24:51.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.331+0000 7f89e39e8640 1 -- 172.21.15.77:0/2819365636 wait complete. 2024-03-16T10:24:51.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.332+0000 7f89e39e8640 1 Processor -- start 2024-03-16T10:24:51.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.332+0000 7f89e39e8640 1 -- start start 2024-03-16T10:24:51.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.332+0000 7f89e39e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc1006e0 0x7f89dc0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.332+0000 7f89e39e8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89dc1018d0 0x7f89dc19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.332+0000 7f89e39e8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89dc102290 0x7f89dc19ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.332+0000 7f89e39e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc10c360 con 0x7f89dc1006e0 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.332+0000 7f89e39e8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc10c1e0 con 0x7f89dc1018d0 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.332+0000 7f89e39e8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc10c4e0 con 0x7f89dc102290 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e1784640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc1006e0 0x7f89dc0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e1784640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc1006e0 0x7f89dc0694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49266/0 (socket says 172.21.15.77:49266) 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e1784640 1 -- 172.21.15.77:0/1043993849 learned_addr learned my addr 172.21.15.77:0/1043993849 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e1f85640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89dc102290 0x7f89dc19ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e0f83640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89dc1018d0 0x7f89dc19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e1784640 1 -- 172.21.15.77:0/1043993849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89dc102290 msgr2=0x7f89dc19ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e1784640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89dc102290 0x7f89dc19ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e1784640 1 -- 172.21.15.77:0/1043993849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89dc1018d0 msgr2=0x7f89dc19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e1784640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89dc1018d0 0x7f89dc19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e1784640 1 -- 172.21.15.77:0/1043993849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc19f7a0 con 0x7f89dc1006e0 2024-03-16T10:24:51.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e1784640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc1006e0 0x7f89dc0694b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f89d800e9c0 tx=0x7f89d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89d27fc640 1 -- 172.21.15.77:0/1043993849 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d800cd80 con 0x7f89dc1006e0 2024-03-16T10:24:51.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89d27fc640 1 -- 172.21.15.77:0/1043993849 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d80035f0 con 0x7f89dc1006e0 2024-03-16T10:24:51.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e39e8640 1 -- 172.21.15.77:0/1043993849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89dc19fa30 con 0x7f89dc1006e0 2024-03-16T10:24:51.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89d27fc640 1 -- 172.21.15.77:0/1043993849 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8010690 con 0x7f89dc1006e0 2024-03-16T10:24:51.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.333+0000 7f89e39e8640 1 -- 172.21.15.77:0/1043993849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89dc19ff00 con 0x7f89dc1006e0 2024-03-16T10:24:51.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.335+0000 7f89d27fc640 1 -- 172.21.15.77:0/1043993849 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f89d80037a0 con 0x7f89dc1006e0 2024-03-16T10:24:51.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.336+0000 7f89d27fc640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f89c80a2800 0x7f89c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.336+0000 7f89e65cb740 1 Processor -- start 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.336+0000 7f89d27fc640 1 -- 172.21.15.77:0/1043993849 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f89d80ced30 con 0x7f89dc1006e0 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.336+0000 7f89e65cb740 1 -- start start 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.336+0000 7f89e0f83640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f89c80a2800 0x7f89c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.336+0000 7f89e65cb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf6c60 0x555bfdbf7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.336+0000 7f89e65cb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbf7650 0x555bfdbfc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.336+0000 7f89e65cb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbfc7c0 0x555bfdbfec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.336+0000 7f89e65cb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555bfdb57490 con 0x555bfdbfc7c0 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.336+0000 7f89e65cb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x555bfdaffc90 con 0x555bfdbf7650 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.336+0000 7f89e65cb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555bfdb53530 con 0x555bfdbf6c60 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e1784640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf6c60 0x555bfdbf7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e1f85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbfc7c0 0x555bfdbfec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e0f83640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbf7650 0x555bfdbfc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e1f85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbfc7c0 0x555bfdbfec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49268/0 (socket says 172.21.15.77:49268) 2024-03-16T10:24:51.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e1784640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf6c60 0x555bfdbf7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:32848/0 (socket says 172.21.15.77:32848) 2024-03-16T10:24:51.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e1f85640 1 -- 172.21.15.77:0/2558203208 learned_addr learned my addr 172.21.15.77:0/2558203208 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e0f83640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f89c80a2800 0x7f89c80a4cc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f89cc02c7b0 tx=0x7f89cc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e1784640 1 -- 172.21.15.77:0/2558203208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbf7650 msgr2=0x555bfdbfc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e1784640 1 --2- 172.21.15.77:0/2558203208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbf7650 0x555bfdbfc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e1784640 1 -- 172.21.15.77:0/2558203208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbfc7c0 msgr2=0x555bfdbfec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e1784640 1 --2- 172.21.15.77:0/2558203208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbfc7c0 0x555bfdbfec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e1784640 1 -- 172.21.15.77:0/2558203208 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bfdbba500 con 0x555bfdbf6c60 2024-03-16T10:24:51.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.337+0000 7f89e1f85640 1 --2- 172.21.15.77:0/2558203208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbfc7c0 0x555bfdbfec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:51.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.338+0000 7f89e1784640 1 --2- 172.21.15.77:0/2558203208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf6c60 0x555bfdbf7080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f89d80cc910 tx=0x7f89d809b0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5228b87e962f6af0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.342 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.339+0000 7f89c77fe640 1 -- 172.21.15.77:0/2558203208 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d80b4070 con 0x555bfdbf6c60 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.339+0000 7f89c77fe640 1 -- 172.21.15.77:0/2558203208 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d809f8b0 con 0x555bfdbf6c60 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.339+0000 7f89c77fe640 1 -- 172.21.15.77:0/2558203208 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d80a3a70 con 0x555bfdbf6c60 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.340+0000 7f89e65cb740 1 -- 172.21.15.77:0/2558203208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf6c60 msgr2=0x555bfdbf7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.340+0000 7f89e65cb740 1 --2- 172.21.15.77:0/2558203208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf6c60 0x555bfdbf7080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f89d80cc910 tx=0x7f89d809b0c0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.340+0000 7f89e65cb740 1 -- 172.21.15.77:0/2558203208 shutdown_connections 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.340+0000 7f89e65cb740 1 --2- 172.21.15.77:0/2558203208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbfc7c0 0x555bfdbfec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.340+0000 7f89e65cb740 1 --2- 172.21.15.77:0/2558203208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbf7650 0x555bfdbfc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.340+0000 7f89e65cb740 1 --2- 172.21.15.77:0/2558203208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf6c60 0x555bfdbf7080 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.340+0000 7f89e65cb740 1 -- 172.21.15.77:0/2558203208 >> 172.21.15.77:0/2558203208 conn(0x555bfdbeddc0 msgr2=0x555bfdbee1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.340+0000 7f89e65cb740 1 -- 172.21.15.77:0/2558203208 shutdown_connections 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.340+0000 7f89e65cb740 1 -- 172.21.15.77:0/2558203208 wait complete. 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.340+0000 7f89e65cb740 1 -- start start 2024-03-16T10:24:51.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.340+0000 7f89e65cb740 1 Processor -- start 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e65cb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 0x555bfdc01230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e65cb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf7650 0x555bfdc01770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e65cb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbfc7c0 0x555bfdc087b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e65cb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555bfdbff770 con 0x555bfdbf6c60 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e65cb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x555bfdb13230 con 0x555bfdbfc7c0 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e65cb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555bfdbff8f0 con 0x555bfdbf7650 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e65cb740 10 client.0 _ll_register_callbacks cb 0x555bfdb4f010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e1784640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 0x555bfdc01230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e1784640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 0x555bfdc01230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49276/0 (socket says 172.21.15.77:49276) 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e1784640 1 -- 172.21.15.77:0/1524706873 learned_addr learned my addr 172.21.15.77:0/1524706873 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e1f85640 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbfc7c0 0x555bfdc087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e0f83640 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf7650 0x555bfdc01770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e1784640 1 -- 172.21.15.77:0/1524706873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf7650 msgr2=0x555bfdc01770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e1784640 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf7650 0x555bfdc01770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.344 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e1784640 1 -- 172.21.15.77:0/1524706873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbfc7c0 msgr2=0x555bfdc087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.341+0000 7f89e1784640 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbfc7c0 0x555bfdc087b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.345 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.342+0000 7f89e1784640 1 -- 172.21.15.77:0/1524706873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bfdb22100 con 0x555bfdbf6c60 2024-03-16T10:24:51.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.342+0000 7f89e1784640 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 0x555bfdc01230 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f89d80a39d0 tx=0x7f89d8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfede00fd46e5a66 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.342+0000 7f89c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:51.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.342+0000 7f89c4ff9640 1 -- 172.21.15.77:0/1524706873 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d807f060 con 0x555bfdbf6c60 2024-03-16T10:24:51.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.343+0000 7f89e65cb740 1 -- 172.21.15.77:0/1524706873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89d809f8b0 con 0x555bfdbf6c60 2024-03-16T10:24:51.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.343+0000 7f89c4ff9640 1 -- 172.21.15.77:0/1524706873 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f89d8099a60 con 0x555bfdbf6c60 2024-03-16T10:24:51.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.343+0000 7f89c4ff9640 1 -- 172.21.15.77:0/1524706873 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f89d80b8030 con 0x555bfdbf6c60 2024-03-16T10:24:51.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.343+0000 7f89e65cb740 10 client.17403 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:51.346 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.343+0000 7f89e65cb740 1 -- 172.21.15.77:0/1524706873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x555bfd86b3f0 con 0x555bfdbf6c60 2024-03-16T10:24:51.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.344+0000 7f89c4ff9640 1 -- 172.21.15.77:0/1524706873 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d809f8b0 con 0x555bfdbf6c60 2024-03-16T10:24:51.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.344+0000 7f89c4ff9640 1 -- 172.21.15.77:0/1524706873 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f89d80ccce0 con 0x555bfdbf6c60 2024-03-16T10:24:51.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.344+0000 7f89e65cb740 10 client.17403 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:51.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.344+0000 7f89e65cb740 10 client.17403 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:51.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.344+0000 7f89e65cb740 20 client.17403 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:51.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.344+0000 7f89e65cb740 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555bfdbf4f70 0x555bfdbf5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.344+0000 7f89e65cb740 4 client.17403 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:51.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.344+0000 7f89e65cb740 1 -- 172.21.15.77:0/1524706873 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bfdb649d0 con 0x555bfdbf4f70 2024-03-16T10:24:51.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.344+0000 7f89e0f83640 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555bfdbf4f70 0x555bfdbf5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.345+0000 7f89e0f83640 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555bfdbf4f70 0x555bfdbf5350 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad0802839d9cc330 server_cookie=f22e66943feb6efb in_seq=0 out_seq=0 2024-03-16T10:24:51.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.345+0000 7f89c4ff9640 10 client.17403 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:51.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.347+0000 7f89c4ff9640 1 -- 172.21.15.77:0/1524706873 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555bfdb649d0 con 0x555bfdbf4f70 2024-03-16T10:24:51.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.347+0000 7f89c4ff9640 10 client.17403 handle_command_reply: tid=0 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.347+0000 7f89e65cb740 1 client.17403 shutdown 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.347+0000 7f89e65cb740 1 -- 172.21.15.77:0/1524706873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 msgr2=0x555bfdc01230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.347+0000 7f89e65cb740 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 0x555bfdc01230 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f89d80a39d0 tx=0x7f89d8005740 comp rx=0 tx=0).stop 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.347+0000 7f89e65cb740 1 -- 172.21.15.77:0/1524706873 shutdown_connections 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.347+0000 7f89e65cb740 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbfc7c0 0x555bfdc087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.347+0000 7f89e65cb740 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf7650 0x555bfdc01770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.347+0000 7f89e65cb740 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555bfdbf4f70 0x555bfdbf5350 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.347+0000 7f89e65cb740 1 --2- 172.21.15.77:0/1524706873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 0x555bfdc01230 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.347+0000 7f89c4ff9640 0 client.17403 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.347+0000 7f89e65cb740 1 -- 172.21.15.77:0/1524706873 >> 172.21.15.77:0/1524706873 conn(0x555bfdbeddc0 msgr2=0x555bfdbecfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.348+0000 7f89e65cb740 1 -- 172.21.15.77:0/1524706873 shutdown_connections 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.348+0000 7f89e65cb740 1 -- 172.21.15.77:0/1524706873 wait complete. 2024-03-16T10:24:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.348+0000 7f89e65cb740 20 client.17403 trim_cache size 0 max 16384 2024-03-16T10:24:51.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e65cb740 1 Processor -- start 2024-03-16T10:24:51.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e65cb740 1 -- start start 2024-03-16T10:24:51.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e65cb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 0x555bfdc014b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e65cb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbf7650 0x555bfdc08770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e65cb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbfc7c0 0x555bfdbffc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e65cb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1a6aa0 con 0x555bfdbf6c60 2024-03-16T10:24:51.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e65cb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1a8670 con 0x555bfdbf7650 2024-03-16T10:24:51.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e65cb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1a71a0 con 0x555bfdbfc7c0 2024-03-16T10:24:51.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e1784640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 0x555bfdc014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e1784640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 0x555bfdc014b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49290/0 (socket says 172.21.15.77:49290) 2024-03-16T10:24:51.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e1784640 1 -- 172.21.15.77:0/285219816 learned_addr learned my addr 172.21.15.77:0/285219816 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e1f85640 1 --2- 172.21.15.77:0/285219816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbfc7c0 0x555bfdbffc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e0f83640 1 --2- 172.21.15.77:0/285219816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbf7650 0x555bfdc08770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e1784640 1 -- 172.21.15.77:0/285219816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbfc7c0 msgr2=0x555bfdbffc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e1784640 1 --2- 172.21.15.77:0/285219816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbfc7c0 0x555bfdbffc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e1784640 1 -- 172.21.15.77:0/285219816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbf7650 msgr2=0x555bfdc08770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e1784640 1 --2- 172.21.15.77:0/285219816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbf7650 0x555bfdc08770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.370+0000 7f89e1784640 1 -- 172.21.15.77:0/285219816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bfd9357d0 con 0x555bfdbf6c60 2024-03-16T10:24:51.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.371+0000 7f89e1784640 1 --2- 172.21.15.77:0/285219816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 0x555bfdc014b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f89d809b2c0 tx=0x7f89d809b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cae1df7a3ab533b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.371+0000 7f89c6ffd640 1 -- 172.21.15.77:0/285219816 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d80b4040 con 0x555bfdbf6c60 2024-03-16T10:24:51.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.371+0000 7f89c6ffd640 1 -- 172.21.15.77:0/285219816 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d809f8b0 con 0x555bfdbf6c60 2024-03-16T10:24:51.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.371+0000 7f89c6ffd640 1 -- 172.21.15.77:0/285219816 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8097dd0 con 0x555bfdbf6c60 2024-03-16T10:24:51.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.372+0000 7f89e65cb740 1 -- 172.21.15.77:0/285219816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 msgr2=0x555bfdc014b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.372+0000 7f89e65cb740 1 --2- 172.21.15.77:0/285219816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 0x555bfdc014b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f89d809b2c0 tx=0x7f89d809b8c0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.372+0000 7f89e65cb740 1 -- 172.21.15.77:0/285219816 shutdown_connections 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.372+0000 7f89e65cb740 1 --2- 172.21.15.77:0/285219816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbfc7c0 0x555bfdbffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.372+0000 7f89e65cb740 1 --2- 172.21.15.77:0/285219816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbf7650 0x555bfdc08770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.372+0000 7f89e65cb740 1 --2- 172.21.15.77:0/285219816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf6c60 0x555bfdc014b0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.372+0000 7f89e65cb740 1 -- 172.21.15.77:0/285219816 >> 172.21.15.77:0/285219816 conn(0x555bfdbeddc0 msgr2=0x555bfdbf18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.373+0000 7f89e65cb740 1 -- 172.21.15.77:0/285219816 shutdown_connections 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.373+0000 7f89e65cb740 1 -- 172.21.15.77:0/285219816 wait complete. 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.373+0000 7f89e65cb740 1 -- start start 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.373+0000 7f89e65cb740 1 Processor -- start 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e65cb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf6c60 0x555bfdc08020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e65cb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf7650 0x555bfdc08560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e65cb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbfc7c0 0x555bfdc04ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e65cb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1a7fa0 con 0x555bfdbf7650 2024-03-16T10:24:51.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e65cb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1a78a0 con 0x555bfdbfc7c0 2024-03-16T10:24:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e65cb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1a4aa0 con 0x555bfdbf6c60 2024-03-16T10:24:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e65cb740 10 client.0 _ll_register_callbacks cb 0x7f89d80b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e1f85640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbfc7c0 0x555bfdc04ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e1784640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf6c60 0x555bfdc08020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e1f85640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbfc7c0 0x555bfdc04ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:33112/0 (socket says 172.21.15.77:33112) 2024-03-16T10:24:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e1f85640 1 -- 172.21.15.77:0/2452896430 learned_addr learned my addr 172.21.15.77:0/2452896430 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e0f83640 1 --2- 172.21.15.77:0/2452896430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf7650 0x555bfdc08560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.374+0000 7f89e1f85640 1 -- 172.21.15.77:0/2452896430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf6c60 msgr2=0x555bfdc08020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.375+0000 7f89e1f85640 1 --2- 172.21.15.77:0/2452896430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf6c60 0x555bfdc08020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.375+0000 7f89e1f85640 1 -- 172.21.15.77:0/2452896430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf7650 msgr2=0x555bfdc08560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.375+0000 7f89e1f85640 1 --2- 172.21.15.77:0/2452896430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf7650 0x555bfdc08560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.375+0000 7f89e1f85640 1 -- 172.21.15.77:0/2452896430 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bfdc05160 con 0x555bfdbfc7c0 2024-03-16T10:24:51.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.377+0000 7f89e1f85640 1 --2- 172.21.15.77:0/2452896430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbfc7c0 0x555bfdc04ac0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f89d4012d20 tx=0x7f89d400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19bdbddb09004e5c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:51.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.377+0000 7f89c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:24:51.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.377+0000 7f89c4ff9640 1 -- 172.21.15.77:0/2452896430 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4022060 con 0x555bfdbfc7c0 2024-03-16T10:24:51.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.377+0000 7f89c4ff9640 1 -- 172.21.15.77:0/2452896430 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f89d4003970 con 0x555bfdbfc7c0 2024-03-16T10:24:51.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.377+0000 7f89e65cb740 1 -- 172.21.15.77:0/2452896430 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89d809f8b0 con 0x555bfdbfc7c0 2024-03-16T10:24:51.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.378+0000 7f89c4ff9640 1 -- 172.21.15.77:0/2452896430 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f89d4003b70 con 0x555bfdbfc7c0 2024-03-16T10:24:51.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.378+0000 7f89e65cb740 10 client.27191 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:51.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.378+0000 7f89e65cb740 1 -- 172.21.15.77:0/2452896430 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x555bfdc76d40 con 0x555bfdbfc7c0 2024-03-16T10:24:51.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.378+0000 7f89c4ff9640 1 -- 172.21.15.77:0/2452896430 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d4003d10 con 0x555bfdbfc7c0 2024-03-16T10:24:51.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.379+0000 7f89c4ff9640 1 -- 172.21.15.77:0/2452896430 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f89d401d550 con 0x555bfdbfc7c0 2024-03-16T10:24:51.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.380+0000 7f89e65cb740 10 client.27191 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:51.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.380+0000 7f89e65cb740 10 client.27191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:51.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.380+0000 7f89e65cb740 20 client.27191 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:51.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.380+0000 7f89e65cb740 1 --2- 172.21.15.77:0/2452896430 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555bfdc776d0 0x555bfdc79af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:51.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.380+0000 7f89e65cb740 4 client.27191 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:24:51.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.380+0000 7f89e65cb740 1 -- 172.21.15.77:0/2452896430 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f89d809f8b0 con 0x555bfdc776d0 2024-03-16T10:24:51.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.380+0000 7f89e1784640 1 --2- 172.21.15.77:0/2452896430 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555bfdc776d0 0x555bfdc79af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:51.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.381+0000 7f89e1784640 1 --2- 172.21.15.77:0/2452896430 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555bfdc776d0 0x555bfdc79af0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a41288de6967dfa8 server_cookie=558c2ddf760354f in_seq=0 out_seq=0 2024-03-16T10:24:51.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.381+0000 7f89c4ff9640 10 client.27191 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:51.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.382+0000 7f89c4ff9640 1 -- 172.21.15.77:0/2452896430 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f89d809f8b0 con 0x555bfdc776d0 2024-03-16T10:24:51.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.382+0000 7f89c4ff9640 10 client.27191 handle_command_reply: tid=0 2024-03-16T10:24:51.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.382+0000 7f89e65cb740 1 client.27191 shutdown 2024-03-16T10:24:51.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.382+0000 7f89e65cb740 1 -- 172.21.15.77:0/2452896430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbfc7c0 msgr2=0x555bfdc04ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.382+0000 7f89e65cb740 1 --2- 172.21.15.77:0/2452896430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbfc7c0 0x555bfdc04ac0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f89d4012d20 tx=0x7f89d400f790 comp rx=0 tx=0).stop 2024-03-16T10:24:51.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.383+0000 7f89e65cb740 1 -- 172.21.15.77:0/2452896430 shutdown_connections 2024-03-16T10:24:51.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.383+0000 7f89e65cb740 1 --2- 172.21.15.77:0/2452896430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555bfdbfc7c0 0x555bfdc04ac0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.383+0000 7f89e65cb740 1 --2- 172.21.15.77:0/2452896430 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x555bfdc776d0 0x555bfdc79af0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.383+0000 7f89e65cb740 1 --2- 172.21.15.77:0/2452896430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555bfdbf7650 0x555bfdc08560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.383+0000 7f89e65cb740 1 --2- 172.21.15.77:0/2452896430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555bfdbf6c60 0x555bfdc08020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.383+0000 7f89c4ff9640 0 client.27191 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:51.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.383+0000 7f89e65cb740 1 -- 172.21.15.77:0/2452896430 >> 172.21.15.77:0/2452896430 conn(0x555bfdbeddc0 msgr2=0x555bfdbf18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.383+0000 7f89e65cb740 1 -- 172.21.15.77:0/2452896430 shutdown_connections 2024-03-16T10:24:51.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.383+0000 7f89e65cb740 1 -- 172.21.15.77:0/2452896430 wait complete. 2024-03-16T10:24:51.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.383+0000 7f89e65cb740 20 client.27191 trim_cache size 0 max 16384 2024-03-16T10:24:51.385 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:24:51.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.384+0000 7f89e39e8640 1 -- 172.21.15.77:0/1043993849 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f89c80a2800 msgr2=0x7f89c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.384+0000 7f89e39e8640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f89c80a2800 0x7f89c80a4cc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f89cc02c7b0 tx=0x7f89cc005fd0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.384+0000 7f89e39e8640 1 -- 172.21.15.77:0/1043993849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc1006e0 msgr2=0x7f89dc0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:51.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.384+0000 7f89e39e8640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc1006e0 0x7f89dc0694b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f89d800e9c0 tx=0x7f89d800ee80 comp rx=0 tx=0).stop 2024-03-16T10:24:51.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.384+0000 7f89e39e8640 1 -- 172.21.15.77:0/1043993849 shutdown_connections 2024-03-16T10:24:51.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.384+0000 7f89e39e8640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f89c80a2800 0x7f89c80a4cc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.384+0000 7f89e39e8640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89dc102290 0x7f89dc19ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.384+0000 7f89e39e8640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89dc1018d0 0x7f89dc19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.384+0000 7f89e39e8640 1 --2- 172.21.15.77:0/1043993849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89dc1006e0 0x7f89dc0694b0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:51.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.384+0000 7f89e39e8640 1 -- 172.21.15.77:0/1043993849 >> 172.21.15.77:0/1043993849 conn(0x7f89dc0fbc70 msgr2=0x7f89dc104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:51.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.384+0000 7f89e39e8640 1 -- 172.21.15.77:0/1043993849 shutdown_connections 2024-03-16T10:24:51.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:51.384+0000 7f89e39e8640 1 -- 172.21.15.77:0/1043993849 wait complete. 2024-03-16T10:24:51.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:51 smithi192 ceph-mon[36354]: pgmap v584: 97 pgs: 97 active+clean; 532 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 780 KiB/s wr, 421 op/s 2024-03-16T10:24:51.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:51 smithi077 ceph-mon[32400]: pgmap v584: 97 pgs: 97 active+clean; 532 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 780 KiB/s wr, 421 op/s 2024-03-16T10:24:52.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:51 smithi193 ceph-mon[36311]: pgmap v584: 97 pgs: 97 active+clean; 532 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 780 KiB/s wr, 421 op/s 2024-03-16T10:24:52.399 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T10:24:52.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:24:52.400 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T10:24:52.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.509+0000 7f1d0fed1640 1 -- 172.21.15.77:0/1086388944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d08104bd0 msgr2=0x7f1d08106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.509+0000 7f1d0fed1640 1 --2- 172.21.15.77:0/1086388944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d08104bd0 0x7f1d08106fc0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009900 tx=0x7f1cfc02bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.509+0000 7f1d0fed1640 1 -- 172.21.15.77:0/1086388944 shutdown_connections 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.509+0000 7f1d0fed1640 1 --2- 172.21.15.77:0/1086388944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d08107500 0x7f1d08109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.509+0000 7f1d0fed1640 1 --2- 172.21.15.77:0/1086388944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d08104bd0 0x7f1d08106fc0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.509+0000 7f1d0fed1640 1 --2- 172.21.15.77:0/1086388944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d081022a0 0x7f1d08104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.509+0000 7f1d0fed1640 1 -- 172.21.15.77:0/1086388944 >> 172.21.15.77:0/1086388944 conn(0x7f1d080fbc70 msgr2=0x7f1d080fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.509+0000 7f1d0fed1640 1 -- 172.21.15.77:0/1086388944 shutdown_connections 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.509+0000 7f1d0fed1640 1 -- 172.21.15.77:0/1086388944 wait complete. 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0fed1640 1 Processor -- start 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0fed1640 1 -- start start 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0fed1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d081022a0 0x7f1d08197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0fed1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d08104bd0 0x7f1d08198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0fed1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d08107500 0x7f1d0819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0fed1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0810d440 con 0x7f1d081022a0 2024-03-16T10:24:52.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0fed1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0810d2c0 con 0x7f1d08107500 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0fed1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0810d5c0 con 0x7f1d08104bd0 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0dc6d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d081022a0 0x7f1d08197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0dc6d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d081022a0 0x7f1d08197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59310/0 (socket says 172.21.15.77:59310) 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0dc6d640 1 -- 172.21.15.77:0/2705387046 learned_addr learned my addr 172.21.15.77:0/2705387046 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0e46e640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d08107500 0x7f1d0819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.510+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d08104bd0 0x7f1d08198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.511+0000 7f1d0dc6d640 1 -- 172.21.15.77:0/2705387046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d08104bd0 msgr2=0x7f1d08198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.511+0000 7f1d0dc6d640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d08104bd0 0x7f1d08198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.511+0000 7f1d0dc6d640 1 -- 172.21.15.77:0/2705387046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d08107500 msgr2=0x7f1d0819f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.511+0000 7f1d0dc6d640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d08107500 0x7f1d0819f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.511+0000 7f1d0dc6d640 1 -- 172.21.15.77:0/2705387046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d0819f790 con 0x7f1d081022a0 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.511+0000 7f1d0dc6d640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d081022a0 0x7f1d08197bc0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400e9f0 tx=0x7f1d0400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:52.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.511+0000 7f1cfaffd640 1 -- 172.21.15.77:0/2705387046 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0400ccc0 con 0x7f1d081022a0 2024-03-16T10:24:52.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.511+0000 7f1cfaffd640 1 -- 172.21.15.77:0/2705387046 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d04010910 con 0x7f1d081022a0 2024-03-16T10:24:52.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.511+0000 7f1d0fed1640 1 -- 172.21.15.77:0/2705387046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d0819fa80 con 0x7f1d081022a0 2024-03-16T10:24:52.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.511+0000 7f1cfaffd640 1 -- 172.21.15.77:0/2705387046 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d040188a0 con 0x7f1d081022a0 2024-03-16T10:24:52.515 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.511+0000 7f1d0fed1640 1 -- 172.21.15.77:0/2705387046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d0819ffa0 con 0x7f1d081022a0 2024-03-16T10:24:52.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.513+0000 7f1cfaffd640 1 -- 172.21.15.77:0/2705387046 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f1d04010ac0 con 0x7f1d081022a0 2024-03-16T10:24:52.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.514+0000 7f1cfaffd640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1cf40a2800 0x7f1cf40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.514+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1cf40a2800 0x7f1cf40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.521 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.514+0000 7f1cfaffd640 1 -- 172.21.15.77:0/2705387046 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f1d040cee30 con 0x7f1d081022a0 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.514+0000 7f1d12b98740 1 Processor -- start 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.514+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1cf40a2800 0x7f1cf40a4cc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc000c00 tx=0x7f1cfc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d12b98740 1 -- start start 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d12b98740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891d7990 0x561e891d7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d12b98740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e891dce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d12b98740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891dd4f0 0x561e891df980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d12b98740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561e8913c0d0 con 0x561e891d8380 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d12b98740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561e890e4c70 con 0x561e891d7990 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d12b98740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561e89138170 con 0x561e891dd4f0 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d0d46c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e891dce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d0d46c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e891dce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59324/0 (socket says 172.21.15.77:59324) 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d0dc6d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891d7990 0x561e891d7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d0d46c640 1 -- 172.21.15.77:0/3530779341 learned_addr learned my addr 172.21.15.77:0/3530779341 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:52.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d0e46e640 1 --2- 172.21.15.77:0/3530779341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891dd4f0 0x561e891df980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d0d46c640 1 -- 172.21.15.77:0/3530779341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891dd4f0 msgr2=0x561e891df980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/3530779341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891dd4f0 0x561e891df980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d0d46c640 1 -- 172.21.15.77:0/3530779341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891d7990 msgr2=0x561e891d7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/3530779341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891d7990 0x561e891d7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d0d46c640 1 -- 172.21.15.77:0/3530779341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e88f187e0 con 0x561e891d8380 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d0dc6d640 1 --2- 172.21.15.77:0/3530779341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891d7990 0x561e891d7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/3530779341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e891dce80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc005ec0 tx=0x7f1cfc03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815063f304648d6e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1cf3fff640 1 -- 172.21.15.77:0/3530779341 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc04c030 con 0x561e891d8380 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1cf3fff640 1 -- 172.21.15.77:0/3530779341 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cfc005530 con 0x561e891d8380 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.515+0000 7f1cf3fff640 1 -- 172.21.15.77:0/3530779341 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc004de0 con 0x561e891d8380 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.516+0000 7f1d12b98740 1 -- 172.21.15.77:0/3530779341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 msgr2=0x561e891dce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.516+0000 7f1d12b98740 1 --2- 172.21.15.77:0/3530779341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e891dce80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc005ec0 tx=0x7f1cfc03e000 comp rx=0 tx=0).stop 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.516+0000 7f1d12b98740 1 -- 172.21.15.77:0/3530779341 shutdown_connections 2024-03-16T10:24:52.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.516+0000 7f1d12b98740 1 --2- 172.21.15.77:0/3530779341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891dd4f0 0x561e891df980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.516+0000 7f1d12b98740 1 --2- 172.21.15.77:0/3530779341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e891dce80 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.516+0000 7f1d12b98740 1 --2- 172.21.15.77:0/3530779341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891d7990 0x561e891d7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.516+0000 7f1d12b98740 1 -- 172.21.15.77:0/3530779341 >> 172.21.15.77:0/3530779341 conn(0x561e8919fc50 msgr2=0x561e891d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.516+0000 7f1d12b98740 1 -- 172.21.15.77:0/3530779341 shutdown_connections 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.516+0000 7f1d12b98740 1 -- 172.21.15.77:0/3530779341 wait complete. 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d12b98740 1 -- start start 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d12b98740 1 Processor -- start 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d12b98740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d7990 0x561e891e1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d12b98740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891d8380 0x561e891e23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d12b98740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891dd4f0 0x561e891e93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d12b98740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561e891e0440 con 0x561e891d7990 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d12b98740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561e890f80d0 con 0x561e891dd4f0 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d12b98740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561e891e05c0 con 0x561e891d8380 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d12b98740 10 client.0 _ll_register_callbacks cb 0x561e890c1760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:52.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d0dc6d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d7990 0x561e891e1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d0dc6d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d7990 0x561e891e1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59338/0 (socket says 172.21.15.77:59338) 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d0dc6d640 1 -- 172.21.15.77:0/912725274 learned_addr learned my addr 172.21.15.77:0/912725274 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891d8380 0x561e891e23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.517+0000 7f1d0e46e640 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891dd4f0 0x561e891e93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1d0dc6d640 1 -- 172.21.15.77:0/912725274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891d8380 msgr2=0x561e891e23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1d0dc6d640 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891d8380 0x561e891e23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1d0dc6d640 1 -- 172.21.15.77:0/912725274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891dd4f0 msgr2=0x561e891e93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1d0dc6d640 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891dd4f0 0x561e891e93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1d0dc6d640 1 -- 172.21.15.77:0/912725274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e88f78f70 con 0x561e891d7990 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891d8380 0x561e891e23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1d0dc6d640 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d7990 0x561e891e1e70 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f1d04098230 tx=0x7f1d0400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8963cf8bca3d582 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1cf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1cf17fa640 1 -- 172.21.15.77:0/912725274 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d040b3020 con 0x561e891d7990 2024-03-16T10:24:52.525 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1cf17fa640 1 -- 172.21.15.77:0/912725274 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f1d040860b0 con 0x561e891d7990 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1d12b98740 1 -- 172.21.15.77:0/912725274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cfc005530 con 0x561e891d7990 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1cf17fa640 1 -- 172.21.15.77:0/912725274 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f1d040862b0 con 0x561e891d7990 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1d12b98740 10 client.17421 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1d12b98740 1 -- 172.21.15.77:0/912725274 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x561e88e10170 con 0x561e891d7990 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1cf17fa640 1 -- 172.21.15.77:0/912725274 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cfc005530 con 0x561e891d7990 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.518+0000 7f1cf17fa640 1 -- 172.21.15.77:0/912725274 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f1d040975b0 con 0x561e891d7990 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.519+0000 7f1d12b98740 10 client.17421 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.519+0000 7f1d12b98740 10 client.17421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.519+0000 7f1d12b98740 20 client.17421 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.519+0000 7f1d12b98740 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561e891ed8c0 0x561e891efd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.519+0000 7f1d12b98740 4 client.17421 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.519+0000 7f1d12b98740 1 -- 172.21.15.77:0/912725274 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e89149610 con 0x561e891ed8c0 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.519+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561e891ed8c0 0x561e891efd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.519+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561e891ed8c0 0x561e891efd10 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf065b67789f338d server_cookie=c494bb46eed6d1a8 in_seq=0 out_seq=0 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.519+0000 7f1cf17fa640 10 client.17421 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1cf17fa640 1 -- 172.21.15.77:0/912725274 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561e89149610 con 0x561e891ed8c0 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1cf17fa640 10 client.17421 handle_command_reply: tid=0 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1d12b98740 1 client.17421 shutdown 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1d12b98740 1 -- 172.21.15.77:0/912725274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d7990 msgr2=0x561e891e1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1d12b98740 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d7990 0x561e891e1e70 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f1d04098230 tx=0x7f1d0400ce60 comp rx=0 tx=0).stop 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1d12b98740 1 -- 172.21.15.77:0/912725274 shutdown_connections 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1d12b98740 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891dd4f0 0x561e891e93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1d12b98740 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891d8380 0x561e891e23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1d12b98740 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561e891ed8c0 0x561e891efd10 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1d12b98740 1 --2- 172.21.15.77:0/912725274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d7990 0x561e891e1e70 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1cf17fa640 0 client.17421 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1d12b98740 1 -- 172.21.15.77:0/912725274 >> 172.21.15.77:0/912725274 conn(0x561e8919fc50 msgr2=0x561e891d24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1d12b98740 1 -- 172.21.15.77:0/912725274 shutdown_connections 2024-03-16T10:24:52.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1d12b98740 1 -- 172.21.15.77:0/912725274 wait complete. 2024-03-16T10:24:52.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.520+0000 7f1d12b98740 20 client.17421 trim_cache size 0 max 16384 2024-03-16T10:24:52.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d12b98740 1 Processor -- start 2024-03-16T10:24:52.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d12b98740 1 -- start start 2024-03-16T10:24:52.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d12b98740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891d7990 0x561e891d6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d12b98740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e891e93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d12b98740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891dd4f0 0x561e891d1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d12b98740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1d081a8b10 con 0x561e891d8380 2024-03-16T10:24:52.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d12b98740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1d081aa6e0 con 0x561e891d7990 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d12b98740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1d081a9210 con 0x561e891dd4f0 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d0d46c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e891e93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d0d46c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e891e93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59344/0 (socket says 172.21.15.77:59344) 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d0e46e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891dd4f0 0x561e891d1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d0d46c640 1 -- 172.21.15.77:0/3155735857 learned_addr learned my addr 172.21.15.77:0/3155735857 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d0dc6d640 1 --2- 172.21.15.77:0/3155735857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891d7990 0x561e891d6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d0e46e640 1 -- 172.21.15.77:0/3155735857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891d7990 msgr2=0x561e891d6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d0e46e640 1 --2- 172.21.15.77:0/3155735857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891d7990 0x561e891d6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d0e46e640 1 -- 172.21.15.77:0/3155735857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 msgr2=0x561e891e93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.554+0000 7f1d0e46e640 1 --2- 172.21.15.77:0/3155735857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e891e93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.555+0000 7f1d0e46e640 1 -- 172.21.15.77:0/3155735857 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e891d38a0 con 0x561e891dd4f0 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.555+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/3155735857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e891e93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.555+0000 7f1d0e46e640 1 --2- 172.21.15.77:0/3155735857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891dd4f0 0x561e891d1c50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x561e891d4250 tx=0x7f1d00009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a513668fa519ca7f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:52.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.555+0000 7f1cf37fe640 1 -- 172.21.15.77:0/3155735857 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d00022070 con 0x561e891dd4f0 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.555+0000 7f1cf37fe640 1 -- 172.21.15.77:0/3155735857 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d00003e50 con 0x561e891dd4f0 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.555+0000 7f1cf37fe640 1 -- 172.21.15.77:0/3155735857 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d000042b0 con 0x561e891dd4f0 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 -- 172.21.15.77:0/3155735857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891dd4f0 msgr2=0x561e891d1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 --2- 172.21.15.77:0/3155735857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891dd4f0 0x561e891d1c50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x561e891d4250 tx=0x7f1d00009500 comp rx=0 tx=0).stop 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 -- 172.21.15.77:0/3155735857 shutdown_connections 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 --2- 172.21.15.77:0/3155735857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891dd4f0 0x561e891d1c50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 --2- 172.21.15.77:0/3155735857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e891e93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 --2- 172.21.15.77:0/3155735857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891d7990 0x561e891d6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 -- 172.21.15.77:0/3155735857 >> 172.21.15.77:0/3155735857 conn(0x561e8919fc50 msgr2=0x561e891d6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 -- 172.21.15.77:0/3155735857 shutdown_connections 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 -- 172.21.15.77:0/3155735857 wait complete. 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 -- start start 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 Processor -- start 2024-03-16T10:24:52.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891d7990 0x561e8924f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e8924f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891dd4f0 0x561e89256680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1d081aa010 con 0x561e891d8380 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1d081a9910 con 0x561e891dd4f0 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1d081a6af0 con 0x561e891d7990 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.556+0000 7f1d12b98740 10 client.0 _ll_register_callbacks cb 0x7f1d04010e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d0d46c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e8924f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d0d46c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e8924f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59350/0 (socket says 172.21.15.77:59350) 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d0d46c640 1 -- 172.21.15.77:0/1458669165 learned_addr learned my addr 172.21.15.77:0/1458669165 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d0dc6d640 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891d7990 0x561e8924f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d0e46e640 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891dd4f0 0x561e89256680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d0d46c640 1 -- 172.21.15.77:0/1458669165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891d7990 msgr2=0x561e8924f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891d7990 0x561e8924f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d0d46c640 1 -- 172.21.15.77:0/1458669165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891dd4f0 msgr2=0x561e89256680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891dd4f0 0x561e89256680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d0d46c640 1 -- 172.21.15.77:0/1458669165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e89256c70 con 0x561e891d8380 2024-03-16T10:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d0d46c640 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e8924f640 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x561e891d8df0 tx=0x7f1cfc004200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0d43a47173697c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1cf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1cf17fa640 1 -- 172.21.15.77:0/1458669165 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc041050 con 0x561e891d8380 2024-03-16T10:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1cf17fa640 1 -- 172.21.15.77:0/1458669165 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f1cfc05a410 con 0x561e891d8380 2024-03-16T10:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d12b98740 1 -- 172.21.15.77:0/1458669165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d00003e50 con 0x561e891d8380 2024-03-16T10:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1cf17fa640 1 -- 172.21.15.77:0/1458669165 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f1cfc04c040 con 0x561e891d8380 2024-03-16T10:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d12b98740 10 client.17427 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:52.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.557+0000 7f1d12b98740 1 -- 172.21.15.77:0/1458669165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x561e89258050 con 0x561e891d8380 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.558+0000 7f1cf17fa640 1 -- 172.21.15.77:0/1458669165 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d00003e50 con 0x561e891d8380 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.558+0000 7f1cf17fa640 1 -- 172.21.15.77:0/1458669165 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f1cfc048440 con 0x561e891d8380 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.558+0000 7f1d12b98740 10 client.17427 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.558+0000 7f1d12b98740 10 client.17427 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.558+0000 7f1d12b98740 20 client.17427 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.558+0000 7f1d12b98740 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561e89258680 0x561e8925aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.558+0000 7f1d12b98740 4 client.17427 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.558+0000 7f1d12b98740 1 -- 172.21.15.77:0/1458669165 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561e89143960 con 0x561e89258680 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.558+0000 7f1d0dc6d640 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561e89258680 0x561e8925aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.559+0000 7f1d0dc6d640 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561e89258680 0x561e8925aa70 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85bec6c1c8dccd2a server_cookie=c7c775f8d109a0be in_seq=0 out_seq=0 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.559+0000 7f1cf17fa640 10 client.17427 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.559+0000 7f1cf17fa640 1 -- 172.21.15.77:0/1458669165 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561e89143960 con 0x561e89258680 2024-03-16T10:24:52.566 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.559+0000 7f1cf17fa640 10 client.17427 handle_command_reply: tid=0 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.559+0000 7f1d12b98740 1 client.17427 shutdown 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.559+0000 7f1d12b98740 1 -- 172.21.15.77:0/1458669165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 msgr2=0x561e8924f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.559+0000 7f1d12b98740 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e8924f640 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x561e891d8df0 tx=0x7f1cfc004200 comp rx=0 tx=0).stop 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.559+0000 7f1d12b98740 1 -- 172.21.15.77:0/1458669165 shutdown_connections 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.559+0000 7f1d12b98740 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e891dd4f0 0x561e89256680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.559+0000 7f1d12b98740 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561e89258680 0x561e8925aa70 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.559+0000 7f1cf17fa640 0 client.17427 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.559+0000 7f1d12b98740 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e891d8380 0x561e8924f640 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d12b98740 1 --2- 172.21.15.77:0/1458669165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e891d7990 0x561e8924f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d12b98740 1 -- 172.21.15.77:0/1458669165 >> 172.21.15.77:0/1458669165 conn(0x561e8919fc50 msgr2=0x561e891d6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d12b98740 1 -- 172.21.15.77:0/1458669165 shutdown_connections 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d12b98740 1 -- 172.21.15.77:0/1458669165 wait complete. 2024-03-16T10:24:52.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d12b98740 20 client.17427 trim_cache size 0 max 16384 2024-03-16T10:24:52.568 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:24:52.568 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T10:24:52.568 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "7f420d11-ebf5-48c7-9aaa-beea512ad90e", 2024-03-16T10:24:52.568 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T10:24:52.568 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:24:52.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d0fed1640 1 -- 172.21.15.77:0/2705387046 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1cf40a2800 msgr2=0x7f1cf40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d0fed1640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1cf40a2800 0x7f1cf40a4cc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc000c00 tx=0x7f1cfc005fd0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d0fed1640 1 -- 172.21.15.77:0/2705387046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d081022a0 msgr2=0x7f1d08197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d0fed1640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d081022a0 0x7f1d08197bc0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400e9f0 tx=0x7f1d0400edf0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d0fed1640 1 -- 172.21.15.77:0/2705387046 shutdown_connections 2024-03-16T10:24:52.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d0fed1640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d08107500 0x7f1d0819f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d0fed1640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1cf40a2800 0x7f1cf40a4cc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d0fed1640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d08104bd0 0x7f1d08198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d0fed1640 1 --2- 172.21.15.77:0/2705387046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d081022a0 0x7f1d08197bc0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d0fed1640 1 -- 172.21.15.77:0/2705387046 >> 172.21.15.77:0/2705387046 conn(0x7f1d080fbc70 msgr2=0x7f1d08102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:52.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d0fed1640 1 -- 172.21.15.77:0/2705387046 shutdown_connections 2024-03-16T10:24:52.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.560+0000 7f1d0fed1640 1 -- 172.21.15.77:0/2705387046 wait complete. 2024-03-16T10:24:52.572 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:24:52.572 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:24:52.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.710+0000 7f768d777640 1 -- 172.21.15.77:0/1354086590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76881006e0 msgr2=0x7f7688100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.710+0000 7f768d777640 1 --2- 172.21.15.77:0/1354086590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76881006e0 0x7f7688100b00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f7680009ca0 tx=0x7f768002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.710+0000 7f768d777640 1 -- 172.21.15.77:0/1354086590 shutdown_connections 2024-03-16T10:24:52.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.710+0000 7f768d777640 1 --2- 172.21.15.77:0/1354086590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7688102290 0x7f7688108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.710+0000 7f768d777640 1 --2- 172.21.15.77:0/1354086590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76881018d0 0x7f7688101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.710+0000 7f768d777640 1 --2- 172.21.15.77:0/1354086590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76881006e0 0x7f7688100b00 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.710+0000 7f768d777640 1 -- 172.21.15.77:0/1354086590 >> 172.21.15.77:0/1354086590 conn(0x7f76880fbc70 msgr2=0x7f76880fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.710+0000 7f768d777640 1 -- 172.21.15.77:0/1354086590 shutdown_connections 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.710+0000 7f768d777640 1 -- 172.21.15.77:0/1354086590 wait complete. 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.711+0000 7f768d777640 1 Processor -- start 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.711+0000 7f768d777640 1 -- start start 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.711+0000 7f768d777640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76881006e0 0x7f7688199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.711+0000 7f768d777640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76881018d0 0x7f768819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.711+0000 7f768d777640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688102290 0x7f76881a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.711+0000 7f768d777640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f768810c360 con 0x7f7688102290 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.711+0000 7f768d777640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f768810c1e0 con 0x7f76881006e0 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.711+0000 7f768d777640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f768810c4e0 con 0x7f76881018d0 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.711+0000 7f76877fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688102290 0x7f76881a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f76877fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688102290 0x7f76881a1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59370/0 (socket says 172.21.15.77:59370) 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f76877fe640 1 -- 172.21.15.77:0/3303794824 learned_addr learned my addr 172.21.15.77:0/3303794824 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f7686ffd640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76881006e0 0x7f7688199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f76867fc640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76881018d0 0x7f768819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f76877fe640 1 -- 172.21.15.77:0/3303794824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76881018d0 msgr2=0x7f768819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f76877fe640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76881018d0 0x7f768819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f76877fe640 1 -- 172.21.15.77:0/3303794824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76881006e0 msgr2=0x7f7688199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f76877fe640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76881006e0 0x7f7688199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f76877fe640 1 -- 172.21.15.77:0/3303794824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76881a1970 con 0x7f7688102290 2024-03-16T10:24:52.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f76877fe640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688102290 0x7f76881a1300 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f767800e970 tx=0x7f767800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:52.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f767ffff640 1 -- 172.21.15.77:0/3303794824 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767800ccc0 con 0x7f7688102290 2024-03-16T10:24:52.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f767ffff640 1 -- 172.21.15.77:0/3303794824 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f767800ce60 con 0x7f7688102290 2024-03-16T10:24:52.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f768d777640 1 -- 172.21.15.77:0/3303794824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76881a1c60 con 0x7f7688102290 2024-03-16T10:24:52.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f767ffff640 1 -- 172.21.15.77:0/3303794824 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7678010490 con 0x7f7688102290 2024-03-16T10:24:52.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.712+0000 7f768d777640 1 -- 172.21.15.77:0/3303794824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76881a21b0 con 0x7f7688102290 2024-03-16T10:24:52.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.714+0000 7f767ffff640 1 -- 172.21.15.77:0/3303794824 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f7678005020 con 0x7f7688102290 2024-03-16T10:24:52.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f7690363740 1 Processor -- start 2024-03-16T10:24:52.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f7690363740 1 -- start start 2024-03-16T10:24:52.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f7690363740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 0x563b14baa590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f7690363740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baab60 0x563b14baf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f7690363740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14bafcd0 0x563b14bb2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f767ffff640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76740a2800 0x7f76740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f7690363740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563b14b0f500 con 0x563b14baa170 2024-03-16T10:24:52.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f7690363740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563b14ab7be0 con 0x563b14bafcd0 2024-03-16T10:24:52.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f7690363740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563b14b0b5a0 con 0x563b14baab60 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f76867fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 0x563b14baa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f76877fe640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76740a2800 0x7f76740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f7686ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baab60 0x563b14baf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f76867fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 0x563b14baa590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59374/0 (socket says 172.21.15.77:59374) 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f76867fc640 1 -- 172.21.15.77:0/289083692 learned_addr learned my addr 172.21.15.77:0/289083692 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f76867fc640 1 --2- 172.21.15.77:0/289083692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14bafcd0 0x563b14bb2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.715+0000 7f767ffff640 1 -- 172.21.15.77:0/3303794824 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f76780ced60 con 0x7f7688102290 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.716+0000 7f76867fc640 1 -- 172.21.15.77:0/289083692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baab60 msgr2=0x563b14baf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.716+0000 7f76867fc640 1 --2- 172.21.15.77:0/289083692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baab60 0x563b14baf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.716+0000 7f76867fc640 1 -- 172.21.15.77:0/289083692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14bafcd0 msgr2=0x563b14bb2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.716+0000 7f76867fc640 1 --2- 172.21.15.77:0/289083692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14bafcd0 0x563b14bb2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.716+0000 7f76867fc640 1 -- 172.21.15.77:0/289083692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b14acb4b0 con 0x563b14baa170 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.716+0000 7f7686ffd640 1 --2- 172.21.15.77:0/289083692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baab60 0x563b14baf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:52.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.716+0000 7f76877fe640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76740a2800 0x7f76740a4cc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7678097bf0 tx=0x7f7678066550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.716+0000 7f76867fc640 1 --2- 172.21.15.77:0/289083692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 0x563b14baa590 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f76700127f0 tx=0x7f767000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f58915ac214c6a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.716+0000 7f767d7fa640 1 -- 172.21.15.77:0/289083692 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7670012d70 con 0x563b14baa170 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.716+0000 7f767d7fa640 1 -- 172.21.15.77:0/289083692 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7670010770 con 0x563b14baa170 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.716+0000 7f767d7fa640 1 -- 172.21.15.77:0/289083692 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76700037f0 con 0x563b14baa170 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.717+0000 7f7690363740 1 -- 172.21.15.77:0/289083692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 msgr2=0x563b14baa590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.717+0000 7f7690363740 1 --2- 172.21.15.77:0/289083692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 0x563b14baa590 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f76700127f0 tx=0x7f767000b810 comp rx=0 tx=0).stop 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.717+0000 7f7690363740 1 -- 172.21.15.77:0/289083692 shutdown_connections 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.717+0000 7f7690363740 1 --2- 172.21.15.77:0/289083692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14bafcd0 0x563b14bb2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.717+0000 7f7690363740 1 --2- 172.21.15.77:0/289083692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baab60 0x563b14baf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.717+0000 7f7690363740 1 --2- 172.21.15.77:0/289083692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 0x563b14baa590 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.717+0000 7f7690363740 1 -- 172.21.15.77:0/289083692 >> 172.21.15.77:0/289083692 conn(0x563b14ba56e0 msgr2=0x563b14ba5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.717+0000 7f7690363740 1 -- 172.21.15.77:0/289083692 shutdown_connections 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.717+0000 7f7690363740 1 -- 172.21.15.77:0/289083692 wait complete. 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.717+0000 7f7690363740 1 -- start start 2024-03-16T10:24:52.724 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.717+0000 7f7690363740 1 Processor -- start 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7690363740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baa170 0x563b14bb4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7690363740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baab60 0x563b14bb4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7690363740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14bafcd0 0x563b14bbbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7690363740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563b14bb2c20 con 0x563b14baab60 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7690363740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563b14acb160 con 0x563b14bafcd0 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7690363740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563b14bb2da0 con 0x563b14baa170 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7690363740 10 client.0 _ll_register_callbacks cb 0x563b14aaf150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f76867fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baa170 0x563b14bb4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7686ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baab60 0x563b14bb4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f76867fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baa170 0x563b14bb4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:54688/0 (socket says 172.21.15.77:54688) 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f76867fc640 1 -- 172.21.15.77:0/1554303806 learned_addr learned my addr 172.21.15.77:0/1554303806 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f76867fc640 1 --2- 172.21.15.77:0/1554303806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14bafcd0 0x563b14bbbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7686ffd640 1 -- 172.21.15.77:0/1554303806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baa170 msgr2=0x563b14bb4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7686ffd640 1 --2- 172.21.15.77:0/1554303806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baa170 0x563b14bb4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7686ffd640 1 -- 172.21.15.77:0/1554303806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14bafcd0 msgr2=0x563b14bbbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7686ffd640 1 --2- 172.21.15.77:0/1554303806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14bafcd0 0x563b14bbbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7686ffd640 1 -- 172.21.15.77:0/1554303806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b148ed7d0 con 0x563b14baab60 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7686ffd640 1 --2- 172.21.15.77:0/1554303806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baab60 0x563b14bb4b70 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f768002c630 tx=0x7f7680009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530a667fd3767cc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f766e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f766e7fc640 1 -- 172.21.15.77:0/1554303806 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680044070 con 0x563b14baab60 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f7690363740 1 -- 172.21.15.77:0/1554303806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7670010770 con 0x563b14baab60 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.718+0000 7f766e7fc640 1 -- 172.21.15.77:0/1554303806 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f7680032890 con 0x563b14baab60 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.719+0000 7f766e7fc640 1 -- 172.21.15.77:0/1554303806 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f7680050050 con 0x563b14baab60 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.719+0000 7f7690363740 10 client.17439 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.719+0000 7f7690363740 1 -- 172.21.15.77:0/1554303806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x563b148233f0 con 0x563b14baab60 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.719+0000 7f766e7fc640 1 -- 172.21.15.77:0/1554303806 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7670010770 con 0x563b14baab60 2024-03-16T10:24:52.726 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.719+0000 7f766e7fc640 1 -- 172.21.15.77:0/1554303806 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f76800043a0 con 0x563b14baab60 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.719+0000 7f7690363740 10 client.17439 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.719+0000 7f7690363740 10 client.17439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.719+0000 7f7690363740 20 client.17439 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.719+0000 7f7690363740 1 --2- 172.21.15.77:0/1554303806 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563b14bc0110 0x563b14bc2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.719+0000 7f7690363740 4 client.17439 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.719+0000 7f7690363740 1 -- 172.21.15.77:0/1554303806 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b14b1ca40 con 0x563b14bc0110 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.719+0000 7f76867fc640 1 --2- 172.21.15.77:0/1554303806 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563b14bc0110 0x563b14bc2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.720+0000 7f766e7fc640 10 client.17439 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.720+0000 7f76867fc640 1 --2- 172.21.15.77:0/1554303806 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563b14bc0110 0x563b14bc2550 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=281377528bb061c1 server_cookie=5a841b4f442af6ed in_seq=0 out_seq=0 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.720+0000 7f766e7fc640 1 -- 172.21.15.77:0/1554303806 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563b14b1ca40 con 0x563b14bc0110 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.720+0000 7f766e7fc640 10 client.17439 handle_command_reply: tid=0 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f7690363740 1 client.17439 shutdown 2024-03-16T10:24:52.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f7690363740 1 -- 172.21.15.77:0/1554303806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baab60 msgr2=0x563b14bb4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f7690363740 1 --2- 172.21.15.77:0/1554303806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baab60 0x563b14bb4b70 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f768002c630 tx=0x7f7680009960 comp rx=0 tx=0).stop 2024-03-16T10:24:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f7690363740 1 -- 172.21.15.77:0/1554303806 shutdown_connections 2024-03-16T10:24:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f7690363740 1 --2- 172.21.15.77:0/1554303806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14bafcd0 0x563b14bbbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f7690363740 1 --2- 172.21.15.77:0/1554303806 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563b14bc0110 0x563b14bc2550 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f7690363740 1 --2- 172.21.15.77:0/1554303806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baab60 0x563b14bb4b70 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f7690363740 1 --2- 172.21.15.77:0/1554303806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baa170 0x563b14bb4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f7690363740 1 -- 172.21.15.77:0/1554303806 >> 172.21.15.77:0/1554303806 conn(0x563b14ba56e0 msgr2=0x563b14ba4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f766e7fc640 0 client.17439 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f7690363740 1 -- 172.21.15.77:0/1554303806 shutdown_connections 2024-03-16T10:24:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f7690363740 1 -- 172.21.15.77:0/1554303806 wait complete. 2024-03-16T10:24:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.721+0000 7f7690363740 20 client.17439 trim_cache size 0 max 16384 2024-03-16T10:24:52.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.743+0000 7f7690363740 1 Processor -- start 2024-03-16T10:24:52.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.743+0000 7f7690363740 1 -- start start 2024-03-16T10:24:52.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f7690363740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baa170 0x563b14bac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f7690363740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14baab60 0x563b14bac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f7690363740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14bafcd0 0x563b14bbbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f7690363740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f76881aae60 con 0x563b14bafcd0 2024-03-16T10:24:52.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f7690363740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f76881aca30 con 0x563b14baab60 2024-03-16T10:24:52.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f7690363740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76881ab560 con 0x563b14baa170 2024-03-16T10:24:52.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f76867fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14bafcd0 0x563b14bbbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f7686ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14baab60 0x563b14bac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f7686ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14baab60 0x563b14bac600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:46190/0 (socket says 172.21.15.77:46190) 2024-03-16T10:24:52.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f7686ffd640 1 -- 172.21.15.77:0/1166345418 learned_addr learned my addr 172.21.15.77:0/1166345418 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:52.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f76867fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14bafcd0 0x563b14bbbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59390/0 (socket says 172.21.15.77:59390) 2024-03-16T10:24:52.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f76867fc640 1 --2- 172.21.15.77:0/1166345418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baa170 0x563b14bac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f76867fc640 1 -- 172.21.15.77:0/1166345418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baa170 msgr2=0x563b14bac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f76867fc640 1 --2- 172.21.15.77:0/1166345418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baa170 0x563b14bac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f76867fc640 1 -- 172.21.15.77:0/1166345418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14baab60 msgr2=0x563b14bac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f76867fc640 1 --2- 172.21.15.77:0/1166345418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14baab60 0x563b14bac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f76867fc640 1 -- 172.21.15.77:0/1166345418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b147e30f0 con 0x563b14bafcd0 2024-03-16T10:24:52.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f7686ffd640 1 --2- 172.21.15.77:0/1166345418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14baab60 0x563b14bac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:24:52.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.744+0000 7f76867fc640 1 --2- 172.21.15.77:0/1166345418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14bafcd0 0x563b14bbbb70 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7670005fd0 tx=0x7f767000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7c474394662d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:52.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.745+0000 7f767cff9640 1 -- 172.21.15.77:0/1166345418 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7670010c80 con 0x563b14bafcd0 2024-03-16T10:24:52.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.745+0000 7f767cff9640 1 -- 172.21.15.77:0/1166345418 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7670004020 con 0x563b14bafcd0 2024-03-16T10:24:52.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.745+0000 7f767cff9640 1 -- 172.21.15.77:0/1166345418 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76700052a0 con 0x563b14bafcd0 2024-03-16T10:24:52.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.745+0000 7f7690363740 1 -- 172.21.15.77:0/1166345418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14bafcd0 msgr2=0x563b14bbbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.745+0000 7f7690363740 1 --2- 172.21.15.77:0/1166345418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14bafcd0 0x563b14bbbb70 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7670005fd0 tx=0x7f767000b810 comp rx=0 tx=0).stop 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.745+0000 7f7690363740 1 -- 172.21.15.77:0/1166345418 shutdown_connections 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.745+0000 7f7690363740 1 --2- 172.21.15.77:0/1166345418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14bafcd0 0x563b14bbbb70 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.745+0000 7f7690363740 1 --2- 172.21.15.77:0/1166345418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14baab60 0x563b14bac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.745+0000 7f7690363740 1 --2- 172.21.15.77:0/1166345418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14baa170 0x563b14bac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.745+0000 7f7690363740 1 -- 172.21.15.77:0/1166345418 >> 172.21.15.77:0/1166345418 conn(0x563b14ba56e0 msgr2=0x563b14bb4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.745+0000 7f7690363740 1 -- 172.21.15.77:0/1166345418 shutdown_connections 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.745+0000 7f7690363740 1 -- 172.21.15.77:0/1166345418 wait complete. 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f7690363740 1 -- start start 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f7690363740 1 Processor -- start 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f7690363740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 0x563b14bb8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f7690363740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14baab60 0x563b14bb9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f7690363740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14bafcd0 0x563b14bba360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f7690363740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f76881ac360 con 0x563b14baa170 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f7690363740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f76881abc60 con 0x563b14baab60 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f7690363740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76881a8e40 con 0x563b14bafcd0 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f7690363740 10 client.0 _ll_register_callbacks cb 0x7f7680009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f76867fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 0x563b14bb8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f76867fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 0x563b14bb8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59394/0 (socket says 172.21.15.77:59394) 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f76867fc640 1 -- 172.21.15.77:0/3442384916 learned_addr learned my addr 172.21.15.77:0/3442384916 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f7686ffd640 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14baab60 0x563b14bb9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f76867fc640 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14bafcd0 0x563b14bba360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f76867fc640 1 -- 172.21.15.77:0/3442384916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14bafcd0 msgr2=0x563b14bba360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f76867fc640 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14bafcd0 0x563b14bba360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f76867fc640 1 -- 172.21.15.77:0/3442384916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14baab60 msgr2=0x563b14bb9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f76867fc640 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14baab60 0x563b14bb9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f76867fc640 1 -- 172.21.15.77:0/3442384916 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b14ad69d0 con 0x563b14baa170 2024-03-16T10:24:52.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f76867fc640 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 0x563b14bb8ed0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f767000b510 tx=0x7f7670010fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b2923e4877aa2e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.746+0000 7f766e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f766e7fc640 1 -- 172.21.15.77:0/3442384916 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76700303d0 con 0x563b14baa170 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f766e7fc640 1 -- 172.21.15.77:0/3442384916 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f7670030570 con 0x563b14baa170 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f7690363740 1 -- 172.21.15.77:0/3442384916 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7670004020 con 0x563b14baa170 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f766e7fc640 1 -- 172.21.15.77:0/3442384916 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f7670030770 con 0x563b14baa170 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f7690363740 10 client.17445 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f7690363740 1 -- 172.21.15.77:0/3442384916 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x563b14bbba90 con 0x563b14baa170 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f766e7fc640 1 -- 172.21.15.77:0/3442384916 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7670004020 con 0x563b14baa170 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f766e7fc640 1 -- 172.21.15.77:0/3442384916 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f7670004890 con 0x563b14baa170 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f7690363740 10 client.17445 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f7690363740 10 client.17445 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f7690363740 20 client.17445 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f7690363740 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563b14c264f0 0x563b14c28910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:52.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f7690363740 4 client.17445 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f7690363740 1 -- 172.21.15.77:0/3442384916 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563b14b16d90 con 0x563b14c264f0 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.747+0000 7f7686ffd640 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563b14c264f0 0x563b14c28910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f766e7fc640 10 client.17445 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f7686ffd640 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563b14c264f0 0x563b14c28910 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5659734abbdc13da server_cookie=97fd61a89839dba7 in_seq=0 out_seq=0 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f766e7fc640 1 -- 172.21.15.77:0/3442384916 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563b14b16d90 con 0x563b14c264f0 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f766e7fc640 10 client.17445 handle_command_reply: tid=0 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f7690363740 1 client.17445 shutdown 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f7690363740 1 -- 172.21.15.77:0/3442384916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 msgr2=0x563b14bb8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f7690363740 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 0x563b14bb8ed0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f767000b510 tx=0x7f7670010fa0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f7690363740 1 -- 172.21.15.77:0/3442384916 shutdown_connections 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f7690363740 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b14bafcd0 0x563b14bba360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f7690363740 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563b14baab60 0x563b14bb9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f7690363740 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x563b14c264f0 0x563b14c28910 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f7690363740 1 --2- 172.21.15.77:0/3442384916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b14baa170 0x563b14bb8ed0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f7690363740 1 -- 172.21.15.77:0/3442384916 >> 172.21.15.77:0/3442384916 conn(0x563b14ba56e0 msgr2=0x563b14ba91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:52.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f766e7fc640 0 client.17445 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:52.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f7690363740 1 -- 172.21.15.77:0/3442384916 shutdown_connections 2024-03-16T10:24:52.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.748+0000 7f7690363740 1 -- 172.21.15.77:0/3442384916 wait complete. 2024-03-16T10:24:52.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f7690363740 20 client.17445 trim_cache size 0 max 16384 2024-03-16T10:24:52.756 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:24:52.756 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:24:52.756 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:24:52.756 INFO:teuthology.orchestra.run.smithi077.stdout: "7f420d11-ebf5-48c7-9aaa-beea512ad90e": { 2024-03-16T10:24:52.756 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:24:52.757 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "7f420d11-ebf5-48c7-9aaa-beea512ad90e", 2024-03-16T10:24:52.757 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:24:52.757 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:24:52.757 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:24:52.757 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:24:52.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f768d777640 1 -- 172.21.15.77:0/3303794824 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76740a2800 msgr2=0x7f76740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f768d777640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76740a2800 0x7f76740a4cc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7678097bf0 tx=0x7f7678066550 comp rx=0 tx=0).stop 2024-03-16T10:24:52.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f768d777640 1 -- 172.21.15.77:0/3303794824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688102290 msgr2=0x7f76881a1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:52.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f768d777640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688102290 0x7f76881a1300 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f767800e970 tx=0x7f767800ed70 comp rx=0 tx=0).stop 2024-03-16T10:24:52.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f768d777640 1 -- 172.21.15.77:0/3303794824 shutdown_connections 2024-03-16T10:24:52.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f768d777640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688102290 0x7f76881a1300 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f768d777640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f76740a2800 0x7f76740a4cc0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f768d777640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76881018d0 0x7f768819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f768d777640 1 --2- 172.21.15.77:0/3303794824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76881006e0 0x7f7688199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:52.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f768d777640 1 -- 172.21.15.77:0/3303794824 >> 172.21.15.77:0/3303794824 conn(0x7f76880fbc70 msgr2=0x7f76880fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:52.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f768d777640 1 -- 172.21.15.77:0/3303794824 shutdown_connections 2024-03-16T10:24:52.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:24:52.749+0000 7f768d777640 1 -- 172.21.15.77:0/3303794824 wait complete. 2024-03-16T10:24:52.761 INFO:tasks.cephfs.filesystem:scrub status for tag:7f420d11-ebf5-48c7-9aaa-beea512ad90e - {'path': '/', 'tag': '7f420d11-ebf5-48c7-9aaa-beea512ad90e', 'options': 'recursive,force'} 2024-03-16T10:24:53.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:52 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:24:53.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:52 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:24:53.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:52 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:24:54.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:53 smithi192 ceph-mon[36354]: scrub queued for path: / 2024-03-16T10:24:54.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:53 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:54.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:53 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:24:54.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:53 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:54.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:53 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:24:54.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:53 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:54.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:53 smithi192 ceph-mon[36354]: pgmap v585: 97 pgs: 97 active+clean; 523 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 445 op/s 2024-03-16T10:24:54.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:53 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:24:54.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:53 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:54.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:53 smithi077 ceph-mon[32400]: scrub queued for path: / 2024-03-16T10:24:54.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:53 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:54.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:53 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:24:54.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:53 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:54.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:53 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:24:54.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:53 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:54.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:53 smithi077 ceph-mon[32400]: pgmap v585: 97 pgs: 97 active+clean; 523 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 445 op/s 2024-03-16T10:24:54.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:53 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:24:54.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:53 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:53 smithi193 ceph-mon[36311]: scrub queued for path: / 2024-03-16T10:24:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:53 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:54.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:53 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:24:54.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:53 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:54.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:53 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:24:54.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:53 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:54.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:53 smithi193 ceph-mon[36311]: pgmap v585: 97 pgs: 97 active+clean; 523 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 445 op/s 2024-03-16T10:24:54.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:53 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:24:54.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:53 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:55.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:54 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:24:55.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:54 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:55.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:54 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:24:55.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:54 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:55.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:54 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:24:55.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:54 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:24:56.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:55 smithi192 ceph-mon[36354]: pgmap v586: 97 pgs: 97 active+clean; 514 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 449 op/s 2024-03-16T10:24:56.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:55 smithi077 ceph-mon[32400]: pgmap v586: 97 pgs: 97 active+clean; 514 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 449 op/s 2024-03-16T10:24:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:55 smithi193 ceph-mon[36311]: pgmap v586: 97 pgs: 97 active+clean; 514 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 449 op/s 2024-03-16T10:24:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.368+0000 7fa53f12f640 1 -- 172.21.15.77:0/3163026275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa538101cd0 msgr2=0x7fa5381020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.368+0000 7fa53f12f640 1 --2- 172.21.15.77:0/3163026275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa538101cd0 0x7fa5381020f0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa52c009900 tx=0x7fa52c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:24:57.371 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.368+0000 7fa53f12f640 1 -- 172.21.15.77:0/3163026275 shutdown_connections 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.368+0000 7fa53f12f640 1 --2- 172.21.15.77:0/3163026275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5381013c0 0x7fa538108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.368+0000 7fa53f12f640 1 --2- 172.21.15.77:0/3163026275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa538100a00 0x7fa538100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.368+0000 7fa53f12f640 1 --2- 172.21.15.77:0/3163026275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa538101cd0 0x7fa5381020f0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.368+0000 7fa53f12f640 1 -- 172.21.15.77:0/3163026275 >> 172.21.15.77:0/3163026275 conn(0x7fa5380fb430 msgr2=0x7fa5380fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.368+0000 7fa53f12f640 1 -- 172.21.15.77:0/3163026275 shutdown_connections 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.368+0000 7fa53f12f640 1 -- 172.21.15.77:0/3163026275 wait complete. 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53f12f640 1 Processor -- start 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53f12f640 1 -- start start 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53f12f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa538100a00 0x7fa538197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53f12f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5381013c0 0x7fa538197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53f12f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa538101cd0 0x7fa53819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53f12f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa53810d4f0 con 0x7fa538100a00 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53f12f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa53810d370 con 0x7fa538101cd0 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53f12f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa53810d670 con 0x7fa5381013c0 2024-03-16T10:24:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53cecb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa538100a00 0x7fa538197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53cecb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa538100a00 0x7fa538197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59408/0 (socket says 172.21.15.77:59408) 2024-03-16T10:24:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53cecb640 1 -- 172.21.15.77:0/2885130367 learned_addr learned my addr 172.21.15.77:0/2885130367 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53d6cc640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa538101cd0 0x7fa53819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa537fff640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5381013c0 0x7fa538197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53cecb640 1 -- 172.21.15.77:0/2885130367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5381013c0 msgr2=0x7fa538197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53cecb640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5381013c0 0x7fa538197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53cecb640 1 -- 172.21.15.77:0/2885130367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa538101cd0 msgr2=0x7fa53819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53cecb640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa538101cd0 0x7fa53819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53cecb640 1 -- 172.21.15.77:0/2885130367 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53819f6b0 con 0x7fa538100a00 2024-03-16T10:24:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53d6cc640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa538101cd0 0x7fa53819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:24:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa537fff640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5381013c0 0x7fa538197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:24:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa53cecb640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa538100a00 0x7fa538197a90 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa52c000c00 tx=0x7fa52c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.369+0000 7fa535ffb640 1 -- 172.21.15.77:0/2885130367 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c041070 con 0x7fa538100a00 2024-03-16T10:24:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.370+0000 7fa535ffb640 1 -- 172.21.15.77:0/2885130367 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa52c004450 con 0x7fa538100a00 2024-03-16T10:24:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.370+0000 7fa53f12f640 1 -- 172.21.15.77:0/2885130367 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa53819f940 con 0x7fa538100a00 2024-03-16T10:24:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.370+0000 7fa535ffb640 1 -- 172.21.15.77:0/2885130367 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c034440 con 0x7fa538100a00 2024-03-16T10:24:57.374 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.370+0000 7fa53f12f640 1 -- 172.21.15.77:0/2885130367 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa53819fe80 con 0x7fa538100a00 2024-03-16T10:24:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.372+0000 7fa535ffb640 1 -- 172.21.15.77:0/2885130367 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fa52c044070 con 0x7fa538100a00 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.372+0000 7fa541df1740 1 Processor -- start 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.372+0000 7fa535ffb640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa5280a2920 0x7fa5280a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.372+0000 7fa541df1740 1 -- start start 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.372+0000 7fa541df1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73340 0x561b03f73760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.372+0000 7fa541df1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73d30 0x561b03f78830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.372+0000 7fa53d6cc640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa5280a2920 0x7fa5280a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.372+0000 7fa541df1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f7b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.372+0000 7fa535ffb640 1 -- 172.21.15.77:0/2885130367 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fa52c0f05a0 con 0x7fa538100a00 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.372+0000 7fa541df1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561b03ed8a40 con 0x561b03f78ea0 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.372+0000 7fa541df1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561b03e81920 con 0x561b03f73d30 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.372+0000 7fa541df1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561b03ed4ae0 con 0x561b03f73340 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa53d6cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f7b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa537fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73d30 0x561b03f78830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa53d6cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f7b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59424/0 (socket says 172.21.15.77:59424) 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa53d6cc640 1 -- 172.21.15.77:0/3096931235 learned_addr learned my addr 172.21.15.77:0/3096931235 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa53cecb640 1 --2- 172.21.15.77:0/3096931235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73340 0x561b03f73760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa537fff640 1 -- 172.21.15.77:0/3096931235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73340 msgr2=0x561b03f73760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa537fff640 1 --2- 172.21.15.77:0/3096931235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73340 0x561b03f73760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa537fff640 1 -- 172.21.15.77:0/3096931235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 msgr2=0x561b03f7b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa537fff640 1 --2- 172.21.15.77:0/3096931235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f7b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa537fff640 1 -- 172.21.15.77:0/3096931235 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b03e99460 con 0x561b03f73d30 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa53d6cc640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa5280a2920 0x7fa5280a4de0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa530012d20 tx=0x7fa530009f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa537fff640 1 --2- 172.21.15.77:0/3096931235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73d30 0x561b03f78830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa524009c00 tx=0x7fa5240117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5845c25d36d2e6e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa522ffd640 1 -- 172.21.15.77:0/3096931235 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa524012450 con 0x561b03f73d30 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa522ffd640 1 -- 172.21.15.77:0/3096931235 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa524012ad0 con 0x561b03f73d30 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.373+0000 7fa522ffd640 1 -- 172.21.15.77:0/3096931235 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52400fa60 con 0x561b03f73d30 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.374+0000 7fa541df1740 1 -- 172.21.15.77:0/3096931235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73d30 msgr2=0x561b03f78830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.374+0000 7fa541df1740 1 --2- 172.21.15.77:0/3096931235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73d30 0x561b03f78830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa524009c00 tx=0x7fa5240117b0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.374+0000 7fa541df1740 1 -- 172.21.15.77:0/3096931235 shutdown_connections 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.374+0000 7fa541df1740 1 --2- 172.21.15.77:0/3096931235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f7b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.374+0000 7fa541df1740 1 --2- 172.21.15.77:0/3096931235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73d30 0x561b03f78830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.374+0000 7fa541df1740 1 --2- 172.21.15.77:0/3096931235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73340 0x561b03f73760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.374+0000 7fa541df1740 1 -- 172.21.15.77:0/3096931235 >> 172.21.15.77:0/3096931235 conn(0x561b03f6e8b0 msgr2=0x561b03f6ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.374+0000 7fa541df1740 1 -- 172.21.15.77:0/3096931235 shutdown_connections 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.374+0000 7fa541df1740 1 -- 172.21.15.77:0/3096931235 wait complete. 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.374+0000 7fa541df1740 1 -- start start 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.374+0000 7fa541df1740 1 Processor -- start 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa541df1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73340 0x561b03f7d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa541df1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73d30 0x561b03f7dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa541df1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f84b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa541df1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561b03f7be50 con 0x561b03f78ea0 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa541df1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x561b03e94d70 con 0x561b03f73340 2024-03-16T10:24:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa541df1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561b03f7bfd0 con 0x561b03f73d30 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa541df1740 10 client.0 _ll_register_callbacks cb 0x561b03beca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa53d6cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f84b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa53d6cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f84b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59428/0 (socket says 172.21.15.77:59428) 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa53d6cc640 1 -- 172.21.15.77:0/2323542835 learned_addr learned my addr 172.21.15.77:0/2323542835 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa537fff640 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73d30 0x561b03f7dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa53cecb640 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73340 0x561b03f7d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa53d6cc640 1 -- 172.21.15.77:0/2323542835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73d30 msgr2=0x561b03f7dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa53d6cc640 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73d30 0x561b03f7dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa53d6cc640 1 -- 172.21.15.77:0/2323542835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73340 msgr2=0x561b03f7d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa53d6cc640 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73340 0x561b03f7d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa53d6cc640 1 -- 172.21.15.77:0/2323542835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b03bed210 con 0x561b03f78ea0 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa53d6cc640 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f84b80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa530009a50 tx=0x7fa53001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3286d50b9d097a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa517fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa517fff640 1 -- 172.21.15.77:0/2323542835 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530026070 con 0x561b03f78ea0 2024-03-16T10:24:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa517fff640 1 -- 172.21.15.77:0/2323542835 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fa530020620 con 0x561b03f78ea0 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.375+0000 7fa541df1740 1 -- 172.21.15.77:0/2323542835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa524012ad0 con 0x561b03f78ea0 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.376+0000 7fa517fff640 1 -- 172.21.15.77:0/2323542835 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fa530020820 con 0x561b03f78ea0 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.376+0000 7fa541df1740 10 client.17457 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.376+0000 7fa541df1740 1 -- 172.21.15.77:0/2323542835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x561b03cb9620 con 0x561b03f78ea0 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.376+0000 7fa517fff640 1 -- 172.21.15.77:0/2323542835 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa524012ad0 con 0x561b03f78ea0 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.376+0000 7fa517fff640 1 -- 172.21.15.77:0/2323542835 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fa53001f050 con 0x561b03f78ea0 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.376+0000 7fa541df1740 10 client.17457 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.376+0000 7fa541df1740 10 client.17457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.376+0000 7fa541df1740 20 client.17457 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.376+0000 7fa541df1740 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561b03f894c0 0x561b03f8b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.376+0000 7fa541df1740 4 client.17457 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.376+0000 7fa541df1740 1 -- 172.21.15.77:0/2323542835 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b03ee5f80 con 0x561b03f894c0 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.376+0000 7fa53cecb640 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561b03f894c0 0x561b03f8b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa517fff640 10 client.17457 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa53cecb640 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561b03f894c0 0x561b03f8b950 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18c6b69058dd1256 server_cookie=8263321df353e650 in_seq=0 out_seq=0 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa517fff640 1 -- 172.21.15.77:0/2323542835 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561b03ee5f80 con 0x561b03f894c0 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa517fff640 10 client.17457 handle_command_reply: tid=0 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa541df1740 1 client.17457 shutdown 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa541df1740 1 -- 172.21.15.77:0/2323542835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 msgr2=0x561b03f84b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa541df1740 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f84b80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa530009a50 tx=0x7fa53001c000 comp rx=0 tx=0).stop 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa541df1740 1 -- 172.21.15.77:0/2323542835 shutdown_connections 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa541df1740 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561b03f894c0 0x561b03f8b950 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa541df1740 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f84b80 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa517fff640 0 client.17457 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa541df1740 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73d30 0x561b03f7dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa541df1740 1 --2- 172.21.15.77:0/2323542835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73340 0x561b03f7d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.377+0000 7fa541df1740 1 -- 172.21.15.77:0/2323542835 >> 172.21.15.77:0/2323542835 conn(0x561b03f6e8b0 msgr2=0x561b03f6dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.378+0000 7fa541df1740 1 -- 172.21.15.77:0/2323542835 shutdown_connections 2024-03-16T10:24:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.378+0000 7fa541df1740 1 -- 172.21.15.77:0/2323542835 wait complete. 2024-03-16T10:24:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.378+0000 7fa541df1740 20 client.17457 trim_cache size 0 max 16384 2024-03-16T10:24:57.397 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.392+0000 7fa541df1740 1 Processor -- start 2024-03-16T10:24:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.392+0000 7fa541df1740 1 -- start start 2024-03-16T10:24:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.392+0000 7fa541df1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73340 0x561b03f90250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.392+0000 7fa541df1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f73d30 0x561b03f84b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.392+0000 7fa541df1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f78ea0 0x561b03f90790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.392+0000 7fa541df1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa5381a8b50 con 0x561b03f73d30 2024-03-16T10:24:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.392+0000 7fa541df1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa5381aa720 con 0x561b03f73340 2024-03-16T10:24:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.392+0000 7fa541df1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa5381a9250 con 0x561b03f78ea0 2024-03-16T10:24:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.392+0000 7fa537fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f73d30 0x561b03f84b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa537fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f73d30 0x561b03f84b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59432/0 (socket says 172.21.15.77:59432) 2024-03-16T10:24:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa537fff640 1 -- 172.21.15.77:0/473050222 learned_addr learned my addr 172.21.15.77:0/473050222 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa53cecb640 1 --2- 172.21.15.77:0/473050222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73340 0x561b03f90250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa53d6cc640 1 --2- 172.21.15.77:0/473050222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f78ea0 0x561b03f90790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa537fff640 1 -- 172.21.15.77:0/473050222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f78ea0 msgr2=0x561b03f90790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa537fff640 1 --2- 172.21.15.77:0/473050222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f78ea0 0x561b03f90790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa537fff640 1 -- 172.21.15.77:0/473050222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73340 msgr2=0x561b03f90250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa537fff640 1 --2- 172.21.15.77:0/473050222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73340 0x561b03f90250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa537fff640 1 -- 172.21.15.77:0/473050222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b03f6f300 con 0x561b03f73d30 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa53cecb640 1 --2- 172.21.15.77:0/473050222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73340 0x561b03f90250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa537fff640 1 --2- 172.21.15.77:0/473050222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f73d30 0x561b03f84b40 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x561b03f747a0 tx=0x7fa524009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b6f1f57ab2b305c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa5227fc640 1 -- 172.21.15.77:0/473050222 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa524029070 con 0x561b03f73d30 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa5227fc640 1 -- 172.21.15.77:0/473050222 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa524005520 con 0x561b03f73d30 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.393+0000 7fa5227fc640 1 -- 172.21.15.77:0/473050222 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa524003d80 con 0x561b03f73d30 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 -- 172.21.15.77:0/473050222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f73d30 msgr2=0x561b03f84b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 --2- 172.21.15.77:0/473050222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f73d30 0x561b03f84b40 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x561b03f747a0 tx=0x7fa524009760 comp rx=0 tx=0).stop 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 -- 172.21.15.77:0/473050222 shutdown_connections 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 --2- 172.21.15.77:0/473050222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f78ea0 0x561b03f90790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 --2- 172.21.15.77:0/473050222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f73d30 0x561b03f84b40 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 --2- 172.21.15.77:0/473050222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73340 0x561b03f90250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 -- 172.21.15.77:0/473050222 >> 172.21.15.77:0/473050222 conn(0x561b03f6e8b0 msgr2=0x561b03f71b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 -- 172.21.15.77:0/473050222 shutdown_connections 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 -- 172.21.15.77:0/473050222 wait complete. 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 -- start start 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 Processor -- start 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73340 0x561b03f845a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73d30 0x561b03f84ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f82cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa5381aa050 con 0x561b03f78ea0 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa5381a9950 con 0x561b03f73d30 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa5381a6b30 con 0x561b03f73340 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.394+0000 7fa541df1740 10 client.0 _ll_register_callbacks cb 0x7fa530005c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:24:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa53d6cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f82cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa53cecb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73340 0x561b03f845a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa53d6cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f82cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59436/0 (socket says 172.21.15.77:59436) 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa53cecb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73340 0x561b03f845a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:54760/0 (socket says 172.21.15.77:54760) 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa53d6cc640 1 -- 172.21.15.77:0/1354450976 learned_addr learned my addr 172.21.15.77:0/1354450976 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa537fff640 1 --2- 172.21.15.77:0/1354450976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73d30 0x561b03f84ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa53d6cc640 1 -- 172.21.15.77:0/1354450976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73340 msgr2=0x561b03f845a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa53d6cc640 1 --2- 172.21.15.77:0/1354450976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73340 0x561b03f845a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa53d6cc640 1 -- 172.21.15.77:0/1354450976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73d30 msgr2=0x561b03f84ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa53d6cc640 1 --2- 172.21.15.77:0/1354450976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73d30 0x561b03f84ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa53d6cc640 1 -- 172.21.15.77:0/1354450976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b03f83500 con 0x561b03f78ea0 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa53d6cc640 1 --2- 172.21.15.77:0/1354450976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f82cd0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fa53002ae20 tx=0x7fa530004bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d0f2fb04bf13cc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa517fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa517fff640 1 -- 172.21.15.77:0/1354450976 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530039030 con 0x561b03f78ea0 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa517fff640 1 -- 172.21.15.77:0/1354450976 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fa53001b040 con 0x561b03f78ea0 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa541df1740 1 -- 172.21.15.77:0/1354450976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa524005520 con 0x561b03f78ea0 2024-03-16T10:24:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa517fff640 1 -- 172.21.15.77:0/1354450976 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fa530026040 con 0x561b03f78ea0 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa541df1740 10 client.17463 fetch_fsmap learned FSMap version 43 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.395+0000 7fa541df1740 1 -- 172.21.15.77:0/1354450976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x561b03ff0460 con 0x561b03f78ea0 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.396+0000 7fa517fff640 1 -- 172.21.15.77:0/1354450976 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa524005520 con 0x561b03f78ea0 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.396+0000 7fa517fff640 1 -- 172.21.15.77:0/1354450976 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fa53002a400 con 0x561b03f78ea0 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.396+0000 7fa541df1740 10 client.17463 fetch_fsmap finished waiting for FSMap version 43 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.396+0000 7fa541df1740 10 client.17463 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.396+0000 7fa541df1740 20 client.17463 populate_metadata read hostname 'smithi077' 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.396+0000 7fa541df1740 1 --2- 172.21.15.77:0/1354450976 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561b03ff0e50 0x561b03ff3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.396+0000 7fa541df1740 4 client.17463 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.396+0000 7fa541df1740 1 -- 172.21.15.77:0/1354450976 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561b03ee02d0 con 0x561b03ff0e50 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.396+0000 7fa53cecb640 1 --2- 172.21.15.77:0/1354450976 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561b03ff0e50 0x561b03ff3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.396+0000 7fa517fff640 10 client.17463 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.396+0000 7fa53cecb640 1 --2- 172.21.15.77:0/1354450976 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561b03ff0e50 0x561b03ff3270 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb88e6dbc3b81d26 server_cookie=3e029765403a3bc in_seq=0 out_seq=0 2024-03-16T10:24:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa517fff640 1 -- 172.21.15.77:0/1354450976 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561b03ee02d0 con 0x561b03ff0e50 2024-03-16T10:24:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa517fff640 10 client.17463 handle_command_reply: tid=0 2024-03-16T10:24:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa541df1740 1 client.17463 shutdown 2024-03-16T10:24:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa541df1740 1 -- 172.21.15.77:0/1354450976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 msgr2=0x561b03f82cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa541df1740 1 --2- 172.21.15.77:0/1354450976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f82cd0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fa53002ae20 tx=0x7fa530004bf0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa541df1740 1 -- 172.21.15.77:0/1354450976 shutdown_connections 2024-03-16T10:24:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa541df1740 1 --2- 172.21.15.77:0/1354450976 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x561b03ff0e50 0x561b03ff3270 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa541df1740 1 --2- 172.21.15.77:0/1354450976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561b03f78ea0 0x561b03f82cd0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa541df1740 1 --2- 172.21.15.77:0/1354450976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b03f73d30 0x561b03f84ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa517fff640 0 client.17463 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:24:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa541df1740 1 --2- 172.21.15.77:0/1354450976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b03f73340 0x561b03f845a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa541df1740 1 -- 172.21.15.77:0/1354450976 >> 172.21.15.77:0/1354450976 conn(0x561b03f6e8b0 msgr2=0x561b03f71b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa541df1740 1 -- 172.21.15.77:0/1354450976 shutdown_connections 2024-03-16T10:24:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa541df1740 1 -- 172.21.15.77:0/1354450976 wait complete. 2024-03-16T10:24:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa541df1740 20 client.17463 trim_cache size 0 max 16384 2024-03-16T10:24:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:24:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:24:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:24:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:24:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa53f12f640 1 -- 172.21.15.77:0/2885130367 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa5280a2920 msgr2=0x7fa5280a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa53f12f640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa5280a2920 0x7fa5280a4de0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa530012d20 tx=0x7fa530009f30 comp rx=0 tx=0).stop 2024-03-16T10:24:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.405+0000 7fa53f12f640 1 -- 172.21.15.77:0/2885130367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa538100a00 msgr2=0x7fa538197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:24:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.406+0000 7fa53f12f640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa538100a00 0x7fa538197a90 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa52c000c00 tx=0x7fa52c02c650 comp rx=0 tx=0).stop 2024-03-16T10:24:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.406+0000 7fa53f12f640 1 -- 172.21.15.77:0/2885130367 shutdown_connections 2024-03-16T10:24:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.406+0000 7fa53f12f640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa538101cd0 0x7fa53819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.406+0000 7fa53f12f640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa5280a2920 0x7fa5280a4de0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.406+0000 7fa53f12f640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5381013c0 0x7fa538197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.406+0000 7fa53f12f640 1 --2- 172.21.15.77:0/2885130367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa538100a00 0x7fa538197a90 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:24:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.406+0000 7fa53f12f640 1 -- 172.21.15.77:0/2885130367 >> 172.21.15.77:0/2885130367 conn(0x7fa5380fb430 msgr2=0x7fa538104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:24:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.406+0000 7fa53f12f640 1 -- 172.21.15.77:0/2885130367 shutdown_connections 2024-03-16T10:24:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:24:57.406+0000 7fa53f12f640 1 -- 172.21.15.77:0/2885130367 wait complete. 2024-03-16T10:24:58.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:57 smithi077 ceph-mon[32400]: pgmap v587: 97 pgs: 97 active+clean; 507 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 973 KiB/s wr, 435 op/s 2024-03-16T10:24:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:57 smithi193 ceph-mon[36311]: pgmap v587: 97 pgs: 97 active+clean; 507 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 973 KiB/s wr, 435 op/s 2024-03-16T10:24:58.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:57 smithi192 ceph-mon[36354]: pgmap v587: 97 pgs: 97 active+clean; 507 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 973 KiB/s wr, 435 op/s 2024-03-16T10:25:00.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:24:59 smithi077 ceph-mon[32400]: pgmap v588: 97 pgs: 97 active+clean; 506 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 492 op/s 2024-03-16T10:25:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:24:59 smithi193 ceph-mon[36311]: pgmap v588: 97 pgs: 97 active+clean; 506 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 492 op/s 2024-03-16T10:25:00.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:24:59 smithi192 ceph-mon[36354]: pgmap v588: 97 pgs: 97 active+clean; 506 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 492 op/s 2024-03-16T10:25:02.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:01 smithi077 ceph-mon[32400]: pgmap v589: 97 pgs: 97 active+clean; 505 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 465 op/s 2024-03-16T10:25:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:01 smithi193 ceph-mon[36311]: pgmap v589: 97 pgs: 97 active+clean; 505 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 465 op/s 2024-03-16T10:25:02.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:01 smithi192 ceph-mon[36354]: pgmap v589: 97 pgs: 97 active+clean; 505 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 465 op/s 2024-03-16T10:25:04.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:03 smithi077 ceph-mon[32400]: pgmap v590: 97 pgs: 97 active+clean; 517 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 459 op/s 2024-03-16T10:25:04.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:03 smithi193 ceph-mon[36311]: pgmap v590: 97 pgs: 97 active+clean; 517 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 459 op/s 2024-03-16T10:25:04.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:03 smithi192 ceph-mon[36354]: pgmap v590: 97 pgs: 97 active+clean; 517 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 459 op/s 2024-03-16T10:25:06.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:05 smithi077 ceph-mon[32400]: pgmap v591: 97 pgs: 97 active+clean; 505 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 429 op/s 2024-03-16T10:25:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:05 smithi193 ceph-mon[36311]: pgmap v591: 97 pgs: 97 active+clean; 505 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 429 op/s 2024-03-16T10:25:06.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:05 smithi192 ceph-mon[36354]: pgmap v591: 97 pgs: 97 active+clean; 505 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 429 op/s 2024-03-16T10:25:08.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:07 smithi077 ceph-mon[32400]: pgmap v592: 97 pgs: 97 active+clean; 507 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 411 op/s 2024-03-16T10:25:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:07 smithi193 ceph-mon[36311]: pgmap v592: 97 pgs: 97 active+clean; 507 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 411 op/s 2024-03-16T10:25:08.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:07 smithi192 ceph-mon[36354]: pgmap v592: 97 pgs: 97 active+clean; 507 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 411 op/s 2024-03-16T10:25:10.218 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:09 smithi192 ceph-mon[36354]: pgmap v593: 97 pgs: 97 active+clean; 493 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 497 op/s 2024-03-16T10:25:10.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:09 smithi077 ceph-mon[32400]: pgmap v593: 97 pgs: 97 active+clean; 493 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 497 op/s 2024-03-16T10:25:10.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:09 smithi193 ceph-mon[36311]: pgmap v593: 97 pgs: 97 active+clean; 493 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 497 op/s 2024-03-16T10:25:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.496+0000 7f6c91eff640 1 -- 172.21.15.77:0/389110025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c1022b0 msgr2=0x7f6c8c108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.496+0000 7f6c91eff640 1 --2- 172.21.15.77:0/389110025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c1022b0 0x7f6c8c108b40 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009ca0 tx=0x7f6c8402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.496+0000 7f6c91eff640 1 -- 172.21.15.77:0/389110025 shutdown_connections 2024-03-16T10:25:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.496+0000 7f6c91eff640 1 --2- 172.21.15.77:0/389110025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c1022b0 0x7f6c8c108b40 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.496+0000 7f6c91eff640 1 --2- 172.21.15.77:0/389110025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c8c1018f0 0x7f6c8c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.496+0000 7f6c91eff640 1 --2- 172.21.15.77:0/389110025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c8c100700 0x7f6c8c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.496+0000 7f6c91eff640 1 -- 172.21.15.77:0/389110025 >> 172.21.15.77:0/389110025 conn(0x7f6c8c0fbc70 msgr2=0x7f6c8c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.496+0000 7f6c91eff640 1 -- 172.21.15.77:0/389110025 shutdown_connections 2024-03-16T10:25:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.496+0000 7f6c91eff640 1 -- 172.21.15.77:0/389110025 wait complete. 2024-03-16T10:25:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c91eff640 1 Processor -- start 2024-03-16T10:25:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c91eff640 1 -- start start 2024-03-16T10:25:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c91eff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c8c100700 0x7f6c8c196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c91eff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c1018f0 0x7f6c8c196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c91eff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c8c1022b0 0x7f6c8c19deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c91eff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c10c380 con 0x7f6c8c1018f0 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c91eff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c10c200 con 0x7f6c8c1022b0 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c91eff640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c10c500 con 0x7f6c8c100700 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c8affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c1018f0 0x7f6c8c196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c8affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c1018f0 0x7f6c8c196e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58680/0 (socket says 172.21.15.77:58680) 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c8affd640 1 -- 172.21.15.77:0/1165281471 learned_addr learned my addr 172.21.15.77:0/1165281471 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c8b7fe640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c8c100700 0x7f6c8c196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.497+0000 7f6c8bfff640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c8c1022b0 0x7f6c8c19deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.498+0000 7f6c8affd640 1 -- 172.21.15.77:0/1165281471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c8c100700 msgr2=0x7f6c8c196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.498+0000 7f6c8affd640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c8c100700 0x7f6c8c196900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.498+0000 7f6c8affd640 1 -- 172.21.15.77:0/1165281471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c8c1022b0 msgr2=0x7f6c8c19deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.498+0000 7f6c8affd640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c8c1022b0 0x7f6c8c19deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.498+0000 7f6c8affd640 1 -- 172.21.15.77:0/1165281471 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c19e520 con 0x7f6c8c1018f0 2024-03-16T10:25:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.498+0000 7f6c8affd640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c1018f0 0x7f6c8c196e40 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000ea60 tx=0x7f6c8000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.498+0000 7f6c88ff9640 1 -- 172.21.15.77:0/1165281471 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8000cde0 con 0x7f6c8c1018f0 2024-03-16T10:25:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.498+0000 7f6c88ff9640 1 -- 172.21.15.77:0/1165281471 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c800035f0 con 0x7f6c8c1018f0 2024-03-16T10:25:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.498+0000 7f6c91eff640 1 -- 172.21.15.77:0/1165281471 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c8c19e810 con 0x7f6c8c1018f0 2024-03-16T10:25:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.498+0000 7f6c88ff9640 1 -- 172.21.15.77:0/1165281471 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c80010670 con 0x7f6c8c1018f0 2024-03-16T10:25:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.498+0000 7f6c91eff640 1 -- 172.21.15.77:0/1165281471 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c8c19ed60 con 0x7f6c8c1018f0 2024-03-16T10:25:10.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c94bc8740 1 Processor -- start 2024-03-16T10:25:10.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c94bc8740 1 -- start start 2024-03-16T10:25:10.508 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c88ff9640 1 -- 172.21.15.77:0/1165281471 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f6c800037a0 con 0x7f6c8c1018f0 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c94bc8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2340 0x5648bc5d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c94bc8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2d30 0x5648bc5d7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c94bc8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d7ea0 0x5648bc5da330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c94bc8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5648bc537a40 con 0x5648bc5d2340 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c94bc8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5648bc4e0920 con 0x5648bc5d2d30 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c94bc8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5648bc533ae0 con 0x5648bc5d7ea0 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c8bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2340 0x5648bc5d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c8affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d7ea0 0x5648bc5da330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c8bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2340 0x5648bc5d2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58694/0 (socket says 172.21.15.77:58694) 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c8bfff640 1 -- 172.21.15.77:0/2474383474 learned_addr learned my addr 172.21.15.77:0/2474383474 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c8b7fe640 1 --2- 172.21.15.77:0/2474383474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2d30 0x5648bc5d7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c8bfff640 1 -- 172.21.15.77:0/2474383474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d7ea0 msgr2=0x5648bc5da330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c8bfff640 1 --2- 172.21.15.77:0/2474383474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d7ea0 0x5648bc5da330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c8bfff640 1 -- 172.21.15.77:0/2474383474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2d30 msgr2=0x5648bc5d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c8bfff640 1 --2- 172.21.15.77:0/2474383474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2d30 0x5648bc5d7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c8bfff640 1 -- 172.21.15.77:0/2474383474 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648bc4f8460 con 0x5648bc5d2340 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.501+0000 7f6c8affd640 1 --2- 172.21.15.77:0/2474383474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d7ea0 0x5648bc5da330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.502+0000 7f6c88ff9640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6c740a2920 0x7f6c740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.502+0000 7f6c8bfff640 1 --2- 172.21.15.77:0/2474383474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2340 0x5648bc5d2760 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009ca0 tx=0x7f6c840323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50c68f14f4d0346 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.502+0000 7f6c8b7fe640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6c740a2920 0x7f6c740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.502+0000 7f6c88ff9640 1 -- 172.21.15.77:0/1165281471 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6c800ced60 con 0x7f6c8c1018f0 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.502+0000 7f6c71ffb640 1 -- 172.21.15.77:0/2474383474 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84043070 con 0x5648bc5d2340 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.502+0000 7f6c71ffb640 1 -- 172.21.15.77:0/2474383474 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c84032c90 con 0x5648bc5d2340 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.502+0000 7f6c71ffb640 1 -- 172.21.15.77:0/2474383474 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84039040 con 0x5648bc5d2340 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2474383474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2340 msgr2=0x5648bc5d2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c8b7fe640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6c740a2920 0x7f6c740a4de0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x5648bc5d3130 tx=0x7f6c7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/2474383474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2340 0x5648bc5d2760 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009ca0 tx=0x7f6c840323c0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2474383474 shutdown_connections 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/2474383474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d7ea0 0x5648bc5da330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/2474383474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2d30 0x5648bc5d7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/2474383474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2340 0x5648bc5d2760 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2474383474 >> 172.21.15.77:0/2474383474 conn(0x5648bc5cd8b0 msgr2=0x5648bc5cdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2474383474 shutdown_connections 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2474383474 wait complete. 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 -- start start 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 Processor -- start 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2340 0x5648bc5dc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 0x5648bc5dcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d7ea0 0x5648bc5e3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5648bc5dadf0 con 0x5648bc5d2d30 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5648bc4f3d70 con 0x5648bc5d2340 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5648bc5daf70 con 0x5648bc5d7ea0 2024-03-16T10:25:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.503+0000 7f6c94bc8740 10 client.0 _ll_register_callbacks cb 0x5648bc24ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 0x5648bc5dcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8b7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2340 0x5648bc5dc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 0x5648bc5dcb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58700/0 (socket says 172.21.15.77:58700) 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8b7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2340 0x5648bc5dc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:60666/0 (socket says 172.21.15.77:60666) 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8affd640 1 -- 172.21.15.77:0/606803122 learned_addr learned my addr 172.21.15.77:0/606803122 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8bfff640 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d7ea0 0x5648bc5e3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8affd640 1 -- 172.21.15.77:0/606803122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d7ea0 msgr2=0x5648bc5e3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8affd640 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d7ea0 0x5648bc5e3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8affd640 1 -- 172.21.15.77:0/606803122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2340 msgr2=0x5648bc5dc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8affd640 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2340 0x5648bc5dc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8affd640 1 -- 172.21.15.77:0/606803122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648bc24c210 con 0x5648bc5d2d30 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8bfff640 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d7ea0 0x5648bc5e3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8b7fe640 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2340 0x5648bc5dc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c8affd640 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 0x5648bc5dcb40 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6c800a3960 tx=0x7f6c80097270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852ac3dd60990905 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c6affd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c6affd640 1 -- 172.21.15.77:0/606803122 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8009b230 con 0x5648bc5d2d30 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c6affd640 1 -- 172.21.15.77:0/606803122 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6c80003b30 con 0x5648bc5d2d30 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c94bc8740 1 -- 172.21.15.77:0/606803122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c84032c90 con 0x5648bc5d2d30 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c6affd640 1 -- 172.21.15.77:0/606803122 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f6c8009b050 con 0x5648bc5d2d30 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c94bc8740 10 client.17475 fetch_fsmap learned FSMap version 44 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.504+0000 7f6c94bc8740 1 -- 172.21.15.77:0/606803122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5648bc318620 con 0x5648bc5d2d30 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.505+0000 7f6c6affd640 1 -- 172.21.15.77:0/606803122 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c84032c90 con 0x5648bc5d2d30 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.505+0000 7f6c6affd640 1 -- 172.21.15.77:0/606803122 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 44) ==== 8382+0+0 (secure 0 0 0) 0x7f6c80097d60 con 0x5648bc5d2d30 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.505+0000 7f6c94bc8740 10 client.17475 fetch_fsmap finished waiting for FSMap version 44 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.505+0000 7f6c94bc8740 10 client.17475 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.505+0000 7f6c94bc8740 20 client.17475 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.505+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5648bc5e83b0 0x5648bc5ea800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.505+0000 7f6c94bc8740 4 client.17475 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.505+0000 7f6c94bc8740 1 -- 172.21.15.77:0/606803122 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648bc544f80 con 0x5648bc5e83b0 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.505+0000 7f6c8b7fe640 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5648bc5e83b0 0x5648bc5ea800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c6affd640 10 client.17475 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c8b7fe640 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5648bc5e83b0 0x5648bc5ea800 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4db5188821e8059e server_cookie=3b81f240b2880a21 in_seq=0 out_seq=0 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c6affd640 1 -- 172.21.15.77:0/606803122 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5648bc544f80 con 0x5648bc5e83b0 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c6affd640 10 client.17475 handle_command_reply: tid=0 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c94bc8740 1 client.17475 shutdown 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c94bc8740 1 -- 172.21.15.77:0/606803122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 msgr2=0x5648bc5dcb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 0x5648bc5dcb40 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6c800a3960 tx=0x7f6c80097270 comp rx=0 tx=0).stop 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c94bc8740 1 -- 172.21.15.77:0/606803122 shutdown_connections 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d7ea0 0x5648bc5e3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5648bc5e83b0 0x5648bc5ea800 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 0x5648bc5dcb40 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/606803122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2340 0x5648bc5dc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c6affd640 0 client.17475 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c94bc8740 1 -- 172.21.15.77:0/606803122 >> 172.21.15.77:0/606803122 conn(0x5648bc5cd8b0 msgr2=0x5648bc5ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c94bc8740 1 -- 172.21.15.77:0/606803122 shutdown_connections 2024-03-16T10:25:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.506+0000 7f6c94bc8740 1 -- 172.21.15.77:0/606803122 wait complete. 2024-03-16T10:25:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.507+0000 7f6c94bc8740 20 client.17475 trim_cache size 0 max 16384 2024-03-16T10:25:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.523+0000 7f6c94bc8740 1 Processor -- start 2024-03-16T10:25:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.523+0000 7f6c94bc8740 1 -- start start 2024-03-16T10:25:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.523+0000 7f6c94bc8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d2340 0x5648bc5d1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.523+0000 7f6c94bc8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2d30 0x5648bc5e3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.523+0000 7f6c94bc8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d7ea0 0x5648bc5ef350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.523+0000 7f6c94bc8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c1a79d0 con 0x5648bc5d7ea0 2024-03-16T10:25:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.523+0000 7f6c94bc8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c1a95a0 con 0x5648bc5d2d30 2024-03-16T10:25:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.523+0000 7f6c94bc8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c1a80d0 con 0x5648bc5d2340 2024-03-16T10:25:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c8bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d7ea0 0x5648bc5ef350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c8bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d7ea0 0x5648bc5ef350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58704/0 (socket says 172.21.15.77:58704) 2024-03-16T10:25:10.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c8bfff640 1 -- 172.21.15.77:0/4287561513 learned_addr learned my addr 172.21.15.77:0/4287561513 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:10.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c8b7fe640 1 --2- 172.21.15.77:0/4287561513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d2340 0x5648bc5d1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c8affd640 1 --2- 172.21.15.77:0/4287561513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2d30 0x5648bc5e3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c8bfff640 1 -- 172.21.15.77:0/4287561513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d2340 msgr2=0x5648bc5d1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:10.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c8bfff640 1 --2- 172.21.15.77:0/4287561513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d2340 0x5648bc5d1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c8bfff640 1 -- 172.21.15.77:0/4287561513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2d30 msgr2=0x5648bc5e3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:10.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c8bfff640 1 --2- 172.21.15.77:0/4287561513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2d30 0x5648bc5e3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c8bfff640 1 -- 172.21.15.77:0/4287561513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648bc5eae80 con 0x5648bc5d7ea0 2024-03-16T10:25:10.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c8bfff640 1 --2- 172.21.15.77:0/4287561513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d7ea0 0x5648bc5ef350 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6c8402c700 tx=0x7f6c8402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465eb496cdca443 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:10.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c717fa640 1 -- 172.21.15.77:0/4287561513 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84039070 con 0x5648bc5d7ea0 2024-03-16T10:25:10.530 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c717fa640 1 -- 172.21.15.77:0/4287561513 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c84009540 con 0x5648bc5d7ea0 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.524+0000 7f6c717fa640 1 -- 172.21.15.77:0/4287561513 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84043040 con 0x5648bc5d7ea0 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.525+0000 7f6c94bc8740 1 -- 172.21.15.77:0/4287561513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d7ea0 msgr2=0x5648bc5ef350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.525+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/4287561513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d7ea0 0x5648bc5ef350 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6c8402c700 tx=0x7f6c8402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.525+0000 7f6c94bc8740 1 -- 172.21.15.77:0/4287561513 shutdown_connections 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.525+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/4287561513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d7ea0 0x5648bc5ef350 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.525+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/4287561513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d2d30 0x5648bc5e3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.525+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/4287561513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d2340 0x5648bc5d1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.525+0000 7f6c94bc8740 1 -- 172.21.15.77:0/4287561513 >> 172.21.15.77:0/4287561513 conn(0x5648bc5cd8b0 msgr2=0x5648bc5d0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.525+0000 7f6c94bc8740 1 -- 172.21.15.77:0/4287561513 shutdown_connections 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.525+0000 7f6c94bc8740 1 -- 172.21.15.77:0/4287561513 wait complete. 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.525+0000 7f6c94bc8740 1 -- start start 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c94bc8740 1 Processor -- start 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c94bc8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d2340 0x5648bc5e19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.531 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c94bc8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 0x5648bc5e1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c94bc8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d7ea0 0x5648bc5e2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c94bc8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c1a8ed0 con 0x5648bc5d2d30 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c94bc8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c1a87d0 con 0x5648bc5d7ea0 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c94bc8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c1a59b0 con 0x5648bc5d2340 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c94bc8740 10 client.0 _ll_register_callbacks cb 0x7f6c8000e9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c8affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 0x5648bc5e1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c8b7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d2340 0x5648bc5e19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c8affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 0x5648bc5e1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58716/0 (socket says 172.21.15.77:58716) 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c8affd640 1 -- 172.21.15.77:0/2692112493 learned_addr learned my addr 172.21.15.77:0/2692112493 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c8bfff640 1 --2- 172.21.15.77:0/2692112493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d7ea0 0x5648bc5e2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c8affd640 1 -- 172.21.15.77:0/2692112493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d2340 msgr2=0x5648bc5e19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c8affd640 1 --2- 172.21.15.77:0/2692112493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d2340 0x5648bc5e19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c8affd640 1 -- 172.21.15.77:0/2692112493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d7ea0 msgr2=0x5648bc5e2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:10.532 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c8affd640 1 --2- 172.21.15.77:0/2692112493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d7ea0 0x5648bc5e2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c8affd640 1 -- 172.21.15.77:0/2692112493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648bc5d1f80 con 0x5648bc5d2d30 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c8affd640 1 --2- 172.21.15.77:0/2692112493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 0x5648bc5e1f00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6c8009ac30 tx=0x7f6c800d2410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b6b732fa92c6f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c6affd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c6affd640 1 -- 172.21.15.77:0/2692112493 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8009b230 con 0x5648bc5d2d30 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c6affd640 1 -- 172.21.15.77:0/2692112493 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6c8009acb0 con 0x5648bc5d2d30 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.526+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2692112493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c84009540 con 0x5648bc5d2d30 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.527+0000 7f6c6affd640 1 -- 172.21.15.77:0/2692112493 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f6c8009bdf0 con 0x5648bc5d2d30 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.527+0000 7f6c94bc8740 10 client.17481 fetch_fsmap learned FSMap version 44 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.527+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2692112493 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5648bc650ce0 con 0x5648bc5d2d30 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.527+0000 7f6c6affd640 1 -- 172.21.15.77:0/2692112493 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c84009540 con 0x5648bc5d2d30 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.527+0000 7f6c6affd640 1 -- 172.21.15.77:0/2692112493 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 44) ==== 8382+0+0 (secure 0 0 0) 0x7f6c80097560 con 0x5648bc5d2d30 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.527+0000 7f6c94bc8740 10 client.17481 fetch_fsmap finished waiting for FSMap version 44 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.527+0000 7f6c94bc8740 10 client.17481 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:10.533 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.527+0000 7f6c94bc8740 20 client.17481 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:10.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.527+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/2692112493 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5648bc651330 0x5648bc653750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:10.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.527+0000 7f6c94bc8740 4 client.17481 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:25:10.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.527+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2692112493 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5648bc53f2d0 con 0x5648bc651330 2024-03-16T10:25:10.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.527+0000 7f6c8b7fe640 1 --2- 172.21.15.77:0/2692112493 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5648bc651330 0x5648bc653750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:10.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.528+0000 7f6c8b7fe640 1 --2- 172.21.15.77:0/2692112493 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5648bc651330 0x5648bc653750 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dc18d14052660bb server_cookie=9a0f8bc2fd0598f0 in_seq=0 out_seq=0 2024-03-16T10:25:10.534 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.528+0000 7f6c6affd640 10 client.17481 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.535+0000 7f6c6affd640 1 -- 172.21.15.77:0/2692112493 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5648bc53f2d0 con 0x5648bc651330 2024-03-16T10:25:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.535+0000 7f6c6affd640 10 client.17481 handle_command_reply: tid=0 2024-03-16T10:25:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.535+0000 7f6c94bc8740 1 client.17481 shutdown 2024-03-16T10:25:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2692112493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 msgr2=0x5648bc5e1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/2692112493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 0x5648bc5e1f00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6c8009ac30 tx=0x7f6c800d2410 comp rx=0 tx=0).stop 2024-03-16T10:25:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2692112493 shutdown_connections 2024-03-16T10:25:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/2692112493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5648bc5d7ea0 0x5648bc5e2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/2692112493 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5648bc651330 0x5648bc653750 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/2692112493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5648bc5d2d30 0x5648bc5e1f00 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c94bc8740 1 --2- 172.21.15.77:0/2692112493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648bc5d2340 0x5648bc5e19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2692112493 >> 172.21.15.77:0/2692112493 conn(0x5648bc5cd8b0 msgr2=0x5648bc5d0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c6affd640 0 client.17481 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2692112493 shutdown_connections 2024-03-16T10:25:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c94bc8740 1 -- 172.21.15.77:0/2692112493 wait complete. 2024-03-16T10:25:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c94bc8740 20 client.17481 trim_cache size 0 max 16384 2024-03-16T10:25:10.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:25:10.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:25:10.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:25:10.539 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:25:10.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c91eff640 1 -- 172.21.15.77:0/1165281471 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6c740a2920 msgr2=0x7f6c740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:10.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c91eff640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6c740a2920 0x7f6c740a4de0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x5648bc5d3130 tx=0x7f6c7c005f90 comp rx=0 tx=0).stop 2024-03-16T10:25:10.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c91eff640 1 -- 172.21.15.77:0/1165281471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c1018f0 msgr2=0x7f6c8c196e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:10.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c91eff640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c1018f0 0x7f6c8c196e40 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000ea60 tx=0x7f6c8000ef20 comp rx=0 tx=0).stop 2024-03-16T10:25:10.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c91eff640 1 -- 172.21.15.77:0/1165281471 shutdown_connections 2024-03-16T10:25:10.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c91eff640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c8c1022b0 0x7f6c8c19deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c91eff640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c8c1018f0 0x7f6c8c196e40 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c91eff640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6c740a2920 0x7f6c740a4de0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c91eff640 1 --2- 172.21.15.77:0/1165281471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c8c100700 0x7f6c8c196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:10.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.536+0000 7f6c91eff640 1 -- 172.21.15.77:0/1165281471 >> 172.21.15.77:0/1165281471 conn(0x7f6c8c0fbc70 msgr2=0x7f6c8c104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:10.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.537+0000 7f6c91eff640 1 -- 172.21.15.77:0/1165281471 shutdown_connections 2024-03-16T10:25:10.540 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:10.537+0000 7f6c91eff640 1 -- 172.21.15.77:0/1165281471 wait complete. 2024-03-16T10:25:11.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:10 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:25:11.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:10 smithi077 ceph-mon[32400]: pgmap v594: 97 pgs: 97 active+clean; 485 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 543 op/s 2024-03-16T10:25:11.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:10 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:25:11.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:10 smithi193 ceph-mon[36311]: pgmap v594: 97 pgs: 97 active+clean; 485 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 543 op/s 2024-03-16T10:25:11.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:10 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:25:11.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:10 smithi192 ceph-mon[36354]: pgmap v594: 97 pgs: 97 active+clean; 485 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 543 op/s 2024-03-16T10:25:13.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:12 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:25:13.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:12 smithi077 ceph-mon[32400]: pgmap v595: 97 pgs: 97 active+clean; 478 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 639 op/s 2024-03-16T10:25:13.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:12 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:25:13.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:12 smithi193 ceph-mon[36311]: pgmap v595: 97 pgs: 97 active+clean; 478 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 639 op/s 2024-03-16T10:25:13.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:12 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:25:13.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:12 smithi192 ceph-mon[36354]: pgmap v595: 97 pgs: 97 active+clean; 478 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 639 op/s 2024-03-16T10:25:16.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:15 smithi077 ceph-mon[32400]: pgmap v596: 97 pgs: 97 active+clean; 475 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 727 op/s 2024-03-16T10:25:16.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:15 smithi193 ceph-mon[36311]: pgmap v596: 97 pgs: 97 active+clean; 475 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 727 op/s 2024-03-16T10:25:16.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:15 smithi192 ceph-mon[36354]: pgmap v596: 97 pgs: 97 active+clean; 475 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 727 op/s 2024-03-16T10:25:18.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:17 smithi077 ceph-mon[32400]: pgmap v597: 97 pgs: 97 active+clean; 473 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 802 op/s 2024-03-16T10:25:18.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:17 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:25:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:17 smithi193 ceph-mon[36311]: pgmap v597: 97 pgs: 97 active+clean; 473 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 802 op/s 2024-03-16T10:25:18.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:17 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:25:18.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:17 smithi192 ceph-mon[36354]: pgmap v597: 97 pgs: 97 active+clean; 473 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 802 op/s 2024-03-16T10:25:18.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:17 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:25:20.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:19 smithi077 ceph-mon[32400]: pgmap v598: 97 pgs: 97 active+clean; 445 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 970 op/s 2024-03-16T10:25:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:19 smithi193 ceph-mon[36311]: pgmap v598: 97 pgs: 97 active+clean; 445 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 970 op/s 2024-03-16T10:25:20.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:19 smithi192 ceph-mon[36354]: pgmap v598: 97 pgs: 97 active+clean; 445 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 970 op/s 2024-03-16T10:25:22.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:21 smithi077 ceph-mon[32400]: pgmap v599: 97 pgs: 97 active+clean; 424 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 945 op/s 2024-03-16T10:25:22.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:21 smithi193 ceph-mon[36311]: pgmap v599: 97 pgs: 97 active+clean; 424 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 945 op/s 2024-03-16T10:25:22.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:21 smithi192 ceph-mon[36354]: pgmap v599: 97 pgs: 97 active+clean; 424 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 945 op/s 2024-03-16T10:25:22.762 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:25:22.763 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:25:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.886+0000 7f3ad2790640 1 -- 172.21.15.77:0/1799739066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc100fb0 msgr2=0x7f3acc101430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.886+0000 7f3ad2790640 1 --2- 172.21.15.77:0/1799739066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc100fb0 0x7f3acc101430 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009900 tx=0x7f3ac002bc30 comp rx=0 tx=0).stop 2024-03-16T10:25:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.886+0000 7f3ad2790640 1 -- 172.21.15.77:0/1799739066 shutdown_connections 2024-03-16T10:25:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.886+0000 7f3ad2790640 1 --2- 172.21.15.77:0/1799739066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3acc101970 0x7f3acc108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.886+0000 7f3ad2790640 1 --2- 172.21.15.77:0/1799739066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc100fb0 0x7f3acc101430 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.886+0000 7f3ad2790640 1 --2- 172.21.15.77:0/1799739066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc0ffe60 0x7f3acc100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.886+0000 7f3ad2790640 1 -- 172.21.15.77:0/1799739066 >> 172.21.15.77:0/1799739066 conn(0x7f3acc0fb3d0 msgr2=0x7f3acc0fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.887+0000 7f3ad2790640 1 -- 172.21.15.77:0/1799739066 shutdown_connections 2024-03-16T10:25:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.887+0000 7f3ad2790640 1 -- 172.21.15.77:0/1799739066 wait complete. 2024-03-16T10:25:22.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.887+0000 7f3ad2790640 1 Processor -- start 2024-03-16T10:25:22.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.887+0000 7f3ad2790640 1 -- start start 2024-03-16T10:25:22.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3ad2790640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc0ffe60 0x7f3acc199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3ad2790640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc100fb0 0x7f3acc19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3ad2790640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3acc101970 0x7f3acc1a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3ad2790640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc10c410 con 0x7f3acc0ffe60 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3ad2790640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc10c290 con 0x7f3acc101970 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3acbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc0ffe60 0x7f3acc199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3acbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc0ffe60 0x7f3acc199dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56866/0 (socket says 172.21.15.77:56866) 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3acbfff640 1 -- 172.21.15.77:0/3361031399 learned_addr learned my addr 172.21.15.77:0/3361031399 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3ad2790640 1 -- 172.21.15.77:0/3361031399 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc10c590 con 0x7f3acc100fb0 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3acb7fe640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc100fb0 0x7f3acc19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3ad0d2d640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3acc101970 0x7f3acc1a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3acbfff640 1 -- 172.21.15.77:0/3361031399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc100fb0 msgr2=0x7f3acc19a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3acbfff640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc100fb0 0x7f3acc19a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3acbfff640 1 -- 172.21.15.77:0/3361031399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3acc101970 msgr2=0x7f3acc1a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3acbfff640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3acc101970 0x7f3acc1a1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3acbfff640 1 -- 172.21.15.77:0/3361031399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3acc1a19f0 con 0x7f3acc0ffe60 2024-03-16T10:25:22.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.888+0000 7f3ad0d2d640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3acc101970 0x7f3acc1a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:25:22.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.889+0000 7f3acbfff640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc0ffe60 0x7f3acc199dd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00e9f0 tx=0x7f3abc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:22.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.889+0000 7f3ac97fa640 1 -- 172.21.15.77:0/3361031399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc00cd80 con 0x7f3acc0ffe60 2024-03-16T10:25:22.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.889+0000 7f3ac97fa640 1 -- 172.21.15.77:0/3361031399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3abc005500 con 0x7f3acc0ffe60 2024-03-16T10:25:22.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.889+0000 7f3ac97fa640 1 -- 172.21.15.77:0/3361031399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc010690 con 0x7f3acc0ffe60 2024-03-16T10:25:22.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.889+0000 7f3ad2790640 1 -- 172.21.15.77:0/3361031399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3acc1a1ce0 con 0x7f3acc0ffe60 2024-03-16T10:25:22.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.889+0000 7f3ad2790640 1 -- 172.21.15.77:0/3361031399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3acc1a21b0 con 0x7f3acc0ffe60 2024-03-16T10:25:22.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ac97fa640 1 -- 172.21.15.77:0/3361031399 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f3abc0046c0 con 0x7f3acc0ffe60 2024-03-16T10:25:22.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ad5378740 1 Processor -- start 2024-03-16T10:25:22.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ad5378740 1 -- start start 2024-03-16T10:25:22.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ad5378740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29a980 0x558a2d29ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ad5378740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29b370 0x558a2d29fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ad5378740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d2a2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ad5378740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558a2d1f7500 con 0x558a2d2a04e0 2024-03-16T10:25:22.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ad5378740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x558a2d19fbe0 con 0x558a2d29b370 2024-03-16T10:25:22.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ad5378740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558a2d1f35a0 con 0x558a2d29a980 2024-03-16T10:25:22.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ad0d2d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29a980 0x558a2d29ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3acb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d2a2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ad0d2d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29a980 0x558a2d29ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:35600/0 (socket says 172.21.15.77:35600) 2024-03-16T10:25:22.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3acb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d2a2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56882/0 (socket says 172.21.15.77:56882) 2024-03-16T10:25:22.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ad0d2d640 1 -- 172.21.15.77:0/752293966 learned_addr learned my addr 172.21.15.77:0/752293966 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:22.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.893+0000 7f3ac97fa640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3ab40a2880 0x7f3ab40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.894+0000 7f3ad0d2d640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3ab40a2880 0x7f3ab40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.894+0000 7f3ac97fa640 1 -- 172.21.15.77:0/3361031399 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f3abc0cef30 con 0x7f3acc0ffe60 2024-03-16T10:25:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.894+0000 7f3acb7fe640 1 -- 172.21.15.77:0/752293966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29a980 msgr2=0x558a2d29ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.894+0000 7f3acb7fe640 1 --2- 172.21.15.77:0/752293966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29a980 0x558a2d29ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.894+0000 7f3acb7fe640 1 -- 172.21.15.77:0/752293966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29b370 msgr2=0x558a2d29fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.894+0000 7f3acb7fe640 1 --2- 172.21.15.77:0/752293966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29b370 0x558a2d29fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.896 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.894+0000 7f3acb7fe640 1 -- 172.21.15.77:0/752293966 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a2d1b34b0 con 0x558a2d2a04e0 2024-03-16T10:25:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.894+0000 7f3ad0d2d640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3ab40a2880 0x7f3ab40a4d40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400cd20 tx=0x7f3ac4009f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.895+0000 7f3acb7fe640 1 --2- 172.21.15.77:0/752293966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d2a2970 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3ac002c170 tx=0x7f3ac00095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3abec4bc99d25531 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.895+0000 7f3ab27fc640 1 -- 172.21.15.77:0/752293966 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0042070 con 0x558a2d2a04e0 2024-03-16T10:25:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.895+0000 7f3ab27fc640 1 -- 172.21.15.77:0/752293966 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac002cb80 con 0x558a2d2a04e0 2024-03-16T10:25:22.897 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.895+0000 7f3ab27fc640 1 -- 172.21.15.77:0/752293966 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac00046c0 con 0x558a2d2a04e0 2024-03-16T10:25:22.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.896+0000 7f3ad5378740 1 -- 172.21.15.77:0/752293966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 msgr2=0x558a2d2a2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:22.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.896+0000 7f3ad5378740 1 --2- 172.21.15.77:0/752293966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d2a2970 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3ac002c170 tx=0x7f3ac00095e0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.898+0000 7f3ad5378740 1 -- 172.21.15.77:0/752293966 shutdown_connections 2024-03-16T10:25:22.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.898+0000 7f3ad5378740 1 --2- 172.21.15.77:0/752293966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d2a2970 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.898+0000 7f3ad5378740 1 --2- 172.21.15.77:0/752293966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29b370 0x558a2d29fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.898+0000 7f3ad5378740 1 --2- 172.21.15.77:0/752293966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29a980 0x558a2d29ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.898+0000 7f3ad5378740 1 -- 172.21.15.77:0/752293966 >> 172.21.15.77:0/752293966 conn(0x558a2d295ef0 msgr2=0x558a2d2962f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:22.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.899+0000 7f3ad5378740 1 -- 172.21.15.77:0/752293966 shutdown_connections 2024-03-16T10:25:22.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.899+0000 7f3ad5378740 1 -- 172.21.15.77:0/752293966 wait complete. 2024-03-16T10:25:22.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.899+0000 7f3ad5378740 1 -- start start 2024-03-16T10:25:22.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3ad5378740 1 Processor -- start 2024-03-16T10:25:22.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3ad5378740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29a980 0x558a2d2a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3ad5378740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29b370 0x558a2d2a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3ad5378740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d2ac400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3ad5378740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558a2d2a3480 con 0x558a2d2a04e0 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3ad5378740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x558a2d1b3160 con 0x558a2d29a980 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3ad5378740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558a2d2a3600 con 0x558a2d29b370 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3ad5378740 10 client.0 _ll_register_callbacks cb 0x558a2d197150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3ad0d2d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d2ac400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3acb7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29b370 0x558a2d2a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3acb7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29b370 0x558a2d2a53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:35616/0 (socket says 172.21.15.77:35616) 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3ad0d2d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d2ac400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56896/0 (socket says 172.21.15.77:56896) 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3acb7fe640 1 -- 172.21.15.77:0/2262586053 learned_addr learned my addr 172.21.15.77:0/2262586053 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3acbfff640 1 --2- 172.21.15.77:0/2262586053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29a980 0x558a2d2a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3acb7fe640 1 -- 172.21.15.77:0/2262586053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29a980 msgr2=0x558a2d2a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3acb7fe640 1 --2- 172.21.15.77:0/2262586053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29a980 0x558a2d2a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3acb7fe640 1 -- 172.21.15.77:0/2262586053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 msgr2=0x558a2d2ac400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:22.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3acb7fe640 1 --2- 172.21.15.77:0/2262586053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d2ac400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.900+0000 7f3acb7fe640 1 -- 172.21.15.77:0/2262586053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a2cfd57d0 con 0x558a2d29b370 2024-03-16T10:25:22.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.901+0000 7f3acb7fe640 1 --2- 172.21.15.77:0/2262586053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29b370 0x558a2d2a53c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0036040 tx=0x7f3ac0005750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9e460cd399041d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:22.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.901+0000 7f3aab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:25:22.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.901+0000 7f3aab7fe640 1 -- 172.21.15.77:0/2262586053 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac004b020 con 0x558a2d29b370 2024-03-16T10:25:22.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.901+0000 7f3aab7fe640 1 -- 172.21.15.77:0/2262586053 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f3ac00046c0 con 0x558a2d29b370 2024-03-16T10:25:22.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.901+0000 7f3ad5378740 1 -- 172.21.15.77:0/2262586053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ac002cb80 con 0x558a2d29b370 2024-03-16T10:25:22.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.901+0000 7f3aab7fe640 1 -- 172.21.15.77:0/2262586053 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f3ac0042070 con 0x558a2d29b370 2024-03-16T10:25:22.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.901+0000 7f3ad5378740 10 client.27247 fetch_fsmap learned FSMap version 45 2024-03-16T10:25:22.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.901+0000 7f3ad5378740 1 -- 172.21.15.77:0/2262586053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x558a2cf0b3f0 con 0x558a2d29b370 2024-03-16T10:25:22.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.902+0000 7f3aab7fe640 1 -- 172.21.15.77:0/2262586053 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac002cb80 con 0x558a2d29b370 2024-03-16T10:25:22.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.902+0000 7f3aab7fe640 1 -- 172.21.15.77:0/2262586053 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 45) ==== 8386+0+0 (secure 0 0 0) 0x7f3ac0003ac0 con 0x558a2d29b370 2024-03-16T10:25:22.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.902+0000 7f3ad5378740 10 client.27247 fetch_fsmap finished waiting for FSMap version 45 2024-03-16T10:25:22.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.902+0000 7f3ad5378740 10 client.27247 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:22.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.902+0000 7f3ad5378740 20 client.27247 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:22.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.902+0000 7f3ad5378740 1 --2- 172.21.15.77:0/2262586053 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558a2d2b0560 0x558a2d2b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.902+0000 7f3ad5378740 4 client.27247 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:22.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.902+0000 7f3ad5378740 1 -- 172.21.15.77:0/2262586053 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a2d204a40 con 0x558a2d2b0560 2024-03-16T10:25:22.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.903+0000 7f3acbfff640 1 --2- 172.21.15.77:0/2262586053 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558a2d2b0560 0x558a2d2b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.903+0000 7f3aab7fe640 10 client.27247 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:22.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.903+0000 7f3acbfff640 1 --2- 172.21.15.77:0/2262586053 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558a2d2b0560 0x558a2d2b2950 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd7b85a2fb1ae043 server_cookie=9865145746a30fa8 in_seq=0 out_seq=0 2024-03-16T10:25:22.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.905+0000 7f3aab7fe640 1 -- 172.21.15.77:0/2262586053 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558a2d204a40 con 0x558a2d2b0560 2024-03-16T10:25:22.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.905+0000 7f3aab7fe640 10 client.27247 handle_command_reply: tid=0 2024-03-16T10:25:22.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.905+0000 7f3ad5378740 1 client.27247 shutdown 2024-03-16T10:25:22.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.905+0000 7f3ad5378740 1 -- 172.21.15.77:0/2262586053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29b370 msgr2=0x558a2d2a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:22.907 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.905+0000 7f3ad5378740 1 --2- 172.21.15.77:0/2262586053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29b370 0x558a2d2a53c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0036040 tx=0x7f3ac0005750 comp rx=0 tx=0).stop 2024-03-16T10:25:22.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.906+0000 7f3ad5378740 1 -- 172.21.15.77:0/2262586053 shutdown_connections 2024-03-16T10:25:22.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.906+0000 7f3ad5378740 1 --2- 172.21.15.77:0/2262586053 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558a2d2b0560 0x558a2d2b2950 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.906+0000 7f3ad5378740 1 --2- 172.21.15.77:0/2262586053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d2ac400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.906+0000 7f3ad5378740 1 --2- 172.21.15.77:0/2262586053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29b370 0x558a2d2a53c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.906+0000 7f3ad5378740 1 --2- 172.21.15.77:0/2262586053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29a980 0x558a2d2a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.906+0000 7f3ad5378740 1 -- 172.21.15.77:0/2262586053 >> 172.21.15.77:0/2262586053 conn(0x558a2d295ef0 msgr2=0x558a2d28c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:22.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.906+0000 7f3aab7fe640 0 client.27247 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:22.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.906+0000 7f3ad5378740 1 -- 172.21.15.77:0/2262586053 shutdown_connections 2024-03-16T10:25:22.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.906+0000 7f3ad5378740 1 -- 172.21.15.77:0/2262586053 wait complete. 2024-03-16T10:25:22.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.906+0000 7f3ad5378740 20 client.27247 trim_cache size 0 max 16384 2024-03-16T10:25:22.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad5378740 1 Processor -- start 2024-03-16T10:25:22.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad5378740 1 -- start start 2024-03-16T10:25:22.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad5378740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29a980 0x558a2d2962d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad5378740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29b370 0x558a2d2ac3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad5378740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d28c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad5378740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1a6930 con 0x558a2d2a04e0 2024-03-16T10:25:22.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad5378740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1a8530 con 0x558a2d29b370 2024-03-16T10:25:22.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad5378740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1a7060 con 0x558a2d29a980 2024-03-16T10:25:22.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad0d2d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d28c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad0d2d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d28c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56898/0 (socket says 172.21.15.77:56898) 2024-03-16T10:25:22.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad0d2d640 1 -- 172.21.15.77:0/508897131 learned_addr learned my addr 172.21.15.77:0/508897131 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:22.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3acbfff640 1 --2- 172.21.15.77:0/508897131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29a980 0x558a2d2962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad0d2d640 1 -- 172.21.15.77:0/508897131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29a980 msgr2=0x558a2d2962d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad0d2d640 1 --2- 172.21.15.77:0/508897131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29a980 0x558a2d2962d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad0d2d640 1 -- 172.21.15.77:0/508897131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29b370 msgr2=0x558a2d2ac3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T10:25:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.936+0000 7f3ad0d2d640 1 --2- 172.21.15.77:0/508897131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29b370 0x558a2d2ac3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.937+0000 7f3ad0d2d640 1 -- 172.21.15.77:0/508897131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a2cecb0f0 con 0x558a2d2a04e0 2024-03-16T10:25:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.937+0000 7f3ad0d2d640 1 --2- 172.21.15.77:0/508897131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d28c070 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400ca60 tx=0x7f3ac401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf64812c614a8fb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.938+0000 7f3ab1ffb640 1 -- 172.21.15.77:0/508897131 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac402a070 con 0x558a2d2a04e0 2024-03-16T10:25:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.938+0000 7f3ab1ffb640 1 -- 172.21.15.77:0/508897131 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac4016040 con 0x558a2d2a04e0 2024-03-16T10:25:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.938+0000 7f3ab1ffb640 1 -- 172.21.15.77:0/508897131 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4015050 con 0x558a2d2a04e0 2024-03-16T10:25:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.939+0000 7f3ad5378740 1 -- 172.21.15.77:0/508897131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 msgr2=0x558a2d28c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.939+0000 7f3ad5378740 1 --2- 172.21.15.77:0/508897131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d28c070 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400ca60 tx=0x7f3ac401c000 comp rx=0 tx=0).stop 2024-03-16T10:25:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.939+0000 7f3ad5378740 1 -- 172.21.15.77:0/508897131 shutdown_connections 2024-03-16T10:25:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.939+0000 7f3ad5378740 1 --2- 172.21.15.77:0/508897131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d2a04e0 0x558a2d28c070 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.939+0000 7f3ad5378740 1 --2- 172.21.15.77:0/508897131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29b370 0x558a2d2ac3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.939+0000 7f3ad5378740 1 --2- 172.21.15.77:0/508897131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d29a980 0x558a2d2962d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.939+0000 7f3ad5378740 1 -- 172.21.15.77:0/508897131 >> 172.21.15.77:0/508897131 conn(0x558a2d295ef0 msgr2=0x558a2d29c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.939+0000 7f3ad5378740 1 -- 172.21.15.77:0/508897131 shutdown_connections 2024-03-16T10:25:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.939+0000 7f3ad5378740 1 -- 172.21.15.77:0/508897131 wait complete. 2024-03-16T10:25:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3ad5378740 1 -- start start 2024-03-16T10:25:22.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3ad5378740 1 Processor -- start 2024-03-16T10:25:22.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3ad5378740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29a980 0x558a2d2aa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3ad5378740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d29b370 0x558a2d2aa580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3ad5378740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d2a04e0 0x558a2d2ab540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3ad5378740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1a7e60 con 0x558a2d29b370 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3ad5378740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1a7760 con 0x558a2d29a980 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3ad5378740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1a4950 con 0x558a2d2a04e0 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3ad5378740 10 client.0 _ll_register_callbacks cb 0x7f3ac40098c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3acbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29a980 0x558a2d2aa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3acb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d29b370 0x558a2d2aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3acbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29a980 0x558a2d2aa040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:60954/0 (socket says 172.21.15.77:60954) 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3acb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d29b370 0x558a2d2aa580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56904/0 (socket says 172.21.15.77:56904) 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3ad0d2d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d2a04e0 0x558a2d2ab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.940+0000 7f3acbfff640 1 -- 172.21.15.77:0/1885013288 learned_addr learned my addr 172.21.15.77:0/1885013288 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.941+0000 7f3acbfff640 1 -- 172.21.15.77:0/1885013288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d2a04e0 msgr2=0x558a2d2ab540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.941+0000 7f3acbfff640 1 --2- 172.21.15.77:0/1885013288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d2a04e0 0x558a2d2ab540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.941+0000 7f3acbfff640 1 -- 172.21.15.77:0/1885013288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d29b370 msgr2=0x558a2d2aa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.941+0000 7f3acbfff640 1 --2- 172.21.15.77:0/1885013288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d29b370 0x558a2d2aa580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.941+0000 7f3acbfff640 1 -- 172.21.15.77:0/1885013288 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a2d1be9d0 con 0x558a2d29a980 2024-03-16T10:25:22.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.941+0000 7f3acbfff640 1 --2- 172.21.15.77:0/1885013288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29a980 0x558a2d2aa040 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0cca20 tx=0x7f3abc086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfc580e880110e9b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:22.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.942+0000 7f3aab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:25:22.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.944+0000 7f3aab7fe640 1 -- 172.21.15.77:0/1885013288 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc0af060 con 0x558a2d29a980 2024-03-16T10:25:22.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.944+0000 7f3aab7fe640 1 -- 172.21.15.77:0/1885013288 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f3abc0ccb30 con 0x558a2d29a980 2024-03-16T10:25:22.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.944+0000 7f3ad5378740 1 -- 172.21.15.77:0/1885013288 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ac4016040 con 0x558a2d29a980 2024-03-16T10:25:22.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.944+0000 7f3aab7fe640 1 -- 172.21.15.77:0/1885013288 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f3abc09fa10 con 0x558a2d29a980 2024-03-16T10:25:22.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.944+0000 7f3ad5378740 10 client.27272 fetch_fsmap learned FSMap version 45 2024-03-16T10:25:22.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.944+0000 7f3ad5378740 1 -- 172.21.15.77:0/1885013288 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x558a2d316b70 con 0x558a2d29a980 2024-03-16T10:25:22.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.944+0000 7f3aab7fe640 1 -- 172.21.15.77:0/1885013288 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac4016040 con 0x558a2d29a980 2024-03-16T10:25:22.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.945+0000 7f3aab7fe640 1 -- 172.21.15.77:0/1885013288 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 45) ==== 8386+0+0 (secure 0 0 0) 0x7f3abc09bbb0 con 0x558a2d29a980 2024-03-16T10:25:22.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.945+0000 7f3ad5378740 10 client.27272 fetch_fsmap finished waiting for FSMap version 45 2024-03-16T10:25:22.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.945+0000 7f3ad5378740 10 client.27272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:22.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.945+0000 7f3ad5378740 20 client.27272 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:22.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.945+0000 7f3ad5378740 1 --2- 172.21.15.77:0/1885013288 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558a2d3174c0 0x558a2d3198e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:22.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.945+0000 7f3ad5378740 4 client.27272 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:25:22.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.945+0000 7f3ad5378740 1 -- 172.21.15.77:0/1885013288 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558a2d1fed90 con 0x558a2d3174c0 2024-03-16T10:25:22.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.945+0000 7f3acb7fe640 1 --2- 172.21.15.77:0/1885013288 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558a2d3174c0 0x558a2d3198e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:22.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.946+0000 7f3acb7fe640 1 --2- 172.21.15.77:0/1885013288 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558a2d3174c0 0x558a2d3198e0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c716c91e358c917 server_cookie=b78ee2afad50605a in_seq=0 out_seq=0 2024-03-16T10:25:22.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.946+0000 7f3aab7fe640 10 client.27272 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:22.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3aab7fe640 1 -- 172.21.15.77:0/1885013288 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558a2d1fed90 con 0x558a2d3174c0 2024-03-16T10:25:22.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3aab7fe640 10 client.27272 handle_command_reply: tid=0 2024-03-16T10:25:22.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3ad5378740 1 client.27272 shutdown 2024-03-16T10:25:22.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3ad5378740 1 -- 172.21.15.77:0/1885013288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29a980 msgr2=0x558a2d2aa040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:22.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3ad5378740 1 --2- 172.21.15.77:0/1885013288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29a980 0x558a2d2aa040 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0cca20 tx=0x7f3abc086040 comp rx=0 tx=0).stop 2024-03-16T10:25:22.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3ad5378740 1 -- 172.21.15.77:0/1885013288 shutdown_connections 2024-03-16T10:25:22.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3ad5378740 1 --2- 172.21.15.77:0/1885013288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a2d2a04e0 0x558a2d2ab540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3ad5378740 1 --2- 172.21.15.77:0/1885013288 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x558a2d3174c0 0x558a2d3198e0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3ad5378740 1 --2- 172.21.15.77:0/1885013288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a2d29b370 0x558a2d2aa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3ad5378740 1 --2- 172.21.15.77:0/1885013288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558a2d29a980 0x558a2d2aa040 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3aab7fe640 0 client.27272 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:22.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3ad5378740 1 -- 172.21.15.77:0/1885013288 >> 172.21.15.77:0/1885013288 conn(0x558a2d295ef0 msgr2=0x558a2d299150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:22.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3ad5378740 1 -- 172.21.15.77:0/1885013288 shutdown_connections 2024-03-16T10:25:22.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3ad5378740 1 -- 172.21.15.77:0/1885013288 wait complete. 2024-03-16T10:25:22.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.947+0000 7f3ad5378740 20 client.27272 trim_cache size 0 max 16384 2024-03-16T10:25:22.949 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:25:22.950 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:25:22.950 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:25:22.950 INFO:teuthology.orchestra.run.smithi077.stdout: "7f420d11-ebf5-48c7-9aaa-beea512ad90e": { 2024-03-16T10:25:22.950 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:25:22.950 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "7f420d11-ebf5-48c7-9aaa-beea512ad90e", 2024-03-16T10:25:22.950 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:25:22.950 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:25:22.950 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:25:22.950 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:25:22.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.948+0000 7f3ad2790640 1 -- 172.21.15.77:0/3361031399 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3ab40a2880 msgr2=0x7f3ab40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:22.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.948+0000 7f3ad2790640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3ab40a2880 0x7f3ab40a4d40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400cd20 tx=0x7f3ac4009f70 comp rx=0 tx=0).stop 2024-03-16T10:25:22.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.948+0000 7f3ad2790640 1 -- 172.21.15.77:0/3361031399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc0ffe60 msgr2=0x7f3acc199dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:22.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.948+0000 7f3ad2790640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc0ffe60 0x7f3acc199dd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00e9f0 tx=0x7f3abc00edf0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.948+0000 7f3ad2790640 1 -- 172.21.15.77:0/3361031399 shutdown_connections 2024-03-16T10:25:22.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.948+0000 7f3ad2790640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3acc101970 0x7f3acc1a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.948+0000 7f3ad2790640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f3ab40a2880 0x7f3ab40a4d40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.948+0000 7f3ad2790640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc100fb0 0x7f3acc19a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.948+0000 7f3ad2790640 1 --2- 172.21.15.77:0/3361031399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc0ffe60 0x7f3acc199dd0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:22.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.948+0000 7f3ad2790640 1 -- 172.21.15.77:0/3361031399 >> 172.21.15.77:0/3361031399 conn(0x7f3acc0fb3d0 msgr2=0x7f3acc108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:22.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.948+0000 7f3ad2790640 1 -- 172.21.15.77:0/3361031399 shutdown_connections 2024-03-16T10:25:22.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:22.948+0000 7f3ad2790640 1 -- 172.21.15.77:0/3361031399 wait complete. 2024-03-16T10:25:22.963 INFO:tasks.cephfs.filesystem:scrub status for tag:7f420d11-ebf5-48c7-9aaa-beea512ad90e - {'path': '/', 'tag': '7f420d11-ebf5-48c7-9aaa-beea512ad90e', 'options': 'recursive,force'} 2024-03-16T10:25:23.640 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.637+0000 7fc4381a9640 1 -- 172.21.15.77:0/2480991955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4301033c0 msgr2=0x7fc430109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.637+0000 7fc4381a9640 1 --2- 172.21.15.77:0/2480991955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4301033c0 0x7fc430109c50 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc428009ca0 tx=0x7fc42802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.637+0000 7fc4381a9640 1 -- 172.21.15.77:0/2480991955 shutdown_connections 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.637+0000 7fc4381a9640 1 --2- 172.21.15.77:0/2480991955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4301033c0 0x7fc430109c50 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.637+0000 7fc4381a9640 1 --2- 172.21.15.77:0/2480991955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc430102a00 0x7fc430102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.637+0000 7fc4381a9640 1 --2- 172.21.15.77:0/2480991955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc430101810 0x7fc430101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.637+0000 7fc4381a9640 1 -- 172.21.15.77:0/2480991955 >> 172.21.15.77:0/2480991955 conn(0x7fc4300fcda0 msgr2=0x7fc4300ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.638+0000 7fc4381a9640 1 -- 172.21.15.77:0/2480991955 shutdown_connections 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.638+0000 7fc4381a9640 1 -- 172.21.15.77:0/2480991955 wait complete. 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.638+0000 7fc4381a9640 1 Processor -- start 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.638+0000 7fc4381a9640 1 -- start start 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.638+0000 7fc4381a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc430101810 0x7fc430197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.638+0000 7fc4381a9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc430102a00 0x7fc430197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.638+0000 7fc4381a9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4301033c0 0x7fc43019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.638+0000 7fc4381a9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc43010d490 con 0x7fc430101810 2024-03-16T10:25:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.638+0000 7fc4381a9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc43010d310 con 0x7fc430102a00 2024-03-16T10:25:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.638+0000 7fc4381a9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc43010d610 con 0x7fc4301033c0 2024-03-16T10:25:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc435f45640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc430101810 0x7fc430197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc435f45640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc430101810 0x7fc430197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56910/0 (socket says 172.21.15.77:56910) 2024-03-16T10:25:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc435f45640 1 -- 172.21.15.77:0/449687014 learned_addr learned my addr 172.21.15.77:0/449687014 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc436746640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4301033c0 0x7fc43019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc435744640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc430102a00 0x7fc430197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc436746640 1 -- 172.21.15.77:0/449687014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc430102a00 msgr2=0x7fc430197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc436746640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc430102a00 0x7fc430197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc436746640 1 -- 172.21.15.77:0/449687014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc430101810 msgr2=0x7fc430197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc436746640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc430101810 0x7fc430197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc436746640 1 -- 172.21.15.77:0/449687014 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc43019f640 con 0x7fc4301033c0 2024-03-16T10:25:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc435f45640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc430101810 0x7fc430197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc436746640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4301033c0 0x7fc43019efd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc42802c630 tx=0x7fc428032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc426ffd640 1 -- 172.21.15.77:0/449687014 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc428041020 con 0x7fc4301033c0 2024-03-16T10:25:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc4381a9640 1 -- 172.21.15.77:0/449687014 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc43019f8d0 con 0x7fc4301033c0 2024-03-16T10:25:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc426ffd640 1 -- 172.21.15.77:0/449687014 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc428034cd0 con 0x7fc4301033c0 2024-03-16T10:25:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc426ffd640 1 -- 172.21.15.77:0/449687014 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc428039040 con 0x7fc4301033c0 2024-03-16T10:25:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.639+0000 7fc4381a9640 1 -- 172.21.15.77:0/449687014 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc43019fd90 con 0x7fc4301033c0 2024-03-16T10:25:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.642+0000 7fc426ffd640 1 -- 172.21.15.77:0/449687014 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fc428044020 con 0x7fc4301033c0 2024-03-16T10:25:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc43ad8c740 1 Processor -- start 2024-03-16T10:25:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc43ad8c740 1 -- start start 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc43ad8c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9acd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc43ad8c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 0x55a7b9ad2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc43ad8c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9ad5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc43ad8c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a7b9a32a40 con 0x55a7b9acdd30 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc43ad8c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a7b99db920 con 0x55a7b9ad2ea0 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc43ad8c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a7b9a2eae0 con 0x55a7b9acd340 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc426ffd640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc41c0a28a0 0x7fc41c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc435f45640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 0x55a7b9ad2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc435744640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9acd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc435744640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9acd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:35662/0 (socket says 172.21.15.77:35662) 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc435744640 1 -- 172.21.15.77:0/3098380750 learned_addr learned my addr 172.21.15.77:0/3098380750 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc435744640 1 --2- 172.21.15.77:0/3098380750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9ad5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.643+0000 7fc436746640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc41c0a28a0 0x7fc41c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.644+0000 7fc435744640 1 -- 172.21.15.77:0/3098380750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 msgr2=0x55a7b9ad5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.644+0000 7fc435744640 1 --2- 172.21.15.77:0/3098380750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9ad5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.644+0000 7fc435744640 1 -- 172.21.15.77:0/3098380750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 msgr2=0x55a7b9ad2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.644+0000 7fc435744640 1 --2- 172.21.15.77:0/3098380750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 0x55a7b9ad2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.644+0000 7fc426ffd640 1 -- 172.21.15.77:0/449687014 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fc4280f0790 con 0x7fc4301033c0 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.644+0000 7fc435744640 1 -- 172.21.15.77:0/3098380750 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7b99f3460 con 0x55a7b9acd340 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.644+0000 7fc435f45640 1 --2- 172.21.15.77:0/3098380750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 0x55a7b9ad2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.644+0000 7fc435744640 1 --2- 172.21.15.77:0/3098380750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9acd760 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc420011810 tx=0x7fc4200097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cadfca2f293213f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.644+0000 7fc436746640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc41c0a28a0 0x7fc41c0a4d60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc4280bd1e0 tx=0x7fc4280e7c10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.644+0000 7fc41bfff640 1 -- 172.21.15.77:0/3098380750 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc420004030 con 0x55a7b9acd340 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.644+0000 7fc41bfff640 1 -- 172.21.15.77:0/3098380750 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc420011e50 con 0x55a7b9acd340 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.644+0000 7fc41bfff640 1 -- 172.21.15.77:0/3098380750 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc420003850 con 0x55a7b9acd340 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.645+0000 7fc43ad8c740 1 -- 172.21.15.77:0/3098380750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 msgr2=0x55a7b9acd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.645+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/3098380750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9acd760 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc420011810 tx=0x7fc4200097d0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.645+0000 7fc43ad8c740 1 -- 172.21.15.77:0/3098380750 shutdown_connections 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.645+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/3098380750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9ad5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.645+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/3098380750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 0x55a7b9ad2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.645+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/3098380750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9acd760 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.645+0000 7fc43ad8c740 1 -- 172.21.15.77:0/3098380750 >> 172.21.15.77:0/3098380750 conn(0x55a7b9ac88b0 msgr2=0x55a7b9ac8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.645+0000 7fc43ad8c740 1 -- 172.21.15.77:0/3098380750 shutdown_connections 2024-03-16T10:25:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.645+0000 7fc43ad8c740 1 -- 172.21.15.77:0/3098380750 wait complete. 2024-03-16T10:25:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.645+0000 7fc43ad8c740 1 -- start start 2024-03-16T10:25:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.645+0000 7fc43ad8c740 1 Processor -- start 2024-03-16T10:25:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc43ad8c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9ad7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc43ad8c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 0x55a7b9ad7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc43ad8c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9adebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc43ad8c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a7b9ad5e50 con 0x55a7b9acdd30 2024-03-16T10:25:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc43ad8c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55a7b99eed70 con 0x55a7b9ad2ea0 2024-03-16T10:25:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc43ad8c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a7b9ad5fd0 con 0x55a7b9acd340 2024-03-16T10:25:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc43ad8c740 10 client.0 _ll_register_callbacks cb 0x55a7b9746a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc435f45640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 0x55a7b9ad7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc435f45640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 0x55a7b9ad7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56934/0 (socket says 172.21.15.77:56934) 2024-03-16T10:25:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc435f45640 1 -- 172.21.15.77:0/1807743410 learned_addr learned my addr 172.21.15.77:0/1807743410 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc435744640 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9ad7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc435744640 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9adebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc435f45640 1 -- 172.21.15.77:0/1807743410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 msgr2=0x55a7b9ad7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc435f45640 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9ad7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc435f45640 1 -- 172.21.15.77:0/1807743410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 msgr2=0x55a7b9adebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc435f45640 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9adebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc435f45640 1 -- 172.21.15.77:0/1807743410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7b9747210 con 0x55a7b9acdd30 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.646+0000 7fc435744640 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9adebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.647+0000 7fc435f45640 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 0x55a7b9ad7bb0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc42c012a00 tx=0x7fc42c012ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d853b57161279e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.647+0000 7fc4197fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.647+0000 7fc4197fa640 1 -- 172.21.15.77:0/1807743410 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c00fa00 con 0x55a7b9acdd30 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.647+0000 7fc43ad8c740 1 -- 172.21.15.77:0/1807743410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc420011e50 con 0x55a7b9acdd30 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.647+0000 7fc4197fa640 1 -- 172.21.15.77:0/1807743410 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fc42c018070 con 0x55a7b9acdd30 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.647+0000 7fc4197fa640 1 -- 172.21.15.77:0/1807743410 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc42c00fa00 con 0x55a7b9acdd30 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.647+0000 7fc43ad8c740 10 client.17511 fetch_fsmap learned FSMap version 45 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.647+0000 7fc43ad8c740 1 -- 172.21.15.77:0/1807743410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55a7b9813620 con 0x55a7b9acdd30 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.647+0000 7fc4197fa640 1 -- 172.21.15.77:0/1807743410 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc420011e50 con 0x55a7b9acdd30 2024-03-16T10:25:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.648+0000 7fc4197fa640 1 -- 172.21.15.77:0/1807743410 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 45) ==== 8386+0+0 (secure 0 0 0) 0x7fc42c003830 con 0x55a7b9acdd30 2024-03-16T10:25:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.648+0000 7fc43ad8c740 10 client.17511 fetch_fsmap finished waiting for FSMap version 45 2024-03-16T10:25:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.648+0000 7fc43ad8c740 10 client.17511 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.648+0000 7fc43ad8c740 20 client.17511 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.648+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b9ae35b0 0x55a7b9ae5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.648+0000 7fc43ad8c740 4 client.17511 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.648+0000 7fc43ad8c740 1 -- 172.21.15.77:0/1807743410 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7b9a3ff80 con 0x55a7b9ae35b0 2024-03-16T10:25:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.648+0000 7fc435744640 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b9ae35b0 0x55a7b9ae5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.648+0000 7fc4197fa640 10 client.17511 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.648+0000 7fc435744640 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b9ae35b0 0x55a7b9ae5a10 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76b17be6a1b6861f server_cookie=31cdaac223d55876 in_seq=0 out_seq=0 2024-03-16T10:25:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc4197fa640 1 -- 172.21.15.77:0/1807743410 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a7b9a3ff80 con 0x55a7b9ae35b0 2024-03-16T10:25:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc4197fa640 10 client.17511 handle_command_reply: tid=0 2024-03-16T10:25:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc43ad8c740 1 client.17511 shutdown 2024-03-16T10:25:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc43ad8c740 1 -- 172.21.15.77:0/1807743410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 msgr2=0x55a7b9ad7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 0x55a7b9ad7bb0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc42c012a00 tx=0x7fc42c012ec0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc43ad8c740 1 -- 172.21.15.77:0/1807743410 shutdown_connections 2024-03-16T10:25:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9adebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b9ae35b0 0x55a7b9ae5a10 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acdd30 0x55a7b9ad7bb0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/1807743410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9ad7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc4197fa640 0 client.17511 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc43ad8c740 1 -- 172.21.15.77:0/1807743410 >> 172.21.15.77:0/1807743410 conn(0x55a7b9ac88b0 msgr2=0x55a7b9ac7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc43ad8c740 1 -- 172.21.15.77:0/1807743410 shutdown_connections 2024-03-16T10:25:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc43ad8c740 1 -- 172.21.15.77:0/1807743410 wait complete. 2024-03-16T10:25:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.649+0000 7fc43ad8c740 20 client.17511 trim_cache size 0 max 16384 2024-03-16T10:25:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.677+0000 7fc43ad8c740 1 Processor -- start 2024-03-16T10:25:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.677+0000 7fc43ad8c740 1 -- start start 2024-03-16T10:25:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.677+0000 7fc43ad8c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acd340 0x55a7b9acbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.677+0000 7fc43ad8c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acdd30 0x55a7b9adebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.677+0000 7fc43ad8c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9acc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.677+0000 7fc43ad8c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc4301a6b20 con 0x55a7b9acd340 2024-03-16T10:25:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.677+0000 7fc43ad8c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc4301a8720 con 0x55a7b9ad2ea0 2024-03-16T10:25:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.677+0000 7fc43ad8c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc4301a7250 con 0x55a7b9acdd30 2024-03-16T10:25:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc435744640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acd340 0x55a7b9acbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc435744640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acd340 0x55a7b9acbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56940/0 (socket says 172.21.15.77:56940) 2024-03-16T10:25:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc435744640 1 -- 172.21.15.77:0/2583304364 learned_addr learned my addr 172.21.15.77:0/2583304364 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc435f45640 1 --2- 172.21.15.77:0/2583304364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acdd30 0x55a7b9adebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc435744640 1 --2- 172.21.15.77:0/2583304364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9acc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc435744640 1 -- 172.21.15.77:0/2583304364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acdd30 msgr2=0x55a7b9adebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc435744640 1 --2- 172.21.15.77:0/2583304364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acdd30 0x55a7b9adebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc435744640 1 -- 172.21.15.77:0/2583304364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 msgr2=0x55a7b9acc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc435744640 1 --2- 172.21.15.77:0/2583304364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9acc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc435744640 1 -- 172.21.15.77:0/2583304364 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7b9ac9300 con 0x55a7b9acd340 2024-03-16T10:25:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc435f45640 1 --2- 172.21.15.77:0/2583304364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acdd30 0x55a7b9adebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc435744640 1 --2- 172.21.15.77:0/2583304364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acd340 0x55a7b9acbb10 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x55a7b9ace590 tx=0x7fc4200097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dafef4f9c5122f64 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc41b7fe640 1 -- 172.21.15.77:0/2583304364 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4200054d0 con 0x55a7b9acd340 2024-03-16T10:25:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc41b7fe640 1 -- 172.21.15.77:0/2583304364 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4200317b0 con 0x55a7b9acd340 2024-03-16T10:25:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.678+0000 7fc41b7fe640 1 -- 172.21.15.77:0/2583304364 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4200239f0 con 0x55a7b9acd340 2024-03-16T10:25:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.679+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2583304364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acd340 msgr2=0x55a7b9acbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.679+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/2583304364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acd340 0x55a7b9acbb10 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x55a7b9ace590 tx=0x7fc4200097d0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.679+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2583304364 shutdown_connections 2024-03-16T10:25:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.679+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/2583304364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9acc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.679+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/2583304364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acdd30 0x55a7b9adebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.679+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/2583304364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9acd340 0x55a7b9acbb10 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.679+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2583304364 >> 172.21.15.77:0/2583304364 conn(0x55a7b9ac88b0 msgr2=0x55a7b9acf050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.679+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2583304364 shutdown_connections 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.679+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2583304364 wait complete. 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc43ad8c740 1 -- start start 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc43ad8c740 1 Processor -- start 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc43ad8c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9adca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc43ad8c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9acdd30 0x55a7b9adcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc43ad8c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9addfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc43ad8c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc4301a8050 con 0x55a7b9ad2ea0 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc43ad8c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc4301a7950 con 0x55a7b9acdd30 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc43ad8c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc4301a4aa0 con 0x55a7b9acd340 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc43ad8c740 10 client.0 _ll_register_callbacks cb 0x7fc42c009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc435744640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9addfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc435744640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9addfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56946/0 (socket says 172.21.15.77:56946) 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc435744640 1 -- 172.21.15.77:0/2048646926 learned_addr learned my addr 172.21.15.77:0/2048646926 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc435f45640 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9acdd30 0x55a7b9adcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.680+0000 7fc435744640 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9adca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc435744640 1 -- 172.21.15.77:0/2048646926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 msgr2=0x55a7b9adca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc435744640 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9adca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc435744640 1 -- 172.21.15.77:0/2048646926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9acdd30 msgr2=0x55a7b9adcfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc435744640 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9acdd30 0x55a7b9adcfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc435744640 1 -- 172.21.15.77:0/2048646926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7b9ade5a0 con 0x55a7b9ad2ea0 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc435f45640 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9acdd30 0x55a7b9adcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc435744640 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9addfb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc420009710 tx=0x7fc420011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac854d35950d701e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc4197fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc4197fa640 1 -- 172.21.15.77:0/2048646926 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc420031ca0 con 0x55a7b9ad2ea0 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc4197fa640 1 -- 172.21.15.77:0/2048646926 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fc4200312d0 con 0x55a7b9ad2ea0 2024-03-16T10:25:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2048646926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4200317b0 con 0x55a7b9ad2ea0 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc4197fa640 1 -- 172.21.15.77:0/2048646926 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc4200314d0 con 0x55a7b9ad2ea0 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc43ad8c740 10 client.17517 fetch_fsmap learned FSMap version 45 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.681+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2048646926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55a7b9b4a080 con 0x55a7b9ad2ea0 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.682+0000 7fc4197fa640 1 -- 172.21.15.77:0/2048646926 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4200317b0 con 0x55a7b9ad2ea0 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.682+0000 7fc4197fa640 1 -- 172.21.15.77:0/2048646926 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 45) ==== 8386+0+0 (secure 0 0 0) 0x7fc42000fa60 con 0x55a7b9ad2ea0 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.682+0000 7fc43ad8c740 10 client.17517 fetch_fsmap finished waiting for FSMap version 45 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.682+0000 7fc43ad8c740 10 client.17517 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.682+0000 7fc43ad8c740 20 client.17517 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.682+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b9b4a8e0 0x55a7b9b4cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.682+0000 7fc43ad8c740 4 client.17517 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.682+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2048646926 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a7b9a3a2d0 con 0x55a7b9b4a8e0 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.682+0000 7fc435f45640 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b9b4a8e0 0x55a7b9b4cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.682+0000 7fc4197fa640 10 client.17517 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.682+0000 7fc435f45640 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b9b4a8e0 0x55a7b9b4cd00 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=688ac9d03d65799a server_cookie=fd177aa0ac8dab49 in_seq=0 out_seq=0 2024-03-16T10:25:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc4197fa640 1 -- 172.21.15.77:0/2048646926 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a7b9a3a2d0 con 0x55a7b9b4a8e0 2024-03-16T10:25:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc4197fa640 10 client.17517 handle_command_reply: tid=0 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc43ad8c740 1 client.17517 shutdown 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2048646926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9ad2ea0 msgr2=0x55a7b9addfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9addfb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc420009710 tx=0x7fc420011e50 comp rx=0 tx=0).stop 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2048646926 shutdown_connections 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55a7b9b4a8e0 0x55a7b9b4cd00 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a7b9ad2ea0 0x55a7b9addfb0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a7b9acdd30 0x55a7b9adcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc4197fa640 0 client.17517 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc43ad8c740 1 --2- 172.21.15.77:0/2048646926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7b9acd340 0x55a7b9adca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2048646926 >> 172.21.15.77:0/2048646926 conn(0x55a7b9ac88b0 msgr2=0x55a7b9acbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2048646926 shutdown_connections 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc43ad8c740 1 -- 172.21.15.77:0/2048646926 wait complete. 2024-03-16T10:25:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.690+0000 7fc43ad8c740 20 client.17517 trim_cache size 0 max 16384 2024-03-16T10:25:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:25:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:25:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:25:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:25:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.691+0000 7fc4381a9640 1 -- 172.21.15.77:0/449687014 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc41c0a28a0 msgr2=0x7fc41c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.691+0000 7fc4381a9640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc41c0a28a0 0x7fc41c0a4d60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc4280bd1e0 tx=0x7fc4280e7c10 comp rx=0 tx=0).stop 2024-03-16T10:25:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.691+0000 7fc4381a9640 1 -- 172.21.15.77:0/449687014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4301033c0 msgr2=0x7fc43019efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.691+0000 7fc4381a9640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4301033c0 0x7fc43019efd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc42802c630 tx=0x7fc428032f20 comp rx=0 tx=0).stop 2024-03-16T10:25:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.691+0000 7fc4381a9640 1 -- 172.21.15.77:0/449687014 shutdown_connections 2024-03-16T10:25:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.691+0000 7fc4381a9640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fc41c0a28a0 0x7fc41c0a4d60 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.691+0000 7fc4381a9640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4301033c0 0x7fc43019efd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.691+0000 7fc4381a9640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc430102a00 0x7fc430197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.691+0000 7fc4381a9640 1 --2- 172.21.15.77:0/449687014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc430101810 0x7fc430197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.691+0000 7fc4381a9640 1 -- 172.21.15.77:0/449687014 >> 172.21.15.77:0/449687014 conn(0x7fc4300fcda0 msgr2=0x7fc430105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.691+0000 7fc4381a9640 1 -- 172.21.15.77:0/449687014 shutdown_connections 2024-03-16T10:25:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:23.691+0000 7fc4381a9640 1 -- 172.21.15.77:0/449687014 wait complete. 2024-03-16T10:25:24.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:23 smithi077 ceph-mon[32400]: pgmap v600: 97 pgs: 97 active+clean; 414 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 876 op/s 2024-03-16T10:25:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:23 smithi193 ceph-mon[36311]: pgmap v600: 97 pgs: 97 active+clean; 414 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 876 op/s 2024-03-16T10:25:24.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:23 smithi192 ceph-mon[36354]: pgmap v600: 97 pgs: 97 active+clean; 414 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 876 op/s 2024-03-16T10:25:26.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:25 smithi077 ceph-mon[32400]: pgmap v601: 97 pgs: 97 active+clean; 410 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 760 op/s 2024-03-16T10:25:26.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:25 smithi193 ceph-mon[36311]: pgmap v601: 97 pgs: 97 active+clean; 410 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 760 op/s 2024-03-16T10:25:26.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:25 smithi192 ceph-mon[36354]: pgmap v601: 97 pgs: 97 active+clean; 410 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 760 op/s 2024-03-16T10:25:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:26 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:25:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:26 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:25:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:26 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:25:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:26 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:25:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:26 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:25:27.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:26 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:25:27.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:26 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:25:27.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:26 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:25:27.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:26 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:25:27.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:26 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:25:27.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:26 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:25:27.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:26 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:25:27.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:26 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:25:27.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:26 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:25:27.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:26 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:25:28.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:27 smithi077 ceph-mon[32400]: pgmap v602: 97 pgs: 97 active+clean; 403 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 655 op/s 2024-03-16T10:25:28.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:27 smithi193 ceph-mon[36311]: pgmap v602: 97 pgs: 97 active+clean; 403 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 655 op/s 2024-03-16T10:25:28.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:27 smithi192 ceph-mon[36354]: pgmap v602: 97 pgs: 97 active+clean; 403 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 655 op/s 2024-03-16T10:25:29.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:28 smithi077 ceph-mon[32400]: pgmap v603: 97 pgs: 97 active+clean; 395 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 625 op/s 2024-03-16T10:25:29.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:28 smithi193 ceph-mon[36311]: pgmap v603: 97 pgs: 97 active+clean; 395 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 625 op/s 2024-03-16T10:25:29.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:28 smithi192 ceph-mon[36354]: pgmap v603: 97 pgs: 97 active+clean; 395 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 625 op/s 2024-03-16T10:25:31.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:30 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:25:31.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:30 smithi077 ceph-mon[32400]: pgmap v604: 97 pgs: 97 active+clean; 389 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 484 op/s 2024-03-16T10:25:31.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:30 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:25:31.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:30 smithi193 ceph-mon[36311]: pgmap v604: 97 pgs: 97 active+clean; 389 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 484 op/s 2024-03-16T10:25:31.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:30 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:25:31.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:30 smithi192 ceph-mon[36354]: pgmap v604: 97 pgs: 97 active+clean; 389 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 484 op/s 2024-03-16T10:25:34.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:33 smithi077 ceph-mon[32400]: pgmap v605: 97 pgs: 97 active+clean; 361 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 485 op/s 2024-03-16T10:25:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:33 smithi193 ceph-mon[36311]: pgmap v605: 97 pgs: 97 active+clean; 361 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 485 op/s 2024-03-16T10:25:34.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:33 smithi192 ceph-mon[36354]: pgmap v605: 97 pgs: 97 active+clean; 361 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 485 op/s 2024-03-16T10:25:35.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:34 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:25:35.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:34 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:25:35.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:34 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:25:36.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:35 smithi077 ceph-mon[32400]: pgmap v606: 97 pgs: 97 active+clean; 349 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 427 op/s 2024-03-16T10:25:36.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:25:36.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:25:36.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:25:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:35 smithi193 ceph-mon[36311]: pgmap v606: 97 pgs: 97 active+clean; 349 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 427 op/s 2024-03-16T10:25:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:25:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:25:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:25:36.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:35 smithi192 ceph-mon[36354]: pgmap v606: 97 pgs: 97 active+clean; 349 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 427 op/s 2024-03-16T10:25:36.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:25:36.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:25:36.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:25:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.779+0000 7f7f9ef90640 1 -- 172.21.15.77:0/3411293240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f981022b0 msgr2=0x7f7f98108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.779+0000 7f7f9ef90640 1 --2- 172.21.15.77:0/3411293240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f981022b0 0x7f7f98108b40 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7f90009ca0 tx=0x7f7f9002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.779+0000 7f7f9ef90640 1 -- 172.21.15.77:0/3411293240 shutdown_connections 2024-03-16T10:25:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.779+0000 7f7f9ef90640 1 --2- 172.21.15.77:0/3411293240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f981022b0 0x7f7f98108b40 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.779+0000 7f7f9ef90640 1 --2- 172.21.15.77:0/3411293240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f981018f0 0x7f7f98101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.779+0000 7f7f9ef90640 1 --2- 172.21.15.77:0/3411293240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f98100700 0x7f7f98100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.779+0000 7f7f9ef90640 1 -- 172.21.15.77:0/3411293240 >> 172.21.15.77:0/3411293240 conn(0x7f7f980fbc70 msgr2=0x7f7f980fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.779+0000 7f7f9ef90640 1 -- 172.21.15.77:0/3411293240 shutdown_connections 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.779+0000 7f7f9ef90640 1 -- 172.21.15.77:0/3411293240 wait complete. 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9ef90640 1 Processor -- start 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9ef90640 1 -- start start 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9ef90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f98100700 0x7f7f98197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9ef90640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f981018f0 0x7f7f98197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9ef90640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f981022b0 0x7f7f9819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9ef90640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9810c380 con 0x7f7f98100700 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9ef90640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9810c200 con 0x7f7f981022b0 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9ef90640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9810c500 con 0x7f7f981018f0 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9cd2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f98100700 0x7f7f98197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9cd2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f98100700 0x7f7f98197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60190/0 (socket says 172.21.15.77:60190) 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9cd2c640 1 -- 172.21.15.77:0/955894035 learned_addr learned my addr 172.21.15.77:0/955894035 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f97fff640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f981018f0 0x7f7f98197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9d52d640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f981022b0 0x7f7f9819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9cd2c640 1 -- 172.21.15.77:0/955894035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f981018f0 msgr2=0x7f7f98197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9cd2c640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f981018f0 0x7f7f98197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9cd2c640 1 -- 172.21.15.77:0/955894035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f981022b0 msgr2=0x7f7f9819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9cd2c640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f981022b0 0x7f7f9819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.780+0000 7f7f9cd2c640 1 -- 172.21.15.77:0/955894035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9819f620 con 0x7f7f98100700 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.781+0000 7f7f9cd2c640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f98100700 0x7f7f98197a00 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00e9c0 tx=0x7f7f8c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.781+0000 7f7f95ffb640 1 -- 172.21.15.77:0/955894035 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c00cd80 con 0x7f7f98100700 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.781+0000 7f7f95ffb640 1 -- 172.21.15.77:0/955894035 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f8c0035f0 con 0x7f7f98100700 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.781+0000 7f7f95ffb640 1 -- 172.21.15.77:0/955894035 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c010690 con 0x7f7f98100700 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.781+0000 7f7f9ef90640 1 -- 172.21.15.77:0/955894035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f9819f910 con 0x7f7f98100700 2024-03-16T10:25:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.781+0000 7f7f9ef90640 1 -- 172.21.15.77:0/955894035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f9819fe10 con 0x7f7f98100700 2024-03-16T10:25:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7f95ffb640 1 -- 172.21.15.77:0/955894035 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f7f8c0037a0 con 0x7f7f98100700 2024-03-16T10:25:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7fa1b7d740 1 Processor -- start 2024-03-16T10:25:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7fa1b7d740 1 -- start start 2024-03-16T10:25:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7f95ffb640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7f880a28a0 0x7f7f880a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7fa1b7d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34340 0x55b8acb34760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7fa1b7d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb34d30 0x55b8acb39830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7fa1b7d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb39ea0 0x55b8acb3c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7fa1b7d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b8aca99a40 con 0x55b8acb39ea0 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7fa1b7d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55b8aca42920 con 0x55b8acb34d30 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7fa1b7d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b8aca95ae0 con 0x55b8acb34340 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7f9d52d640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7f880a28a0 0x7f7f880a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7f9cd2c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34340 0x55b8acb34760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7f97fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb34d30 0x55b8acb39830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7f97fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb34d30 0x55b8acb39830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:41266/0 (socket says 172.21.15.77:41266) 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7f97fff640 1 -- 172.21.15.77:0/4029663177 learned_addr learned my addr 172.21.15.77:0/4029663177 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7f9cd2c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34340 0x55b8acb34760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:37790/0 (socket says 172.21.15.77:37790) 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7f9d52d640 1 --2- 172.21.15.77:0/4029663177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb39ea0 0x55b8acb3c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.784+0000 7f7f95ffb640 1 -- 172.21.15.77:0/955894035 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f7f8c0cedd0 con 0x7f7f98100700 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.785+0000 7f7f97fff640 1 -- 172.21.15.77:0/4029663177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34340 msgr2=0x55b8acb34760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.785+0000 7f7f97fff640 1 --2- 172.21.15.77:0/4029663177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34340 0x55b8acb34760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.785+0000 7f7f97fff640 1 -- 172.21.15.77:0/4029663177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb39ea0 msgr2=0x55b8acb3c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.785+0000 7f7f97fff640 1 --2- 172.21.15.77:0/4029663177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb39ea0 0x55b8acb3c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.785+0000 7f7f97fff640 1 -- 172.21.15.77:0/4029663177 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8aca5a460 con 0x55b8acb34d30 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.785+0000 7f7f9d52d640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7f880a28a0 0x7f7f880a4d60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7f90005e30 tx=0x7f7f90009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.785+0000 7f7f97fff640 1 --2- 172.21.15.77:0/4029663177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb34d30 0x55b8acb39830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009c90 tx=0x7f7f84011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d165532960e7bb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.785+0000 7f7f82ffd640 1 -- 172.21.15.77:0/4029663177 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8401e070 con 0x55b8acb34d30 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.785+0000 7f7f82ffd640 1 -- 172.21.15.77:0/4029663177 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f84019b10 con 0x55b8acb34d30 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.785+0000 7f7f82ffd640 1 -- 172.21.15.77:0/4029663177 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8400e9e0 con 0x55b8acb34d30 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.786+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/4029663177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb34d30 msgr2=0x55b8acb39830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.786+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/4029663177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb34d30 0x55b8acb39830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009c90 tx=0x7f7f84011810 comp rx=0 tx=0).stop 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.786+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/4029663177 shutdown_connections 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.786+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/4029663177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb39ea0 0x55b8acb3c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.786+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/4029663177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb34d30 0x55b8acb39830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.786+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/4029663177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34340 0x55b8acb34760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.786+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/4029663177 >> 172.21.15.77:0/4029663177 conn(0x55b8acb2f8b0 msgr2=0x55b8acb2fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.786+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/4029663177 shutdown_connections 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.786+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/4029663177 wait complete. 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.786+0000 7f7fa1b7d740 1 -- start start 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.786+0000 7f7fa1b7d740 1 Processor -- start 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7fa1b7d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 0x55b8acb3e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7fa1b7d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb34d30 0x55b8acb3eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7fa1b7d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb39ea0 0x55b8acb45bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7fa1b7d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b8acb3ce50 con 0x55b8acb34340 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7fa1b7d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55b8aca55d70 con 0x55b8acb34d30 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7fa1b7d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b8acb3cfd0 con 0x55b8acb39ea0 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7fa1b7d740 10 client.0 _ll_register_callbacks cb 0x55b8ac7ada00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f9cd2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 0x55b8acb3e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f97fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb34d30 0x55b8acb3eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f9cd2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 0x55b8acb3e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60214/0 (socket says 172.21.15.77:60214) 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f9cd2c640 1 -- 172.21.15.77:0/1847837476 learned_addr learned my addr 172.21.15.77:0/1847837476 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f9d52d640 1 --2- 172.21.15.77:0/1847837476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb39ea0 0x55b8acb45bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f9cd2c640 1 -- 172.21.15.77:0/1847837476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb39ea0 msgr2=0x55b8acb45bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f9cd2c640 1 --2- 172.21.15.77:0/1847837476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb39ea0 0x55b8acb45bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f9cd2c640 1 -- 172.21.15.77:0/1847837476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb34d30 msgr2=0x55b8acb3eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f9cd2c640 1 --2- 172.21.15.77:0/1847837476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb34d30 0x55b8acb3eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f9cd2c640 1 -- 172.21.15.77:0/1847837476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8ac7ae210 con 0x55b8acb34340 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f9cd2c640 1 --2- 172.21.15.77:0/1847837476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 0x55b8acb3e640 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c003f50 tx=0x7f7f8c09b360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e157fc9b2c57ba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f77fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f77fff640 1 -- 172.21.15.77:0/1847837476 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c0b4060 con 0x55b8acb34340 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f77fff640 1 -- 172.21.15.77:0/1847837476 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f7f8c0cc9d0 con 0x55b8acb34340 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/1847837476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f84019b10 con 0x55b8acb34340 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7f77fff640 1 -- 172.21.15.77:0/1847837476 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f7f8c09b760 con 0x55b8acb34340 2024-03-16T10:25:36.795 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7fa1b7d740 10 client.17529 fetch_fsmap learned FSMap version 46 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.787+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/1847837476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55b8ac87a620 con 0x55b8acb34340 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.788+0000 7f7f77fff640 1 -- 172.21.15.77:0/1847837476 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f84019b10 con 0x55b8acb34340 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.788+0000 7f7f77fff640 1 -- 172.21.15.77:0/1847837476 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 46) ==== 8382+0+0 (secure 0 0 0) 0x7f7f8c0ccce0 con 0x55b8acb34340 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.788+0000 7f7fa1b7d740 10 client.17529 fetch_fsmap finished waiting for FSMap version 46 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.788+0000 7f7fa1b7d740 10 client.17529 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.788+0000 7f7fa1b7d740 20 client.17529 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.788+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/1847837476 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b8acb4a570 0x55b8acb4c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.788+0000 7f7fa1b7d740 4 client.17529 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.788+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/1847837476 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8acaa6f80 con 0x55b8acb4a570 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.788+0000 7f7f97fff640 1 --2- 172.21.15.77:0/1847837476 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b8acb4a570 0x55b8acb4c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.789+0000 7f7f77fff640 10 client.17529 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.789+0000 7f7f97fff640 1 --2- 172.21.15.77:0/1847837476 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b8acb4a570 0x55b8acb4c9a0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e23ad8cb8211b64 server_cookie=ab911e715a899e7c in_seq=0 out_seq=0 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.789+0000 7f7f77fff640 1 -- 172.21.15.77:0/1847837476 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b8acaa6f80 con 0x55b8acb4a570 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.789+0000 7f7f77fff640 10 client.17529 handle_command_reply: tid=0 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.789+0000 7f7fa1b7d740 1 client.17529 shutdown 2024-03-16T10:25:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.790+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/1847837476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 msgr2=0x55b8acb3e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.790+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/1847837476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 0x55b8acb3e640 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c003f50 tx=0x7f7f8c09b360 comp rx=0 tx=0).stop 2024-03-16T10:25:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.790+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/1847837476 shutdown_connections 2024-03-16T10:25:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.790+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/1847837476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb39ea0 0x55b8acb45bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.790+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/1847837476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb34d30 0x55b8acb3eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.790+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/1847837476 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b8acb4a570 0x55b8acb4c9a0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.790+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/1847837476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 0x55b8acb3e640 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.790+0000 7f7f77fff640 0 client.17529 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.790+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/1847837476 >> 172.21.15.77:0/1847837476 conn(0x55b8acb2f8b0 msgr2=0x55b8acb2ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.790+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/1847837476 shutdown_connections 2024-03-16T10:25:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.790+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/1847837476 wait complete. 2024-03-16T10:25:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.790+0000 7f7fa1b7d740 20 client.17529 trim_cache size 0 max 16384 2024-03-16T10:25:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.805+0000 7f7fa1b7d740 1 Processor -- start 2024-03-16T10:25:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.805+0000 7f7fa1b7d740 1 -- start start 2024-03-16T10:25:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7fa1b7d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb2f8b0 0x55b8acb3e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7fa1b7d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 0x55b8acb45b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7fa1b7d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34d30 0x55b8acb512a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7fa1b7d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7f981a8a80 con 0x55b8acb34340 2024-03-16T10:25:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7fa1b7d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7f981aa680 con 0x55b8acb2f8b0 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7fa1b7d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7f981a91b0 con 0x55b8acb34d30 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 0x55b8acb45b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 0x55b8acb45b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60228/0 (socket says 172.21.15.77:60228) 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f97fff640 1 -- 172.21.15.77:0/2421931477 learned_addr learned my addr 172.21.15.77:0/2421931477 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f9cd2c640 1 --2- 172.21.15.77:0/2421931477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb2f8b0 0x55b8acb3e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f9d52d640 1 --2- 172.21.15.77:0/2421931477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34d30 0x55b8acb512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f97fff640 1 -- 172.21.15.77:0/2421931477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34d30 msgr2=0x55b8acb512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f97fff640 1 --2- 172.21.15.77:0/2421931477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34d30 0x55b8acb512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f97fff640 1 -- 172.21.15.77:0/2421931477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb2f8b0 msgr2=0x55b8acb3e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f97fff640 1 --2- 172.21.15.77:0/2421931477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb2f8b0 0x55b8acb3e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f97fff640 1 -- 172.21.15.77:0/2421931477 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8acb30620 con 0x55b8acb34340 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f97fff640 1 --2- 172.21.15.77:0/2421931477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 0x55b8acb45b80 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009250 tx=0x7f7f84009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c63f7465c95e74 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f827fc640 1 -- 172.21.15.77:0/2421931477 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8401e040 con 0x55b8acb34340 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f827fc640 1 -- 172.21.15.77:0/2421931477 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f84005040 con 0x55b8acb34340 2024-03-16T10:25:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.806+0000 7f7f827fc640 1 -- 172.21.15.77:0/2421931477 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f84019690 con 0x55b8acb34340 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.807+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/2421931477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 msgr2=0x55b8acb45b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.807+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/2421931477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 0x55b8acb45b80 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009250 tx=0x7f7f84009850 comp rx=0 tx=0).stop 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.807+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/2421931477 shutdown_connections 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.807+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/2421931477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34d30 0x55b8acb512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.807+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/2421931477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34340 0x55b8acb45b80 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.807+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/2421931477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb2f8b0 0x55b8acb3e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.807+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/2421931477 >> 172.21.15.77:0/2421931477 conn(0x55b8acb39970 msgr2=0x55b8acb3e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.807+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/2421931477 shutdown_connections 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.807+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/2421931477 wait complete. 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.807+0000 7f7fa1b7d740 1 -- start start 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.807+0000 7f7fa1b7d740 1 Processor -- start 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7fa1b7d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb2f8b0 0x55b8acb439c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7fa1b7d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34340 0x55b8acb43f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7fa1b7d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34d30 0x55b8acb44e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7fa1b7d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7f981a9fb0 con 0x55b8acb34d30 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7fa1b7d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7f981a98b0 con 0x55b8acb2f8b0 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7fa1b7d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7f981a6a60 con 0x55b8acb34340 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7fa1b7d740 10 client.0 _ll_register_callbacks cb 0x7f7f8c0096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f9d52d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34d30 0x55b8acb44e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f9cd2c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb2f8b0 0x55b8acb439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f9d52d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34d30 0x55b8acb44e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60244/0 (socket says 172.21.15.77:60244) 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f9d52d640 1 -- 172.21.15.77:0/3605190985 learned_addr learned my addr 172.21.15.77:0/3605190985 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f97fff640 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34340 0x55b8acb43f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f9d52d640 1 -- 172.21.15.77:0/3605190985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34340 msgr2=0x55b8acb43f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f9d52d640 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34340 0x55b8acb43f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f9d52d640 1 -- 172.21.15.77:0/3605190985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb2f8b0 msgr2=0x55b8acb439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f9d52d640 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb2f8b0 0x55b8acb439c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f9d52d640 1 -- 172.21.15.77:0/3605190985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8acb45440 con 0x55b8acb34d30 2024-03-16T10:25:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f9cd2c640 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb2f8b0 0x55b8acb439c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f9d52d640 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34d30 0x55b8acb44e50 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7f900324c0 tx=0x7f7f90032590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbb04698b555b0b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.808+0000 7f7f77fff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7f77fff640 1 -- 172.21.15.77:0/3605190985 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90052020 con 0x55b8acb34d30 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7f77fff640 1 -- 172.21.15.77:0/3605190985 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f7f90003e00 con 0x55b8acb34d30 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/3605190985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f84005040 con 0x55b8acb34d30 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7f77fff640 1 -- 172.21.15.77:0/3605190985 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f7f90039040 con 0x55b8acb34d30 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7fa1b7d740 10 client.17535 fetch_fsmap learned FSMap version 46 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/3605190985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55b8acbb0f80 con 0x55b8acb34d30 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7f77fff640 1 -- 172.21.15.77:0/3605190985 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f84005040 con 0x55b8acb34d30 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7f77fff640 1 -- 172.21.15.77:0/3605190985 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 46) ==== 8382+0+0 (secure 0 0 0) 0x7f7f9003f040 con 0x55b8acb34d30 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7fa1b7d740 10 client.17535 fetch_fsmap finished waiting for FSMap version 46 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7fa1b7d740 10 client.17535 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7fa1b7d740 20 client.17535 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b8acb39ea0 0x55b8acbb3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7fa1b7d740 4 client.17535 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:25:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.809+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/3605190985 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b8acaa12d0 con 0x55b8acb39ea0 2024-03-16T10:25:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.810+0000 7f7f9cd2c640 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b8acb39ea0 0x55b8acbb3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.810+0000 7f7f77fff640 10 client.17535 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.810+0000 7f7f9cd2c640 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b8acb39ea0 0x55b8acbb3a70 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e92945058467ef8 server_cookie=61b4b99bb93aec9 in_seq=0 out_seq=0 2024-03-16T10:25:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.817+0000 7f7f77fff640 1 -- 172.21.15.77:0/3605190985 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b8acaa12d0 con 0x55b8acb39ea0 2024-03-16T10:25:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.817+0000 7f7f77fff640 10 client.17535 handle_command_reply: tid=0 2024-03-16T10:25:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.817+0000 7f7fa1b7d740 1 client.17535 shutdown 2024-03-16T10:25:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/3605190985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34d30 msgr2=0x55b8acb44e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34d30 0x55b8acb44e50 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7f900324c0 tx=0x7f7f90032590 comp rx=0 tx=0).stop 2024-03-16T10:25:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/3605190985 shutdown_connections 2024-03-16T10:25:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55b8acb39ea0 0x55b8acbb3a70 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8acb34d30 0x55b8acb44e50 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b8acb34340 0x55b8acb43f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7fa1b7d740 1 --2- 172.21.15.77:0/3605190985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b8acb2f8b0 0x55b8acb439c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7f77fff640 0 client.17535 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/3605190985 >> 172.21.15.77:0/3605190985 conn(0x55b8acb39970 msgr2=0x55b8acb2e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/3605190985 shutdown_connections 2024-03-16T10:25:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7fa1b7d740 1 -- 172.21.15.77:0/3605190985 wait complete. 2024-03-16T10:25:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7fa1b7d740 20 client.17535 trim_cache size 0 max 16384 2024-03-16T10:25:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:25:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:25:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:25:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:25:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7f9ef90640 1 -- 172.21.15.77:0/955894035 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7f880a28a0 msgr2=0x7f7f880a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7f9ef90640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7f880a28a0 0x7f7f880a4d60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7f90005e30 tx=0x7f7f90009800 comp rx=0 tx=0).stop 2024-03-16T10:25:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7f9ef90640 1 -- 172.21.15.77:0/955894035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f98100700 msgr2=0x7f7f98197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7f9ef90640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f98100700 0x7f7f98197a00 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00e9c0 tx=0x7f7f8c00ee80 comp rx=0 tx=0).stop 2024-03-16T10:25:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7f9ef90640 1 -- 172.21.15.77:0/955894035 shutdown_connections 2024-03-16T10:25:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7f9ef90640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f981022b0 0x7f7f9819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7f9ef90640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f7f880a28a0 0x7f7f880a4d60 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7f9ef90640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f981018f0 0x7f7f98197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7f9ef90640 1 --2- 172.21.15.77:0/955894035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f98100700 0x7f7f98197a00 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.818+0000 7f7f9ef90640 1 -- 172.21.15.77:0/955894035 >> 172.21.15.77:0/955894035 conn(0x7f7f980fbc70 msgr2=0x7f7f980fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.819+0000 7f7f9ef90640 1 -- 172.21.15.77:0/955894035 shutdown_connections 2024-03-16T10:25:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:36.819+0000 7f7f9ef90640 1 -- 172.21.15.77:0/955894035 wait complete. 2024-03-16T10:25:38.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:37 smithi077 ceph-mon[32400]: pgmap v607: 97 pgs: 97 active+clean; 345 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 425 op/s 2024-03-16T10:25:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:37 smithi193 ceph-mon[36311]: pgmap v607: 97 pgs: 97 active+clean; 345 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 425 op/s 2024-03-16T10:25:38.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:37 smithi192 ceph-mon[36354]: pgmap v607: 97 pgs: 97 active+clean; 345 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 425 op/s 2024-03-16T10:25:40.218 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:39 smithi192 ceph-mon[36354]: pgmap v608: 97 pgs: 97 active+clean; 331 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 467 op/s 2024-03-16T10:25:40.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:39 smithi077 ceph-mon[32400]: pgmap v608: 97 pgs: 97 active+clean; 331 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 467 op/s 2024-03-16T10:25:40.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:39 smithi193 ceph-mon[36311]: pgmap v608: 97 pgs: 97 active+clean; 331 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 467 op/s 2024-03-16T10:25:41.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:40 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:25:41.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:40 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:25:41.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:40 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:25:42.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:41 smithi077 ceph-mon[32400]: pgmap v609: 97 pgs: 97 active+clean; 316 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 422 op/s 2024-03-16T10:25:42.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:41 smithi193 ceph-mon[36311]: pgmap v609: 97 pgs: 97 active+clean; 316 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 422 op/s 2024-03-16T10:25:42.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:41 smithi192 ceph-mon[36354]: pgmap v609: 97 pgs: 97 active+clean; 316 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 422 op/s 2024-03-16T10:25:43.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:42 smithi077 ceph-mon[32400]: pgmap v610: 97 pgs: 97 active+clean; 308 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 429 op/s 2024-03-16T10:25:43.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:42 smithi193 ceph-mon[36311]: pgmap v610: 97 pgs: 97 active+clean; 308 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 429 op/s 2024-03-16T10:25:43.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:42 smithi192 ceph-mon[36354]: pgmap v610: 97 pgs: 97 active+clean; 308 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 429 op/s 2024-03-16T10:25:46.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:45 smithi077 ceph-mon[32400]: pgmap v611: 97 pgs: 97 active+clean; 303 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 336 op/s 2024-03-16T10:25:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:45 smithi193 ceph-mon[36311]: pgmap v611: 97 pgs: 97 active+clean; 303 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 336 op/s 2024-03-16T10:25:46.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:45 smithi192 ceph-mon[36354]: pgmap v611: 97 pgs: 97 active+clean; 303 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 336 op/s 2024-03-16T10:25:48.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:47 smithi077 ceph-mon[32400]: pgmap v612: 97 pgs: 97 active+clean; 296 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 359 op/s 2024-03-16T10:25:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:47 smithi193 ceph-mon[36311]: pgmap v612: 97 pgs: 97 active+clean; 296 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 359 op/s 2024-03-16T10:25:48.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:47 smithi192 ceph-mon[36354]: pgmap v612: 97 pgs: 97 active+clean; 296 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 359 op/s 2024-03-16T10:25:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.909+0000 7f299490a640 1 -- 172.21.15.77:0/2648245483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29901018f0 msgr2=0x7f2990101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.909+0000 7f299490a640 1 --2- 172.21.15.77:0/2648245483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29901018f0 0x7f2990101d70 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f2980009900 tx=0x7f298002bc30 comp rx=0 tx=0).stop 2024-03-16T10:25:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.909+0000 7f299490a640 1 -- 172.21.15.77:0/2648245483 shutdown_connections 2024-03-16T10:25:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.909+0000 7f299490a640 1 --2- 172.21.15.77:0/2648245483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29901022b0 0x7f2990108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.909+0000 7f299490a640 1 --2- 172.21.15.77:0/2648245483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29901018f0 0x7f2990101d70 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.909+0000 7f299490a640 1 --2- 172.21.15.77:0/2648245483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2990100700 0x7f2990100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.909+0000 7f299490a640 1 -- 172.21.15.77:0/2648245483 >> 172.21.15.77:0/2648245483 conn(0x7f29900fbc70 msgr2=0x7f29900fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.909+0000 7f299490a640 1 -- 172.21.15.77:0/2648245483 shutdown_connections 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.909+0000 7f299490a640 1 -- 172.21.15.77:0/2648245483 wait complete. 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f299490a640 1 Processor -- start 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f299490a640 1 -- start start 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f299490a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2990100700 0x7f2990196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f299490a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29901018f0 0x7f2990196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f299490a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29901022b0 0x7f299019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f299490a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f299010c360 con 0x7f29901022b0 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f299490a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f299010c1e0 con 0x7f2990100700 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f299490a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f299010c4e0 con 0x7f29901018f0 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f298dd9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29901018f0 0x7f2990196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f298ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29901022b0 0x7f299019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f298e59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2990100700 0x7f2990196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f298dd9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29901018f0 0x7f2990196e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:43230/0 (socket says 172.21.15.77:43230) 2024-03-16T10:25:49.913 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f298e59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2990100700 0x7f2990196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:53324/0 (socket says 172.21.15.77:53324) 2024-03-16T10:25:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f298dd9b640 1 -- 172.21.15.77:0/229192633 learned_addr learned my addr 172.21.15.77:0/229192633 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f298ed9d640 1 -- 172.21.15.77:0/229192633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29901018f0 msgr2=0x7f2990196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f298ed9d640 1 --2- 172.21.15.77:0/229192633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29901018f0 0x7f2990196e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f298ed9d640 1 -- 172.21.15.77:0/229192633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2990100700 msgr2=0x7f2990196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f298ed9d640 1 --2- 172.21.15.77:0/229192633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2990100700 0x7f2990196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.910+0000 7f298ed9d640 1 -- 172.21.15.77:0/229192633 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299019e560 con 0x7f29901022b0 2024-03-16T10:25:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.911+0000 7f298ed9d640 1 --2- 172.21.15.77:0/229192633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29901022b0 0x7f299019def0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f2988009fd0 tx=0x7f298800b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.911+0000 7f297f7fe640 1 -- 172.21.15.77:0/229192633 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298800be40 con 0x7f29901022b0 2024-03-16T10:25:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.911+0000 7f297f7fe640 1 -- 172.21.15.77:0/229192633 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2988017450 con 0x7f29901022b0 2024-03-16T10:25:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.911+0000 7f299490a640 1 -- 172.21.15.77:0/229192633 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2990068ec0 con 0x7f29901022b0 2024-03-16T10:25:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.911+0000 7f297f7fe640 1 -- 172.21.15.77:0/229192633 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988005280 con 0x7f29901022b0 2024-03-16T10:25:49.914 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.911+0000 7f299490a640 1 -- 172.21.15.77:0/229192633 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2990069390 con 0x7f29901022b0 2024-03-16T10:25:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.913+0000 7f29975cd740 1 Processor -- start 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f297f7fe640 1 -- 172.21.15.77:0/229192633 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f2988004a40 con 0x7f29901022b0 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f29975cd740 1 -- start start 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f29975cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e3682fd340 0x55e3682fd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f29975cd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368302830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f29975cd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e368302ea0 0x55e368305330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f29975cd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e368262a40 con 0x55e3682fdd30 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f29975cd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55e36820b920 con 0x55e368302ea0 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f29975cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e36825eae0 con 0x55e3682fd340 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f298e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368302830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f298e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368302830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60050/0 (socket says 172.21.15.77:60050) 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f298e59c640 1 -- 172.21.15.77:0/1336430968 learned_addr learned my addr 172.21.15.77:0/1336430968 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f298dd9b640 1 --2- 172.21.15.77:0/1336430968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e3682fd340 0x55e3682fd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f298dd9b640 1 --2- 172.21.15.77:0/1336430968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e368302ea0 0x55e368305330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f297f7fe640 1 --2- 172.21.15.77:0/229192633 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f29780a28a0 0x7f29780a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f298e59c640 1 -- 172.21.15.77:0/1336430968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e3682fd340 msgr2=0x55e3682fd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f298e59c640 1 --2- 172.21.15.77:0/1336430968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e3682fd340 0x55e3682fd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f298e59c640 1 -- 172.21.15.77:0/1336430968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e368302ea0 msgr2=0x55e368305330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f298e59c640 1 --2- 172.21.15.77:0/1336430968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e368302ea0 0x55e368305330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.914+0000 7f298e59c640 1 -- 172.21.15.77:0/1336430968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e368223460 con 0x55e3682fdd30 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.915+0000 7f298ed9d640 1 --2- 172.21.15.77:0/229192633 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f29780a28a0 0x7f29780a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.915+0000 7f297f7fe640 1 -- 172.21.15.77:0/229192633 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f29880d05b0 con 0x7f29901022b0 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.915+0000 7f298e59c640 1 --2- 172.21.15.77:0/1336430968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368302830 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f2984009ca0 tx=0x7f2984011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf80470a9f724dde server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.915+0000 7f297cff9640 1 -- 172.21.15.77:0/1336430968 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984009e50 con 0x55e3682fdd30 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.915+0000 7f297cff9640 1 -- 172.21.15.77:0/1336430968 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29840035f0 con 0x55e3682fdd30 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.915+0000 7f297cff9640 1 -- 172.21.15.77:0/1336430968 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984019670 con 0x55e3682fdd30 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f298ed9d640 1 --2- 172.21.15.77:0/229192633 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f29780a28a0 0x7f29780a4d60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f29880cd6f0 tx=0x7f2988067e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 -- 172.21.15.77:0/1336430968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 msgr2=0x55e368302830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 --2- 172.21.15.77:0/1336430968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368302830 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f2984009ca0 tx=0x7f2984011810 comp rx=0 tx=0).stop 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 -- 172.21.15.77:0/1336430968 shutdown_connections 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 --2- 172.21.15.77:0/1336430968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e368302ea0 0x55e368305330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 --2- 172.21.15.77:0/1336430968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368302830 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 --2- 172.21.15.77:0/1336430968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e3682fd340 0x55e3682fd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 -- 172.21.15.77:0/1336430968 >> 172.21.15.77:0/1336430968 conn(0x55e3682f88b0 msgr2=0x55e3682f8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 -- 172.21.15.77:0/1336430968 shutdown_connections 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 -- 172.21.15.77:0/1336430968 wait complete. 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 -- start start 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 Processor -- start 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e3682fd340 0x55e36830ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fdd30 0x55e36830bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e368302ea0 0x55e368312f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e368305ea0 con 0x55e368302ea0 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55e36821ed70 con 0x55e3682fdd30 2024-03-16T10:25:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.916+0000 7f29975cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e368306020 con 0x55e3682fd340 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f29975cd740 10 client.0 _ll_register_callbacks cb 0x55e367f76a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f298dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e368302ea0 0x55e368312f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f298dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e368302ea0 0x55e368312f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60058/0 (socket says 172.21.15.77:60058) 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f298dd9b640 1 -- 172.21.15.77:0/2982926792 learned_addr learned my addr 172.21.15.77:0/2982926792 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f298e59c640 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fdd30 0x55e36830bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f298dd9b640 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e3682fd340 0x55e36830ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f298dd9b640 1 -- 172.21.15.77:0/2982926792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e3682fd340 msgr2=0x55e36830ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f298dd9b640 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e3682fd340 0x55e36830ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f298dd9b640 1 -- 172.21.15.77:0/2982926792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fdd30 msgr2=0x55e36830bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f298dd9b640 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fdd30 0x55e36830bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f298dd9b640 1 -- 172.21.15.77:0/2982926792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e367f77210 con 0x55e368302ea0 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f298dd9b640 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e368302ea0 0x55e368312f90 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2980005c40 tx=0x7f2980009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36f3c2029ee7615 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f2975ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f2975ffb640 1 -- 172.21.15.77:0/2982926792 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980004260 con 0x55e368302ea0 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f2975ffb640 1 -- 172.21.15.77:0/2982926792 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f2980004400 con 0x55e368302ea0 2024-03-16T10:25:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f29975cd740 1 -- 172.21.15.77:0/2982926792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29840035f0 con 0x55e368302ea0 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f2975ffb640 1 -- 172.21.15.77:0/2982926792 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f2980056050 con 0x55e368302ea0 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f29975cd740 10 client.17547 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f29975cd740 1 -- 172.21.15.77:0/2982926792 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55e368043620 con 0x55e368302ea0 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.917+0000 7f2975ffb640 1 -- 172.21.15.77:0/2982926792 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29840035f0 con 0x55e368302ea0 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.918+0000 7f2975ffb640 1 -- 172.21.15.77:0/2982926792 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f298004d070 con 0x55e368302ea0 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.918+0000 7f29975cd740 10 client.17547 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.918+0000 7f29975cd740 10 client.17547 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.918+0000 7f29975cd740 20 client.17547 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.918+0000 7f29975cd740 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e368317820 0x55e368319c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.918+0000 7f29975cd740 4 client.17547 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.918+0000 7f29975cd740 1 -- 172.21.15.77:0/2982926792 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e36826ff80 con 0x55e368317820 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.918+0000 7f298e59c640 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e368317820 0x55e368319c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.918+0000 7f2975ffb640 10 client.17547 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.918+0000 7f298e59c640 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e368317820 0x55e368319c90 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36810850c88e959f server_cookie=13a1c981f6e21b5e in_seq=0 out_seq=0 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.919+0000 7f2975ffb640 1 -- 172.21.15.77:0/2982926792 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e36826ff80 con 0x55e368317820 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.919+0000 7f2975ffb640 10 client.17547 handle_command_reply: tid=0 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.919+0000 7f29975cd740 1 client.17547 shutdown 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.919+0000 7f29975cd740 1 -- 172.21.15.77:0/2982926792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e368302ea0 msgr2=0x55e368312f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.919+0000 7f29975cd740 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e368302ea0 0x55e368312f90 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2980005c40 tx=0x7f2980009260 comp rx=0 tx=0).stop 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.919+0000 7f29975cd740 1 -- 172.21.15.77:0/2982926792 shutdown_connections 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.919+0000 7f29975cd740 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e368317820 0x55e368319c90 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.919+0000 7f29975cd740 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e368302ea0 0x55e368312f90 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.919+0000 7f2975ffb640 0 client.17547 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.919+0000 7f29975cd740 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fdd30 0x55e36830bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.919+0000 7f29975cd740 1 --2- 172.21.15.77:0/2982926792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e3682fd340 0x55e36830ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.920+0000 7f29975cd740 1 -- 172.21.15.77:0/2982926792 >> 172.21.15.77:0/2982926792 conn(0x55e3682f88b0 msgr2=0x55e3682f7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.920+0000 7f29975cd740 1 -- 172.21.15.77:0/2982926792 shutdown_connections 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.920+0000 7f29975cd740 1 -- 172.21.15.77:0/2982926792 wait complete. 2024-03-16T10:25:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.920+0000 7f29975cd740 20 client.17547 trim_cache size 0 max 16384 2024-03-16T10:25:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.937+0000 7f29975cd740 1 Processor -- start 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f29975cd740 1 -- start start 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f29975cd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fd340 0x55e3682fc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f29975cd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368312f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f29975cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e368302ea0 0x55e36831e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f29975cd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f29901a68a0 con 0x55e3682fdd30 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f29975cd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f29901a84a0 con 0x55e3682fd340 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f29975cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f29901a6fd0 con 0x55e368302ea0 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f298e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368312f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f298e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368312f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60072/0 (socket says 172.21.15.77:60072) 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f298e59c640 1 -- 172.21.15.77:0/2667488699 learned_addr learned my addr 172.21.15.77:0/2667488699 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f298dd9b640 1 --2- 172.21.15.77:0/2667488699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fd340 0x55e3682fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f298dd9b640 1 --2- 172.21.15.77:0/2667488699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e368302ea0 0x55e36831e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f298e59c640 1 -- 172.21.15.77:0/2667488699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e368302ea0 msgr2=0x55e36831e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f298e59c640 1 --2- 172.21.15.77:0/2667488699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e368302ea0 0x55e36831e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f298e59c640 1 -- 172.21.15.77:0/2667488699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fd340 msgr2=0x55e3682fc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f298e59c640 1 --2- 172.21.15.77:0/2667488699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fd340 0x55e3682fc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f298e59c640 1 -- 172.21.15.77:0/2667488699 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3682feb00 con 0x55e3682fdd30 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f298dd9b640 1 --2- 172.21.15.77:0/2667488699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fd340 0x55e3682fc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f298e59c640 1 --2- 172.21.15.77:0/2667488699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368312f50 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2984011d10 tx=0x7f2984005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebbf7910204ff294 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.938+0000 7f2977fff640 1 -- 172.21.15.77:0/2667488699 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984005230 con 0x55e3682fdd30 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f2977fff640 1 -- 172.21.15.77:0/2667488699 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29840053d0 con 0x55e3682fdd30 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f2977fff640 1 -- 172.21.15.77:0/2667488699 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984026510 con 0x55e3682fdd30 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f29975cd740 1 -- 172.21.15.77:0/2667488699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 msgr2=0x55e368312f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f29975cd740 1 --2- 172.21.15.77:0/2667488699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368312f50 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2984011d10 tx=0x7f2984005850 comp rx=0 tx=0).stop 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f29975cd740 1 -- 172.21.15.77:0/2667488699 shutdown_connections 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f29975cd740 1 --2- 172.21.15.77:0/2667488699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e368302ea0 0x55e36831e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f29975cd740 1 --2- 172.21.15.77:0/2667488699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368312f50 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f29975cd740 1 --2- 172.21.15.77:0/2667488699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fd340 0x55e3682fc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f29975cd740 1 -- 172.21.15.77:0/2667488699 >> 172.21.15.77:0/2667488699 conn(0x55e3682f88b0 msgr2=0x55e3682fbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f29975cd740 1 -- 172.21.15.77:0/2667488699 shutdown_connections 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f29975cd740 1 -- 172.21.15.77:0/2667488699 wait complete. 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f29975cd740 1 -- start start 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.939+0000 7f29975cd740 1 Processor -- start 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f29975cd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fd340 0x55e3683127d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f29975cd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368312d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f29975cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e368302ea0 0x55e368310de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f29975cd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f29901a7dd0 con 0x55e3682fdd30 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f29975cd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f29901a76d0 con 0x55e3682fd340 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f29975cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f29901a4880 con 0x55e368302ea0 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f29975cd740 10 client.0 _ll_register_callbacks cb 0x7f2980004b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f298e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368312d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f298e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368312d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60082/0 (socket says 172.21.15.77:60082) 2024-03-16T10:25:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f298e59c640 1 -- 172.21.15.77:0/3896957909 learned_addr learned my addr 172.21.15.77:0/3896957909 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f298dd9b640 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fd340 0x55e3683127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f298dd9b640 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e368302ea0 0x55e368310de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f298e59c640 1 -- 172.21.15.77:0/3896957909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e368302ea0 msgr2=0x55e368310de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f298e59c640 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e368302ea0 0x55e368310de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f298e59c640 1 -- 172.21.15.77:0/3896957909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fd340 msgr2=0x55e3683127d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f298e59c640 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fd340 0x55e3683127d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f298e59c640 1 -- 172.21.15.77:0/3896957909 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e368311450 con 0x55e3682fdd30 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f298dd9b640 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e368302ea0 0x55e368310de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f298e59c640 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368312d10 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2984009f40 tx=0x7f2984011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81e12613c1d577 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f2975ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f2975ffb640 1 -- 172.21.15.77:0/3896957909 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984037070 con 0x55e3682fdd30 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.940+0000 7f29975cd740 1 -- 172.21.15.77:0/3896957909 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29840053d0 con 0x55e3682fdd30 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.941+0000 7f2975ffb640 1 -- 172.21.15.77:0/3896957909 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f2984016040 con 0x55e3682fdd30 2024-03-16T10:25:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.941+0000 7f2975ffb640 1 -- 172.21.15.77:0/3896957909 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f29840035f0 con 0x55e3682fdd30 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.941+0000 7f29975cd740 10 client.17553 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.941+0000 7f29975cd740 1 -- 172.21.15.77:0/3896957909 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55e36837e540 con 0x55e3682fdd30 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.941+0000 7f2975ffb640 1 -- 172.21.15.77:0/3896957909 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29840053d0 con 0x55e3682fdd30 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.941+0000 7f2975ffb640 1 -- 172.21.15.77:0/3896957909 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f298403a030 con 0x55e3682fdd30 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.941+0000 7f29975cd740 10 client.17553 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.941+0000 7f29975cd740 10 client.17553 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.941+0000 7f29975cd740 20 client.17553 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.941+0000 7f29975cd740 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e36837edc0 0x55e3683811e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.941+0000 7f29975cd740 4 client.17553 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.941+0000 7f29975cd740 1 -- 172.21.15.77:0/3896957909 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e36826a2d0 con 0x55e36837edc0 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.942+0000 7f298dd9b640 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e36837edc0 0x55e3683811e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.942+0000 7f2975ffb640 10 client.17553 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.942+0000 7f298dd9b640 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e36837edc0 0x55e3683811e0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=945b63bda67730ff server_cookie=22a6809037eb8077 in_seq=0 out_seq=0 2024-03-16T10:25:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f2975ffb640 1 -- 172.21.15.77:0/3896957909 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e36826a2d0 con 0x55e36837edc0 2024-03-16T10:25:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f2975ffb640 10 client.17553 handle_command_reply: tid=0 2024-03-16T10:25:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f29975cd740 1 client.17553 shutdown 2024-03-16T10:25:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f29975cd740 1 -- 172.21.15.77:0/3896957909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 msgr2=0x55e368312d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f29975cd740 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368312d10 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2984009f40 tx=0x7f2984011db0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f29975cd740 1 -- 172.21.15.77:0/3896957909 shutdown_connections 2024-03-16T10:25:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f29975cd740 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e368302ea0 0x55e368310de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f29975cd740 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e36837edc0 0x55e3683811e0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f29975cd740 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3682fdd30 0x55e368312d10 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f29975cd740 1 --2- 172.21.15.77:0/3896957909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3682fd340 0x55e3683127d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f2975ffb640 0 client.17553 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f29975cd740 1 -- 172.21.15.77:0/3896957909 >> 172.21.15.77:0/3896957909 conn(0x55e3682f88b0 msgr2=0x55e3682fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f29975cd740 1 -- 172.21.15.77:0/3896957909 shutdown_connections 2024-03-16T10:25:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f29975cd740 1 -- 172.21.15.77:0/3896957909 wait complete. 2024-03-16T10:25:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.950+0000 7f29975cd740 20 client.17553 trim_cache size 0 max 16384 2024-03-16T10:25:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:{ 2024-03-16T10:25:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "message": "", 2024-03-16T10:25:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout: "return_code": 0 2024-03-16T10:25:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stdout:} 2024-03-16T10:25:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.951+0000 7f299490a640 1 -- 172.21.15.77:0/229192633 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f29780a28a0 msgr2=0x7f29780a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.951+0000 7f299490a640 1 --2- 172.21.15.77:0/229192633 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f29780a28a0 0x7f29780a4d60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f29880cd6f0 tx=0x7f2988067e50 comp rx=0 tx=0).stop 2024-03-16T10:25:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.951+0000 7f299490a640 1 -- 172.21.15.77:0/229192633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29901022b0 msgr2=0x7f299019def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.951+0000 7f299490a640 1 --2- 172.21.15.77:0/229192633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29901022b0 0x7f299019def0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f2988009fd0 tx=0x7f298800b690 comp rx=0 tx=0).stop 2024-03-16T10:25:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.951+0000 7f299490a640 1 -- 172.21.15.77:0/229192633 shutdown_connections 2024-03-16T10:25:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.951+0000 7f299490a640 1 --2- 172.21.15.77:0/229192633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29901022b0 0x7f299019def0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.951+0000 7f299490a640 1 --2- 172.21.15.77:0/229192633 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f29780a28a0 0x7f29780a4d60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.951+0000 7f299490a640 1 --2- 172.21.15.77:0/229192633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f29901018f0 0x7f2990196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.951+0000 7f299490a640 1 --2- 172.21.15.77:0/229192633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2990100700 0x7f2990196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.951+0000 7f299490a640 1 -- 172.21.15.77:0/229192633 >> 172.21.15.77:0/229192633 conn(0x7f29900fbc70 msgr2=0x7f2990104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.951+0000 7f299490a640 1 -- 172.21.15.77:0/229192633 shutdown_connections 2024-03-16T10:25:49.955 INFO:teuthology.task.background_exec.ubuntu@smithi077.front.sepia.ceph.com.smithi077.stderr:2024-03-16T10:25:49.951+0000 7f299490a640 1 -- 172.21.15.77:0/229192633 wait complete. 2024-03-16T10:25:50.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:49 smithi077 ceph-mon[32400]: pgmap v613: 97 pgs: 97 active+clean; 288 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 409 op/s 2024-03-16T10:25:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:49 smithi193 ceph-mon[36311]: pgmap v613: 97 pgs: 97 active+clean; 288 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 409 op/s 2024-03-16T10:25:50.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:49 smithi192 ceph-mon[36354]: pgmap v613: 97 pgs: 97 active+clean; 288 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 409 op/s 2024-03-16T10:25:52.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:51 smithi077 ceph-mon[32400]: pgmap v614: 97 pgs: 97 active+clean; 274 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 384 op/s 2024-03-16T10:25:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:51 smithi193 ceph-mon[36311]: pgmap v614: 97 pgs: 97 active+clean; 274 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 384 op/s 2024-03-16T10:25:52.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:51 smithi192 ceph-mon[36354]: pgmap v614: 97 pgs: 97 active+clean; 274 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 384 op/s 2024-03-16T10:25:52.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:25:52.964 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:25:53.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.073+0000 7f6e0bd90640 1 -- 172.21.15.77:0/72009423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e04100700 msgr2=0x7f6e04100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.073+0000 7f6e0bd90640 1 --2- 172.21.15.77:0/72009423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e04100700 0x7f6e04100b20 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009900 tx=0x7f6e0002bc30 comp rx=0 tx=0).stop 2024-03-16T10:25:53.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.073+0000 7f6e0bd90640 1 -- 172.21.15.77:0/72009423 shutdown_connections 2024-03-16T10:25:53.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.073+0000 7f6e0bd90640 1 --2- 172.21.15.77:0/72009423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e041022b0 0x7f6e04108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.073+0000 7f6e0bd90640 1 --2- 172.21.15.77:0/72009423 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e041018f0 0x7f6e04101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.073+0000 7f6e0bd90640 1 --2- 172.21.15.77:0/72009423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e04100700 0x7f6e04100b20 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.073+0000 7f6e0bd90640 1 -- 172.21.15.77:0/72009423 >> 172.21.15.77:0/72009423 conn(0x7f6e040fbc90 msgr2=0x7f6e040fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.073+0000 7f6e0bd90640 1 -- 172.21.15.77:0/72009423 shutdown_connections 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0bd90640 1 -- 172.21.15.77:0/72009423 wait complete. 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0bd90640 1 Processor -- start 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0bd90640 1 -- start start 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0bd90640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e04100700 0x7f6e0410b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0bd90640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e041018f0 0x7f6e041096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0bd90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e041022b0 0x7f6e04109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0bd90640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0410d4a0 con 0x7f6e041022b0 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0bd90640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0410d320 con 0x7f6e04100700 2024-03-16T10:25:53.077 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0bd90640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0410d620 con 0x7f6e041018f0 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0a32d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e041022b0 0x7f6e04109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0a32d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e041022b0 0x7f6e04109c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41350/0 (socket says 172.21.15.77:41350) 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0a32d640 1 -- 172.21.15.77:0/1623031200 learned_addr learned my addr 172.21.15.77:0/1623031200 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e09b2c640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e04100700 0x7f6e0410b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.074+0000 7f6e0932b640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e041018f0 0x7f6e041096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.075+0000 7f6e0a32d640 1 -- 172.21.15.77:0/1623031200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e041018f0 msgr2=0x7f6e041096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.075+0000 7f6e0a32d640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e041018f0 0x7f6e041096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.075+0000 7f6e0a32d640 1 -- 172.21.15.77:0/1623031200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e04100700 msgr2=0x7f6e0410b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.075+0000 7f6e0a32d640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e04100700 0x7f6e0410b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.075+0000 7f6e0a32d640 1 -- 172.21.15.77:0/1623031200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0410a140 con 0x7f6e041022b0 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.075+0000 7f6e0a32d640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e041022b0 0x7f6e04109c00 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00eb10 tx=0x7f6dfc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.075+0000 7f6dfaffd640 1 -- 172.21.15.77:0/1623031200 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc004b20 con 0x7f6e041022b0 2024-03-16T10:25:53.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.075+0000 7f6e0bd90640 1 -- 172.21.15.77:0/1623031200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e0410a430 con 0x7f6e041022b0 2024-03-16T10:25:53.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.075+0000 7f6dfaffd640 1 -- 172.21.15.77:0/1623031200 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dfc004cc0 con 0x7f6e041022b0 2024-03-16T10:25:53.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.075+0000 7f6dfaffd640 1 -- 172.21.15.77:0/1623031200 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc010760 con 0x7f6e041022b0 2024-03-16T10:25:53.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.075+0000 7f6e0bd90640 1 -- 172.21.15.77:0/1623031200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e041a6600 con 0x7f6e041022b0 2024-03-16T10:25:53.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.078+0000 7f6dfaffd640 1 -- 172.21.15.77:0/1623031200 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f6dfc005020 con 0x7f6e041022b0 2024-03-16T10:25:53.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.078+0000 7f6e0e977740 1 Processor -- start 2024-03-16T10:25:53.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.078+0000 7f6e0e977740 1 -- start start 2024-03-16T10:25:53.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.078+0000 7f6e0e977740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7170 0x55f7223e7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.078+0000 7f6e0e977740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223e7b60 0x55f7223ec660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.078+0000 7f6e0e977740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223eccd0 0x55f7223ef160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.078+0000 7f6e0e977740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f72234c500 con 0x55f7223e7b60 2024-03-16T10:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.078+0000 7f6e0e977740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55f7222f4be0 con 0x55f7223eccd0 2024-03-16T10:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.078+0000 7f6e0e977740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f7223485a0 con 0x55f7223e7170 2024-03-16T10:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6e09b2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223e7b60 0x55f7223ec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6e09b2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223e7b60 0x55f7223ec660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41360/0 (socket says 172.21.15.77:41360) 2024-03-16T10:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6e09b2c640 1 -- 172.21.15.77:0/2314352773 learned_addr learned my addr 172.21.15.77:0/2314352773 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6e0932b640 1 --2- 172.21.15.77:0/2314352773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7170 0x55f7223e7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6e0932b640 1 --2- 172.21.15.77:0/2314352773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223eccd0 0x55f7223ef160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6dfaffd640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6df00a2800 0x7f6df00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6e09b2c640 1 -- 172.21.15.77:0/2314352773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7170 msgr2=0x55f7223e7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6e09b2c640 1 --2- 172.21.15.77:0/2314352773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7170 0x55f7223e7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6e09b2c640 1 -- 172.21.15.77:0/2314352773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223eccd0 msgr2=0x55f7223ef160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6e09b2c640 1 --2- 172.21.15.77:0/2314352773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223eccd0 0x55f7223ef160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6e09b2c640 1 -- 172.21.15.77:0/2314352773 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7223084b0 con 0x55f7223e7b60 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6e0a32d640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6df00a2800 0x7f6df00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6e09b2c640 1 --2- 172.21.15.77:0/2314352773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223e7b60 0x55f7223ec660 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009900 tx=0x7f6e00009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425290baebee9009 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6dfaffd640 1 -- 172.21.15.77:0/1623031200 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6dfc0cf0b0 con 0x7f6e041022b0 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6deffff640 1 -- 172.21.15.77:0/2314352773 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e00005d50 con 0x55f7223e7b60 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6deffff640 1 -- 172.21.15.77:0/2314352773 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e0002cb80 con 0x55f7223e7b60 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.079+0000 7f6deffff640 1 -- 172.21.15.77:0/2314352773 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e00004920 con 0x55f7223e7b60 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.080+0000 7f6e0e977740 1 -- 172.21.15.77:0/2314352773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223e7b60 msgr2=0x55f7223ec660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.080+0000 7f6e0a32d640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6df00a2800 0x7f6df00a4cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc021000 tx=0x7f6dfc0c6530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.080+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2314352773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223e7b60 0x55f7223ec660 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009900 tx=0x7f6e00009560 comp rx=0 tx=0).stop 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.080+0000 7f6e0e977740 1 -- 172.21.15.77:0/2314352773 shutdown_connections 2024-03-16T10:25:53.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.080+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2314352773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223eccd0 0x55f7223ef160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.080+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2314352773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223e7b60 0x55f7223ec660 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.080+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2314352773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7170 0x55f7223e7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.080+0000 7f6e0e977740 1 -- 172.21.15.77:0/2314352773 >> 172.21.15.77:0/2314352773 conn(0x55f7223e26e0 msgr2=0x55f7223e2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.080+0000 7f6e0e977740 1 -- 172.21.15.77:0/2314352773 shutdown_connections 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.080+0000 7f6e0e977740 1 -- 172.21.15.77:0/2314352773 wait complete. 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0e977740 1 -- start start 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0e977740 1 Processor -- start 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0e977740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7170 0x55f7223f1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0e977740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7b60 0x55f7223f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0e977740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223f8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0e977740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f7223efc20 con 0x55f7223eccd0 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0e977740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55f722308160 con 0x55f7223e7170 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0e977740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f7223efda0 con 0x55f7223e7b60 2024-03-16T10:25:53.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0e977740 10 client.0 _ll_register_callbacks cb 0x55f7222ec150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0932b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223f8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e09b2c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7b60 0x55f7223f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e09b2c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7b60 0x55f7223f1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50240/0 (socket says 172.21.15.77:50240) 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e09b2c640 1 -- 172.21.15.77:0/2442030552 learned_addr learned my addr 172.21.15.77:0/2442030552 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0932b640 1 --2- 172.21.15.77:0/2442030552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7170 0x55f7223f1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0932b640 1 -- 172.21.15.77:0/2442030552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7b60 msgr2=0x55f7223f1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.081+0000 7f6e0932b640 1 --2- 172.21.15.77:0/2442030552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7b60 0x55f7223f1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0932b640 1 -- 172.21.15.77:0/2442030552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7170 msgr2=0x55f7223f1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0932b640 1 --2- 172.21.15.77:0/2442030552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7170 0x55f7223f1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0932b640 1 -- 172.21.15.77:0/2442030552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f72212a7d0 con 0x55f7223eccd0 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0932b640 1 --2- 172.21.15.77:0/2442030552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223f8be0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009fd0 tx=0x7f6df40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a51a3834253722 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6ded7fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6ded7fa640 1 -- 172.21.15.77:0/2442030552 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df400fe30 con 0x55f7223eccd0 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6ded7fa640 1 -- 172.21.15.77:0/2442030552 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6df4004050 con 0x55f7223eccd0 2024-03-16T10:25:53.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0e977740 1 -- 172.21.15.77:0/2442030552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e0002cb80 con 0x55f7223eccd0 2024-03-16T10:25:53.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6ded7fa640 1 -- 172.21.15.77:0/2442030552 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f6df4004250 con 0x55f7223eccd0 2024-03-16T10:25:53.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0e977740 10 client.17565 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:53.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0e977740 1 -- 172.21.15.77:0/2442030552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f7220603f0 con 0x55f7223eccd0 2024-03-16T10:25:53.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6ded7fa640 1 -- 172.21.15.77:0/2442030552 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e0002cb80 con 0x55f7223eccd0 2024-03-16T10:25:53.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6ded7fa640 1 -- 172.21.15.77:0/2442030552 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f6df4022470 con 0x55f7223eccd0 2024-03-16T10:25:53.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0e977740 10 client.17565 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:53.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0e977740 10 client.17565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:53.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0e977740 20 client.17565 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:53.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2442030552 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f7223fd160 0x55f7223ff600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0e977740 4 client.17565 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:53.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.082+0000 7f6e0e977740 1 -- 172.21.15.77:0/2442030552 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f722359a40 con 0x55f7223fd160 2024-03-16T10:25:53.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.083+0000 7f6e09b2c640 1 --2- 172.21.15.77:0/2442030552 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f7223fd160 0x55f7223ff600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.083+0000 7f6ded7fa640 10 client.17565 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.083+0000 7f6e09b2c640 1 --2- 172.21.15.77:0/2442030552 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f7223fd160 0x55f7223ff600 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0b23522c4b7078e server_cookie=9f5a00271e1ab0eb in_seq=0 out_seq=0 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6ded7fa640 1 -- 172.21.15.77:0/2442030552 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f722359a40 con 0x55f7223fd160 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6ded7fa640 10 client.17565 handle_command_reply: tid=0 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6e0e977740 1 client.17565 shutdown 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6e0e977740 1 -- 172.21.15.77:0/2442030552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 msgr2=0x55f7223f8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2442030552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223f8be0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009fd0 tx=0x7f6df40046c0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6e0e977740 1 -- 172.21.15.77:0/2442030552 shutdown_connections 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2442030552 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f7223fd160 0x55f7223ff600 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2442030552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223f8be0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2442030552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7b60 0x55f7223f1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2442030552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7170 0x55f7223f1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6ded7fa640 0 client.17565 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:53.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6e0e977740 1 -- 172.21.15.77:0/2442030552 >> 172.21.15.77:0/2442030552 conn(0x55f7223e26e0 msgr2=0x55f7223e1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6e0e977740 1 -- 172.21.15.77:0/2442030552 shutdown_connections 2024-03-16T10:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6e0e977740 1 -- 172.21.15.77:0/2442030552 wait complete. 2024-03-16T10:25:53.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.084+0000 7f6e0e977740 20 client.17565 trim_cache size 0 max 16384 2024-03-16T10:25:53.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.114+0000 7f6e0e977740 1 Processor -- start 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.114+0000 7f6e0e977740 1 -- start start 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0e977740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7170 0x55f7223e1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0e977740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7b60 0x55f7223f8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0e977740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223e15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0e977740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6e041af1b0 con 0x55f7223eccd0 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0e977740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6e041b0db0 con 0x55f7223e7170 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0e977740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6e041af8e0 con 0x55f7223e7b60 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0932b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223e15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0932b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223e15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41380/0 (socket says 172.21.15.77:41380) 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0932b640 1 -- 172.21.15.77:0/2119160442 learned_addr learned my addr 172.21.15.77:0/2119160442 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e09b2c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7b60 0x55f7223f8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0932b640 1 --2- 172.21.15.77:0/2119160442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7170 0x55f7223e1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0932b640 1 -- 172.21.15.77:0/2119160442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7b60 msgr2=0x55f7223f8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0932b640 1 --2- 172.21.15.77:0/2119160442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7b60 0x55f7223f8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0932b640 1 -- 172.21.15.77:0/2119160442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7170 msgr2=0x55f7223e1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0932b640 1 --2- 172.21.15.77:0/2119160442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7170 0x55f7223e1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0932b640 1 -- 172.21.15.77:0/2119160442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7220200f0 con 0x55f7223eccd0 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e09b2c640 1 --2- 172.21.15.77:0/2119160442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7b60 0x55f7223f8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6e0932b640 1 --2- 172.21.15.77:0/2119160442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223e15b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6df40114c0 tx=0x7f6df4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03b354c1d65fd84 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6def7fe640 1 -- 172.21.15.77:0/2119160442 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4031030 con 0x55f7223eccd0 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6def7fe640 1 -- 172.21.15.77:0/2119160442 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6df4032040 con 0x55f7223eccd0 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.115+0000 7f6def7fe640 1 -- 172.21.15.77:0/2119160442 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df402d420 con 0x55f7223eccd0 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.116+0000 7f6e0e977740 1 -- 172.21.15.77:0/2119160442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 msgr2=0x55f7223e15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.116+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2119160442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223e15b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6df40114c0 tx=0x7f6df4005f90 comp rx=0 tx=0).stop 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.116+0000 7f6e0e977740 1 -- 172.21.15.77:0/2119160442 shutdown_connections 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.116+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2119160442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223e15b0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.116+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2119160442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7b60 0x55f7223f8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.116+0000 7f6e0e977740 1 --2- 172.21.15.77:0/2119160442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7170 0x55f7223e1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.116+0000 7f6e0e977740 1 -- 172.21.15.77:0/2119160442 >> 172.21.15.77:0/2119160442 conn(0x55f7223e26e0 msgr2=0x55f7223e90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.116+0000 7f6e0e977740 1 -- 172.21.15.77:0/2119160442 shutdown_connections 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.116+0000 7f6e0e977740 1 -- 172.21.15.77:0/2119160442 wait complete. 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.116+0000 7f6e0e977740 1 -- start start 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.116+0000 7f6e0e977740 1 Processor -- start 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e0e977740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7170 0x55f7223f6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e0e977740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7b60 0x55f7223f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e0e977740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223f7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e0e977740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6e041b06e0 con 0x55f7223eccd0 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e0e977740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6e041affe0 con 0x55f7223e7b60 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e0e977740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6e041ad1d0 con 0x55f7223e7170 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e0e977740 10 client.0 _ll_register_callbacks cb 0x7f6df4011550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e0932b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e0932b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223f7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41388/0 (socket says 172.21.15.77:41388) 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e09b2c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7b60 0x55f7223f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e0932b640 1 -- 172.21.15.77:0/57438949 learned_addr learned my addr 172.21.15.77:0/57438949 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e0932b640 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7170 0x55f7223f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e09b2c640 1 -- 172.21.15.77:0/57438949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7170 msgr2=0x55f7223f6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e09b2c640 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7170 0x55f7223f6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e09b2c640 1 -- 172.21.15.77:0/57438949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 msgr2=0x55f7223f7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e09b2c640 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223f7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e09b2c640 1 -- 172.21.15.77:0/57438949 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7223139d0 con 0x55f7223e7b60 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e0932b640 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223f7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6e09b2c640 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7b60 0x55f7223f65c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6e00000c00 tx=0x7f6e00005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc728173115b5af9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.117+0000 7f6ded7fa640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6ded7fa640 1 -- 172.21.15.77:0/57438949 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0004a040 con 0x55f7223e7b60 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6e0e977740 1 -- 172.21.15.77:0/57438949 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6df4032040 con 0x55f7223e7b60 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6ded7fa640 1 -- 172.21.15.77:0/57438949 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6e0002ca00 con 0x55f7223e7b60 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6ded7fa640 1 -- 172.21.15.77:0/57438949 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f6e0004f9c0 con 0x55f7223e7b60 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6e0e977740 10 client.27344 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6e0e977740 1 -- 172.21.15.77:0/57438949 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f7223f8d20 con 0x55f7223e7b60 2024-03-16T10:25:53.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6ded7fa640 1 -- 172.21.15.77:0/57438949 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6df4032040 con 0x55f7223e7b60 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6ded7fa640 1 -- 172.21.15.77:0/57438949 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f6e00058050 con 0x55f7223e7b60 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6e0e977740 10 client.27344 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6e0e977740 10 client.27344 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6e0e977740 20 client.27344 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6e0e977740 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f722463760 0x55f722465b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6e0e977740 4 client.27344 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6e0e977740 1 -- 172.21.15.77:0/57438949 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f722353d90 con 0x55f722463760 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.118+0000 7f6e0932b640 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f722463760 0x55f722465b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6ded7fa640 10 client.27344 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0932b640 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f722463760 0x55f722465b50 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=804d7b213c69c9b0 server_cookie=428c326cfa55a55e in_seq=0 out_seq=0 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6ded7fa640 1 -- 172.21.15.77:0/57438949 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f722353d90 con 0x55f722463760 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6ded7fa640 10 client.27344 handle_command_reply: tid=0 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0e977740 1 client.27344 shutdown 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0e977740 1 -- 172.21.15.77:0/57438949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7b60 msgr2=0x55f7223f65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0e977740 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7b60 0x55f7223f65c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6e00000c00 tx=0x7f6e00005a20 comp rx=0 tx=0).stop 2024-03-16T10:25:53.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0e977740 1 -- 172.21.15.77:0/57438949 shutdown_connections 2024-03-16T10:25:53.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0e977740 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55f722463760 0x55f722465b50 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0e977740 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7223eccd0 0x55f7223f7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0e977740 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7223e7b60 0x55f7223f65c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6ded7fa640 0 client.27344 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:53.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0e977740 1 --2- 172.21.15.77:0/57438949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7223e7170 0x55f7223f6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0e977740 1 -- 172.21.15.77:0/57438949 >> 172.21.15.77:0/57438949 conn(0x55f7223e26e0 msgr2=0x55f7223e5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0e977740 1 -- 172.21.15.77:0/57438949 shutdown_connections 2024-03-16T10:25:53.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0e977740 1 -- 172.21.15.77:0/57438949 wait complete. 2024-03-16T10:25:53.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.119+0000 7f6e0e977740 20 client.27344 trim_cache size 0 max 16384 2024-03-16T10:25:53.126 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:25:53.126 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T10:25:53.127 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T10:25:53.127 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:25:53.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.120+0000 7f6e0bd90640 1 -- 172.21.15.77:0/1623031200 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6df00a2800 msgr2=0x7f6df00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.120+0000 7f6e0bd90640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6df00a2800 0x7f6df00a4cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc021000 tx=0x7f6dfc0c6530 comp rx=0 tx=0).stop 2024-03-16T10:25:53.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.120+0000 7f6e0bd90640 1 -- 172.21.15.77:0/1623031200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e041022b0 msgr2=0x7f6e04109c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.120+0000 7f6e0bd90640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e041022b0 0x7f6e04109c00 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00eb10 tx=0x7f6dfc00c2d0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.120+0000 7f6e0bd90640 1 -- 172.21.15.77:0/1623031200 shutdown_connections 2024-03-16T10:25:53.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.120+0000 7f6e0bd90640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e041022b0 0x7f6e04109c00 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.120+0000 7f6e0bd90640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6df00a2800 0x7f6df00a4cc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.120+0000 7f6e0bd90640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e041018f0 0x7f6e041096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.120+0000 7f6e0bd90640 1 --2- 172.21.15.77:0/1623031200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e04100700 0x7f6e0410b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.120+0000 7f6e0bd90640 1 -- 172.21.15.77:0/1623031200 >> 172.21.15.77:0/1623031200 conn(0x7f6e040fbc90 msgr2=0x7f6e04104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.120+0000 7f6e0bd90640 1 -- 172.21.15.77:0/1623031200 shutdown_connections 2024-03-16T10:25:53.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.120+0000 7f6e0bd90640 1 -- 172.21.15.77:0/1623031200 wait complete. 2024-03-16T10:25:53.132 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T10:25:53.132 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:25:53.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.236+0000 7f1e49034640 1 -- 172.21.15.77:0/3745029969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e441022b0 msgr2=0x7f1e44108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.236+0000 7f1e49034640 1 --2- 172.21.15.77:0/3745029969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e441022b0 0x7f1e44108b40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009ca0 tx=0x7f1e3c02c0f0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.236+0000 7f1e49034640 1 -- 172.21.15.77:0/3745029969 shutdown_connections 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.236+0000 7f1e49034640 1 --2- 172.21.15.77:0/3745029969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e441022b0 0x7f1e44108b40 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.236+0000 7f1e49034640 1 --2- 172.21.15.77:0/3745029969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e441018f0 0x7f1e44101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.236+0000 7f1e49034640 1 --2- 172.21.15.77:0/3745029969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e44100700 0x7f1e44100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.236+0000 7f1e49034640 1 -- 172.21.15.77:0/3745029969 >> 172.21.15.77:0/3745029969 conn(0x7f1e440fbc70 msgr2=0x7f1e440fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.236+0000 7f1e49034640 1 -- 172.21.15.77:0/3745029969 shutdown_connections 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.236+0000 7f1e49034640 1 -- 172.21.15.77:0/3745029969 wait complete. 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.236+0000 7f1e49034640 1 Processor -- start 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.236+0000 7f1e49034640 1 -- start start 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e49034640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e44100700 0x7f1e440694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e49034640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e441018f0 0x7f1e4419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e49034640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e441022b0 0x7f1e4419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e49034640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4410c380 con 0x7f1e441022b0 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e49034640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4410c200 con 0x7f1e44100700 2024-03-16T10:25:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e49034640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4410c500 con 0x7f1e441018f0 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e4359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e441022b0 0x7f1e4419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e4359e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e441022b0 0x7f1e4419ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41400/0 (socket says 172.21.15.77:41400) 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e4359e640 1 -- 172.21.15.77:0/716999728 learned_addr learned my addr 172.21.15.77:0/716999728 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e42d9d640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e44100700 0x7f1e440694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e3bfff640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e441018f0 0x7f1e4419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e4359e640 1 -- 172.21.15.77:0/716999728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e441018f0 msgr2=0x7f1e4419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e4359e640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e441018f0 0x7f1e4419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e4359e640 1 -- 172.21.15.77:0/716999728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e44100700 msgr2=0x7f1e440694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e4359e640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e44100700 0x7f1e440694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e4359e640 1 -- 172.21.15.77:0/716999728 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e4419f770 con 0x7f1e441022b0 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e4359e640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e441022b0 0x7f1e4419ee60 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c037040 tx=0x7f1e3c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e40d99640 1 -- 172.21.15.77:0/716999728 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c041020 con 0x7f1e441022b0 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e49034640 1 -- 172.21.15.77:0/716999728 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e4419f9a0 con 0x7f1e441022b0 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e40d99640 1 -- 172.21.15.77:0/716999728 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e3c034cd0 con 0x7f1e441022b0 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.237+0000 7f1e40d99640 1 -- 172.21.15.77:0/716999728 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c039040 con 0x7f1e441022b0 2024-03-16T10:25:53.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.238+0000 7f1e49034640 1 -- 172.21.15.77:0/716999728 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e4419fee0 con 0x7f1e441022b0 2024-03-16T10:25:53.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.238+0000 7f1e49034640 1 -- 172.21.15.77:0/716999728 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e440685d0 con 0x7f1e441022b0 2024-03-16T10:25:53.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.241+0000 7f1e40d99640 1 -- 172.21.15.77:0/716999728 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f1e3c044020 con 0x7f1e441022b0 2024-03-16T10:25:53.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.241+0000 7f1e40d99640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1e300a2800 0x7f1e300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.241+0000 7f1e40d99640 1 -- 172.21.15.77:0/716999728 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f1e3c0f0c30 con 0x7f1e441022b0 2024-03-16T10:25:53.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.241+0000 7f1e42d9d640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1e300a2800 0x7f1e300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.242+0000 7f1e42d9d640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1e300a2800 0x7f1e300a4cc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1e44101750 tx=0x7f1e34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.244+0000 7f1e40d99640 1 -- 172.21.15.77:0/716999728 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1e3c04b020 con 0x7f1e441022b0 2024-03-16T10:25:53.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.506+0000 7f1e49034640 1 -- 172.21.15.77:0/716999728 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e44107a20 con 0x7f1e441022b0 2024-03-16T10:25:53.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.507+0000 7f1e40d99640 1 -- 172.21.15.77:0/716999728 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+11942 (secure 0 0 0) 0x7f1e44107a20 con 0x7f1e441022b0 2024-03-16T10:25:53.507 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:25:53.508 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":47,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:25:39.752500+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26800":{"gid":26800,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/946505511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":946505511},{"type":"v1","addr":"172.21.15.193:6835","nonce":946505511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24475,24454,24482,14658,24455]},"id":1}]} 2024-03-16T10:25:53.508 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 47 2024-03-16T10:25:53.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.508+0000 7f1e49034640 1 -- 172.21.15.77:0/716999728 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1e300a2800 msgr2=0x7f1e300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.508+0000 7f1e49034640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1e300a2800 0x7f1e300a4cc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1e44101750 tx=0x7f1e34005f90 comp rx=0 tx=0).stop 2024-03-16T10:25:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.508+0000 7f1e49034640 1 -- 172.21.15.77:0/716999728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e441022b0 msgr2=0x7f1e4419ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.508+0000 7f1e49034640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e441022b0 0x7f1e4419ee60 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c037040 tx=0x7f1e3c032f40 comp rx=0 tx=0).stop 2024-03-16T10:25:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.508+0000 7f1e49034640 1 -- 172.21.15.77:0/716999728 shutdown_connections 2024-03-16T10:25:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.508+0000 7f1e49034640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e441022b0 0x7f1e4419ee60 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.508+0000 7f1e49034640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1e300a2800 0x7f1e300a4cc0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.508+0000 7f1e49034640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e441018f0 0x7f1e4419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.508+0000 7f1e49034640 1 --2- 172.21.15.77:0/716999728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e44100700 0x7f1e440694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.508+0000 7f1e49034640 1 -- 172.21.15.77:0/716999728 >> 172.21.15.77:0/716999728 conn(0x7f1e440fbc70 msgr2=0x7f1e44104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.509+0000 7f1e49034640 1 -- 172.21.15.77:0/716999728 shutdown_connections 2024-03-16T10:25:53.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.509+0000 7f1e49034640 1 -- 172.21.15.77:0/716999728 wait complete. 2024-03-16T10:25:53.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-16T10:25:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.624+0000 7f6d7c362640 1 -- 172.21.15.77:0/419193903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d741006e0 msgr2=0x7f6d74100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.624+0000 7f6d7c362640 1 --2- 172.21.15.77:0/419193903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d741006e0 0x7f6d74100b00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c009900 tx=0x7f6d6c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 -- 172.21.15.77:0/419193903 shutdown_connections 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 --2- 172.21.15.77:0/419193903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d74102290 0x7f6d74108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 --2- 172.21.15.77:0/419193903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d741018d0 0x7f6d74101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 --2- 172.21.15.77:0/419193903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d741006e0 0x7f6d74100b00 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 -- 172.21.15.77:0/419193903 >> 172.21.15.77:0/419193903 conn(0x7f6d740fbc70 msgr2=0x7f6d740fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 -- 172.21.15.77:0/419193903 shutdown_connections 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 -- 172.21.15.77:0/419193903 wait complete. 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 Processor -- start 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 -- start start 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d741006e0 0x7f6d74197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d741018d0 0x7f6d741980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d74102290 0x7f6d7419f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7410c330 con 0x7f6d741006e0 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7410c1b0 con 0x7f6d74102290 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7c362640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7410c4b0 con 0x7f6d741018d0 2024-03-16T10:25:53.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7a0fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d741006e0 0x7f6d74197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7a0fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d741006e0 0x7f6d74197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41428/0 (socket says 172.21.15.77:41428) 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7a0fe640 1 -- 172.21.15.77:0/3398296848 learned_addr learned my addr 172.21.15.77:0/3398296848 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d798fd640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d741018d0 0x7f6d741980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.625+0000 7f6d7a8ff640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d74102290 0x7f6d7419f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.626+0000 7f6d7a0fe640 1 -- 172.21.15.77:0/3398296848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d741018d0 msgr2=0x7f6d741980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.626+0000 7f6d7a0fe640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d741018d0 0x7f6d741980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.626+0000 7f6d7a0fe640 1 -- 172.21.15.77:0/3398296848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d74102290 msgr2=0x7f6d7419f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.626+0000 7f6d7a0fe640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d74102290 0x7f6d7419f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.626+0000 7f6d7a0fe640 1 -- 172.21.15.77:0/3398296848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7419f7a0 con 0x7f6d741006e0 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.626+0000 7f6d7a0fe640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d741006e0 0x7f6d74197b80 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c000c00 tx=0x7f6d6c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.626+0000 7f6d6b7fe640 1 -- 172.21.15.77:0/3398296848 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c041070 con 0x7f6d741006e0 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.626+0000 7f6d7c362640 1 -- 172.21.15.77:0/3398296848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d7419fa30 con 0x7f6d741006e0 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.626+0000 7f6d6b7fe640 1 -- 172.21.15.77:0/3398296848 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d6c0044d0 con 0x7f6d741006e0 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.626+0000 7f6d6b7fe640 1 -- 172.21.15.77:0/3398296848 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c034440 con 0x7f6d741006e0 2024-03-16T10:25:53.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.626+0000 7f6d7c362640 1 -- 172.21.15.77:0/3398296848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d7419ff20 con 0x7f6d741006e0 2024-03-16T10:25:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.628+0000 7f6d6b7fe640 1 -- 172.21.15.77:0/3398296848 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f6d6c044070 con 0x7f6d741006e0 2024-03-16T10:25:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.629+0000 7f6d6b7fe640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6d600a2800 0x7f6d600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.629+0000 7f6d798fd640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6d600a2800 0x7f6d600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.629+0000 7f6d6b7fe640 1 -- 172.21.15.77:0/3398296848 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6d6c0f04c0 con 0x7f6d741006e0 2024-03-16T10:25:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.629+0000 7f6d798fd640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6d600a2800 0x7f6d600a4cc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6d74199090 tx=0x7f6d64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d7f02d740 1 Processor -- start 2024-03-16T10:25:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d7f02d740 1 -- start start 2024-03-16T10:25:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d7f02d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e11850 0x5559a7e11c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d7f02d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e12240 0x5559a7e16d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d7f02d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e173b0 0x5559a7e19840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d7f02d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5559a7d76490 con 0x5559a7e11850 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d7f02d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5559a7d1ec90 con 0x5559a7e12240 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d7f02d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5559a7d72530 con 0x5559a7e173b0 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d7a0fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e11850 0x5559a7e11c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d7a8ff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e173b0 0x5559a7e19840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d798fd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e12240 0x5559a7e16d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d7a0fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e11850 0x5559a7e11c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41440/0 (socket says 172.21.15.77:41440) 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d7a0fe640 1 -- 172.21.15.77:0/2936585630 learned_addr learned my addr 172.21.15.77:0/2936585630 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d798fd640 1 -- 172.21.15.77:0/2936585630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e173b0 msgr2=0x5559a7e19840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d798fd640 1 --2- 172.21.15.77:0/2936585630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e173b0 0x5559a7e19840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d798fd640 1 -- 172.21.15.77:0/2936585630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e11850 msgr2=0x5559a7e11c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d798fd640 1 --2- 172.21.15.77:0/2936585630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e11850 0x5559a7e11c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.630+0000 7f6d798fd640 1 -- 172.21.15.77:0/2936585630 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559a7dd9500 con 0x5559a7e12240 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.631+0000 7f6d7a8ff640 1 --2- 172.21.15.77:0/2936585630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e173b0 0x5559a7e19840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.631+0000 7f6d7a0fe640 1 --2- 172.21.15.77:0/2936585630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e11850 0x5559a7e11c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:53.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.631+0000 7f6d798fd640 1 --2- 172.21.15.77:0/2936585630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e12240 0x5559a7e16d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6d640118d0 tx=0x7f6d64014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=103df4b866237925 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.631+0000 7f6d68bfe640 1 -- 172.21.15.77:0/2936585630 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d64004030 con 0x5559a7e12240 2024-03-16T10:25:53.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.631+0000 7f6d68bfe640 1 -- 172.21.15.77:0/2936585630 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d640041d0 con 0x5559a7e12240 2024-03-16T10:25:53.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.631+0000 7f6d68bfe640 1 -- 172.21.15.77:0/2936585630 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d64026760 con 0x5559a7e12240 2024-03-16T10:25:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.632+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2936585630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e12240 msgr2=0x5559a7e16d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.632+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2936585630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e12240 0x5559a7e16d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6d640118d0 tx=0x7f6d64014000 comp rx=0 tx=0).stop 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.632+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2936585630 shutdown_connections 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.632+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2936585630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e173b0 0x5559a7e19840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.632+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2936585630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e12240 0x5559a7e16d40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.632+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2936585630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e11850 0x5559a7e11c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.632+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2936585630 >> 172.21.15.77:0/2936585630 conn(0x5559a7e0cdc0 msgr2=0x5559a7e0d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.632+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2936585630 shutdown_connections 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.632+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2936585630 wait complete. 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.632+0000 7f6d7f02d740 1 -- start start 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.632+0000 7f6d7f02d740 1 Processor -- start 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d7f02d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e11850 0x5559a7e1bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d7f02d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 0x5559a7e1c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d7f02d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e173b0 0x5559a7e23320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d7f02d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5559a7e1a360 con 0x5559a7e12240 2024-03-16T10:25:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d7f02d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5559a7d32230 con 0x5559a7e173b0 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d7f02d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5559a7e1a4e0 con 0x5559a7e11850 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d7f02d740 10 client.0 _ll_register_callbacks cb 0x5559a7d6e010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d798fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 0x5559a7e1c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d798fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 0x5559a7e1c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41448/0 (socket says 172.21.15.77:41448) 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d798fd640 1 -- 172.21.15.77:0/3335904136 learned_addr learned my addr 172.21.15.77:0/3335904136 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d7a0fe640 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e11850 0x5559a7e1bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d7a8ff640 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e173b0 0x5559a7e23320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d798fd640 1 -- 172.21.15.77:0/3335904136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e11850 msgr2=0x5559a7e1bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d798fd640 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e11850 0x5559a7e1bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d798fd640 1 -- 172.21.15.77:0/3335904136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e173b0 msgr2=0x5559a7e23320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.633+0000 7f6d798fd640 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e173b0 0x5559a7e23320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.634+0000 7f6d798fd640 1 -- 172.21.15.77:0/3335904136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559a7d41100 con 0x5559a7e12240 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.634+0000 7f6d7a8ff640 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e173b0 0x5559a7e23320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.634+0000 7f6d7a0fe640 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e11850 0x5559a7e1bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.634+0000 7f6d798fd640 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 0x5559a7e1c2e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6d64009a90 tx=0x7f6d64004030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e597010fa334fa9e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.634+0000 7f6d5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.634+0000 7f6d5dffb640 1 -- 172.21.15.77:0/3335904136 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d64004400 con 0x5559a7e12240 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.634+0000 7f6d5dffb640 1 -- 172.21.15.77:0/3335904136 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6d640035f0 con 0x5559a7e12240 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.634+0000 7f6d7f02d740 1 -- 172.21.15.77:0/3335904136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d640041d0 con 0x5559a7e12240 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.634+0000 7f6d5dffb640 1 -- 172.21.15.77:0/3335904136 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f6d640037f0 con 0x5559a7e12240 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.634+0000 7f6d7f02d740 10 client.17589 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.634+0000 7f6d7f02d740 1 -- 172.21.15.77:0/3335904136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5559a7a8a3f0 con 0x5559a7e12240 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.634+0000 7f6d5dffb640 1 -- 172.21.15.77:0/3335904136 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d640041d0 con 0x5559a7e12240 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.635+0000 7f6d5dffb640 1 -- 172.21.15.77:0/3335904136 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f6d64031410 con 0x5559a7e12240 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.635+0000 7f6d7f02d740 10 client.17589 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.635+0000 7f6d7f02d740 10 client.17589 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.635+0000 7f6d7f02d740 20 client.17589 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.635+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x5559a7e27870 0x5559a7e29c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.635+0000 7f6d7f02d740 4 client.17589 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.635+0000 7f6d7f02d740 1 -- 172.21.15.77:0/3335904136 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559a7d839d0 con 0x5559a7e27870 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.635+0000 7f6d7a0fe640 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x5559a7e27870 0x5559a7e29c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.635+0000 7f6d7a0fe640 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x5559a7e27870 0x5559a7e29c60 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f6f60d3fcbeb63f5 server_cookie=cc2a3f6922ac7143 in_seq=0 out_seq=0 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.635+0000 7f6d5dffb640 10 client.17589 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d5dffb640 1 -- 172.21.15.77:0/3335904136 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5559a7d839d0 con 0x5559a7e27870 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d5dffb640 10 client.17589 handle_command_reply: tid=0 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d7f02d740 1 client.17589 shutdown 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d7f02d740 1 -- 172.21.15.77:0/3335904136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 msgr2=0x5559a7e1c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 0x5559a7e1c2e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6d64009a90 tx=0x7f6d64004030 comp rx=0 tx=0).stop 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d7f02d740 1 -- 172.21.15.77:0/3335904136 shutdown_connections 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e173b0 0x5559a7e23320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 0x5559a7e1c2e0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x5559a7e27870 0x5559a7e29c60 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/3335904136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e11850 0x5559a7e1bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d5dffb640 0 client.17589 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:25:53.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d7f02d740 1 -- 172.21.15.77:0/3335904136 >> 172.21.15.77:0/3335904136 conn(0x5559a7e0cdc0 msgr2=0x5559a7e0bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d7f02d740 1 -- 172.21.15.77:0/3335904136 shutdown_connections 2024-03-16T10:25:53.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d7f02d740 1 -- 172.21.15.77:0/3335904136 wait complete. 2024-03-16T10:25:53.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.636+0000 7f6d7f02d740 20 client.17589 trim_cache size 0 max 16384 2024-03-16T10:25:53.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.649+0000 7f6d7f02d740 1 Processor -- start 2024-03-16T10:25:53.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.649+0000 7f6d7f02d740 1 -- start start 2024-03-16T10:25:53.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7f02d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e11850 0x5559a7e0b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7f02d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e12240 0x5559a7e232e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7f02d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e173b0 0x5559a7e0bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7f02d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d741a8c10 con 0x5559a7e173b0 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7f02d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6d741aa810 con 0x5559a7e12240 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7f02d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6d741a9340 con 0x5559a7e11850 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7a8ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e173b0 0x5559a7e0bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7a8ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e173b0 0x5559a7e0bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41462/0 (socket says 172.21.15.77:41462) 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7a8ff640 1 -- 172.21.15.77:0/2014276796 learned_addr learned my addr 172.21.15.77:0/2014276796 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d798fd640 1 --2- 172.21.15.77:0/2014276796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e12240 0x5559a7e232e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7a0fe640 1 --2- 172.21.15.77:0/2014276796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e11850 0x5559a7e0b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7a8ff640 1 -- 172.21.15.77:0/2014276796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e11850 msgr2=0x5559a7e0b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7a8ff640 1 --2- 172.21.15.77:0/2014276796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e11850 0x5559a7e0b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.656 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7a8ff640 1 -- 172.21.15.77:0/2014276796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e12240 msgr2=0x5559a7e232e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7a8ff640 1 --2- 172.21.15.77:0/2014276796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e12240 0x5559a7e232e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7a8ff640 1 -- 172.21.15.77:0/2014276796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559a7b547d0 con 0x5559a7e173b0 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d7a8ff640 1 --2- 172.21.15.77:0/2014276796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e173b0 0x5559a7e0bc90 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f6d70003970 tx=0x7f6d700096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1d1a6c18f29d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d5ffff640 1 -- 172.21.15.77:0/2014276796 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d70004820 con 0x5559a7e173b0 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d5ffff640 1 -- 172.21.15.77:0/2014276796 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d70011920 con 0x5559a7e173b0 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.650+0000 7f6d5ffff640 1 -- 172.21.15.77:0/2014276796 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7001bba0 con 0x5559a7e173b0 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.651+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2014276796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e173b0 msgr2=0x5559a7e0bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.651+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2014276796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e173b0 0x5559a7e0bc90 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f6d70003970 tx=0x7f6d700096a0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.651+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2014276796 shutdown_connections 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.651+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2014276796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e173b0 0x5559a7e0bc90 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.651+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2014276796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e12240 0x5559a7e232e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.651+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2014276796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e11850 0x5559a7e0b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.651+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2014276796 >> 172.21.15.77:0/2014276796 conn(0x5559a7e0cdc0 msgr2=0x5559a7e137d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.651+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2014276796 shutdown_connections 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.651+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2014276796 wait complete. 2024-03-16T10:25:53.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.651+0000 7f6d7f02d740 1 -- start start 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.651+0000 7f6d7f02d740 1 Processor -- start 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d7f02d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e11850 0x5559a7e20b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d7f02d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 0x5559a7e21060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d7f02d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e173b0 0x5559a7e22070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d7f02d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d741aa140 con 0x5559a7e12240 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d7f02d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6d741a9a40 con 0x5559a7e11850 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d7f02d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6d741a6bf0 con 0x5559a7e173b0 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d7f02d740 10 client.0 _ll_register_callbacks cb 0x7f6d6c005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d798fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 0x5559a7e21060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d7a0fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e11850 0x5559a7e20b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d798fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 0x5559a7e21060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41476/0 (socket says 172.21.15.77:41476) 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d7a0fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e11850 0x5559a7e20b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:55610/0 (socket says 172.21.15.77:55610) 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d798fd640 1 -- 172.21.15.77:0/2356819038 learned_addr learned my addr 172.21.15.77:0/2356819038 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d7a8ff640 1 --2- 172.21.15.77:0/2356819038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e173b0 0x5559a7e22070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d798fd640 1 -- 172.21.15.77:0/2356819038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e173b0 msgr2=0x5559a7e22070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d798fd640 1 --2- 172.21.15.77:0/2356819038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e173b0 0x5559a7e22070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d798fd640 1 -- 172.21.15.77:0/2356819038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e11850 msgr2=0x5559a7e20b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d798fd640 1 --2- 172.21.15.77:0/2356819038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e11850 0x5559a7e20b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d798fd640 1 -- 172.21.15.77:0/2356819038 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559a7e22660 con 0x5559a7e12240 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d798fd640 1 --2- 172.21.15.77:0/2356819038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 0x5559a7e21060 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f6d64031a20 tx=0x7f6d64031f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2110f5dd3cb8d4c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d5dffb640 1 -- 172.21.15.77:0/2356819038 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d64040070 con 0x5559a7e12240 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d5dffb640 1 -- 172.21.15.77:0/2356819038 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6d64031410 con 0x5559a7e12240 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.652+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2356819038 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d70011920 con 0x5559a7e12240 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d5dffb640 1 -- 172.21.15.77:0/2356819038 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f6d6404c050 con 0x5559a7e12240 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d7f02d740 10 client.17595 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2356819038 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5559a7e8dc10 con 0x5559a7e12240 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d5dffb640 1 -- 172.21.15.77:0/2356819038 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d70011920 con 0x5559a7e12240 2024-03-16T10:25:53.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d5dffb640 1 -- 172.21.15.77:0/2356819038 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f6d6403c930 con 0x5559a7e12240 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d7f02d740 10 client.17595 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d7f02d740 10 client.17595 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d7f02d740 20 client.17595 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2356819038 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x5559a7e8e5c0 0x5559a7e909e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d7f02d740 4 client.17595 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2356819038 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5559a7d7dd20 con 0x5559a7e8e5c0 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d7a0fe640 1 --2- 172.21.15.77:0/2356819038 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x5559a7e8e5c0 0x5559a7e909e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d7a0fe640 1 --2- 172.21.15.77:0/2356819038 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x5559a7e8e5c0 0x5559a7e909e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=68903ee609e393fc server_cookie=245453ea2751a834 in_seq=0 out_seq=0 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.653+0000 7f6d5dffb640 10 client.17595 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d5dffb640 1 -- 172.21.15.77:0/2356819038 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5559a7d7dd20 con 0x5559a7e8e5c0 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d5dffb640 10 client.17595 handle_command_reply: tid=0 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7f02d740 1 client.17595 shutdown 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2356819038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 msgr2=0x5559a7e21060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2356819038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 0x5559a7e21060 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f6d64031a20 tx=0x7f6d64031f90 comp rx=0 tx=0).stop 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2356819038 shutdown_connections 2024-03-16T10:25:53.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2356819038 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x5559a7e8e5c0 0x5559a7e909e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2356819038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559a7e173b0 0x5559a7e22070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2356819038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559a7e12240 0x5559a7e21060 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d5dffb640 0 client.17595 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:25:53.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7f02d740 1 --2- 172.21.15.77:0/2356819038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559a7e11850 0x5559a7e20b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2356819038 >> 172.21.15.77:0/2356819038 conn(0x5559a7e0cdc0 msgr2=0x5559a7e10020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2356819038 shutdown_connections 2024-03-16T10:25:53.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7f02d740 1 -- 172.21.15.77:0/2356819038 wait complete. 2024-03-16T10:25:53.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7f02d740 20 client.17595 trim_cache size 0 max 16384 2024-03-16T10:25:53.661 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:25:53.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7c362640 1 -- 172.21.15.77:0/3398296848 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6d600a2800 msgr2=0x7f6d600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7c362640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6d600a2800 0x7f6d600a4cc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6d74199090 tx=0x7f6d64005f90 comp rx=0 tx=0).stop 2024-03-16T10:25:53.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7c362640 1 -- 172.21.15.77:0/3398296848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d741006e0 msgr2=0x7f6d74197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7c362640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d741006e0 0x7f6d74197b80 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c000c00 tx=0x7f6d6c005d50 comp rx=0 tx=0).stop 2024-03-16T10:25:53.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7c362640 1 -- 172.21.15.77:0/3398296848 shutdown_connections 2024-03-16T10:25:53.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7c362640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d74102290 0x7f6d7419f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7c362640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6d600a2800 0x7f6d600a4cc0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7c362640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d741018d0 0x7f6d741980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7c362640 1 --2- 172.21.15.77:0/3398296848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d741006e0 0x7f6d74197b80 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.654+0000 7f6d7c362640 1 -- 172.21.15.77:0/3398296848 >> 172.21.15.77:0/3398296848 conn(0x7f6d740fbc70 msgr2=0x7f6d74068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.655+0000 7f6d7c362640 1 -- 172.21.15.77:0/3398296848 shutdown_connections 2024-03-16T10:25:53.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.655+0000 7f6d7c362640 1 -- 172.21.15.77:0/3398296848 wait complete. 2024-03-16T10:25:53.666 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-16T10:25:53.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.772+0000 7f15db31b640 1 -- 172.21.15.77:0/2549853115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15d41029e0 msgr2=0x7f15d4102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.772+0000 7f15db31b640 1 --2- 172.21.15.77:0/2549853115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15d41029e0 0x7f15d4102e60 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009900 tx=0x7f15c402bc30 comp rx=0 tx=0).stop 2024-03-16T10:25:53.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.772+0000 7f15db31b640 1 -- 172.21.15.77:0/2549853115 shutdown_connections 2024-03-16T10:25:53.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.772+0000 7f15db31b640 1 --2- 172.21.15.77:0/2549853115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15d41033a0 0x7f15d4109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.772+0000 7f15db31b640 1 --2- 172.21.15.77:0/2549853115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15d41029e0 0x7f15d4102e60 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.772+0000 7f15db31b640 1 --2- 172.21.15.77:0/2549853115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d41017f0 0x7f15d4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.772+0000 7f15db31b640 1 -- 172.21.15.77:0/2549853115 >> 172.21.15.77:0/2549853115 conn(0x7f15d40fcd80 msgr2=0x7f15d40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.772+0000 7f15db31b640 1 -- 172.21.15.77:0/2549853115 shutdown_connections 2024-03-16T10:25:53.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.772+0000 7f15db31b640 1 -- 172.21.15.77:0/2549853115 wait complete. 2024-03-16T10:25:53.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.772+0000 7f15db31b640 1 Processor -- start 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15db31b640 1 -- start start 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15db31b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15d41017f0 0x7f15d4197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15db31b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d41029e0 0x7f15d4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15db31b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15d41033a0 0x7f15d419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15db31b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f15d410d450 con 0x7f15d41017f0 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15db31b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f15d410d2d0 con 0x7f15d41029e0 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15db31b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f15d410d5d0 con 0x7f15d41033a0 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15d90b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15d41017f0 0x7f15d4197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15d90b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15d41017f0 0x7f15d4197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41494/0 (socket says 172.21.15.77:41494) 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15d90b7640 1 -- 172.21.15.77:0/4243324994 learned_addr learned my addr 172.21.15.77:0/4243324994 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15d98b8640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15d41033a0 0x7f15d419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15d88b6640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d41029e0 0x7f15d4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15d90b7640 1 -- 172.21.15.77:0/4243324994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15d41033a0 msgr2=0x7f15d419f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15d90b7640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15d41033a0 0x7f15d419f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15d90b7640 1 -- 172.21.15.77:0/4243324994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d41029e0 msgr2=0x7f15d4198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15d90b7640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d41029e0 0x7f15d4198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15d90b7640 1 -- 172.21.15.77:0/4243324994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15d419f730 con 0x7f15d41017f0 2024-03-16T10:25:53.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15d90b7640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15d41017f0 0x7f15d4197b10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f15d000e9c0 tx=0x7f15d000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15ca7fc640 1 -- 172.21.15.77:0/4243324994 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d000cd80 con 0x7f15d41017f0 2024-03-16T10:25:53.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15ca7fc640 1 -- 172.21.15.77:0/4243324994 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15d00035f0 con 0x7f15d41017f0 2024-03-16T10:25:53.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15db31b640 1 -- 172.21.15.77:0/4243324994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15d419fa20 con 0x7f15d41017f0 2024-03-16T10:25:53.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.773+0000 7f15ca7fc640 1 -- 172.21.15.77:0/4243324994 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0010690 con 0x7f15d41017f0 2024-03-16T10:25:53.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.774+0000 7f15db31b640 1 -- 172.21.15.77:0/4243324994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15d419ff40 con 0x7f15d41017f0 2024-03-16T10:25:53.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15ca7fc640 1 -- 172.21.15.77:0/4243324994 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f15d00037a0 con 0x7f15d41017f0 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15ddfe5740 1 Processor -- start 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15ddfe5740 1 -- start start 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15ddfe5740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b2850 0x5628b38b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15ddfe5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b3240 0x5628b38b7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15ddfe5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b83b0 0x5628b38ba840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15ddfe5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5628b3817490 con 0x5628b38b3240 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15ddfe5740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5628b37bfc90 con 0x5628b38b2850 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15ddfe5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5628b3813530 con 0x5628b38b83b0 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15d90b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b3240 0x5628b38b7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15d90b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b3240 0x5628b38b7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41498/0 (socket says 172.21.15.77:41498) 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15d90b7640 1 -- 172.21.15.77:0/1743350186 learned_addr learned my addr 172.21.15.77:0/1743350186 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15d98b8640 1 --2- 172.21.15.77:0/1743350186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b2850 0x5628b38b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15ca7fc640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f15c00a2800 0x7f15c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.776+0000 7f15d88b6640 1 --2- 172.21.15.77:0/1743350186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b83b0 0x5628b38ba840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15d98b8640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f15c00a2800 0x7f15c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15ca7fc640 1 -- 172.21.15.77:0/4243324994 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f15d00ced80 con 0x7f15d41017f0 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15d88b6640 1 -- 172.21.15.77:0/1743350186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b2850 msgr2=0x5628b38b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15d88b6640 1 --2- 172.21.15.77:0/1743350186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b2850 0x5628b38b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15d88b6640 1 -- 172.21.15.77:0/1743350186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b3240 msgr2=0x5628b38b7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15d88b6640 1 --2- 172.21.15.77:0/1743350186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b3240 0x5628b38b7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15d88b6640 1 -- 172.21.15.77:0/1743350186 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628b387a500 con 0x5628b38b83b0 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15d90b7640 1 --2- 172.21.15.77:0/1743350186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b3240 0x5628b38b7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15d88b6640 1 --2- 172.21.15.77:0/1743350186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b83b0 0x5628b38ba840 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009230 tx=0x7f15c4009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8adfab26997a051 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15d98b8640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f15c00a2800 0x7f15c00a4cc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x5628b38b3aa0 tx=0x7f15cc004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15bf7fe640 1 -- 172.21.15.77:0/1743350186 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c402cb80 con 0x5628b38b83b0 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15bf7fe640 1 -- 172.21.15.77:0/1743350186 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15c4005040 con 0x5628b38b83b0 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.777+0000 7f15bf7fe640 1 -- 172.21.15.77:0/1743350186 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c40409d0 con 0x5628b38b83b0 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.778+0000 7f15ddfe5740 1 -- 172.21.15.77:0/1743350186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b83b0 msgr2=0x5628b38ba840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.778+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/1743350186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b83b0 0x5628b38ba840 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009230 tx=0x7f15c4009560 comp rx=0 tx=0).stop 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.778+0000 7f15ddfe5740 1 -- 172.21.15.77:0/1743350186 shutdown_connections 2024-03-16T10:25:53.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.778+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/1743350186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b83b0 0x5628b38ba840 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.778+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/1743350186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b3240 0x5628b38b7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.778+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/1743350186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b2850 0x5628b38b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.778+0000 7f15ddfe5740 1 -- 172.21.15.77:0/1743350186 >> 172.21.15.77:0/1743350186 conn(0x5628b38addc0 msgr2=0x5628b38ae1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.778+0000 7f15ddfe5740 1 -- 172.21.15.77:0/1743350186 shutdown_connections 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.778+0000 7f15ddfe5740 1 -- 172.21.15.77:0/1743350186 wait complete. 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.778+0000 7f15ddfe5740 1 -- start start 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.778+0000 7f15ddfe5740 1 Processor -- start 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15ddfe5740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b2850 0x5628b38bcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15ddfe5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b3240 0x5628b38bd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15ddfe5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 0x5628b38c4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15ddfe5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5628b38bb300 con 0x5628b38b83b0 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15ddfe5740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5628b37d3230 con 0x5628b38b2850 2024-03-16T10:25:53.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15ddfe5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5628b38bb480 con 0x5628b38b3240 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15ddfe5740 10 client.0 _ll_register_callbacks cb 0x5628b380f010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15d98b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 0x5628b38c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15d88b6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b3240 0x5628b38bd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15d98b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 0x5628b38c4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41500/0 (socket says 172.21.15.77:41500) 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15d90b7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b2850 0x5628b38bcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15d98b8640 1 -- 172.21.15.77:0/798170059 learned_addr learned my addr 172.21.15.77:0/798170059 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15d98b8640 1 -- 172.21.15.77:0/798170059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b3240 msgr2=0x5628b38bd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15d98b8640 1 --2- 172.21.15.77:0/798170059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b3240 0x5628b38bd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15d98b8640 1 -- 172.21.15.77:0/798170059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b2850 msgr2=0x5628b38bcce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15d98b8640 1 --2- 172.21.15.77:0/798170059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b2850 0x5628b38bcce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15d98b8640 1 -- 172.21.15.77:0/798170059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628b37e2100 con 0x5628b38b83b0 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15d88b6640 1 --2- 172.21.15.77:0/798170059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b3240 0x5628b38bd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15d98b8640 1 --2- 172.21.15.77:0/798170059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 0x5628b38c4250 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f15cc003cc0 tx=0x7f15cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293c3f0e3faf11a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:53.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.779+0000 7f15bcff9640 1 -- 172.21.15.77:0/798170059 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc025030 con 0x5628b38b83b0 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15bcff9640 1 -- 172.21.15.77:0/798170059 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f15cc00fe00 con 0x5628b38b83b0 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15ddfe5740 1 -- 172.21.15.77:0/798170059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15c4005040 con 0x5628b38b83b0 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15bcff9640 1 -- 172.21.15.77:0/798170059 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f15cc0045d0 con 0x5628b38b83b0 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15ddfe5740 10 client.17607 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15ddfe5740 1 -- 172.21.15.77:0/798170059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5628b352b3f0 con 0x5628b38b83b0 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15bcff9640 1 -- 172.21.15.77:0/798170059 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15c4005040 con 0x5628b38b83b0 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15bcff9640 1 -- 172.21.15.77:0/798170059 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f15cc01f040 con 0x5628b38b83b0 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15ddfe5740 10 client.17607 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15ddfe5740 10 client.17607 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15ddfe5740 20 client.17607 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/798170059 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5628b38c8bc0 0x5628b38cb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15ddfe5740 4 client.17607 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15ddfe5740 1 -- 172.21.15.77:0/798170059 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628b38249d0 con 0x5628b38c8bc0 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.780+0000 7f15d90b7640 1 --2- 172.21.15.77:0/798170059 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5628b38c8bc0 0x5628b38cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.781+0000 7f15bcff9640 10 client.17607 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:25:53.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.781+0000 7f15d90b7640 1 --2- 172.21.15.77:0/798170059 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5628b38c8bc0 0x5628b38cb060 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6581ee22d46c2dd6 server_cookie=1fe244bb0094330f in_seq=0 out_seq=0 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.781+0000 7f15bcff9640 1 -- 172.21.15.77:0/798170059 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5628b38249d0 con 0x5628b38c8bc0 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.781+0000 7f15bcff9640 10 client.17607 handle_command_reply: tid=0 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.781+0000 7f15ddfe5740 1 client.17607 shutdown 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.781+0000 7f15ddfe5740 1 -- 172.21.15.77:0/798170059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 msgr2=0x5628b38c4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.781+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/798170059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 0x5628b38c4250 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f15cc003cc0 tx=0x7f15cc004a40 comp rx=0 tx=0).stop 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.781+0000 7f15ddfe5740 1 -- 172.21.15.77:0/798170059 shutdown_connections 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.781+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/798170059 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5628b38c8bc0 0x5628b38cb060 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.782+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/798170059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 0x5628b38c4250 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.782+0000 7f15bcff9640 0 client.17607 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.782+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/798170059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b3240 0x5628b38bd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.782+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/798170059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b2850 0x5628b38bcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.782+0000 7f15ddfe5740 1 -- 172.21.15.77:0/798170059 >> 172.21.15.77:0/798170059 conn(0x5628b38addc0 msgr2=0x5628b38acfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.782+0000 7f15ddfe5740 1 -- 172.21.15.77:0/798170059 shutdown_connections 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.782+0000 7f15ddfe5740 1 -- 172.21.15.77:0/798170059 wait complete. 2024-03-16T10:25:53.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.782+0000 7f15ddfe5740 20 client.17607 trim_cache size 0 max 16384 2024-03-16T10:25:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15ddfe5740 1 Processor -- start 2024-03-16T10:25:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15ddfe5740 1 -- start start 2024-03-16T10:25:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15ddfe5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b2850 0x5628b38b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15ddfe5740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b3240 0x5628b38b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15ddfe5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b83b0 0x5628b38c4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15ddfe5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41a69a0 con 0x5628b38b2850 2024-03-16T10:25:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15ddfe5740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41a85a0 con 0x5628b38b3240 2024-03-16T10:25:53.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15ddfe5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41a70d0 con 0x5628b38b83b0 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15d90b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b2850 0x5628b38b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15d90b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b2850 0x5628b38b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41508/0 (socket says 172.21.15.77:41508) 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15d90b7640 1 -- 172.21.15.77:0/136347491 learned_addr learned my addr 172.21.15.77:0/136347491 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15d98b8640 1 --2- 172.21.15.77:0/136347491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b83b0 0x5628b38c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15d88b6640 1 --2- 172.21.15.77:0/136347491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b3240 0x5628b38b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15d90b7640 1 -- 172.21.15.77:0/136347491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b83b0 msgr2=0x5628b38c4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15d90b7640 1 --2- 172.21.15.77:0/136347491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b83b0 0x5628b38c4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15d90b7640 1 -- 172.21.15.77:0/136347491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b3240 msgr2=0x5628b38b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15d90b7640 1 --2- 172.21.15.77:0/136347491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b3240 0x5628b38b4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15d90b7640 1 -- 172.21.15.77:0/136347491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628b35f57d0 con 0x5628b38b2850 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.804+0000 7f15d88b6640 1 --2- 172.21.15.77:0/136347491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b3240 0x5628b38b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15d90b7640 1 --2- 172.21.15.77:0/136347491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b2850 0x5628b38b4700 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f15d00cc960 tx=0x7f15d0097b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29656a95dcdc3f1f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15beffd640 1 -- 172.21.15.77:0/136347491 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d009f900 con 0x5628b38b2850 2024-03-16T10:25:53.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15beffd640 1 -- 172.21.15.77:0/136347491 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15d00a3f50 con 0x5628b38b2850 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15beffd640 1 -- 172.21.15.77:0/136347491 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d00860c0 con 0x5628b38b2850 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15ddfe5740 1 -- 172.21.15.77:0/136347491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b2850 msgr2=0x5628b38b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/136347491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b2850 0x5628b38b4700 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f15d00cc960 tx=0x7f15d0097b80 comp rx=0 tx=0).stop 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15ddfe5740 1 -- 172.21.15.77:0/136347491 shutdown_connections 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/136347491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b83b0 0x5628b38c4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/136347491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b3240 0x5628b38b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/136347491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b2850 0x5628b38b4700 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15ddfe5740 1 -- 172.21.15.77:0/136347491 >> 172.21.15.77:0/136347491 conn(0x5628b38addc0 msgr2=0x5628b38ae1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15ddfe5740 1 -- 172.21.15.77:0/136347491 shutdown_connections 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.805+0000 7f15ddfe5740 1 -- 172.21.15.77:0/136347491 wait complete. 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15ddfe5740 1 -- start start 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15ddfe5740 1 Processor -- start 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15ddfe5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b2850 0x5628b38c4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15ddfe5740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b3240 0x5628b38c0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15ddfe5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 0x5628b38c1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.812 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15ddfe5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41a7ed0 con 0x5628b38b83b0 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15ddfe5740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41a77d0 con 0x5628b38b3240 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15ddfe5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41a4980 con 0x5628b38b2850 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15ddfe5740 10 client.0 _ll_register_callbacks cb 0x7f15cc003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15d98b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 0x5628b38c1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15d90b7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b2850 0x5628b38c4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15d88b6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b3240 0x5628b38c0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15d90b7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b2850 0x5628b38c4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50398/0 (socket says 172.21.15.77:50398) 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15d90b7640 1 -- 172.21.15.77:0/2239001095 learned_addr learned my addr 172.21.15.77:0/2239001095 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15d98b8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 0x5628b38c1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41520/0 (socket says 172.21.15.77:41520) 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15d90b7640 1 -- 172.21.15.77:0/2239001095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b3240 msgr2=0x5628b38c0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15d90b7640 1 --2- 172.21.15.77:0/2239001095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b3240 0x5628b38c0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.806+0000 7f15d90b7640 1 -- 172.21.15.77:0/2239001095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 msgr2=0x5628b38c1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15d90b7640 1 --2- 172.21.15.77:0/2239001095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 0x5628b38c1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15d90b7640 1 -- 172.21.15.77:0/2239001095 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628b392cf50 con 0x5628b38b2850 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15d98b8640 1 --2- 172.21.15.77:0/2239001095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 0x5628b38c1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15d90b7640 1 --2- 172.21.15.77:0/2239001095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b2850 0x5628b38c4120 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f15d00058c0 tx=0x7f15d009b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=809a42c71cc571fb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15bcff9640 1 -- 172.21.15.77:0/2239001095 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0099850 con 0x5628b38b2850 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15ddfe5740 1 -- 172.21.15.77:0/2239001095 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15d00a3f50 con 0x5628b38b2850 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15bcff9640 1 -- 172.21.15.77:0/2239001095 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f15d0086050 con 0x5628b38b2850 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15bcff9640 1 -- 172.21.15.77:0/2239001095 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f15d009f900 con 0x5628b38b2850 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15ddfe5740 10 client.27367 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15ddfe5740 1 -- 172.21.15.77:0/2239001095 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5628b392e1d0 con 0x5628b38b2850 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15bcff9640 1 -- 172.21.15.77:0/2239001095 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15d00a3f50 con 0x5628b38b2850 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15bcff9640 1 -- 172.21.15.77:0/2239001095 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f15d006cbd0 con 0x5628b38b2850 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15ddfe5740 10 client.27367 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:53.814 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15ddfe5740 10 client.27367 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15ddfe5740 20 client.27367 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/2239001095 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5628b392eb80 0x5628b3930fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15ddfe5740 4 client.27367 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.807+0000 7f15ddfe5740 1 -- 172.21.15.77:0/2239001095 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628b381ed20 con 0x5628b392eb80 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15d88b6640 1 --2- 172.21.15.77:0/2239001095 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5628b392eb80 0x5628b3930fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15bcff9640 10 client.27367 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15d88b6640 1 --2- 172.21.15.77:0/2239001095 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5628b392eb80 0x5628b3930fa0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=57589d9598426de8 server_cookie=77607750def5020 in_seq=0 out_seq=0 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15bcff9640 1 -- 172.21.15.77:0/2239001095 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628b381ed20 con 0x5628b392eb80 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15bcff9640 10 client.27367 handle_command_reply: tid=0 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15ddfe5740 1 client.27367 shutdown 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15ddfe5740 1 -- 172.21.15.77:0/2239001095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b2850 msgr2=0x5628b38c4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/2239001095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b2850 0x5628b38c4120 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f15d00058c0 tx=0x7f15d009b900 comp rx=0 tx=0).stop 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15ddfe5740 1 -- 172.21.15.77:0/2239001095 shutdown_connections 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/2239001095 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x5628b392eb80 0x5628b3930fa0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/2239001095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628b38b83b0 0x5628b38c1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/2239001095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5628b38b3240 0x5628b38c0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15bcff9640 0 client.27367 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:25:53.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15ddfe5740 1 --2- 172.21.15.77:0/2239001095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628b38b2850 0x5628b38c4120 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15ddfe5740 1 -- 172.21.15.77:0/2239001095 >> 172.21.15.77:0/2239001095 conn(0x5628b38addc0 msgr2=0x5628b38bb820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15ddfe5740 1 -- 172.21.15.77:0/2239001095 shutdown_connections 2024-03-16T10:25:53.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15ddfe5740 1 -- 172.21.15.77:0/2239001095 wait complete. 2024-03-16T10:25:53.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.808+0000 7f15ddfe5740 20 client.27367 trim_cache size 0 max 16384 2024-03-16T10:25:53.816 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:25:53.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.809+0000 7f15db31b640 1 -- 172.21.15.77:0/4243324994 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f15c00a2800 msgr2=0x7f15c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.809+0000 7f15db31b640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f15c00a2800 0x7f15c00a4cc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x5628b38b3aa0 tx=0x7f15cc004ab0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.809+0000 7f15db31b640 1 -- 172.21.15.77:0/4243324994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15d41017f0 msgr2=0x7f15d4197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.809+0000 7f15db31b640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15d41017f0 0x7f15d4197b10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f15d000e9c0 tx=0x7f15d000ee80 comp rx=0 tx=0).stop 2024-03-16T10:25:53.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.809+0000 7f15db31b640 1 -- 172.21.15.77:0/4243324994 shutdown_connections 2024-03-16T10:25:53.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.809+0000 7f15db31b640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f15c00a2800 0x7f15c00a4cc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.809+0000 7f15db31b640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15d41033a0 0x7f15d419f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.809+0000 7f15db31b640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15d41029e0 0x7f15d4198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.809+0000 7f15db31b640 1 --2- 172.21.15.77:0/4243324994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15d41017f0 0x7f15d4197b10 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.809+0000 7f15db31b640 1 -- 172.21.15.77:0/4243324994 >> 172.21.15.77:0/4243324994 conn(0x7f15d40fcd80 msgr2=0x7f15d4105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.809+0000 7f15db31b640 1 -- 172.21.15.77:0/4243324994 shutdown_connections 2024-03-16T10:25:53.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.809+0000 7f15db31b640 1 -- 172.21.15.77:0/4243324994 wait complete. 2024-03-16T10:25:53.820 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T10:25:53.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.929+0000 7fac59612640 1 -- 172.21.15.77:0/3992518611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac541033c0 msgr2=0x7fac54109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.929+0000 7fac59612640 1 --2- 172.21.15.77:0/3992518611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac541033c0 0x7fac54109c50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fac44009be0 tx=0x7fac4402bef0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.929+0000 7fac59612640 1 -- 172.21.15.77:0/3992518611 shutdown_connections 2024-03-16T10:25:53.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.929+0000 7fac59612640 1 --2- 172.21.15.77:0/3992518611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac541033c0 0x7fac54109c50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.929+0000 7fac59612640 1 --2- 172.21.15.77:0/3992518611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac54102a00 0x7fac54102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.929+0000 7fac59612640 1 --2- 172.21.15.77:0/3992518611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac54101810 0x7fac54101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.929+0000 7fac59612640 1 -- 172.21.15.77:0/3992518611 >> 172.21.15.77:0/3992518611 conn(0x7fac540fcd80 msgr2=0x7fac540ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.929+0000 7fac59612640 1 -- 172.21.15.77:0/3992518611 shutdown_connections 2024-03-16T10:25:53.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.929+0000 7fac59612640 1 -- 172.21.15.77:0/3992518611 wait complete. 2024-03-16T10:25:53.931 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.930+0000 7fac59612640 1 Processor -- start 2024-03-16T10:25:53.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.930+0000 7fac59612640 1 -- start start 2024-03-16T10:25:53.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.930+0000 7fac59612640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac54101810 0x7fac54197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.930+0000 7fac59612640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac54102a00 0x7fac54198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.930+0000 7fac59612640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac541033c0 0x7fac5419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.930+0000 7fac59612640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fac5410d490 con 0x7fac54101810 2024-03-16T10:25:53.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.930+0000 7fac59612640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fac5410d310 con 0x7fac541033c0 2024-03-16T10:25:53.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.930+0000 7fac59612640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fac5410d610 con 0x7fac54102a00 2024-03-16T10:25:53.933 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.930+0000 7fac52ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac54101810 0x7fac54197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac52ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac54101810 0x7fac54197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41544/0 (socket says 172.21.15.77:41544) 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac52ffd640 1 -- 172.21.15.77:0/2459049765 learned_addr learned my addr 172.21.15.77:0/2459049765 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac537fe640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac541033c0 0x7fac5419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac527fc640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac54102a00 0x7fac54198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac52ffd640 1 -- 172.21.15.77:0/2459049765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac54102a00 msgr2=0x7fac54198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac52ffd640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac54102a00 0x7fac54198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac52ffd640 1 -- 172.21.15.77:0/2459049765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac541033c0 msgr2=0x7fac5419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac52ffd640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac541033c0 0x7fac5419f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac52ffd640 1 -- 172.21.15.77:0/2459049765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5419f720 con 0x7fac54101810 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac52ffd640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac54101810 0x7fac54197b00 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00e9f0 tx=0x7fac4c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac3bfff640 1 -- 172.21.15.77:0/2459049765 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00ccc0 con 0x7fac54101810 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac59612640 1 -- 172.21.15.77:0/2459049765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5419fa10 con 0x7fac54101810 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac3bfff640 1 -- 172.21.15.77:0/2459049765 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac4c0035f0 con 0x7fac54101810 2024-03-16T10:25:53.934 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac3bfff640 1 -- 172.21.15.77:0/2459049765 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00ce60 con 0x7fac54101810 2024-03-16T10:25:53.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.931+0000 7fac59612640 1 -- 172.21.15.77:0/2459049765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5419ffb0 con 0x7fac54101810 2024-03-16T10:25:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac5c203740 1 Processor -- start 2024-03-16T10:25:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac5c203740 1 -- start start 2024-03-16T10:25:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac5c203740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eabc60 0x55cc22eac080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac5c203740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eac650 0x55cc22eb1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac5c203740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eb17c0 0x55cc22eb3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac5c203740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cc22e0c490 con 0x55cc22eac650 2024-03-16T10:25:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac5c203740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55cc22db4c90 con 0x55cc22eabc60 2024-03-16T10:25:53.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac5c203740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cc22e08530 con 0x55cc22eb17c0 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac537fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eabc60 0x55cc22eac080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac3bfff640 1 -- 172.21.15.77:0/2459049765 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fac4c0037a0 con 0x7fac54101810 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac537fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eabc60 0x55cc22eac080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:55696/0 (socket says 172.21.15.77:55696) 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac537fe640 1 -- 172.21.15.77:0/3102396035 learned_addr learned my addr 172.21.15.77:0/3102396035 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac527fc640 1 --2- 172.21.15.77:0/3102396035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eb17c0 0x55cc22eb3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac52ffd640 1 --2- 172.21.15.77:0/3102396035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eac650 0x55cc22eb1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac537fe640 1 -- 172.21.15.77:0/3102396035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eb17c0 msgr2=0x55cc22eb3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac537fe640 1 --2- 172.21.15.77:0/3102396035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eb17c0 0x55cc22eb3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac537fe640 1 -- 172.21.15.77:0/3102396035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eac650 msgr2=0x55cc22eb1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac537fe640 1 --2- 172.21.15.77:0/3102396035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eac650 0x55cc22eb1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac537fe640 1 -- 172.21.15.77:0/3102396035 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc22e6f500 con 0x55cc22eabc60 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.934+0000 7fac527fc640 1 --2- 172.21.15.77:0/3102396035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eb17c0 0x55cc22eb3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.935+0000 7fac537fe640 1 --2- 172.21.15.77:0/3102396035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eabc60 0x55cc22eac080 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fac44009be0 tx=0x7fac44005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8a9daf04c2525a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.935+0000 7fac3bfff640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fac3c0a2800 0x7fac3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.935+0000 7fac397fa640 1 -- 172.21.15.77:0/3102396035 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44043030 con 0x55cc22eabc60 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.935+0000 7fac52ffd640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fac3c0a2800 0x7fac3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.935+0000 7fac397fa640 1 -- 172.21.15.77:0/3102396035 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac44033ab0 con 0x55cc22eabc60 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.935+0000 7fac397fa640 1 -- 172.21.15.77:0/3102396035 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4402c890 con 0x55cc22eabc60 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.935+0000 7fac3bfff640 1 -- 172.21.15.77:0/2459049765 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fac4c0cec30 con 0x7fac54101810 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 -- 172.21.15.77:0/3102396035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eabc60 msgr2=0x55cc22eac080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac52ffd640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fac3c0a2800 0x7fac3c0a4cc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55cc22eaca50 tx=0x7fac4c0045d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 --2- 172.21.15.77:0/3102396035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eabc60 0x55cc22eac080 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fac44009be0 tx=0x7fac44005bd0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 -- 172.21.15.77:0/3102396035 shutdown_connections 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 --2- 172.21.15.77:0/3102396035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eb17c0 0x55cc22eb3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 --2- 172.21.15.77:0/3102396035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eac650 0x55cc22eb1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 --2- 172.21.15.77:0/3102396035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eabc60 0x55cc22eac080 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 -- 172.21.15.77:0/3102396035 >> 172.21.15.77:0/3102396035 conn(0x55cc22ea2dc0 msgr2=0x55cc22ea31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 -- 172.21.15.77:0/3102396035 shutdown_connections 2024-03-16T10:25:53.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 -- 172.21.15.77:0/3102396035 wait complete. 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 -- start start 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 Processor -- start 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eabc60 0x55cc22eb6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eac650 0x55cc22eb6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eb17c0 0x55cc22ebd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cc22eb4710 con 0x55cc22eac650 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55cc22dc8230 con 0x55cc22eb17c0 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cc22eb4890 con 0x55cc22eabc60 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac5c203740 10 client.0 _ll_register_callbacks cb 0x55cc22e04010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac527fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eac650 0x55cc22eb6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.936+0000 7fac537fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eabc60 0x55cc22eb6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac537fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eabc60 0x55cc22eb6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50432/0 (socket says 172.21.15.77:50432) 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac537fe640 1 -- 172.21.15.77:0/2820787051 learned_addr learned my addr 172.21.15.77:0/2820787051 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac537fe640 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eb17c0 0x55cc22ebd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac537fe640 1 -- 172.21.15.77:0/2820787051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eb17c0 msgr2=0x55cc22ebd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac537fe640 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eb17c0 0x55cc22ebd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac537fe640 1 -- 172.21.15.77:0/2820787051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eac650 msgr2=0x55cc22eb6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac537fe640 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eac650 0x55cc22eb6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac537fe640 1 -- 172.21.15.77:0/2820787051 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc22dd7100 con 0x55cc22eabc60 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac527fc640 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eac650 0x55cc22eb6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac537fe640 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eabc60 0x55cc22eb6110 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fac44003c10 tx=0x7fac44009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=232fcc53a1607207 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac36ff5640 1 -- 172.21.15.77:0/2820787051 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44043040 con 0x55cc22eabc60 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac36ff5640 1 -- 172.21.15.77:0/2820787051 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fac44004b40 con 0x55cc22eabc60 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac5c203740 1 -- 172.21.15.77:0/2820787051 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac44033ab0 con 0x55cc22eabc60 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac36ff5640 1 -- 172.21.15.77:0/2820787051 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fac4402ca10 con 0x55cc22eabc60 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac5c203740 10 client.27379 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.937+0000 7fac5c203740 1 -- 172.21.15.77:0/2820787051 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55cc22b203f0 con 0x55cc22eabc60 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.938+0000 7fac36ff5640 1 -- 172.21.15.77:0/2820787051 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac44033ab0 con 0x55cc22eabc60 2024-03-16T10:25:53.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.938+0000 7fac36ff5640 1 -- 172.21.15.77:0/2820787051 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7fac440344d0 con 0x55cc22eabc60 2024-03-16T10:25:53.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.938+0000 7fac5c203740 10 client.27379 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:53.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.938+0000 7fac5c203740 10 client.27379 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:25:53.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.938+0000 7fac5c203740 20 client.27379 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:53.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.938+0000 7fac5c203740 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55cc22ec1b60 0x55cc22ec3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.938+0000 7fac5c203740 4 client.27379 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:53.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.938+0000 7fac5c203740 1 -- 172.21.15.77:0/2820787051 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc22e199d0 con 0x55cc22ec1b60 2024-03-16T10:25:53.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.938+0000 7fac527fc640 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55cc22ec1b60 0x55cc22ec3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.938+0000 7fac36ff5640 10 client.27379 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:25:53.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.938+0000 7fac527fc640 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55cc22ec1b60 0x55cc22ec3f50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=85db1186cb4d3939 server_cookie=b09808285821d78a in_seq=0 out_seq=0 2024-03-16T10:25:53.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac36ff5640 1 -- 172.21.15.77:0/2820787051 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cc22e199d0 con 0x55cc22ec1b60 2024-03-16T10:25:53.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac36ff5640 10 client.27379 handle_command_reply: tid=0 2024-03-16T10:25:53.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac5c203740 1 client.27379 shutdown 2024-03-16T10:25:53.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac5c203740 1 -- 172.21.15.77:0/2820787051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eabc60 msgr2=0x55cc22eb6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac5c203740 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eabc60 0x55cc22eb6110 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fac44003c10 tx=0x7fac44009500 comp rx=0 tx=0).stop 2024-03-16T10:25:53.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac5c203740 1 -- 172.21.15.77:0/2820787051 shutdown_connections 2024-03-16T10:25:53.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac5c203740 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55cc22ec1b60 0x55cc22ec3f50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac5c203740 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eb17c0 0x55cc22ebd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac36ff5640 0 client.27379 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:25:53.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac5c203740 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eac650 0x55cc22eb6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac5c203740 1 --2- 172.21.15.77:0/2820787051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eabc60 0x55cc22eb6110 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac5c203740 1 -- 172.21.15.77:0/2820787051 >> 172.21.15.77:0/2820787051 conn(0x55cc22ea2dc0 msgr2=0x55cc22ea1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac5c203740 1 -- 172.21.15.77:0/2820787051 shutdown_connections 2024-03-16T10:25:53.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac5c203740 1 -- 172.21.15.77:0/2820787051 wait complete. 2024-03-16T10:25:53.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.940+0000 7fac5c203740 20 client.27379 trim_cache size 0 max 16384 2024-03-16T10:25:53.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.954+0000 7fac5c203740 1 Processor -- start 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.954+0000 7fac5c203740 1 -- start start 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac5c203740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 0x55cc22eb6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac5c203740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eac650 0x55cc22ebd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac5c203740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eb17c0 0x55cc22ea1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac5c203740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fac541a69a0 con 0x55cc22eabc60 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac5c203740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fac541a85a0 con 0x55cc22eb17c0 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac5c203740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fac541a70d0 con 0x55cc22eac650 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 0x55cc22eb6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 0x55cc22eb6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41568/0 (socket says 172.21.15.77:41568) 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac537fe640 1 -- 172.21.15.77:0/2058104100 learned_addr learned my addr 172.21.15.77:0/2058104100 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac527fc640 1 --2- 172.21.15.77:0/2058104100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eac650 0x55cc22ebd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac537fe640 1 --2- 172.21.15.77:0/2058104100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eb17c0 0x55cc22ea1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac537fe640 1 -- 172.21.15.77:0/2058104100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eac650 msgr2=0x55cc22ebd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac537fe640 1 --2- 172.21.15.77:0/2058104100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eac650 0x55cc22ebd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac537fe640 1 -- 172.21.15.77:0/2058104100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eb17c0 msgr2=0x55cc22ea1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac537fe640 1 --2- 172.21.15.77:0/2058104100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eb17c0 0x55cc22ea1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac537fe640 1 -- 172.21.15.77:0/2058104100 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc22bea7d0 con 0x55cc22eabc60 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac527fc640 1 --2- 172.21.15.77:0/2058104100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eac650 0x55cc22ebd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac537fe640 1 --2- 172.21.15.77:0/2058104100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 0x55cc22eb6430 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fac44002230 tx=0x7fac4402bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c106ee1257d75db0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac38ff9640 1 -- 172.21.15.77:0/2058104100 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44043070 con 0x55cc22eabc60 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac38ff9640 1 -- 172.21.15.77:0/2058104100 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac44034b60 con 0x55cc22eabc60 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.955+0000 7fac38ff9640 1 -- 172.21.15.77:0/2058104100 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4405a040 con 0x55cc22eabc60 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.956+0000 7fac5c203740 1 -- 172.21.15.77:0/2058104100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 msgr2=0x55cc22eb6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.956+0000 7fac5c203740 1 --2- 172.21.15.77:0/2058104100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 0x55cc22eb6430 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fac44002230 tx=0x7fac4402bef0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.956+0000 7fac5c203740 1 -- 172.21.15.77:0/2058104100 shutdown_connections 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.956+0000 7fac5c203740 1 --2- 172.21.15.77:0/2058104100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eb17c0 0x55cc22ea1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.956+0000 7fac5c203740 1 --2- 172.21.15.77:0/2058104100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eac650 0x55cc22ebd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.956+0000 7fac5c203740 1 --2- 172.21.15.77:0/2058104100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 0x55cc22eb6430 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.956+0000 7fac5c203740 1 -- 172.21.15.77:0/2058104100 >> 172.21.15.77:0/2058104100 conn(0x55cc22ea2dc0 msgr2=0x55cc22eb5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.956+0000 7fac5c203740 1 -- 172.21.15.77:0/2058104100 shutdown_connections 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.956+0000 7fac5c203740 1 -- 172.21.15.77:0/2058104100 wait complete. 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac5c203740 1 -- start start 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac5c203740 1 Processor -- start 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac5c203740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 0x55cc22eba500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac5c203740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eac650 0x55cc22ebaa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac5c203740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eb17c0 0x55cc22f25d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac5c203740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fac541a7ed0 con 0x55cc22eabc60 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac5c203740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fac541a77d0 con 0x55cc22eac650 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac5c203740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fac541a49d0 con 0x55cc22eb17c0 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac5c203740 10 client.0 _ll_register_callbacks cb 0x7fac48009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 0x55cc22eba500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 0x55cc22eba500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41578/0 (socket says 172.21.15.77:41578) 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac537fe640 1 -- 172.21.15.77:0/2796413761 learned_addr learned my addr 172.21.15.77:0/2796413761 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac527fc640 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eac650 0x55cc22ebaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac537fe640 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eb17c0 0x55cc22f25d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac537fe640 1 -- 172.21.15.77:0/2796413761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eb17c0 msgr2=0x55cc22f25d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac537fe640 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eb17c0 0x55cc22f25d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac537fe640 1 -- 172.21.15.77:0/2796413761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eac650 msgr2=0x55cc22ebaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac537fe640 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eac650 0x55cc22ebaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac537fe640 1 -- 172.21.15.77:0/2796413761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc22f262c0 con 0x55cc22eabc60 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac537fe640 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 0x55cc22eba500 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fac44034ae0 tx=0x7fac4402c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f7724a8304eb4f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.957+0000 7fac36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac36ff5640 1 -- 172.21.15.77:0/2796413761 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4405a030 con 0x55cc22eabc60 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac36ff5640 1 -- 172.21.15.77:0/2796413761 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fac4402cdb0 con 0x55cc22eabc60 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac5c203740 1 -- 172.21.15.77:0/2796413761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac44034b60 con 0x55cc22eabc60 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac36ff5640 1 -- 172.21.15.77:0/2796413761 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fac44043070 con 0x55cc22eabc60 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac5c203740 10 client.17631 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac5c203740 1 -- 172.21.15.77:0/2796413761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55cc22f27440 con 0x55cc22eabc60 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac36ff5640 1 -- 172.21.15.77:0/2796413761 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac44034b60 con 0x55cc22eabc60 2024-03-16T10:25:53.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac36ff5640 1 -- 172.21.15.77:0/2796413761 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7fac44004560 con 0x55cc22eabc60 2024-03-16T10:25:53.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac5c203740 10 client.17631 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:53.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac5c203740 10 client.17631 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:25:53.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac5c203740 20 client.17631 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:53.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac5c203740 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55cc22ec1b60 0x55cc22f29db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:53.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac5c203740 4 client.17631 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:25:53.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac5c203740 1 -- 172.21.15.77:0/2796413761 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cc22ec44f0 con 0x55cc22ec1b60 2024-03-16T10:25:53.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.958+0000 7fac527fc640 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55cc22ec1b60 0x55cc22f29db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:53.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.959+0000 7fac36ff5640 10 client.17631 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:25:53.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.959+0000 7fac527fc640 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55cc22ec1b60 0x55cc22f29db0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dd796c3027798b30 server_cookie=a39bc1a8ff222add in_seq=0 out_seq=0 2024-03-16T10:25:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.960+0000 7fac36ff5640 1 -- 172.21.15.77:0/2796413761 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cc22ec44f0 con 0x55cc22ec1b60 2024-03-16T10:25:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.960+0000 7fac36ff5640 10 client.17631 handle_command_reply: tid=0 2024-03-16T10:25:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.960+0000 7fac5c203740 1 client.17631 shutdown 2024-03-16T10:25:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.960+0000 7fac5c203740 1 -- 172.21.15.77:0/2796413761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 msgr2=0x55cc22eba500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.960+0000 7fac5c203740 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 0x55cc22eba500 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fac44034ae0 tx=0x7fac4402c890 comp rx=0 tx=0).stop 2024-03-16T10:25:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.960+0000 7fac5c203740 1 -- 172.21.15.77:0/2796413761 shutdown_connections 2024-03-16T10:25:53.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.960+0000 7fac5c203740 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc22eb17c0 0x55cc22f25d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.960+0000 7fac5c203740 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x55cc22ec1b60 0x55cc22f29db0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.960+0000 7fac5c203740 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cc22eac650 0x55cc22ebaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.960+0000 7fac36ff5640 0 client.17631 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:25:53.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.960+0000 7fac5c203740 1 --2- 172.21.15.77:0/2796413761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cc22eabc60 0x55cc22eba500 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.960+0000 7fac5c203740 1 -- 172.21.15.77:0/2796413761 >> 172.21.15.77:0/2796413761 conn(0x55cc22ea2dc0 msgr2=0x55cc22ea6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac5c203740 1 -- 172.21.15.77:0/2796413761 shutdown_connections 2024-03-16T10:25:53.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac5c203740 1 -- 172.21.15.77:0/2796413761 wait complete. 2024-03-16T10:25:53.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac5c203740 20 client.17631 trim_cache size 0 max 16384 2024-03-16T10:25:53.966 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:25:53.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac59612640 1 -- 172.21.15.77:0/2459049765 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fac3c0a2800 msgr2=0x7fac3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac59612640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fac3c0a2800 0x7fac3c0a4cc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55cc22eaca50 tx=0x7fac4c0045d0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac59612640 1 -- 172.21.15.77:0/2459049765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac54101810 msgr2=0x7fac54197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:53.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac59612640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac54101810 0x7fac54197b00 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00e9f0 tx=0x7fac4c00eeb0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac59612640 1 -- 172.21.15.77:0/2459049765 shutdown_connections 2024-03-16T10:25:53.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac59612640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac541033c0 0x7fac5419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac59612640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fac3c0a2800 0x7fac3c0a4cc0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac59612640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac54102a00 0x7fac54198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac59612640 1 --2- 172.21.15.77:0/2459049765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac54101810 0x7fac54197b00 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:53.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac59612640 1 -- 172.21.15.77:0/2459049765 >> 172.21.15.77:0/2459049765 conn(0x7fac540fcd80 msgr2=0x7fac54105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:53.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac59612640 1 -- 172.21.15.77:0/2459049765 shutdown_connections 2024-03-16T10:25:53.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:53.961+0000 7fac59612640 1 -- 172.21.15.77:0/2459049765 wait complete. 2024-03-16T10:25:53.972 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T10:25:54.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.088+0000 7f6d84859640 1 -- 172.21.15.77:0/2060555783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d801006e0 msgr2=0x7f6d80100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.088+0000 7f6d84859640 1 --2- 172.21.15.77:0/2060555783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d801006e0 0x7f6d80100b00 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009ca0 tx=0x7f6d7002c100 comp rx=0 tx=0).stop 2024-03-16T10:25:54.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.088+0000 7f6d84859640 1 -- 172.21.15.77:0/2060555783 shutdown_connections 2024-03-16T10:25:54.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.088+0000 7f6d84859640 1 --2- 172.21.15.77:0/2060555783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d80102290 0x7f6d80108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.088+0000 7f6d84859640 1 --2- 172.21.15.77:0/2060555783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d801018d0 0x7f6d80101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.088+0000 7f6d84859640 1 --2- 172.21.15.77:0/2060555783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d801006e0 0x7f6d80100b00 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.088+0000 7f6d84859640 1 -- 172.21.15.77:0/2060555783 >> 172.21.15.77:0/2060555783 conn(0x7f6d800fbc70 msgr2=0x7f6d800fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:54.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.089+0000 7f6d84859640 1 -- 172.21.15.77:0/2060555783 shutdown_connections 2024-03-16T10:25:54.090 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.089+0000 7f6d84859640 1 -- 172.21.15.77:0/2060555783 wait complete. 2024-03-16T10:25:54.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.089+0000 7f6d84859640 1 Processor -- start 2024-03-16T10:25:54.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.089+0000 7f6d84859640 1 -- start start 2024-03-16T10:25:54.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.089+0000 7f6d84859640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d801006e0 0x7f6d80197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.089+0000 7f6d84859640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d801018d0 0x7f6d801980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.089+0000 7f6d84859640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d80102290 0x7f6d8019f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.089+0000 7f6d84859640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8010c330 con 0x7f6d80102290 2024-03-16T10:25:54.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.089+0000 7f6d84859640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8010c1b0 con 0x7f6d801018d0 2024-03-16T10:25:54.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.089+0000 7f6d84859640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8010c4b0 con 0x7f6d801006e0 2024-03-16T10:25:54.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d7ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d80102290 0x7f6d8019f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d7ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d80102290 0x7f6d8019f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41606/0 (socket says 172.21.15.77:41606) 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d7ed9d640 1 -- 172.21.15.77:0/3824500959 learned_addr learned my addr 172.21.15.77:0/3824500959 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d801018d0 0x7f6d801980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d7e59c640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d801006e0 0x7f6d80197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d7ed9d640 1 -- 172.21.15.77:0/3824500959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d801006e0 msgr2=0x7f6d80197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d7ed9d640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d801006e0 0x7f6d80197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d7ed9d640 1 -- 172.21.15.77:0/3824500959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d801018d0 msgr2=0x7f6d801980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d7ed9d640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d801018d0 0x7f6d801980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d7ed9d640 1 -- 172.21.15.77:0/3824500959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d8019f7a0 con 0x7f6d80102290 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d7ed9d640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d80102290 0x7f6d8019f130 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800d950 tx=0x7f6d7800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d6f7fe640 1 -- 172.21.15.77:0/3824500959 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78017030 con 0x7f6d80102290 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d84859640 1 -- 172.21.15.77:0/3824500959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d8019fa90 con 0x7f6d80102290 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d6f7fe640 1 -- 172.21.15.77:0/3824500959 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d78004020 con 0x7f6d80102290 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d6f7fe640 1 -- 172.21.15.77:0/3824500959 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78003c00 con 0x7f6d80102290 2024-03-16T10:25:54.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.090+0000 7f6d84859640 1 -- 172.21.15.77:0/3824500959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d8019ff90 con 0x7f6d80102290 2024-03-16T10:25:54.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d6f7fe640 1 -- 172.21.15.77:0/3824500959 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f6d7801a070 con 0x7f6d80102290 2024-03-16T10:25:54.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d87443740 1 Processor -- start 2024-03-16T10:25:54.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d87443740 1 -- start start 2024-03-16T10:25:54.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d87443740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf2850 0x55829fbf2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d87443740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf3240 0x55829fbf7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d87443740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf83b0 0x55829fbfa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d87443740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55829fb57490 con 0x55829fbf83b0 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d87443740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55829faffc90 con 0x55829fbf2850 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d87443740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55829fb53530 con 0x55829fbf3240 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d7e59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf3240 0x55829fbf7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d7dd9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf2850 0x55829fbf2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d7dd9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf2850 0x55829fbf2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:55750/0 (socket says 172.21.15.77:55750) 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d7e59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf3240 0x55829fbf7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50474/0 (socket says 172.21.15.77:50474) 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.093+0000 7f6d7dd9b640 1 -- 172.21.15.77:0/2256889046 learned_addr learned my addr 172.21.15.77:0/2256889046 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/2256889046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf83b0 0x55829fbfa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d6f7fe640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6d680a2800 0x7f6d680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d7dd9b640 1 -- 172.21.15.77:0/2256889046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf3240 msgr2=0x55829fbf7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/2256889046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf3240 0x55829fbf7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d7dd9b640 1 -- 172.21.15.77:0/2256889046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf83b0 msgr2=0x55829fbfa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/2256889046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf83b0 0x55829fbfa840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d7dd9b640 1 -- 172.21.15.77:0/2256889046 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55829fbba500 con 0x55829fbf2850 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d7e59c640 1 --2- 172.21.15.77:0/2256889046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf3240 0x55829fbf7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d7ed9d640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6d680a2800 0x7f6d680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d6f7fe640 1 -- 172.21.15.77:0/3824500959 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6d780ce280 con 0x7f6d80102290 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/2256889046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf2850 0x55829fbf2c70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6d740127c0 tx=0x7f6d7400b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f19152cf3165a6c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d6cff9640 1 -- 172.21.15.77:0/2256889046 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d74012e00 con 0x55829fbf2850 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d6cff9640 1 -- 172.21.15.77:0/2256889046 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d74010c50 con 0x55829fbf2850 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d6cff9640 1 -- 172.21.15.77:0/2256889046 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7401eb70 con 0x55829fbf2850 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.094+0000 7f6d7ed9d640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6d680a2800 0x7f6d680a4cc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6d780970b0 tx=0x7f6d78065a10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.095+0000 7f6d87443740 1 -- 172.21.15.77:0/2256889046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf2850 msgr2=0x55829fbf2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.095+0000 7f6d87443740 1 --2- 172.21.15.77:0/2256889046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf2850 0x55829fbf2c70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6d740127c0 tx=0x7f6d7400b7c0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.095+0000 7f6d87443740 1 -- 172.21.15.77:0/2256889046 shutdown_connections 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.095+0000 7f6d87443740 1 --2- 172.21.15.77:0/2256889046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf83b0 0x55829fbfa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.095+0000 7f6d87443740 1 --2- 172.21.15.77:0/2256889046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf3240 0x55829fbf7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.095+0000 7f6d87443740 1 --2- 172.21.15.77:0/2256889046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf2850 0x55829fbf2c70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.095+0000 7f6d87443740 1 -- 172.21.15.77:0/2256889046 >> 172.21.15.77:0/2256889046 conn(0x55829fbeddc0 msgr2=0x55829fbee1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.095+0000 7f6d87443740 1 -- 172.21.15.77:0/2256889046 shutdown_connections 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.095+0000 7f6d87443740 1 -- 172.21.15.77:0/2256889046 wait complete. 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.095+0000 7f6d87443740 1 -- start start 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d87443740 1 Processor -- start 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d87443740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf2850 0x55829fbfcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d87443740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf3240 0x55829fbfd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d87443740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf83b0 0x55829fc04290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d87443740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55829fbfb360 con 0x55829fbf2850 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d87443740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55829fb13230 con 0x55829fbf83b0 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d87443740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55829fbfb4e0 con 0x55829fbf3240 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d87443740 10 client.0 _ll_register_callbacks cb 0x55829fb4f010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d7dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf2850 0x55829fbfcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d7dd9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf2850 0x55829fbfcd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41614/0 (socket says 172.21.15.77:41614) 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d7dd9b640 1 -- 172.21.15.77:0/2540036481 learned_addr learned my addr 172.21.15.77:0/2540036481 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d7e59c640 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf3240 0x55829fbfd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf83b0 0x55829fc04290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d7dd9b640 1 -- 172.21.15.77:0/2540036481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf3240 msgr2=0x55829fbfd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf3240 0x55829fbfd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d7dd9b640 1 -- 172.21.15.77:0/2540036481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf83b0 msgr2=0x55829fc04290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf83b0 0x55829fc04290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d7dd9b640 1 -- 172.21.15.77:0/2540036481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55829fb22100 con 0x55829fbf2850 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf2850 0x55829fbfcd50 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6d7400b720 tx=0x7f6d74005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692e0bd5775f6059 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.096+0000 7f6d65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d65ffb640 1 -- 172.21.15.77:0/2540036481 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d74024040 con 0x55829fbf2850 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d65ffb640 1 -- 172.21.15.77:0/2540036481 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6d74010950 con 0x55829fbf2850 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d87443740 1 -- 172.21.15.77:0/2540036481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d74010c50 con 0x55829fbf2850 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d65ffb640 1 -- 172.21.15.77:0/2540036481 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f6d74010e00 con 0x55829fbf2850 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d87443740 10 client.17643 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d87443740 1 -- 172.21.15.77:0/2540036481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55829f86b3f0 con 0x55829fbf2850 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d65ffb640 1 -- 172.21.15.77:0/2540036481 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d74010c50 con 0x55829fbf2850 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d65ffb640 1 -- 172.21.15.77:0/2540036481 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f6d740054a0 con 0x55829fbf2850 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d87443740 10 client.17643 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:54.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d87443740 10 client.17643 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d87443740 20 client.17643 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d87443740 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55829fc08960 0x55829fc0ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d87443740 4 client.17643 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d87443740 1 -- 172.21.15.77:0/2540036481 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55829fb649d0 con 0x55829fc08960 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.097+0000 7f6d7e59c640 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55829fc08960 0x55829fc0ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.098+0000 7f6d65ffb640 10 client.17643 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.098+0000 7f6d7e59c640 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55829fc08960 0x55829fc0ada0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=80b12a1ddbc59493 server_cookie=461b5ef84583aa32 in_seq=0 out_seq=0 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.098+0000 7f6d65ffb640 1 -- 172.21.15.77:0/2540036481 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55829fb649d0 con 0x55829fc08960 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.098+0000 7f6d65ffb640 10 client.17643 handle_command_reply: tid=0 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.098+0000 7f6d87443740 1 client.17643 shutdown 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.098+0000 7f6d87443740 1 -- 172.21.15.77:0/2540036481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf2850 msgr2=0x55829fbfcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.098+0000 7f6d87443740 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf2850 0x55829fbfcd50 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6d7400b720 tx=0x7f6d74005c90 comp rx=0 tx=0).stop 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.099+0000 7f6d87443740 1 -- 172.21.15.77:0/2540036481 shutdown_connections 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.099+0000 7f6d87443740 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55829fc08960 0x55829fc0ada0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.099+0000 7f6d87443740 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf83b0 0x55829fc04290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.099+0000 7f6d87443740 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf3240 0x55829fbfd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.099+0000 7f6d65ffb640 0 client.17643 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:25:54.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.099+0000 7f6d87443740 1 --2- 172.21.15.77:0/2540036481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf2850 0x55829fbfcd50 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.099+0000 7f6d87443740 1 -- 172.21.15.77:0/2540036481 >> 172.21.15.77:0/2540036481 conn(0x55829fbeddc0 msgr2=0x55829fbecfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:54.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.099+0000 7f6d87443740 1 -- 172.21.15.77:0/2540036481 shutdown_connections 2024-03-16T10:25:54.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.099+0000 7f6d87443740 1 -- 172.21.15.77:0/2540036481 wait complete. 2024-03-16T10:25:54.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.099+0000 7f6d87443740 20 client.17643 trim_cache size 0 max 16384 2024-03-16T10:25:54.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d87443740 1 Processor -- start 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d87443740 1 -- start start 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d87443740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 0x55829fbfccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d87443740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf3240 0x55829fbfd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d87443740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf83b0 0x55829fc04250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d87443740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d801a8b90 con 0x55829fbf83b0 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d87443740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6d801aa790 con 0x55829fbf3240 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d87443740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6d801a92c0 con 0x55829fbf2850 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d7dd9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 0x55829fbfccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d7dd9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 0x55829fbfccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50494/0 (socket says 172.21.15.77:50494) 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d7e59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf3240 0x55829fbfd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d7dd9b640 1 -- 172.21.15.77:0/170749563 learned_addr learned my addr 172.21.15.77:0/170749563 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/170749563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf83b0 0x55829fc04250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d7dd9b640 1 -- 172.21.15.77:0/170749563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf3240 msgr2=0x55829fbfd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/170749563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf3240 0x55829fbfd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d7dd9b640 1 -- 172.21.15.77:0/170749563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf83b0 msgr2=0x55829fc04250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/170749563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf83b0 0x55829fc04250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.114+0000 7f6d7dd9b640 1 -- 172.21.15.77:0/170749563 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55829f9357d0 con 0x55829fbf2850 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/170749563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 0x55829fbfccf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6d74001eb0 tx=0x7f6d74005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0b485b7b49ef41c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d67fff640 1 -- 172.21.15.77:0/170749563 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d74004cb0 con 0x55829fbf2850 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d67fff640 1 -- 172.21.15.77:0/170749563 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d74016030 con 0x55829fbf2850 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d67fff640 1 -- 172.21.15.77:0/170749563 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d74010bb0 con 0x55829fbf2850 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d87443740 1 -- 172.21.15.77:0/170749563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 msgr2=0x55829fbfccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d87443740 1 --2- 172.21.15.77:0/170749563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 0x55829fbfccf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6d74001eb0 tx=0x7f6d74005c90 comp rx=0 tx=0).stop 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d87443740 1 -- 172.21.15.77:0/170749563 shutdown_connections 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d87443740 1 --2- 172.21.15.77:0/170749563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf83b0 0x55829fc04250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d87443740 1 --2- 172.21.15.77:0/170749563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf3240 0x55829fbfd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d87443740 1 --2- 172.21.15.77:0/170749563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 0x55829fbfccf0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d87443740 1 -- 172.21.15.77:0/170749563 >> 172.21.15.77:0/170749563 conn(0x55829fbeddc0 msgr2=0x55829fbf4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d87443740 1 -- 172.21.15.77:0/170749563 shutdown_connections 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.115+0000 7f6d87443740 1 -- 172.21.15.77:0/170749563 wait complete. 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d87443740 1 -- start start 2024-03-16T10:25:54.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d87443740 1 Processor -- start 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d87443740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 0x55829fc01030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d87443740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf3240 0x55829fc01570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d87443740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf83b0 0x55829fc6ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d87443740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d801aa0c0 con 0x55829fbf3240 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d87443740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6d801a99c0 con 0x55829fbf83b0 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d87443740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6d801a6b70 con 0x55829fbf2850 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d87443740 10 client.0 _ll_register_callbacks cb 0x7f6d7400be90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d7e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf3240 0x55829fc01570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d7dd9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 0x55829fc01030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d7e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf3240 0x55829fc01570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41632/0 (socket says 172.21.15.77:41632) 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d7e59c640 1 -- 172.21.15.77:0/936470190 learned_addr learned my addr 172.21.15.77:0/936470190 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.116+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf83b0 0x55829fc6ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d7e59c640 1 -- 172.21.15.77:0/936470190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 msgr2=0x55829fc01030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d7e59c640 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 0x55829fc01030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d7e59c640 1 -- 172.21.15.77:0/936470190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf83b0 msgr2=0x55829fc6ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d7e59c640 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf83b0 0x55829fc6ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d7e59c640 1 -- 172.21.15.77:0/936470190 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55829fc01b80 con 0x55829fbf3240 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 0x55829fc01030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d7e59c640 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf3240 0x55829fc01570 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009ca0 tx=0x7f6d70005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd89dcb0d5d21a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d65ffb640 1 -- 172.21.15.77:0/936470190 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d70051060 con 0x55829fbf3240 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d65ffb640 1 -- 172.21.15.77:0/936470190 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f6d70032750 con 0x55829fbf3240 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d87443740 1 -- 172.21.15.77:0/936470190 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d74016030 con 0x55829fbf3240 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d65ffb640 1 -- 172.21.15.77:0/936470190 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f6d70039040 con 0x55829fbf3240 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d87443740 10 client.17649 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d87443740 1 -- 172.21.15.77:0/936470190 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55829fc6e3a0 con 0x55829fbf3240 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d65ffb640 1 -- 172.21.15.77:0/936470190 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d74016030 con 0x55829fbf3240 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.117+0000 7f6d65ffb640 1 -- 172.21.15.77:0/936470190 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f6d7004b070 con 0x55829fbf3240 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 10 client.17649 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:54.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 10 client.17649 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 20 client.17649 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55829fc08960 0x55829fc70d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 4 client.17649 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 1 -- 172.21.15.77:0/936470190 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55829fb5ed20 con 0x55829fc08960 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55829fc08960 0x55829fc70d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d65ffb640 10 client.17649 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d7dd9b640 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55829fc08960 0x55829fc70d50 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d67fe96ff57136e3 server_cookie=2ab29af23ceb4f0c in_seq=0 out_seq=0 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d65ffb640 1 -- 172.21.15.77:0/936470190 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55829fb5ed20 con 0x55829fc08960 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d65ffb640 10 client.17649 handle_command_reply: tid=0 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 1 client.17649 shutdown 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 1 -- 172.21.15.77:0/936470190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf3240 msgr2=0x55829fc01570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf3240 0x55829fc01570 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009ca0 tx=0x7f6d70005fd0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 1 -- 172.21.15.77:0/936470190 shutdown_connections 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x55829fc08960 0x55829fc70d50 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829fbf83b0 0x55829fc6ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.124 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55829fbf3240 0x55829fc01570 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d65ffb640 0 client.17649 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:25:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 1 --2- 172.21.15.77:0/936470190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55829fbf2850 0x55829fc01030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 1 -- 172.21.15.77:0/936470190 >> 172.21.15.77:0/936470190 conn(0x55829fbeddc0 msgr2=0x55829fbf4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.118+0000 7f6d87443740 1 -- 172.21.15.77:0/936470190 shutdown_connections 2024-03-16T10:25:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d87443740 1 -- 172.21.15.77:0/936470190 wait complete. 2024-03-16T10:25:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d87443740 20 client.17649 trim_cache size 0 max 16384 2024-03-16T10:25:54.125 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:25:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d84859640 1 -- 172.21.15.77:0/3824500959 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6d680a2800 msgr2=0x7f6d680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d84859640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6d680a2800 0x7f6d680a4cc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6d780970b0 tx=0x7f6d78065a10 comp rx=0 tx=0).stop 2024-03-16T10:25:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d84859640 1 -- 172.21.15.77:0/3824500959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d80102290 msgr2=0x7f6d8019f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d84859640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d80102290 0x7f6d8019f130 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800d950 tx=0x7f6d7800de10 comp rx=0 tx=0).stop 2024-03-16T10:25:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d84859640 1 -- 172.21.15.77:0/3824500959 shutdown_connections 2024-03-16T10:25:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d84859640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d80102290 0x7f6d8019f130 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d84859640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d801018d0 0x7f6d801980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d84859640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f6d680a2800 0x7f6d680a4cc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d84859640 1 --2- 172.21.15.77:0/3824500959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d801006e0 0x7f6d80197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d84859640 1 -- 172.21.15.77:0/3824500959 >> 172.21.15.77:0/3824500959 conn(0x7f6d800fbc70 msgr2=0x7f6d80068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d84859640 1 -- 172.21.15.77:0/3824500959 shutdown_connections 2024-03-16T10:25:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.119+0000 7f6d84859640 1 -- 172.21.15.77:0/3824500959 wait complete. 2024-03-16T10:25:54.129 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:53 smithi077 ceph-mon[32400]: pgmap v615: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 429 op/s 2024-03-16T10:25:54.129 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:53 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/716999728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:25:54.131 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T10:25:54.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.238+0000 7fa502577640 1 -- 172.21.15.77:0/997060890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4fc1022b0 msgr2=0x7fa4fc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.238+0000 7fa502577640 1 --2- 172.21.15.77:0/997060890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4fc1022b0 0x7fa4fc108b40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4009ca0 tx=0x7fa4f402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.238+0000 7fa502577640 1 -- 172.21.15.77:0/997060890 shutdown_connections 2024-03-16T10:25:54.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.238+0000 7fa502577640 1 --2- 172.21.15.77:0/997060890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4fc1022b0 0x7fa4fc108b40 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.238+0000 7fa502577640 1 --2- 172.21.15.77:0/997060890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4fc1018f0 0x7fa4fc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.238+0000 7fa502577640 1 --2- 172.21.15.77:0/997060890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4fc100700 0x7fa4fc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.238+0000 7fa502577640 1 -- 172.21.15.77:0/997060890 >> 172.21.15.77:0/997060890 conn(0x7fa4fc0fbc90 msgr2=0x7fa4fc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:54.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.238+0000 7fa502577640 1 -- 172.21.15.77:0/997060890 shutdown_connections 2024-03-16T10:25:54.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.238+0000 7fa502577640 1 -- 172.21.15.77:0/997060890 wait complete. 2024-03-16T10:25:54.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa502577640 1 Processor -- start 2024-03-16T10:25:54.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa502577640 1 -- start start 2024-03-16T10:25:54.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa502577640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4fc100700 0x7fa4fc104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa502577640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4fc1018f0 0x7fa4fc102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa502577640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4fc1022b0 0x7fa4fc103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa502577640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc10c360 con 0x7fa4fc1018f0 2024-03-16T10:25:54.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa502577640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc10c1e0 con 0x7fa4fc100700 2024-03-16T10:25:54.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa502577640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc10c4e0 con 0x7fa4fc1022b0 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa4fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4fc1018f0 0x7fa4fc102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa500b14640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4fc1022b0 0x7fa4fc103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa4fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4fc1018f0 0x7fa4fc102cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41648/0 (socket says 172.21.15.77:41648) 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa4fb7fe640 1 -- 172.21.15.77:0/555103003 learned_addr learned my addr 172.21.15.77:0/555103003 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.239+0000 7fa4fbfff640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4fc100700 0x7fa4fc104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.240+0000 7fa500b14640 1 -- 172.21.15.77:0/555103003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4fc100700 msgr2=0x7fa4fc104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.240+0000 7fa500b14640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4fc100700 0x7fa4fc104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.240+0000 7fa500b14640 1 -- 172.21.15.77:0/555103003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4fc1018f0 msgr2=0x7fa4fc102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.240+0000 7fa500b14640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4fc1018f0 0x7fa4fc102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.240+0000 7fa500b14640 1 -- 172.21.15.77:0/555103003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4fc103aa0 con 0x7fa4fc1022b0 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.240+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4fc1018f0 0x7fa4fc102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.240+0000 7fa500b14640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4fc1022b0 0x7fa4fc103210 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa4f402c630 tx=0x7fa4f4032f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.240+0000 7fa4f97fa640 1 -- 172.21.15.77:0/555103003 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4039040 con 0x7fa4fc1022b0 2024-03-16T10:25:54.243 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.240+0000 7fa502577640 1 -- 172.21.15.77:0/555103003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4fc1a1b80 con 0x7fa4fc1022b0 2024-03-16T10:25:54.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.240+0000 7fa4f97fa640 1 -- 172.21.15.77:0/555103003 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4f4044070 con 0x7fa4fc1022b0 2024-03-16T10:25:54.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.240+0000 7fa4f97fa640 1 -- 172.21.15.77:0/555103003 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f40042a0 con 0x7fa4fc1022b0 2024-03-16T10:25:54.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.240+0000 7fa502577640 1 -- 172.21.15.77:0/555103003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4fc1a20f0 con 0x7fa4fc1022b0 2024-03-16T10:25:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.242+0000 7fa4f97fa640 1 -- 172.21.15.77:0/555103003 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fa4f4003970 con 0x7fa4fc1022b0 2024-03-16T10:25:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa505164740 1 Processor -- start 2024-03-16T10:25:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa505164740 1 -- start start 2024-03-16T10:25:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa4f97fa640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4e40a2800 0x7fa4e40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa505164740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e56850 0x55e083e56c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa505164740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 0x55e083e5bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa505164740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e5c3b0 0x55e083e5e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa505164740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e083dbb490 con 0x55e083e5c3b0 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa505164740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55e083d63c90 con 0x55e083e57240 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa505164740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e083db7530 con 0x55e083e56850 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4e40a2800 0x7fa4e40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa4fbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e56850 0x55e083e56c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa4fb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 0x55e083e5bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa4fbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e56850 0x55e083e56c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50520/0 (socket says 172.21.15.77:50520) 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa4fbfff640 1 -- 172.21.15.77:0/1154488391 learned_addr learned my addr 172.21.15.77:0/1154488391 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/1154488391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 0x55e083e5bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:55798/0 (socket says 172.21.15.77:55798) 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa500b14640 1 --2- 172.21.15.77:0/1154488391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e5c3b0 0x55e083e5e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa4f97fa640 1 -- 172.21.15.77:0/555103003 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fa4f40f0d80 con 0x7fa4fc1022b0 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa4fb7fe640 1 -- 172.21.15.77:0/1154488391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e56850 msgr2=0x55e083e56c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/1154488391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e56850 0x55e083e56c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa4fb7fe640 1 -- 172.21.15.77:0/1154488391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e5c3b0 msgr2=0x55e083e5e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.243+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/1154488391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e5c3b0 0x55e083e5e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.244+0000 7fa4fb7fe640 1 -- 172.21.15.77:0/1154488391 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e083e1e500 con 0x55e083e57240 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.244+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4e40a2800 0x7fa4e40a4cc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc104400 tx=0x7fa4f0012980 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.244+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/1154488391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 0x55e083e5bd40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00058c0 tx=0x7fa4f001c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dedb7dfcbd1344a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.244+0000 7fa4e27fc640 1 -- 172.21.15.77:0/1154488391 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0024070 con 0x55e083e57240 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.244+0000 7fa4e27fc640 1 -- 172.21.15.77:0/1154488391 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4f001fc00 con 0x55e083e57240 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.244+0000 7fa4e27fc640 1 -- 172.21.15.77:0/1154488391 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f001e3f0 con 0x55e083e57240 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.245+0000 7fa505164740 1 -- 172.21.15.77:0/1154488391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 msgr2=0x55e083e5bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.245+0000 7fa505164740 1 --2- 172.21.15.77:0/1154488391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 0x55e083e5bd40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00058c0 tx=0x7fa4f001c000 comp rx=0 tx=0).stop 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.245+0000 7fa505164740 1 -- 172.21.15.77:0/1154488391 shutdown_connections 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.245+0000 7fa505164740 1 --2- 172.21.15.77:0/1154488391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e5c3b0 0x55e083e5e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.245+0000 7fa505164740 1 --2- 172.21.15.77:0/1154488391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 0x55e083e5bd40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.245+0000 7fa505164740 1 --2- 172.21.15.77:0/1154488391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e56850 0x55e083e56c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.245+0000 7fa505164740 1 -- 172.21.15.77:0/1154488391 >> 172.21.15.77:0/1154488391 conn(0x55e083e51dc0 msgr2=0x55e083e521c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:54.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.245+0000 7fa505164740 1 -- 172.21.15.77:0/1154488391 shutdown_connections 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.245+0000 7fa505164740 1 -- 172.21.15.77:0/1154488391 wait complete. 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.245+0000 7fa505164740 1 -- start start 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.245+0000 7fa505164740 1 Processor -- start 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa505164740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e56850 0x55e083e60dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa505164740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e57240 0x55e083e61310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa505164740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e5c3b0 0x55e083e68310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa505164740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e083e5f300 con 0x55e083e56850 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa505164740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55e083d77230 con 0x55e083e5c3b0 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa505164740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e083e5f480 con 0x55e083e57240 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa505164740 10 client.0 _ll_register_callbacks cb 0x55e083db3010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4fbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e56850 0x55e083e60dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4fbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e56850 0x55e083e60dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41668/0 (socket says 172.21.15.77:41668) 2024-03-16T10:25:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4fbfff640 1 -- 172.21.15.77:0/2305336569 learned_addr learned my addr 172.21.15.77:0/2305336569 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e57240 0x55e083e61310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa500b14640 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e5c3b0 0x55e083e68310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4fbfff640 1 -- 172.21.15.77:0/2305336569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e57240 msgr2=0x55e083e61310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4fbfff640 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e57240 0x55e083e61310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4fbfff640 1 -- 172.21.15.77:0/2305336569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e5c3b0 msgr2=0x55e083e68310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4fbfff640 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e5c3b0 0x55e083e68310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4fbfff640 1 -- 172.21.15.77:0/2305336569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e083d86100 con 0x55e083e56850 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4fbfff640 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e56850 0x55e083e60dd0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00bb10 tx=0x7fa4ec0045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3e3013309557d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4db7fe640 1 -- 172.21.15.77:0/2305336569 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec011440 con 0x55e083e56850 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa4db7fe640 1 -- 172.21.15.77:0/2305336569 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fa4ec0115e0 con 0x55e083e56850 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.246+0000 7fa505164740 1 -- 172.21.15.77:0/2305336569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4f001fc00 con 0x55e083e56850 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.247+0000 7fa4db7fe640 1 -- 172.21.15.77:0/2305336569 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa4ec0117e0 con 0x55e083e56850 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.247+0000 7fa505164740 10 client.17661 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.247+0000 7fa505164740 1 -- 172.21.15.77:0/2305336569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55e083acf3f0 con 0x55e083e56850 2024-03-16T10:25:54.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.247+0000 7fa4db7fe640 1 -- 172.21.15.77:0/2305336569 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4f001fc00 con 0x55e083e56850 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.247+0000 7fa4db7fe640 1 -- 172.21.15.77:0/2305336569 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7fa4ec024050 con 0x55e083e56850 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.247+0000 7fa505164740 10 client.17661 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.247+0000 7fa505164740 10 client.17661 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.247+0000 7fa505164740 20 client.17661 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.247+0000 7fa505164740 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e083e6cc20 0x55e083e6f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.247+0000 7fa505164740 4 client.17661 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.247+0000 7fa505164740 1 -- 172.21.15.77:0/2305336569 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e083dc89d0 con 0x55e083e6cc20 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.247+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e083e6cc20 0x55e083e6f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.248+0000 7fa4db7fe640 10 client.17661 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.248+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e083e6cc20 0x55e083e6f0c0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6438da0710024a12 server_cookie=3a5af8366b506269 in_seq=0 out_seq=0 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa4db7fe640 1 -- 172.21.15.77:0/2305336569 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e083dc89d0 con 0x55e083e6cc20 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa4db7fe640 10 client.17661 handle_command_reply: tid=0 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa505164740 1 client.17661 shutdown 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa505164740 1 -- 172.21.15.77:0/2305336569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e56850 msgr2=0x55e083e60dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa505164740 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e56850 0x55e083e60dd0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00bb10 tx=0x7fa4ec0045f0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa505164740 1 -- 172.21.15.77:0/2305336569 shutdown_connections 2024-03-16T10:25:54.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa505164740 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e5c3b0 0x55e083e68310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa505164740 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e57240 0x55e083e61310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa505164740 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e083e6cc20 0x55e083e6f0c0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa505164740 1 --2- 172.21.15.77:0/2305336569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e56850 0x55e083e60dd0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa4db7fe640 0 client.17661 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:54.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa505164740 1 -- 172.21.15.77:0/2305336569 >> 172.21.15.77:0/2305336569 conn(0x55e083e51dc0 msgr2=0x55e083e50fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:54.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa505164740 1 -- 172.21.15.77:0/2305336569 shutdown_connections 2024-03-16T10:25:54.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa505164740 1 -- 172.21.15.77:0/2305336569 wait complete. 2024-03-16T10:25:54.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.249+0000 7fa505164740 20 client.17661 trim_cache size 0 max 16384 2024-03-16T10:25:54.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:53 smithi193 ceph-mon[36311]: pgmap v615: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 429 op/s 2024-03-16T10:25:54.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:53 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/716999728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:25:54.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.269+0000 7fa505164740 1 Processor -- start 2024-03-16T10:25:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.269+0000 7fa505164740 1 -- start start 2024-03-16T10:25:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa505164740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e56850 0x55e083e61270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa505164740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e57240 0x55e083e682d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa505164740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e5c3b0 0x55e083e50750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa505164740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc1aad30 con 0x55e083e57240 2024-03-16T10:25:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa505164740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc1ac930 con 0x55e083e56850 2024-03-16T10:25:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa505164740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc1ab460 con 0x55e083e5c3b0 2024-03-16T10:25:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e57240 0x55e083e682d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e57240 0x55e083e682d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41672/0 (socket says 172.21.15.77:41672) 2024-03-16T10:25:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4fb7fe640 1 -- 172.21.15.77:0/1402645306 learned_addr learned my addr 172.21.15.77:0/1402645306 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4fbfff640 1 --2- 172.21.15.77:0/1402645306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e56850 0x55e083e61270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa500b14640 1 --2- 172.21.15.77:0/1402645306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e5c3b0 0x55e083e50750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4fb7fe640 1 -- 172.21.15.77:0/1402645306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e5c3b0 msgr2=0x55e083e50750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/1402645306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e5c3b0 0x55e083e50750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4fb7fe640 1 -- 172.21.15.77:0/1402645306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e56850 msgr2=0x55e083e61270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/1402645306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e56850 0x55e083e61270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4fb7fe640 1 -- 172.21.15.77:0/1402645306 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e083b997d0 con 0x55e083e57240 2024-03-16T10:25:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/1402645306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e57240 0x55e083e682d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0005fa0 tx=0x7fa4f0014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9bb2f16f950909 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4e1ffb640 1 -- 172.21.15.77:0/1402645306 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0024040 con 0x55e083e57240 2024-03-16T10:25:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4e1ffb640 1 -- 172.21.15.77:0/1402645306 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4f0034030 con 0x55e083e57240 2024-03-16T10:25:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.270+0000 7fa4e1ffb640 1 -- 172.21.15.77:0/1402645306 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0005500 con 0x55e083e57240 2024-03-16T10:25:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.271+0000 7fa505164740 1 -- 172.21.15.77:0/1402645306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e57240 msgr2=0x55e083e682d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.271+0000 7fa505164740 1 --2- 172.21.15.77:0/1402645306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e57240 0x55e083e682d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0005fa0 tx=0x7fa4f0014040 comp rx=0 tx=0).stop 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.271+0000 7fa505164740 1 -- 172.21.15.77:0/1402645306 shutdown_connections 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.271+0000 7fa505164740 1 --2- 172.21.15.77:0/1402645306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e5c3b0 0x55e083e50750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.271+0000 7fa505164740 1 --2- 172.21.15.77:0/1402645306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e57240 0x55e083e682d0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.271+0000 7fa505164740 1 --2- 172.21.15.77:0/1402645306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e56850 0x55e083e61270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.271+0000 7fa505164740 1 -- 172.21.15.77:0/1402645306 >> 172.21.15.77:0/1402645306 conn(0x55e083e51dc0 msgr2=0x55e083e60b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.271+0000 7fa505164740 1 -- 172.21.15.77:0/1402645306 shutdown_connections 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.271+0000 7fa505164740 1 -- 172.21.15.77:0/1402645306 wait complete. 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.271+0000 7fa505164740 1 -- start start 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa505164740 1 Processor -- start 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa505164740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e56850 0x55e083e65330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa505164740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 0x55e083e65870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa505164740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e5c3b0 0x55e083ed0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa505164740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc1ac260 con 0x55e083e5c3b0 2024-03-16T10:25:54.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa505164740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc1abb60 con 0x55e083e57240 2024-03-16T10:25:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa505164740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc1a8d10 con 0x55e083e56850 2024-03-16T10:25:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa505164740 10 client.0 _ll_register_callbacks cb 0x7fa4ec004210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa500b14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e5c3b0 0x55e083ed0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa4fb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 0x55e083e65870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa4fbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e56850 0x55e083e65330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa4fb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 0x55e083e65870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:55844/0 (socket says 172.21.15.77:55844) 2024-03-16T10:25:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa4fb7fe640 1 -- 172.21.15.77:0/2311215790 learned_addr learned my addr 172.21.15.77:0/2311215790 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa500b14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e5c3b0 0x55e083ed0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41676/0 (socket says 172.21.15.77:41676) 2024-03-16T10:25:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa4fb7fe640 1 -- 172.21.15.77:0/2311215790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e56850 msgr2=0x55e083e65330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e56850 0x55e083e65330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.280 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa4fb7fe640 1 -- 172.21.15.77:0/2311215790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e5c3b0 msgr2=0x55e083ed0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e5c3b0 0x55e083ed0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa4fb7fe640 1 -- 172.21.15.77:0/2311215790 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e083ed15e0 con 0x55e083e57240 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa4fbfff640 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e56850 0x55e083e65330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.272+0000 7fa500b14640 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e5c3b0 0x55e083ed0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa4fb7fe640 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 0x55e083e65870 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0005080 tx=0x7fa4f0010c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dce7398a2d4c365 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa4db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa4db7fe640 1 -- 172.21.15.77:0/2311215790 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0024040 con 0x55e083e57240 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa505164740 1 -- 172.21.15.77:0/2311215790 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4f0034030 con 0x55e083e57240 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa4db7fe640 1 -- 172.21.15.77:0/2311215790 <== mon.2 v2:172.21.15.192:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fa4f0005280 con 0x55e083e57240 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa4db7fe640 1 -- 172.21.15.77:0/2311215790 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa4f0005480 con 0x55e083e57240 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa505164740 10 client.27440 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa505164740 1 -- 172.21.15.77:0/2311215790 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55e083ed2960 con 0x55e083e57240 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa4db7fe640 1 -- 172.21.15.77:0/2311215790 <== mon.2 v2:172.21.15.192:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4f0034030 con 0x55e083e57240 2024-03-16T10:25:54.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa4db7fe640 1 -- 172.21.15.77:0/2311215790 <== mon.2 v2:172.21.15.192:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7fa4f001f050 con 0x55e083e57240 2024-03-16T10:25:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa505164740 10 client.27440 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa505164740 10 client.27440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa505164740 20 client.27440 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa505164740 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e083ed3220 0x55e083ed5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa505164740 4 client.27440 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:25:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.273+0000 7fa505164740 1 -- 172.21.15.77:0/2311215790 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e083dc2d20 con 0x55e083ed3220 2024-03-16T10:25:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa4fbfff640 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e083ed3220 0x55e083ed5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa4fbfff640 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e083ed3220 0x55e083ed5610 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe40bae79e5e9600 server_cookie=1a6281100888b42 in_seq=0 out_seq=0 2024-03-16T10:25:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa4db7fe640 10 client.27440 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa4db7fe640 1 -- 172.21.15.77:0/2311215790 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e083dc2d20 con 0x55e083ed3220 2024-03-16T10:25:54.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa4db7fe640 10 client.27440 handle_command_reply: tid=0 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa505164740 1 client.27440 shutdown 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa505164740 1 -- 172.21.15.77:0/2311215790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 msgr2=0x55e083e65870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa505164740 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 0x55e083e65870 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0005080 tx=0x7fa4f0010c20 comp rx=0 tx=0).stop 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa505164740 1 -- 172.21.15.77:0/2311215790 shutdown_connections 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa505164740 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55e083ed3220 0x55e083ed5610 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa505164740 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e083e5c3b0 0x55e083ed0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa505164740 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e083e57240 0x55e083e65870 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa505164740 1 --2- 172.21.15.77:0/2311215790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e083e56850 0x55e083e65330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa4db7fe640 0 client.27440 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa505164740 1 -- 172.21.15.77:0/2311215790 >> 172.21.15.77:0/2311215790 conn(0x55e083e51dc0 msgr2=0x55e083e558b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa505164740 1 -- 172.21.15.77:0/2311215790 shutdown_connections 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa505164740 1 -- 172.21.15.77:0/2311215790 wait complete. 2024-03-16T10:25:54.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.274+0000 7fa505164740 20 client.27440 trim_cache size 0 max 16384 2024-03-16T10:25:54.284 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:25:54.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.275+0000 7fa502577640 1 -- 172.21.15.77:0/555103003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4e40a2800 msgr2=0x7fa4e40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.275+0000 7fa502577640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4e40a2800 0x7fa4e40a4cc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc104400 tx=0x7fa4f0012980 comp rx=0 tx=0).stop 2024-03-16T10:25:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.275+0000 7fa502577640 1 -- 172.21.15.77:0/555103003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4fc1022b0 msgr2=0x7fa4fc103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.275+0000 7fa502577640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4fc1022b0 0x7fa4fc103210 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa4f402c630 tx=0x7fa4f4032f40 comp rx=0 tx=0).stop 2024-03-16T10:25:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.275+0000 7fa502577640 1 -- 172.21.15.77:0/555103003 shutdown_connections 2024-03-16T10:25:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.275+0000 7fa502577640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa4e40a2800 0x7fa4e40a4cc0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.275+0000 7fa502577640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4fc1022b0 0x7fa4fc103210 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.275+0000 7fa502577640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa4fc1018f0 0x7fa4fc102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.275+0000 7fa502577640 1 --2- 172.21.15.77:0/555103003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4fc100700 0x7fa4fc104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.275+0000 7fa502577640 1 -- 172.21.15.77:0/555103003 >> 172.21.15.77:0/555103003 conn(0x7fa4fc0fbc90 msgr2=0x7fa4fc0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.275+0000 7fa502577640 1 -- 172.21.15.77:0/555103003 shutdown_connections 2024-03-16T10:25:54.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:54.275+0000 7fa502577640 1 -- 172.21.15.77:0/555103003 wait complete. 2024-03-16T10:25:54.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:53 smithi192 ceph-mon[36354]: pgmap v615: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 429 op/s 2024-03-16T10:25:54.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:53 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/716999728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:25:55.287 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-16T10:25:55.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:25:55.288 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-16T10:25:55.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.423+0000 7f62a3234640 1 -- 172.21.15.77:0/2884540332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f629c1022a0 msgr2=0x7f629c108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.423+0000 7f62a3234640 1 --2- 172.21.15.77:0/2884540332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f629c1022a0 0x7f629c108b30 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f6294009ca0 tx=0x7f629402c100 comp rx=0 tx=0).stop 2024-03-16T10:25:55.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.424+0000 7f62a3234640 1 -- 172.21.15.77:0/2884540332 shutdown_connections 2024-03-16T10:25:55.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.424+0000 7f62a3234640 1 --2- 172.21.15.77:0/2884540332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f629c1022a0 0x7f629c108b30 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.424+0000 7f62a3234640 1 --2- 172.21.15.77:0/2884540332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f629c1018e0 0x7f629c101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.424+0000 7f62a3234640 1 --2- 172.21.15.77:0/2884540332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f629c1006f0 0x7f629c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.424+0000 7f62a3234640 1 -- 172.21.15.77:0/2884540332 >> 172.21.15.77:0/2884540332 conn(0x7f629c0fbc80 msgr2=0x7f629c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:55.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.424+0000 7f62a3234640 1 -- 172.21.15.77:0/2884540332 shutdown_connections 2024-03-16T10:25:55.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.424+0000 7f62a3234640 1 -- 172.21.15.77:0/2884540332 wait complete. 2024-03-16T10:25:55.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.424+0000 7f62a3234640 1 Processor -- start 2024-03-16T10:25:55.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.424+0000 7f62a3234640 1 -- start start 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f62a3234640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f629c1006f0 0x7f629c199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f62a3234640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f629c1018e0 0x7f629c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f62a3234640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f629c1022a0 0x7f629c1a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f62a3234640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f629c10a9c0 con 0x7f629c1018e0 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f62a3234640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f629c10a840 con 0x7f629c1006f0 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f62a3234640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f629c10ab40 con 0x7f629c1022a0 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f629bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f629c1018e0 0x7f629c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f629bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f629c1018e0 0x7f629c19a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41688/0 (socket says 172.21.15.77:41688) 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f629bfff640 1 -- 172.21.15.77:0/932474916 learned_addr learned my addr 172.21.15.77:0/932474916 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f62a17d1640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f629c1022a0 0x7f629c1a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f62a0fd0640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f629c1006f0 0x7f629c199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f629bfff640 1 -- 172.21.15.77:0/932474916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f629c1022a0 msgr2=0x7f629c1a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f629bfff640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f629c1022a0 0x7f629c1a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f629bfff640 1 -- 172.21.15.77:0/932474916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f629c1006f0 msgr2=0x7f629c199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f629bfff640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f629c1006f0 0x7f629c199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f629bfff640 1 -- 172.21.15.77:0/932474916 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629c1a18f0 con 0x7f629c1018e0 2024-03-16T10:25:55.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f62a0fd0640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f629c1006f0 0x7f629c199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:25:55.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f629bfff640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f629c1018e0 0x7f629c19a1f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f628800d950 tx=0x7f628800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:55.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.425+0000 7f6299ffb640 1 -- 172.21.15.77:0/932474916 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6288017070 con 0x7f629c1018e0 2024-03-16T10:25:55.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.426+0000 7f6299ffb640 1 -- 172.21.15.77:0/932474916 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6288009c30 con 0x7f629c1018e0 2024-03-16T10:25:55.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.426+0000 7f6299ffb640 1 -- 172.21.15.77:0/932474916 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628800f6b0 con 0x7f629c1018e0 2024-03-16T10:25:55.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.426+0000 7f62a3234640 1 -- 172.21.15.77:0/932474916 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f629c1a1be0 con 0x7f629c1018e0 2024-03-16T10:25:55.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.427+0000 7f62a3234640 1 -- 172.21.15.77:0/932474916 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f629c1a2180 con 0x7f629c1018e0 2024-03-16T10:25:55.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f62a5e1a740 1 Processor -- start 2024-03-16T10:25:55.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f62a5e1a740 1 -- start start 2024-03-16T10:25:55.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f6299ffb640 1 -- 172.21.15.77:0/932474916 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f62880109d0 con 0x7f629c1018e0 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f62a5e1a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 0x56093752edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f62a5e1a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56093752f380 0x560937533e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f62a5e1a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 0x560937536980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f62a5e1a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5609374930d0 con 0x56093752e990 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f62a5e1a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56093743bc70 con 0x56093752f380 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f62a5e1a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56093748f170 con 0x5609375344f0 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f629bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 0x560937536980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f629bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 0x560937536980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50576/0 (socket says 172.21.15.77:50576) 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f629bfff640 1 -- 172.21.15.77:0/1024375114 learned_addr learned my addr 172.21.15.77:0/1024375114 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.430+0000 7f62a17d1640 1 --2- 172.21.15.77:0/1024375114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 0x56093752edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f62a0fd0640 1 --2- 172.21.15.77:0/1024375114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56093752f380 0x560937533e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f629bfff640 1 -- 172.21.15.77:0/1024375114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56093752f380 msgr2=0x560937533e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f629bfff640 1 --2- 172.21.15.77:0/1024375114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56093752f380 0x560937533e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f629bfff640 1 -- 172.21.15.77:0/1024375114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 msgr2=0x56093752edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f629bfff640 1 --2- 172.21.15.77:0/1024375114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 0x56093752edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f629bfff640 1 -- 172.21.15.77:0/1024375114 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56093726f7e0 con 0x5609375344f0 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f6299ffb640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f628c0a2880 0x7f628c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f62a0fd0640 1 --2- 172.21.15.77:0/1024375114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56093752f380 0x560937533e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f62a17d1640 1 --2- 172.21.15.77:0/1024375114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 0x56093752edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f62a0fd0640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f628c0a2880 0x7f628c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f6299ffb640 1 -- 172.21.15.77:0/932474916 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f62880ce5c0 con 0x7f629c1018e0 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f629bfff640 1 --2- 172.21.15.77:0/1024375114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 0x560937536980 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f62880d4bf0 tx=0x7f6288097290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60712e68cb788ee0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f6286ffd640 1 -- 172.21.15.77:0/1024375114 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628809adc0 con 0x5609375344f0 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f6286ffd640 1 -- 172.21.15.77:0/1024375114 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f628809b6f0 con 0x5609375344f0 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f6286ffd640 1 -- 172.21.15.77:0/1024375114 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628809da60 con 0x5609375344f0 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.431+0000 7f62a0fd0640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f628c0a2880 0x7f628c0a4d40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x56093752f780 tx=0x7f629000b300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:55.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.432+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1024375114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 msgr2=0x560937536980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.432+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1024375114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 0x560937536980 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f62880d4bf0 tx=0x7f6288097290 comp rx=0 tx=0).stop 2024-03-16T10:25:55.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.432+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1024375114 shutdown_connections 2024-03-16T10:25:55.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.432+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1024375114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 0x560937536980 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.432+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1024375114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56093752f380 0x560937533e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.432+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1024375114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 0x56093752edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.432+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1024375114 >> 172.21.15.77:0/1024375114 conn(0x5609374f6c50 msgr2=0x56093752a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:55.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.432+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1024375114 shutdown_connections 2024-03-16T10:25:55.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.432+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1024375114 wait complete. 2024-03-16T10:25:55.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f62a5e1a740 1 -- start start 2024-03-16T10:25:55.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f62a5e1a740 1 Processor -- start 2024-03-16T10:25:55.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f62a5e1a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56093752e990 0x560937538f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f62a5e1a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 0x5609375394a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f62a5e1a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609375344f0 0x560937540470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f62a5e1a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5609375374a0 con 0x56093752f380 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f62a5e1a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x56093744f0d0 con 0x5609375344f0 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f62a5e1a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560937537620 con 0x56093752e990 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f629bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 0x5609375394a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f62a5e1a740 10 client.0 _ll_register_callbacks cb 0x560937418760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f629bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 0x5609375394a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41712/0 (socket says 172.21.15.77:41712) 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f629bfff640 1 -- 172.21.15.77:0/1564364661 learned_addr learned my addr 172.21.15.77:0/1564364661 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.433+0000 7f62a0fd0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56093752e990 0x560937538f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f62a17d1640 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609375344f0 0x560937540470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f629bfff640 1 -- 172.21.15.77:0/1564364661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56093752e990 msgr2=0x560937538f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f629bfff640 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56093752e990 0x560937538f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f629bfff640 1 -- 172.21.15.77:0/1564364661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609375344f0 msgr2=0x560937540470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f629bfff640 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609375344f0 0x560937540470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f629bfff640 1 -- 172.21.15.77:0/1564364661 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609372cff70 con 0x56093752f380 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f62a17d1640 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609375344f0 0x560937540470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f629bfff640 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 0x5609375394a0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f62880979c0 tx=0x7f628800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641d6e5b7ffc67e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f627bfff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f627bfff640 1 -- 172.21.15.77:0/1564364661 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628809a950 con 0x56093752f380 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f627bfff640 1 -- 172.21.15.77:0/1564364661 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f62880d4730 con 0x56093752f380 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1564364661 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f628809b6f0 con 0x56093752f380 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f627bfff640 1 -- 172.21.15.77:0/1564364661 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f628809ab30 con 0x56093752f380 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f62a5e1a740 10 client.17679 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.434+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1564364661 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x560937167170 con 0x56093752f380 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.435+0000 7f627bfff640 1 -- 172.21.15.77:0/1564364661 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f628809b6f0 con 0x56093752f380 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.435+0000 7f627bfff640 1 -- 172.21.15.77:0/1564364661 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f6288023060 con 0x56093752f380 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.435+0000 7f62a5e1a740 10 client.17679 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:55.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.435+0000 7f62a5e1a740 10 client.17679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.435+0000 7f62a5e1a740 20 client.17679 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.435+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560937544540 0x560937546930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.435+0000 7f62a5e1a740 4 client.17679 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.435+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1564364661 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609374a0610 con 0x560937544540 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.435+0000 7f62a0fd0640 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560937544540 0x560937546930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.435+0000 7f627bfff640 10 client.17679 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.435+0000 7f62a0fd0640 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560937544540 0x560937546930 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=526213b8807277ef server_cookie=3e8534a83c7435a3 in_seq=0 out_seq=0 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.436+0000 7f627bfff640 1 -- 172.21.15.77:0/1564364661 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5609374a0610 con 0x560937544540 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.436+0000 7f627bfff640 10 client.17679 handle_command_reply: tid=0 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.436+0000 7f62a5e1a740 1 client.17679 shutdown 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.436+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1564364661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 msgr2=0x5609375394a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.436+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 0x5609375394a0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f62880979c0 tx=0x7f628800bf60 comp rx=0 tx=0).stop 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.436+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1564364661 shutdown_connections 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.436+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609375344f0 0x560937540470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.436+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x560937544540 0x560937546930 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.436+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 0x5609375394a0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.436+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1564364661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56093752e990 0x560937538f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.436+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1564364661 >> 172.21.15.77:0/1564364661 conn(0x5609374f6c50 msgr2=0x5609375294e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:55.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.436+0000 7f627bfff640 0 client.17679 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:55.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.437+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1564364661 shutdown_connections 2024-03-16T10:25:55.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.437+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1564364661 wait complete. 2024-03-16T10:25:55.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.437+0000 7f62a5e1a740 20 client.17679 trim_cache size 0 max 16384 2024-03-16T10:25:55.457 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f62a5e1a740 1 Processor -- start 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f62a5e1a740 1 -- start start 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f62a5e1a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56093752e990 0x5609375379c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f62a5e1a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 0x560937540430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f62a5e1a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 0x560937537f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f62a5e1a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f629c1aadb0 con 0x56093752f380 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f62a5e1a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f629c1ac9b0 con 0x56093752e990 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f62a5e1a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f629c1ab4e0 con 0x5609375344f0 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f629bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 0x560937540430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f629bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 0x560937540430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41724/0 (socket says 172.21.15.77:41724) 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f629bfff640 1 -- 172.21.15.77:0/368931256 learned_addr learned my addr 172.21.15.77:0/368931256 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f62a0fd0640 1 --2- 172.21.15.77:0/368931256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56093752e990 0x5609375379c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f62a17d1640 1 --2- 172.21.15.77:0/368931256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 0x560937537f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f629bfff640 1 -- 172.21.15.77:0/368931256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 msgr2=0x560937537f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f629bfff640 1 --2- 172.21.15.77:0/368931256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 0x560937537f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f629bfff640 1 -- 172.21.15.77:0/368931256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56093752e990 msgr2=0x5609375379c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f629bfff640 1 --2- 172.21.15.77:0/368931256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56093752e990 0x5609375379c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f629bfff640 1 -- 172.21.15.77:0/368931256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560937533c00 con 0x56093752f380 2024-03-16T10:25:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f62a17d1640 1 --2- 172.21.15.77:0/368931256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 0x560937537f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.455+0000 7f629bfff640 1 --2- 172.21.15.77:0/368931256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 0x560937540430 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f628809acf0 tx=0x7f628809aff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1a120cecc9b276 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.456+0000 7f62867fc640 1 -- 172.21.15.77:0/368931256 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628809a950 con 0x56093752f380 2024-03-16T10:25:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.456+0000 7f62867fc640 1 -- 172.21.15.77:0/368931256 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62880d4e60 con 0x56093752f380 2024-03-16T10:25:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.456+0000 7f62867fc640 1 -- 172.21.15.77:0/368931256 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62880d41e0 con 0x56093752f380 2024-03-16T10:25:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.456+0000 7f62a5e1a740 1 -- 172.21.15.77:0/368931256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 msgr2=0x560937540430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.456+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/368931256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 0x560937540430 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f628809acf0 tx=0x7f628809aff0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.456+0000 7f62a5e1a740 1 -- 172.21.15.77:0/368931256 shutdown_connections 2024-03-16T10:25:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.456+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/368931256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609375344f0 0x560937537f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.456+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/368931256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752f380 0x560937540430 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.456+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/368931256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56093752e990 0x5609375379c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.456+0000 7f62a5e1a740 1 -- 172.21.15.77:0/368931256 >> 172.21.15.77:0/368931256 conn(0x5609374f6c50 msgr2=0x560937530910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.456+0000 7f62a5e1a740 1 -- 172.21.15.77:0/368931256 shutdown_connections 2024-03-16T10:25:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.456+0000 7f62a5e1a740 1 -- 172.21.15.77:0/368931256 wait complete. 2024-03-16T10:25:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a5e1a740 1 -- start start 2024-03-16T10:25:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a5e1a740 1 Processor -- start 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a5e1a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 0x56093753d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a5e1a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56093752f380 0x56093753dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a5e1a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609375344f0 0x56093753ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a5e1a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f629c1ac2e0 con 0x56093752e990 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a5e1a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f629c1abbe0 con 0x5609375344f0 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a5e1a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f629c1a8dd0 con 0x56093752f380 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a5e1a740 10 client.0 _ll_register_callbacks cb 0x7f6294009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a0fd0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 0x56093753d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a0fd0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 0x56093753d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41738/0 (socket says 172.21.15.77:41738) 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a0fd0640 1 -- 172.21.15.77:0/1278569808 learned_addr learned my addr 172.21.15.77:0/1278569808 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f629bfff640 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56093752f380 0x56093753dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a17d1640 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609375344f0 0x56093753ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a0fd0640 1 -- 172.21.15.77:0/1278569808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56093752f380 msgr2=0x56093753dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a0fd0640 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56093752f380 0x56093753dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a0fd0640 1 -- 172.21.15.77:0/1278569808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609375344f0 msgr2=0x56093753ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a0fd0640 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609375344f0 0x56093753ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.457+0000 7f62a0fd0640 1 -- 172.21.15.77:0/1278569808 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56093753f280 con 0x56093752e990 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f62a0fd0640 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 0x56093753d740 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f629c19ab80 tx=0x7f62900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b3b8682550c13ff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f627bfff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f627bfff640 1 -- 172.21.15.77:0/1278569808 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290004020 con 0x56093752e990 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f627bfff640 1 -- 172.21.15.77:0/1278569808 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f62900041c0 con 0x56093752e990 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1278569808 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56093749a960 con 0x56093752e990 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f627bfff640 1 -- 172.21.15.77:0/1278569808 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f62900043c0 con 0x56093752e990 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f62a5e1a740 10 client.17685 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1278569808 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x560937540690 con 0x56093752e990 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f627bfff640 1 -- 172.21.15.77:0/1278569808 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56093749a960 con 0x56093752e990 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f627bfff640 1 -- 172.21.15.77:0/1278569808 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f629000f040 con 0x56093752e990 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f62a5e1a740 10 client.17685 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f62a5e1a740 10 client.17685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f62a5e1a740 20 client.17685 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5609375aaea0 0x5609375ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f62a5e1a740 4 client.17685 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.458+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1278569808 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560937418be0 con 0x5609375aaea0 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.459+0000 7f629bfff640 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5609375aaea0 0x5609375ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.459+0000 7f627bfff640 10 client.17685 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.459+0000 7f629bfff640 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5609375aaea0 0x5609375ad2c0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92c05d1adb42e1e server_cookie=e2b3a40474df2fd4 in_seq=0 out_seq=0 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f627bfff640 1 -- 172.21.15.77:0/1278569808 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560937418be0 con 0x5609375aaea0 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f627bfff640 10 client.17685 handle_command_reply: tid=0 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f62a5e1a740 1 client.17685 shutdown 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1278569808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 msgr2=0x56093753d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 0x56093753d740 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f629c19ab80 tx=0x7f62900046c0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1278569808 shutdown_connections 2024-03-16T10:25:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609375344f0 0x56093753ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56093752f380 0x56093753dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x5609375aaea0 0x5609375ad2c0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f62a5e1a740 1 --2- 172.21.15.77:0/1278569808 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56093752e990 0x56093753d740 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f627bfff640 0 client.17685 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:55.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1278569808 >> 172.21.15.77:0/1278569808 conn(0x5609374f6c50 msgr2=0x5609375379a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:55.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1278569808 shutdown_connections 2024-03-16T10:25:55.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f62a5e1a740 1 -- 172.21.15.77:0/1278569808 wait complete. 2024-03-16T10:25:55.468 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.460+0000 7f62a5e1a740 20 client.17685 trim_cache size 0 max 16384 2024-03-16T10:25:55.468 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:25:55.469 INFO:teuthology.orchestra.run.smithi077.stdout: "return_code": 0, 2024-03-16T10:25:55.469 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_tag": "09db8d3a-435c-4bb3-b3c5-a2f6a042810d", 2024-03-16T10:25:55.469 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": "asynchronous" 2024-03-16T10:25:55.469 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:25:55.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.461+0000 7f62a3234640 1 -- 172.21.15.77:0/932474916 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f628c0a2880 msgr2=0x7f628c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.461+0000 7f62a3234640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f628c0a2880 0x7f628c0a4d40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x56093752f780 tx=0x7f629000b300 comp rx=0 tx=0).stop 2024-03-16T10:25:55.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.461+0000 7f62a3234640 1 -- 172.21.15.77:0/932474916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f629c1018e0 msgr2=0x7f629c19a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.461+0000 7f62a3234640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f629c1018e0 0x7f629c19a1f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f628800d950 tx=0x7f628800de10 comp rx=0 tx=0).stop 2024-03-16T10:25:55.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.461+0000 7f62a3234640 1 -- 172.21.15.77:0/932474916 shutdown_connections 2024-03-16T10:25:55.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.461+0000 7f62a3234640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f628c0a2880 0x7f628c0a4d40 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.461+0000 7f62a3234640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f629c1022a0 0x7f629c1a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.461+0000 7f62a3234640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f629c1018e0 0x7f629c19a1f0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.461+0000 7f62a3234640 1 --2- 172.21.15.77:0/932474916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f629c1006f0 0x7f629c199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.470 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.461+0000 7f62a3234640 1 -- 172.21.15.77:0/932474916 >> 172.21.15.77:0/932474916 conn(0x7f629c0fbc80 msgr2=0x7f629c104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:55.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.461+0000 7f62a3234640 1 -- 172.21.15.77:0/932474916 shutdown_connections 2024-03-16T10:25:55.471 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.461+0000 7f62a3234640 1 -- 172.21.15.77:0/932474916 wait complete. 2024-03-16T10:25:55.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:25:55.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:25:55.587 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.583+0000 7f4b2bc80640 1 -- 172.21.15.77:0/2898093166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b241029f0 msgr2=0x7f4b24102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.583+0000 7f4b2bc80640 1 --2- 172.21.15.77:0/2898093166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b241029f0 0x7f4b24102e70 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f4b14009900 tx=0x7f4b1402bc30 comp rx=0 tx=0).stop 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.583+0000 7f4b2bc80640 1 -- 172.21.15.77:0/2898093166 shutdown_connections 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.583+0000 7f4b2bc80640 1 --2- 172.21.15.77:0/2898093166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b241033b0 0x7f4b24109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.583+0000 7f4b2bc80640 1 --2- 172.21.15.77:0/2898093166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b241029f0 0x7f4b24102e70 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.583+0000 7f4b2bc80640 1 --2- 172.21.15.77:0/2898093166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b24101800 0x7f4b24101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.583+0000 7f4b2bc80640 1 -- 172.21.15.77:0/2898093166 >> 172.21.15.77:0/2898093166 conn(0x7f4b240fcd90 msgr2=0x7f4b240ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.583+0000 7f4b2bc80640 1 -- 172.21.15.77:0/2898093166 shutdown_connections 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.583+0000 7f4b2bc80640 1 -- 172.21.15.77:0/2898093166 wait complete. 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b2bc80640 1 Processor -- start 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b2bc80640 1 -- start start 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b2bc80640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b24101800 0x7f4b24197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b2bc80640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b241029f0 0x7f4b24198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b2bc80640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b241033b0 0x7f4b2419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b2bc80640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2410d480 con 0x7f4b241033b0 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b2bc80640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2410d300 con 0x7f4b241029f0 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b2bc80640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2410d600 con 0x7f4b24101800 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b2a21d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b241033b0 0x7f4b2419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b2a21d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b241033b0 0x7f4b2419f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41762/0 (socket says 172.21.15.77:41762) 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b2a21d640 1 -- 172.21.15.77:0/4156332777 learned_addr learned my addr 172.21.15.77:0/4156332777 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b24101800 0x7f4b24197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.584+0000 7f4b2921b640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b241029f0 0x7f4b24198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.585+0000 7f4b2a21d640 1 -- 172.21.15.77:0/4156332777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b24101800 msgr2=0x7f4b24197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.585+0000 7f4b2a21d640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b24101800 0x7f4b24197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.585+0000 7f4b2a21d640 1 -- 172.21.15.77:0/4156332777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b241029f0 msgr2=0x7f4b24198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.585+0000 7f4b2a21d640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b241029f0 0x7f4b24198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.585+0000 7f4b2a21d640 1 -- 172.21.15.77:0/4156332777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2419f720 con 0x7f4b241033b0 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.585+0000 7f4b2a21d640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b241033b0 0x7f4b2419f0b0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c00eb10 tx=0x7f4b1c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.585+0000 7f4b1affd640 1 -- 172.21.15.77:0/4156332777 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c004b20 con 0x7f4b241033b0 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.585+0000 7f4b1affd640 1 -- 172.21.15.77:0/4156332777 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b1c004cc0 con 0x7f4b241033b0 2024-03-16T10:25:55.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.585+0000 7f4b1affd640 1 -- 172.21.15.77:0/4156332777 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c010760 con 0x7f4b241033b0 2024-03-16T10:25:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.585+0000 7f4b2bc80640 1 -- 172.21.15.77:0/4156332777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b2419fa10 con 0x7f4b241033b0 2024-03-16T10:25:55.590 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.585+0000 7f4b2bc80640 1 -- 172.21.15.77:0/4156332777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b2419ff80 con 0x7f4b241033b0 2024-03-16T10:25:55.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.588+0000 7f4b1affd640 1 -- 172.21.15.77:0/4156332777 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f4b1c005020 con 0x7f4b241033b0 2024-03-16T10:25:55.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.588+0000 7f4b2e860740 1 Processor -- start 2024-03-16T10:25:55.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.588+0000 7f4b2e860740 1 -- start start 2024-03-16T10:25:55.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.588+0000 7f4b2e860740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817170 0x556bfd817590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.588+0000 7f4b2e860740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd81c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.588+0000 7f4b2e860740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd81ccd0 0x556bfd81f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b2e860740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556bfd77c500 con 0x556bfd81ccd0 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b2e860740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x556bfd724be0 con 0x556bfd817170 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b2e860740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556bfd7785a0 con 0x556bfd817b60 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b2921b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817170 0x556bfd817590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b2921b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817170 0x556bfd817590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:55928/0 (socket says 172.21.15.77:55928) 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b2921b640 1 -- 172.21.15.77:0/128645740 learned_addr learned my addr 172.21.15.77:0/128645740 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/128645740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd81c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b2921b640 1 --2- 172.21.15.77:0/128645740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd81ccd0 0x556bfd81f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b1affd640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4b100a2880 0x7f4b100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b29a1c640 1 -- 172.21.15.77:0/128645740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817170 msgr2=0x556bfd817590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/128645740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817170 0x556bfd817590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b29a1c640 1 -- 172.21.15.77:0/128645740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd81ccd0 msgr2=0x556bfd81f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/128645740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd81ccd0 0x556bfd81f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b29a1c640 1 -- 172.21.15.77:0/128645740 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bfd7384b0 con 0x556bfd817b60 2024-03-16T10:25:55.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b2a21d640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4b100a2880 0x7f4b100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b1affd640 1 -- 172.21.15.77:0/4156332777 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f4b1c0cf020 con 0x7f4b241033b0 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/128645740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd81c660 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4b20009bf0 tx=0x7f4b20011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c5e3e68ad2e8693 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b0ffff640 1 -- 172.21.15.77:0/128645740 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20009da0 con 0x556bfd817b60 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.589+0000 7f4b0ffff640 1 -- 172.21.15.77:0/128645740 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b20005020 con 0x556bfd817b60 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.590+0000 7f4b0ffff640 1 -- 172.21.15.77:0/128645740 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20004a70 con 0x556bfd817b60 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.590+0000 7f4b2e860740 1 -- 172.21.15.77:0/128645740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 msgr2=0x556bfd81c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.590+0000 7f4b2a21d640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4b100a2880 0x7f4b100a4d40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009200 tx=0x7f4b1c04b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.590+0000 7f4b2e860740 1 --2- 172.21.15.77:0/128645740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd81c660 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4b20009bf0 tx=0x7f4b20011810 comp rx=0 tx=0).stop 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.590+0000 7f4b2e860740 1 -- 172.21.15.77:0/128645740 shutdown_connections 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.590+0000 7f4b2e860740 1 --2- 172.21.15.77:0/128645740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd81ccd0 0x556bfd81f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.590+0000 7f4b2e860740 1 --2- 172.21.15.77:0/128645740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd81c660 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.590+0000 7f4b2e860740 1 --2- 172.21.15.77:0/128645740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817170 0x556bfd817590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.590+0000 7f4b2e860740 1 -- 172.21.15.77:0/128645740 >> 172.21.15.77:0/128645740 conn(0x556bfd8126e0 msgr2=0x556bfd812ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.590+0000 7f4b2e860740 1 -- 172.21.15.77:0/128645740 shutdown_connections 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.590+0000 7f4b2e860740 1 -- 172.21.15.77:0/128645740 wait complete. 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2e860740 1 -- start start 2024-03-16T10:25:55.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2e860740 1 Processor -- start 2024-03-16T10:25:55.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2e860740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 0x556bfd8216f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2e860740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd821c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2e860740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd81ccd0 0x556bfd828c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2e860740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556bfd81fc80 con 0x556bfd817170 2024-03-16T10:25:55.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2e860740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x556bfd738160 con 0x556bfd81ccd0 2024-03-16T10:25:55.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2e860740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556bfd81fe00 con 0x556bfd817b60 2024-03-16T10:25:55.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2e860740 10 client.0 _ll_register_callbacks cb 0x556bfd71c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:55.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2921b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 0x556bfd8216f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b29a1c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd821c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2921b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 0x556bfd8216f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41780/0 (socket says 172.21.15.77:41780) 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2921b640 1 -- 172.21.15.77:0/522362442 learned_addr learned my addr 172.21.15.77:0/522362442 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b2921b640 1 --2- 172.21.15.77:0/522362442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd81ccd0 0x556bfd828c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b29a1c640 1 -- 172.21.15.77:0/522362442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd81ccd0 msgr2=0x556bfd828c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/522362442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd81ccd0 0x556bfd828c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b29a1c640 1 -- 172.21.15.77:0/522362442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 msgr2=0x556bfd8216f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/522362442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 0x556bfd8216f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.591+0000 7f4b29a1c640 1 -- 172.21.15.77:0/522362442 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bfd55a7d0 con 0x556bfd817b60 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/522362442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd821c30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x556bfd813650 tx=0x7f4b20004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67ae010c02d7e91 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b0d7fa640 1 -- 172.21.15.77:0/522362442 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20029020 con 0x556bfd817b60 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b0d7fa640 1 -- 172.21.15.77:0/522362442 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f4b20004dc0 con 0x556bfd817b60 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b2e860740 1 -- 172.21.15.77:0/522362442 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b20005020 con 0x556bfd817b60 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b0d7fa640 1 -- 172.21.15.77:0/522362442 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f4b20034050 con 0x556bfd817b60 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b2e860740 10 client.27451 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:55.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b2e860740 1 -- 172.21.15.77:0/522362442 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x556bfd4903f0 con 0x556bfd817b60 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b0d7fa640 1 -- 172.21.15.77:0/522362442 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b20005020 con 0x556bfd817b60 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b0d7fa640 1 -- 172.21.15.77:0/522362442 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f4b20019450 con 0x556bfd817b60 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b2e860740 10 client.27451 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b2e860740 10 client.27451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b2e860740 20 client.27451 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b2e860740 1 --2- 172.21.15.77:0/522362442 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556bfd82cfb0 0x556bfd82f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b2e860740 4 client.27451 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.592+0000 7f4b2e860740 1 -- 172.21.15.77:0/522362442 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bfd789a40 con 0x556bfd82cfb0 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.593+0000 7f4b2921b640 1 --2- 172.21.15.77:0/522362442 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556bfd82cfb0 0x556bfd82f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.593+0000 7f4b0d7fa640 10 client.27451 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.593+0000 7f4b2921b640 1 --2- 172.21.15.77:0/522362442 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556bfd82cfb0 0x556bfd82f400 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51e58cbc23603cf2 server_cookie=ee1514399b750b78 in_seq=0 out_seq=0 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b0d7fa640 1 -- 172.21.15.77:0/522362442 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556bfd789a40 con 0x556bfd82cfb0 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b0d7fa640 10 client.27451 handle_command_reply: tid=0 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b2e860740 1 client.27451 shutdown 2024-03-16T10:25:55.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b2e860740 1 -- 172.21.15.77:0/522362442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 msgr2=0x556bfd821c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b2e860740 1 --2- 172.21.15.77:0/522362442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd821c30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x556bfd813650 tx=0x7f4b20004a40 comp rx=0 tx=0).stop 2024-03-16T10:25:55.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b2e860740 1 -- 172.21.15.77:0/522362442 shutdown_connections 2024-03-16T10:25:55.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b2e860740 1 --2- 172.21.15.77:0/522362442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd81ccd0 0x556bfd828c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b2e860740 1 --2- 172.21.15.77:0/522362442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd821c30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b2e860740 1 --2- 172.21.15.77:0/522362442 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556bfd82cfb0 0x556bfd82f400 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b2e860740 1 --2- 172.21.15.77:0/522362442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 0x556bfd8216f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b0d7fa640 0 client.27451 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:55.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b2e860740 1 -- 172.21.15.77:0/522362442 >> 172.21.15.77:0/522362442 conn(0x556bfd8126e0 msgr2=0x556bfd811900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:55.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b2e860740 1 -- 172.21.15.77:0/522362442 shutdown_connections 2024-03-16T10:25:55.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b2e860740 1 -- 172.21.15.77:0/522362442 wait complete. 2024-03-16T10:25:55.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.594+0000 7f4b2e860740 20 client.27451 trim_cache size 0 max 16384 2024-03-16T10:25:55.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2e860740 1 Processor -- start 2024-03-16T10:25:55.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2e860740 1 -- start start 2024-03-16T10:25:55.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2e860740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 0x556bfd816050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2e860740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817b60 0x556bfd828bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.619 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2e860740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd81ccd0 0x556bfd833cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2e860740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4b241a8bd0 con 0x556bfd817170 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2e860740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4b241aa7d0 con 0x556bfd817b60 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2e860740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b241a9300 con 0x556bfd81ccd0 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2921b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 0x556bfd816050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b29a1c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817b60 0x556bfd828bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2921b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 0x556bfd816050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41796/0 (socket says 172.21.15.77:41796) 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2921b640 1 -- 172.21.15.77:0/3198776673 learned_addr learned my addr 172.21.15.77:0/3198776673 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2921b640 1 --2- 172.21.15.77:0/3198776673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd81ccd0 0x556bfd833cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b29a1c640 1 -- 172.21.15.77:0/3198776673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd81ccd0 msgr2=0x556bfd833cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/3198776673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd81ccd0 0x556bfd833cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b29a1c640 1 -- 172.21.15.77:0/3198776673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 msgr2=0x556bfd816050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/3198776673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 0x556bfd816050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b29a1c640 1 -- 172.21.15.77:0/3198776673 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bfd4500f0 con 0x556bfd817b60 2024-03-16T10:25:55.620 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.612+0000 7f4b2921b640 1 --2- 172.21.15.77:0/3198776673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 0x556bfd816050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.613+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/3198776673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817b60 0x556bfd828bc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4b20019ab0 tx=0x7f4b20004b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=109ec1210ab8093d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.613+0000 7f4b0f7fe640 1 -- 172.21.15.77:0/3198776673 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20029040 con 0x556bfd817b60 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.613+0000 7f4b0f7fe640 1 -- 172.21.15.77:0/3198776673 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b20016070 con 0x556bfd817b60 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.613+0000 7f4b0f7fe640 1 -- 172.21.15.77:0/3198776673 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20004dc0 con 0x556bfd817b60 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 -- 172.21.15.77:0/3198776673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817b60 msgr2=0x556bfd828bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 --2- 172.21.15.77:0/3198776673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817b60 0x556bfd828bc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4b20019ab0 tx=0x7f4b20004b30 comp rx=0 tx=0).stop 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 -- 172.21.15.77:0/3198776673 shutdown_connections 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 --2- 172.21.15.77:0/3198776673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd81ccd0 0x556bfd833cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 --2- 172.21.15.77:0/3198776673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817b60 0x556bfd828bc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 --2- 172.21.15.77:0/3198776673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd817170 0x556bfd816050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 -- 172.21.15.77:0/3198776673 >> 172.21.15.77:0/3198776673 conn(0x556bfd8126e0 msgr2=0x556bfd815940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 -- 172.21.15.77:0/3198776673 shutdown_connections 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 -- 172.21.15.77:0/3198776673 wait complete. 2024-03-16T10:25:55.621 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 -- start start 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 Processor -- start 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817170 0x556bfd826430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd826970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd81ccd0 0x556bfd827930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4b241aa100 con 0x556bfd81ccd0 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4b241a9a00 con 0x556bfd817170 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b241a6bb0 con 0x556bfd817b60 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.614+0000 7f4b2e860740 10 client.0 _ll_register_callbacks cb 0x7f4b14009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b29a1c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd826970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b2921b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd81ccd0 0x556bfd827930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b29a1c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd826970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:50640/0 (socket says 172.21.15.77:50640) 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b29a1c640 1 -- 172.21.15.77:0/2787523140 learned_addr learned my addr 172.21.15.77:0/2787523140 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b2921b640 1 --2- 172.21.15.77:0/2787523140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817170 0x556bfd826430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b29a1c640 1 -- 172.21.15.77:0/2787523140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817170 msgr2=0x556bfd826430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.622 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/2787523140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817170 0x556bfd826430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b29a1c640 1 -- 172.21.15.77:0/2787523140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd81ccd0 msgr2=0x556bfd827930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/2787523140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd81ccd0 0x556bfd827930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b29a1c640 1 -- 172.21.15.77:0/2787523140 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bfd7439d0 con 0x556bfd817b60 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b29a1c640 1 --2- 172.21.15.77:0/2787523140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd826970 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4b200047e0 tx=0x7f4b200092e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60ea0741c55e4916 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b0d7fa640 1 -- 172.21.15.77:0/2787523140 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20029070 con 0x556bfd817b60 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b0d7fa640 1 -- 172.21.15.77:0/2787523140 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f4b20004fc0 con 0x556bfd817b60 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b2e860740 1 -- 172.21.15.77:0/2787523140 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b20016070 con 0x556bfd817b60 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b0d7fa640 1 -- 172.21.15.77:0/2787523140 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f4b20019450 con 0x556bfd817b60 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b2e860740 10 client.27457 fetch_fsmap learned FSMap version 47 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.615+0000 7f4b2e860740 1 -- 172.21.15.77:0/2787523140 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x556bfd893300 con 0x556bfd817b60 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.616+0000 7f4b0d7fa640 1 -- 172.21.15.77:0/2787523140 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b20016070 con 0x556bfd817b60 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.616+0000 7f4b0d7fa640 1 -- 172.21.15.77:0/2787523140 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f4b20003900 con 0x556bfd817b60 2024-03-16T10:25:55.623 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.616+0000 7f4b2e860740 10 client.27457 fetch_fsmap finished waiting for FSMap version 47 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.616+0000 7f4b2e860740 10 client.27457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.616+0000 7f4b2e860740 20 client.27457 populate_metadata read hostname 'smithi077' 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.616+0000 7f4b2e860740 1 --2- 172.21.15.77:0/2787523140 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556bfd893be0 0x556bfd896000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.616+0000 7f4b2e860740 4 client.27457 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.616+0000 7f4b2e860740 1 -- 172.21.15.77:0/2787523140 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556bfd783d90 con 0x556bfd893be0 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.616+0000 7f4b2921b640 1 --2- 172.21.15.77:0/2787523140 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556bfd893be0 0x556bfd896000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.616+0000 7f4b0d7fa640 10 client.27457 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.616+0000 7f4b2921b640 1 --2- 172.21.15.77:0/2787523140 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556bfd893be0 0x556bfd896000 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8019b1ba2d63250 server_cookie=8666f03606af6704 in_seq=0 out_seq=0 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b0d7fa640 1 -- 172.21.15.77:0/2787523140 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556bfd783d90 con 0x556bfd893be0 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b0d7fa640 10 client.27457 handle_command_reply: tid=0 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2e860740 1 client.27457 shutdown 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2e860740 1 -- 172.21.15.77:0/2787523140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 msgr2=0x556bfd826970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2e860740 1 --2- 172.21.15.77:0/2787523140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd826970 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4b200047e0 tx=0x7f4b200092e0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2e860740 1 -- 172.21.15.77:0/2787523140 shutdown_connections 2024-03-16T10:25:55.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2e860740 1 --2- 172.21.15.77:0/2787523140 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x556bfd893be0 0x556bfd896000 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2e860740 1 --2- 172.21.15.77:0/2787523140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556bfd81ccd0 0x556bfd827930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2e860740 1 --2- 172.21.15.77:0/2787523140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556bfd817b60 0x556bfd826970 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b0d7fa640 0 client.27457 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2e860740 1 --2- 172.21.15.77:0/2787523140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556bfd817170 0x556bfd826430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2e860740 1 -- 172.21.15.77:0/2787523140 >> 172.21.15.77:0/2787523140 conn(0x556bfd8126e0 msgr2=0x556bfd8161d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2e860740 1 -- 172.21.15.77:0/2787523140 shutdown_connections 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2e860740 1 -- 172.21.15.77:0/2787523140 wait complete. 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2e860740 20 client.27457 trim_cache size 0 max 16384 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": { 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stdout: "09db8d3a-435c-4bb3-b3c5-a2f6a042810d": { 2024-03-16T10:25:55.625 INFO:teuthology.orchestra.run.smithi077.stdout: "path": "/", 2024-03-16T10:25:55.626 INFO:teuthology.orchestra.run.smithi077.stdout: "tag": "09db8d3a-435c-4bb3-b3c5-a2f6a042810d", 2024-03-16T10:25:55.626 INFO:teuthology.orchestra.run.smithi077.stdout: "options": "recursive,force" 2024-03-16T10:25:55.626 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:25:55.626 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:25:55.626 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:25:55.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2bc80640 1 -- 172.21.15.77:0/4156332777 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4b100a2880 msgr2=0x7f4b100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2bc80640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4b100a2880 0x7f4b100a4d40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009200 tx=0x7f4b1c04b040 comp rx=0 tx=0).stop 2024-03-16T10:25:55.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2bc80640 1 -- 172.21.15.77:0/4156332777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b241033b0 msgr2=0x7f4b2419f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:25:55.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2bc80640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b241033b0 0x7f4b2419f0b0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c00eb10 tx=0x7f4b1c00c2d0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2bc80640 1 -- 172.21.15.77:0/4156332777 shutdown_connections 2024-03-16T10:25:55.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2bc80640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b241033b0 0x7f4b2419f0b0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2bc80640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b241029f0 0x7f4b24198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2bc80640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4b100a2880 0x7f4b100a4d40 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2bc80640 1 --2- 172.21.15.77:0/4156332777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b24101800 0x7f4b24197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:25:55.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.617+0000 7f4b2bc80640 1 -- 172.21.15.77:0/4156332777 >> 172.21.15.77:0/4156332777 conn(0x7f4b240fcd90 msgr2=0x7f4b24105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:25:55.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.618+0000 7f4b2bc80640 1 -- 172.21.15.77:0/4156332777 shutdown_connections 2024-03-16T10:25:55.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:25:55.618+0000 7f4b2bc80640 1 -- 172.21.15.77:0/4156332777 wait complete. 2024-03-16T10:25:55.630 INFO:tasks.cephfs.filesystem:scrub status for tag:09db8d3a-435c-4bb3-b3c5-a2f6a042810d - {'path': '/', 'tag': '09db8d3a-435c-4bb3-b3c5-a2f6a042810d', 'options': 'recursive,force'} 2024-03-16T10:25:56.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:55 smithi077 ceph-mon[32400]: pgmap v616: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 370 op/s 2024-03-16T10:25:56.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:55 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:25:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:55 smithi193 ceph-mon[36311]: pgmap v616: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 370 op/s 2024-03-16T10:25:56.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:55 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:25:56.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:55 smithi192 ceph-mon[36354]: pgmap v616: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 370 op/s 2024-03-16T10:25:56.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:55 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:25:57.006 INFO:tasks.workunit:Stopping ['suites/blogbench.sh'] on client.0... 2024-03-16T10:25:57.006 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-16T10:25:57.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:56 smithi077 ceph-mon[32400]: scrub queued for path: / 2024-03-16T10:25:57.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:56 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:25:57.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:56 smithi077 ceph-mon[32400]: scrub summary: active paths [/] 2024-03-16T10:25:57.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:56 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:25:57.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:56 smithi077 ceph-mon[32400]: scrub summary: idle+waiting paths [/] 2024-03-16T10:25:57.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:56 smithi193 ceph-mon[36311]: scrub queued for path: / 2024-03-16T10:25:57.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:56 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:25:57.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:56 smithi193 ceph-mon[36311]: scrub summary: active paths [/] 2024-03-16T10:25:57.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:56 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:25:57.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:56 smithi193 ceph-mon[36311]: scrub summary: idle+waiting paths [/] 2024-03-16T10:25:57.288 DEBUG:teuthology.parallel:result is None 2024-03-16T10:25:57.289 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-16T10:25:57.395 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-16T10:25:57.396 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-16T10:25:57.407 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T10:25:57.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:56 smithi192 ceph-mon[36354]: scrub queued for path: / 2024-03-16T10:25:57.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:56 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:25:57.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:56 smithi192 ceph-mon[36354]: scrub summary: active paths [/] 2024-03-16T10:25:57.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:56 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:25:57.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:56 smithi192 ceph-mon[36354]: scrub summary: idle+waiting paths [/] 2024-03-16T10:25:57.418 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-16T10:25:57.428 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-16T10:25:58.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:57 smithi077 ceph-mon[32400]: pgmap v617: 97 pgs: 97 active+clean; 236 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 393 op/s 2024-03-16T10:25:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:57 smithi193 ceph-mon[36311]: pgmap v617: 97 pgs: 97 active+clean; 236 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 393 op/s 2024-03-16T10:25:58.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:57 smithi192 ceph-mon[36354]: pgmap v617: 97 pgs: 97 active+clean; 236 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 393 op/s 2024-03-16T10:25:59.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:25:58 smithi077 ceph-mon[32400]: pgmap v618: 97 pgs: 97 active+clean; 229 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 431 op/s 2024-03-16T10:25:59.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:25:58 smithi193 ceph-mon[36311]: pgmap v618: 97 pgs: 97 active+clean; 229 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 431 op/s 2024-03-16T10:25:59.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:25:58 smithi192 ceph-mon[36354]: pgmap v618: 97 pgs: 97 active+clean; 229 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 431 op/s 2024-03-16T10:26:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:00 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:26:01.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:00 smithi077 ceph-mon[32400]: pgmap v619: 97 pgs: 97 active+clean; 222 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 378 op/s 2024-03-16T10:26:01.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:00 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:26:01.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:00 smithi193 ceph-mon[36311]: pgmap v619: 97 pgs: 97 active+clean; 222 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 378 op/s 2024-03-16T10:26:01.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:00 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:26:01.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:00 smithi192 ceph-mon[36354]: pgmap v619: 97 pgs: 97 active+clean; 222 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 378 op/s 2024-03-16T10:26:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:01 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:26:02.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:01 smithi077 ceph-mon[32400]: scrub summary: idle 2024-03-16T10:26:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:01 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:26:02.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:01 smithi193 ceph-mon[36311]: scrub summary: idle 2024-03-16T10:26:02.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:01 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:26:02.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:01 smithi192 ceph-mon[36354]: scrub summary: idle 2024-03-16T10:26:03.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:02 smithi077 ceph-mon[32400]: pgmap v620: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 335 op/s 2024-03-16T10:26:03.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:02 smithi193 ceph-mon[36311]: pgmap v620: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 335 op/s 2024-03-16T10:26:03.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:02 smithi192 ceph-mon[36354]: pgmap v620: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 335 op/s 2024-03-16T10:26:06.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:05 smithi077 ceph-mon[32400]: pgmap v621: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 486 KiB/s wr, 236 op/s 2024-03-16T10:26:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:05 smithi193 ceph-mon[36311]: pgmap v621: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 486 KiB/s wr, 236 op/s 2024-03-16T10:26:06.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:05 smithi192 ceph-mon[36354]: pgmap v621: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 486 KiB/s wr, 236 op/s 2024-03-16T10:26:08.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:07 smithi077 ceph-mon[32400]: pgmap v622: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 486 KiB/s wr, 208 op/s 2024-03-16T10:26:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:07 smithi193 ceph-mon[36311]: pgmap v622: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 486 KiB/s wr, 208 op/s 2024-03-16T10:26:08.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:07 smithi192 ceph-mon[36354]: pgmap v622: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 486 KiB/s wr, 208 op/s 2024-03-16T10:26:10.219 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:09 smithi192 ceph-mon[36354]: pgmap v623: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 487 KiB/s wr, 162 op/s 2024-03-16T10:26:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:09 smithi077 ceph-mon[32400]: pgmap v623: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 487 KiB/s wr, 162 op/s 2024-03-16T10:26:10.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:09 smithi193 ceph-mon[36311]: pgmap v623: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 487 KiB/s wr, 162 op/s 2024-03-16T10:26:11.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:10 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:26:11.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:10 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:26:11.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:10 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:26:12.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:11 smithi077 ceph-mon[32400]: pgmap v624: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.2 KiB/s wr, 73 op/s 2024-03-16T10:26:12.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:11 smithi193 ceph-mon[36311]: pgmap v624: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.2 KiB/s wr, 73 op/s 2024-03-16T10:26:12.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:11 smithi192 ceph-mon[36354]: pgmap v624: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.2 KiB/s wr, 73 op/s 2024-03-16T10:26:13.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:12 smithi077 ceph-mon[32400]: pgmap v625: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.2 KiB/s wr, 32 op/s 2024-03-16T10:26:13.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:12 smithi193 ceph-mon[36311]: pgmap v625: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.2 KiB/s wr, 32 op/s 2024-03-16T10:26:13.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:12 smithi192 ceph-mon[36354]: pgmap v625: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.2 KiB/s wr, 32 op/s 2024-03-16T10:26:16.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:15 smithi077 ceph-mon[32400]: pgmap v626: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-16T10:26:16.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:15 smithi193 ceph-mon[36311]: pgmap v626: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-16T10:26:16.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:15 smithi192 ceph-mon[36354]: pgmap v626: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-16T10:26:18.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:17 smithi077 ceph-mon[32400]: pgmap v627: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-16T10:26:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:17 smithi193 ceph-mon[36311]: pgmap v627: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-16T10:26:18.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:17 smithi192 ceph-mon[36354]: pgmap v627: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-16T10:26:20.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:19 smithi077 ceph-mon[32400]: pgmap v628: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-16T10:26:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:19 smithi193 ceph-mon[36311]: pgmap v628: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-16T10:26:20.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:19 smithi192 ceph-mon[36354]: pgmap v628: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-16T10:26:22.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:21 smithi077 ceph-mon[32400]: pgmap v629: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-16T10:26:22.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:21 smithi193 ceph-mon[36311]: pgmap v629: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-16T10:26:22.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:21 smithi192 ceph-mon[36354]: pgmap v629: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-16T10:26:24.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:23 smithi077 ceph-mon[32400]: pgmap v630: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 13 op/s 2024-03-16T10:26:24.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:23 smithi193 ceph-mon[36311]: pgmap v630: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 13 op/s 2024-03-16T10:26:24.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:23 smithi192 ceph-mon[36354]: pgmap v630: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 13 op/s 2024-03-16T10:26:25.631 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-16T10:26:25.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-16T10:26:25.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.736+0000 7f10b6227640 1 -- 172.21.15.77:0/2066361888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b0107500 msgr2=0x7f10b0109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:25.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.736+0000 7f10b6227640 1 --2- 172.21.15.77:0/2066361888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b0107500 0x7f10b0109910 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009ca0 tx=0x7f10a802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.736+0000 7f10b6227640 1 -- 172.21.15.77:0/2066361888 shutdown_connections 2024-03-16T10:26:25.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.736+0000 7f10b6227640 1 --2- 172.21.15.77:0/2066361888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b0107500 0x7f10b0109910 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.736+0000 7f10b6227640 1 --2- 172.21.15.77:0/2066361888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10b0104bd0 0x7f10b0106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.736+0000 7f10b6227640 1 --2- 172.21.15.77:0/2066361888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10b01022a0 0x7f10b0104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.736+0000 7f10b6227640 1 -- 172.21.15.77:0/2066361888 >> 172.21.15.77:0/2066361888 conn(0x7f10b00fbc70 msgr2=0x7f10b00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:25.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.736+0000 7f10b6227640 1 -- 172.21.15.77:0/2066361888 shutdown_connections 2024-03-16T10:26:25.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.736+0000 7f10b6227640 1 -- 172.21.15.77:0/2066361888 wait complete. 2024-03-16T10:26:25.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10b6227640 1 Processor -- start 2024-03-16T10:26:25.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10b6227640 1 -- start start 2024-03-16T10:26:25.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10b6227640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10b01022a0 0x7f10b0197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10b6227640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b0104bd0 0x7f10b0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10b6227640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10b0107500 0x7f10b019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10b6227640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f10b010d460 con 0x7f10b0104bd0 2024-03-16T10:26:25.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10b6227640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f10b010d2e0 con 0x7f10b01022a0 2024-03-16T10:26:25.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10b6227640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f10b010d5e0 con 0x7f10b0107500 2024-03-16T10:26:25.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10aeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b0104bd0 0x7f10b0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10af7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10b01022a0 0x7f10b0197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10aeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b0104bd0 0x7f10b0198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46724/0 (socket says 172.21.15.77:46724) 2024-03-16T10:26:25.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10aeffd640 1 -- 172.21.15.77:0/3431086044 learned_addr learned my addr 172.21.15.77:0/3431086044 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:25.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10affff640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10b0107500 0x7f10b019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10aeffd640 1 -- 172.21.15.77:0/3431086044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10b0107500 msgr2=0x7f10b019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:25.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10aeffd640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10b0107500 0x7f10b019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.737+0000 7f10aeffd640 1 -- 172.21.15.77:0/3431086044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10b01022a0 msgr2=0x7f10b0197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:25.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.738+0000 7f10aeffd640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10b01022a0 0x7f10b0197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.738+0000 7f10aeffd640 1 -- 172.21.15.77:0/3431086044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b019f720 con 0x7f10b0104bd0 2024-03-16T10:26:25.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.738+0000 7f10af7fe640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10b01022a0 0x7f10b0197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.738+0000 7f10affff640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10b0107500 0x7f10b019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.738+0000 7f10aeffd640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b0104bd0 0x7f10b0198040 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f10a000e9c0 tx=0x7f10a000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.738+0000 7f10acff9640 1 -- 172.21.15.77:0/3431086044 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a000cd80 con 0x7f10b0104bd0 2024-03-16T10:26:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.738+0000 7f10acff9640 1 -- 172.21.15.77:0/3431086044 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10a0005500 con 0x7f10b0104bd0 2024-03-16T10:26:25.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.738+0000 7f10b6227640 1 -- 172.21.15.77:0/3431086044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b019fa10 con 0x7f10b0104bd0 2024-03-16T10:26:25.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.738+0000 7f10acff9640 1 -- 172.21.15.77:0/3431086044 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0010690 con 0x7f10b0104bd0 2024-03-16T10:26:25.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.738+0000 7f10b6227640 1 -- 172.21.15.77:0/3431086044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b019ffb0 con 0x7f10b0104bd0 2024-03-16T10:26:25.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.740+0000 7f10acff9640 1 -- 172.21.15.77:0/3431086044 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f10a00046c0 con 0x7f10b0104bd0 2024-03-16T10:26:25.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.741+0000 7f10acff9640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f10980a2880 0x7f10980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.741+0000 7f10af7fe640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f10980a2880 0x7f10980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.741+0000 7f10acff9640 1 -- 172.21.15.77:0/3431086044 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f10a00cecd0 con 0x7f10b0104bd0 2024-03-16T10:26:25.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.741+0000 7f10af7fe640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f10980a2880 0x7f10980a4d40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009fc0 tx=0x7f10a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:25.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.741+0000 7f10b8e13740 1 Processor -- start 2024-03-16T10:26:25.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.741+0000 7f10b8e13740 1 -- start start 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.741+0000 7f10b8e13740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf5291590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.741+0000 7f10b8e13740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf5296660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.741+0000 7f10b8e13740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5296cd0 0x559bf5299160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.741+0000 7f10b8e13740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559bf51f6500 con 0x559bf5291170 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.741+0000 7f10b8e13740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559bf519ebe0 con 0x559bf5296cd0 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.741+0000 7f10b8e13740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559bf51f25a0 con 0x559bf5291b60 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f10af7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf5291590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f10af7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf5291590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46730/0 (socket says 172.21.15.77:46730) 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f10af7fe640 1 -- 172.21.15.77:0/728118587 learned_addr learned my addr 172.21.15.77:0/728118587 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f10aeffd640 1 --2- 172.21.15.77:0/728118587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf5296660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f10affff640 1 --2- 172.21.15.77:0/728118587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5296cd0 0x559bf5299160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f10af7fe640 1 -- 172.21.15.77:0/728118587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 msgr2=0x559bf5296660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f10af7fe640 1 --2- 172.21.15.77:0/728118587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf5296660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f10af7fe640 1 -- 172.21.15.77:0/728118587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5296cd0 msgr2=0x559bf5299160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:25.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f10af7fe640 1 --2- 172.21.15.77:0/728118587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5296cd0 0x559bf5299160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f10af7fe640 1 -- 172.21.15.77:0/728118587 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bf51b24b0 con 0x559bf5291170 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f10aeffd640 1 --2- 172.21.15.77:0/728118587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf5296660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f10af7fe640 1 --2- 172.21.15.77:0/728118587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf5291590 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f10a40042c0 tx=0x7f10a4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48ade33ffdc4189 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f1095ffb640 1 -- 172.21.15.77:0/728118587 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a400f480 con 0x559bf5291170 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f1095ffb640 1 -- 172.21.15.77:0/728118587 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10a400fb00 con 0x559bf5291170 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.742+0000 7f1095ffb640 1 -- 172.21.15.77:0/728118587 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a401cbe0 con 0x559bf5291170 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.743+0000 7f10b8e13740 1 -- 172.21.15.77:0/728118587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 msgr2=0x559bf5291590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.743+0000 7f10b8e13740 1 --2- 172.21.15.77:0/728118587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf5291590 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f10a40042c0 tx=0x7f10a4014000 comp rx=0 tx=0).stop 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.743+0000 7f10b8e13740 1 -- 172.21.15.77:0/728118587 shutdown_connections 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.743+0000 7f10b8e13740 1 --2- 172.21.15.77:0/728118587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5296cd0 0x559bf5299160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.743+0000 7f10b8e13740 1 --2- 172.21.15.77:0/728118587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf5296660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.743+0000 7f10b8e13740 1 --2- 172.21.15.77:0/728118587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf5291590 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.743+0000 7f10b8e13740 1 -- 172.21.15.77:0/728118587 >> 172.21.15.77:0/728118587 conn(0x559bf528c6e0 msgr2=0x559bf528cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.743+0000 7f10b8e13740 1 -- 172.21.15.77:0/728118587 shutdown_connections 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.743+0000 7f10b8e13740 1 -- 172.21.15.77:0/728118587 wait complete. 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.743+0000 7f10b8e13740 1 -- start start 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.743+0000 7f10b8e13740 1 Processor -- start 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10b8e13740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5291170 0x559bf529b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10b8e13740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf529bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10b8e13740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5296cd0 0x559bf52a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10b8e13740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559bf5299c80 con 0x559bf5296cd0 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10b8e13740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559bf51b2160 con 0x559bf5291170 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10b8e13740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559bf5299e00 con 0x559bf5291b60 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10b8e13740 10 client.0 _ll_register_callbacks cb 0x559bf5196150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5296cd0 0x559bf52a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10af7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5291170 0x559bf529b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10affff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5296cd0 0x559bf52a2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46746/0 (socket says 172.21.15.77:46746) 2024-03-16T10:26:25.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10affff640 1 -- 172.21.15.77:0/4231131442 learned_addr learned my addr 172.21.15.77:0/4231131442 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:25.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10af7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5291170 0x559bf529b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:43526/0 (socket says 172.21.15.77:43526) 2024-03-16T10:26:25.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10aeffd640 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf529bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10affff640 1 -- 172.21.15.77:0/4231131442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 msgr2=0x559bf529bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:25.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10affff640 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf529bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10affff640 1 -- 172.21.15.77:0/4231131442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5291170 msgr2=0x559bf529b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:25.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10affff640 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5291170 0x559bf529b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10affff640 1 -- 172.21.15.77:0/4231131442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bf4fd47d0 con 0x559bf5296cd0 2024-03-16T10:26:25.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10af7fe640 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5291170 0x559bf529b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:25.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10affff640 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5296cd0 0x559bf52a2ba0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009ca0 tx=0x7f10a8009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa6f3ee1516b59 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:25.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f108effd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:26:25.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f108effd640 1 -- 172.21.15.77:0/4231131442 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8044070 con 0x559bf5296cd0 2024-03-16T10:26:25.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f108effd640 1 -- 172.21.15.77:0/4231131442 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f10a8032890 con 0x559bf5296cd0 2024-03-16T10:26:25.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.744+0000 7f10b8e13740 1 -- 172.21.15.77:0/4231131442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10a400fb00 con 0x559bf5296cd0 2024-03-16T10:26:25.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.745+0000 7f108effd640 1 -- 172.21.15.77:0/4231131442 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f10a8039040 con 0x559bf5296cd0 2024-03-16T10:26:25.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.745+0000 7f10b8e13740 10 client.17715 fetch_fsmap learned FSMap version 49 2024-03-16T10:26:25.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.745+0000 7f10b8e13740 1 -- 172.21.15.77:0/4231131442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x559bf4f0a3f0 con 0x559bf5296cd0 2024-03-16T10:26:25.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.745+0000 7f108effd640 1 -- 172.21.15.77:0/4231131442 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10a400fb00 con 0x559bf5296cd0 2024-03-16T10:26:25.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.745+0000 7f108effd640 1 -- 172.21.15.77:0/4231131442 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 49) ==== 8378+0+0 (secure 0 0 0) 0x7f10a8004040 con 0x559bf5296cd0 2024-03-16T10:26:25.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.745+0000 7f10b8e13740 10 client.17715 fetch_fsmap finished waiting for FSMap version 49 2024-03-16T10:26:25.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.745+0000 7f10b8e13740 10 client.17715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:26:25.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.745+0000 7f10b8e13740 20 client.17715 populate_metadata read hostname 'smithi077' 2024-03-16T10:26:25.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.745+0000 7f10b8e13740 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559bf52a7140 0x559bf52a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.745+0000 7f10b8e13740 4 client.17715 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:26:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.745+0000 7f10b8e13740 1 -- 172.21.15.77:0/4231131442 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559bf5203a40 con 0x559bf52a7140 2024-03-16T10:26:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.745+0000 7f10af7fe640 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559bf52a7140 0x559bf52a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f10af7fe640 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559bf52a7140 0x559bf52a95e0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42df1be3ffc50719 server_cookie=4550d390e5a166f7 in_seq=0 out_seq=0 2024-03-16T10:26:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f108effd640 10 client.17715 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:26:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f108effd640 1 -- 172.21.15.77:0/4231131442 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559bf5203a40 con 0x559bf52a7140 2024-03-16T10:26:25.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f108effd640 10 client.17715 handle_command_reply: tid=0 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f10b8e13740 1 client.17715 shutdown 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f10b8e13740 1 -- 172.21.15.77:0/4231131442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5296cd0 msgr2=0x559bf52a2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f10b8e13740 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5296cd0 0x559bf52a2ba0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009ca0 tx=0x7f10a8009870 comp rx=0 tx=0).stop 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f10b8e13740 1 -- 172.21.15.77:0/4231131442 shutdown_connections 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f10b8e13740 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559bf52a7140 0x559bf52a95e0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f10b8e13740 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5296cd0 0x559bf52a2ba0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f10b8e13740 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf529bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f108effd640 0 client.17715 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f10b8e13740 1 --2- 172.21.15.77:0/4231131442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5291170 0x559bf529b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.746+0000 7f10b8e13740 1 -- 172.21.15.77:0/4231131442 >> 172.21.15.77:0/4231131442 conn(0x559bf528c6e0 msgr2=0x559bf528b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.747+0000 7f10b8e13740 1 -- 172.21.15.77:0/4231131442 shutdown_connections 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.747+0000 7f10b8e13740 1 -- 172.21.15.77:0/4231131442 wait complete. 2024-03-16T10:26:25.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.747+0000 7f10b8e13740 20 client.17715 trim_cache size 0 max 16384 2024-03-16T10:26:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10b8e13740 1 Processor -- start 2024-03-16T10:26:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10b8e13740 1 -- start start 2024-03-16T10:26:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10b8e13740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf528b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10b8e13740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf52a2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10b8e13740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5296cd0 0x559bf528b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10b8e13740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01a8b70 con 0x559bf5291170 2024-03-16T10:26:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10b8e13740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01aa770 con 0x559bf5296cd0 2024-03-16T10:26:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10b8e13740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01a92a0 con 0x559bf5291b60 2024-03-16T10:26:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10af7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf528b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10af7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf528b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46760/0 (socket says 172.21.15.77:46760) 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10af7fe640 1 -- 172.21.15.77:0/2513172205 learned_addr learned my addr 172.21.15.77:0/2513172205 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10aeffd640 1 --2- 172.21.15.77:0/2513172205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf52a2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10affff640 1 --2- 172.21.15.77:0/2513172205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5296cd0 0x559bf528b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10af7fe640 1 -- 172.21.15.77:0/2513172205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 msgr2=0x559bf52a2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10af7fe640 1 --2- 172.21.15.77:0/2513172205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf52a2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10af7fe640 1 -- 172.21.15.77:0/2513172205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5296cd0 msgr2=0x559bf528b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10af7fe640 1 --2- 172.21.15.77:0/2513172205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5296cd0 0x559bf528b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.764+0000 7f10af7fe640 1 -- 172.21.15.77:0/2513172205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bf4eca0f0 con 0x559bf5291170 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10af7fe640 1 --2- 172.21.15.77:0/2513172205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf528b070 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f10a40042c0 tx=0x7f10a4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=977c534e6893a71f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10957fa640 1 -- 172.21.15.77:0/2513172205 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a40043b0 con 0x559bf5291170 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10957fa640 1 -- 172.21.15.77:0/2513172205 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10a4005520 con 0x559bf5291170 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10957fa640 1 -- 172.21.15.77:0/2513172205 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a4034810 con 0x559bf5291170 2024-03-16T10:26:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10b8e13740 1 -- 172.21.15.77:0/2513172205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 msgr2=0x559bf528b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10b8e13740 1 --2- 172.21.15.77:0/2513172205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf528b070 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f10a40042c0 tx=0x7f10a4018000 comp rx=0 tx=0).stop 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10b8e13740 1 -- 172.21.15.77:0/2513172205 shutdown_connections 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10b8e13740 1 --2- 172.21.15.77:0/2513172205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5296cd0 0x559bf528b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10b8e13740 1 --2- 172.21.15.77:0/2513172205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5291b60 0x559bf52a2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10b8e13740 1 --2- 172.21.15.77:0/2513172205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf528b070 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10b8e13740 1 -- 172.21.15.77:0/2513172205 >> 172.21.15.77:0/2513172205 conn(0x559bf528c6e0 msgr2=0x559bf5292fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10b8e13740 1 -- 172.21.15.77:0/2513172205 shutdown_connections 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.765+0000 7f10b8e13740 1 -- 172.21.15.77:0/2513172205 wait complete. 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10b8e13740 1 -- start start 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10b8e13740 1 Processor -- start 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10b8e13740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf52a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10b8e13740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5291b60 0x559bf52a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10b8e13740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5296cd0 0x559bf52a1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10b8e13740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01aa0a0 con 0x559bf5291170 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10b8e13740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01a99a0 con 0x559bf5291b60 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10b8e13740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f10b01a6b50 con 0x559bf5296cd0 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10b8e13740 10 client.0 _ll_register_callbacks cb 0x7f10a4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10af7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf52a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10af7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf52a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46768/0 (socket says 172.21.15.77:46768) 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10af7fe640 1 -- 172.21.15.77:0/2015327943 learned_addr learned my addr 172.21.15.77:0/2015327943 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10aeffd640 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5291b60 0x559bf52a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10affff640 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5296cd0 0x559bf52a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10af7fe640 1 -- 172.21.15.77:0/2015327943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5296cd0 msgr2=0x559bf52a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10af7fe640 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5296cd0 0x559bf52a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10af7fe640 1 -- 172.21.15.77:0/2015327943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5291b60 msgr2=0x559bf52a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10af7fe640 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5291b60 0x559bf52a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.766+0000 7f10af7fe640 1 -- 172.21.15.77:0/2015327943 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bf51bd9d0 con 0x559bf5291170 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f10af7fe640 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf52a01b0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f10a4036040 tx=0x7f10a4003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452c3b0820ff75dd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f108effd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f108effd640 1 -- 172.21.15.77:0/2015327943 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a40043b0 con 0x559bf5291170 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f108effd640 1 -- 172.21.15.77:0/2015327943 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f10a400fd30 con 0x559bf5291170 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f10b8e13740 1 -- 172.21.15.77:0/2015327943 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10a4005520 con 0x559bf5291170 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f108effd640 1 -- 172.21.15.77:0/2015327943 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f10a4034590 con 0x559bf5291170 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f10b8e13740 10 client.17721 fetch_fsmap learned FSMap version 49 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f10b8e13740 1 -- 172.21.15.77:0/2015327943 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x559bf52a2e30 con 0x559bf5291170 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f108effd640 1 -- 172.21.15.77:0/2015327943 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10a4005520 con 0x559bf5291170 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f108effd640 1 -- 172.21.15.77:0/2015327943 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 49) ==== 8378+0+0 (secure 0 0 0) 0x7f10a401a070 con 0x559bf5291170 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f10b8e13740 10 client.17721 fetch_fsmap finished waiting for FSMap version 49 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f10b8e13740 10 client.17721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:26:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f10b8e13740 20 client.17721 populate_metadata read hostname 'smithi077' 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f10b8e13740 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559bf530d9a0 0x559bf530fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f10b8e13740 4 client.17721 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f10b8e13740 1 -- 172.21.15.77:0/2015327943 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559bf51fdd90 con 0x559bf530d9a0 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.767+0000 7f10aeffd640 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559bf530d9a0 0x559bf530fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f108effd640 10 client.17721 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10aeffd640 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559bf530d9a0 0x559bf530fdc0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb5b968026806215 server_cookie=b1c35affda0c1188 in_seq=0 out_seq=0 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f108effd640 1 -- 172.21.15.77:0/2015327943 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559bf51fdd90 con 0x559bf530d9a0 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f108effd640 10 client.17721 handle_command_reply: tid=0 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10b8e13740 1 client.17721 shutdown 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10b8e13740 1 -- 172.21.15.77:0/2015327943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 msgr2=0x559bf52a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10b8e13740 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf52a01b0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f10a4036040 tx=0x7f10a4003f90 comp rx=0 tx=0).stop 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10b8e13740 1 -- 172.21.15.77:0/2015327943 shutdown_connections 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10b8e13740 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bf5296cd0 0x559bf52a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10b8e13740 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559bf5291b60 0x559bf52a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10b8e13740 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x559bf530d9a0 0x559bf530fdc0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10b8e13740 1 --2- 172.21.15.77:0/2015327943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559bf5291170 0x559bf52a01b0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10b8e13740 1 -- 172.21.15.77:0/2015327943 >> 172.21.15.77:0/2015327943 conn(0x559bf528c6e0 msgr2=0x559bf528f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:25.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f108effd640 0 client.17721 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:26:25.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10b8e13740 1 -- 172.21.15.77:0/2015327943 shutdown_connections 2024-03-16T10:26:25.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10b8e13740 1 -- 172.21.15.77:0/2015327943 wait complete. 2024-03-16T10:26:25.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.768+0000 7f10b8e13740 20 client.17721 trim_cache size 0 max 16384 2024-03-16T10:26:25.777 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:26:25.777 INFO:teuthology.orchestra.run.smithi077.stdout: "status": "no active scrubs running", 2024-03-16T10:26:25.778 INFO:teuthology.orchestra.run.smithi077.stdout: "scrubs": {} 2024-03-16T10:26:25.778 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:26:25.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.769+0000 7f10b6227640 1 -- 172.21.15.77:0/3431086044 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f10980a2880 msgr2=0x7f10980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.769+0000 7f10b6227640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f10980a2880 0x7f10980a4d40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009fc0 tx=0x7f10a4005f90 comp rx=0 tx=0).stop 2024-03-16T10:26:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.769+0000 7f10b6227640 1 -- 172.21.15.77:0/3431086044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b0104bd0 msgr2=0x7f10b0198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.769+0000 7f10b6227640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b0104bd0 0x7f10b0198040 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f10a000e9c0 tx=0x7f10a000ee80 comp rx=0 tx=0).stop 2024-03-16T10:26:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.769+0000 7f10b6227640 1 -- 172.21.15.77:0/3431086044 shutdown_connections 2024-03-16T10:26:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.769+0000 7f10b6227640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f10980a2880 0x7f10980a4d40 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.769+0000 7f10b6227640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10b0107500 0x7f10b019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.769+0000 7f10b6227640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10b0104bd0 0x7f10b0198040 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.769+0000 7f10b6227640 1 --2- 172.21.15.77:0/3431086044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10b01022a0 0x7f10b0197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.769+0000 7f10b6227640 1 -- 172.21.15.77:0/3431086044 >> 172.21.15.77:0/3431086044 conn(0x7f10b00fbc70 msgr2=0x7f10b0102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.769+0000 7f10b6227640 1 -- 172.21.15.77:0/3431086044 shutdown_connections 2024-03-16T10:26:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.769+0000 7f10b6227640 1 -- 172.21.15.77:0/3431086044 wait complete. 2024-03-16T10:26:25.780 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-16T10:26:25.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:26:25.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.884+0000 7f49476e8640 1 -- 172.21.15.77:0/2037336290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49401022b0 msgr2=0x7f4940108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:25.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.884+0000 7f49476e8640 1 --2- 172.21.15.77:0/2037336290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49401022b0 0x7f4940108b40 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4938009ca0 tx=0x7f493802c0f0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 -- 172.21.15.77:0/2037336290 shutdown_connections 2024-03-16T10:26:25.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 --2- 172.21.15.77:0/2037336290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49401022b0 0x7f4940108b40 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 --2- 172.21.15.77:0/2037336290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49401018f0 0x7f4940101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 --2- 172.21.15.77:0/2037336290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4940100700 0x7f4940100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 -- 172.21.15.77:0/2037336290 >> 172.21.15.77:0/2037336290 conn(0x7f49400fbc70 msgr2=0x7f49400fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:25.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 -- 172.21.15.77:0/2037336290 shutdown_connections 2024-03-16T10:26:25.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 -- 172.21.15.77:0/2037336290 wait complete. 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 Processor -- start 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 -- start start 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4940100700 0x7f4940197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49401018f0 0x7f4940198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49401022b0 0x7f494019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f494010c380 con 0x7f49401018f0 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f494010c200 con 0x7f4940100700 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f49476e8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f494010c500 con 0x7f49401022b0 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f4944c83640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49401018f0 0x7f4940198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f4944c83640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49401018f0 0x7f4940198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46778/0 (socket says 172.21.15.77:46778) 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f4944c83640 1 -- 172.21.15.77:0/1102572036 learned_addr learned my addr 172.21.15.77:0/1102572036 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.885+0000 7f4945484640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4940100700 0x7f4940197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.886+0000 7f4945c85640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49401022b0 0x7f494019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.886+0000 7f4944c83640 1 -- 172.21.15.77:0/1102572036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49401022b0 msgr2=0x7f494019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:25.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.886+0000 7f4944c83640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49401022b0 0x7f494019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.886+0000 7f4944c83640 1 -- 172.21.15.77:0/1102572036 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4940100700 msgr2=0x7f4940197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:25.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.886+0000 7f4944c83640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4940100700 0x7f4940197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:25.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.886+0000 7f4944c83640 1 -- 172.21.15.77:0/1102572036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f494019f6d0 con 0x7f49401018f0 2024-03-16T10:26:25.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.886+0000 7f4944c83640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49401018f0 0x7f4940198040 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f493000e9f0 tx=0x7f493000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:25.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.886+0000 7f49367fc640 1 -- 172.21.15.77:0/1102572036 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493000cd80 con 0x7f49401018f0 2024-03-16T10:26:25.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.886+0000 7f49367fc640 1 -- 172.21.15.77:0/1102572036 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4930010910 con 0x7f49401018f0 2024-03-16T10:26:25.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.886+0000 7f49476e8640 1 -- 172.21.15.77:0/1102572036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f494019f9c0 con 0x7f49401018f0 2024-03-16T10:26:25.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.886+0000 7f49367fc640 1 -- 172.21.15.77:0/1102572036 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930018990 con 0x7f49401018f0 2024-03-16T10:26:25.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.886+0000 7f49476e8640 1 -- 172.21.15.77:0/1102572036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f494019ff60 con 0x7f49401018f0 2024-03-16T10:26:25.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.887+0000 7f49476e8640 1 -- 172.21.15.77:0/1102572036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49401a01f0 con 0x7f49401018f0 2024-03-16T10:26:25.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.888+0000 7f49367fc640 1 -- 172.21.15.77:0/1102572036 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f4930010ac0 con 0x7f49401018f0 2024-03-16T10:26:25.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.889+0000 7f49367fc640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f492c0a2880 0x7f492c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:25.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.889+0000 7f4945484640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f492c0a2880 0x7f492c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:25.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.889+0000 7f49367fc640 1 -- 172.21.15.77:0/1102572036 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f49300ceca0 con 0x7f49401018f0 2024-03-16T10:26:25.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.889+0000 7f4945484640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f492c0a2880 0x7f492c0a4d40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4940101750 tx=0x7f493c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:25.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:25.898+0000 7f49367fc640 1 -- 172.21.15.77:0/1102572036 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f493009b780 con 0x7f49401018f0 2024-03-16T10:26:26.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.129+0000 7f49476e8640 1 -- 172.21.15.77:0/1102572036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49401a0460 con 0x7f49401018f0 2024-03-16T10:26:26.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.130+0000 7f49367fc640 1 -- 172.21.15.77:0/1102572036 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11942 (secure 0 0 0) 0x7f493009f820 con 0x7f49401018f0 2024-03-16T10:26:26.131 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:26:26.131 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:26:09.755353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26800":{"gid":26800,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/946505511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":946505511},{"type":"v1","addr":"172.21.15.193:6835","nonce":946505511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[24454,24475,24482,24455,14658]},"id":1}]} 2024-03-16T10:26:26.132 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 49 2024-03-16T10:26:26.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.133+0000 7f49476e8640 1 -- 172.21.15.77:0/1102572036 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f492c0a2880 msgr2=0x7f492c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.133+0000 7f49476e8640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f492c0a2880 0x7f492c0a4d40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4940101750 tx=0x7f493c005f90 comp rx=0 tx=0).stop 2024-03-16T10:26:26.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.133+0000 7f49476e8640 1 -- 172.21.15.77:0/1102572036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49401018f0 msgr2=0x7f4940198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.133+0000 7f49476e8640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49401018f0 0x7f4940198040 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f493000e9f0 tx=0x7f493000eeb0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.133+0000 7f49476e8640 1 -- 172.21.15.77:0/1102572036 shutdown_connections 2024-03-16T10:26:26.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.133+0000 7f49476e8640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f492c0a2880 0x7f492c0a4d40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.133+0000 7f49476e8640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49401022b0 0x7f494019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.133+0000 7f49476e8640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49401018f0 0x7f4940198040 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.133+0000 7f49476e8640 1 --2- 172.21.15.77:0/1102572036 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4940100700 0x7f4940197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.133+0000 7f49476e8640 1 -- 172.21.15.77:0/1102572036 >> 172.21.15.77:0/1102572036 conn(0x7f49400fbc70 msgr2=0x7f49400fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.133+0000 7f49476e8640 1 -- 172.21.15.77:0/1102572036 shutdown_connections 2024-03-16T10:26:26.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.133+0000 7f49476e8640 1 -- 172.21.15.77:0/1102572036 wait complete. 2024-03-16T10:26:26.145 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-16T10:26:26.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:25 smithi077 ceph-mon[32400]: pgmap v631: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T10:26:26.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:25 smithi193 ceph-mon[36311]: pgmap v631: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T10:26:26.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.252+0000 7f0fad4b2640 1 -- 172.21.15.77:0/2755227250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fa81018d0 msgr2=0x7f0fa8101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.252+0000 7f0fad4b2640 1 --2- 172.21.15.77:0/2755227250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fa81018d0 0x7f0fa8101d50 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f0f90009900 tx=0x7f0f9002bc30 comp rx=0 tx=0).stop 2024-03-16T10:26:26.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.252+0000 7f0fad4b2640 1 -- 172.21.15.77:0/2755227250 shutdown_connections 2024-03-16T10:26:26.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.252+0000 7f0fad4b2640 1 --2- 172.21.15.77:0/2755227250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fa8102290 0x7f0fa8108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.252+0000 7f0fad4b2640 1 --2- 172.21.15.77:0/2755227250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fa81018d0 0x7f0fa8101d50 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.252+0000 7f0fad4b2640 1 --2- 172.21.15.77:0/2755227250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fa81006e0 0x7f0fa8100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.252+0000 7f0fad4b2640 1 -- 172.21.15.77:0/2755227250 >> 172.21.15.77:0/2755227250 conn(0x7f0fa80fbc70 msgr2=0x7f0fa80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.252+0000 7f0fad4b2640 1 -- 172.21.15.77:0/2755227250 shutdown_connections 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.252+0000 7f0fad4b2640 1 -- 172.21.15.77:0/2755227250 wait complete. 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.252+0000 7f0fad4b2640 1 Processor -- start 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fad4b2640 1 -- start start 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fad4b2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fa81006e0 0x7f0fa8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fad4b2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fa81018d0 0x7f0fa81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fad4b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fa8102290 0x7f0fa819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fad4b2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa810c360 con 0x7f0fa8102290 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fad4b2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa810c1e0 con 0x7f0fa81018d0 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fad4b2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa810c4e0 con 0x7f0fa81006e0 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fa77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fa8102290 0x7f0fa819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fa77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fa8102290 0x7f0fa819f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46806/0 (socket says 172.21.15.77:46806) 2024-03-16T10:26:26.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fa77fe640 1 -- 172.21.15.77:0/1337249951 learned_addr learned my addr 172.21.15.77:0/1337249951 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fa6ffd640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fa81006e0 0x7f0fa8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fa67fc640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fa81018d0 0x7f0fa81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fa77fe640 1 -- 172.21.15.77:0/1337249951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fa81006e0 msgr2=0x7f0fa8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fa81006e0 0x7f0fa8197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fa77fe640 1 -- 172.21.15.77:0/1337249951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fa81018d0 msgr2=0x7f0fa81980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fa81018d0 0x7f0fa81980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fa77fe640 1 -- 172.21.15.77:0/1337249951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa819f710 con 0x7f0fa8102290 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fa8102290 0x7f0fa819f0a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f0fa000e970 tx=0x7f0fa000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0f9ffff640 1 -- 172.21.15.77:0/1337249951 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa000ccc0 con 0x7f0fa8102290 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0fad4b2640 1 -- 172.21.15.77:0/1337249951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fa819fa00 con 0x7f0fa8102290 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0f9ffff640 1 -- 172.21.15.77:0/1337249951 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fa000ce60 con 0x7f0fa8102290 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.253+0000 7f0f9ffff640 1 -- 172.21.15.77:0/1337249951 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0010450 con 0x7f0fa8102290 2024-03-16T10:26:26.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.254+0000 7f0fad4b2640 1 -- 172.21.15.77:0/1337249951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fa819ff20 con 0x7f0fa8102290 2024-03-16T10:26:26.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.256+0000 7f0f9ffff640 1 -- 172.21.15.77:0/1337249951 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f0fa0005020 con 0x7f0fa8102290 2024-03-16T10:26:26.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.256+0000 7f0f9ffff640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0f940a2800 0x7f0f940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.256+0000 7f0fb0180740 1 Processor -- start 2024-03-16T10:26:26.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.256+0000 7f0fb0180740 1 -- start start 2024-03-16T10:26:26.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.256+0000 7f0fa6ffd640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0f940a2800 0x7f0f940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0f9ffff640 1 -- 172.21.15.77:0/1337249951 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f0fa00cedd0 con 0x7f0fa8102290 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fb0180740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41262f850 0x55c41262fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fb0180740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c412630240 0x55c412634d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fb0180740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c4126353b0 0x55c412637840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fb0180740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c412594490 con 0x55c412630240 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fb0180740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c41253cc90 con 0x55c4126353b0 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fb0180740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c412590530 con 0x55c41262f850 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c412630240 0x55c412634d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa67fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c412630240 0x55c412634d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46818/0 (socket says 172.21.15.77:46818) 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa6ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41262f850 0x55c41262fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa67fc640 1 -- 172.21.15.77:0/3300743181 learned_addr learned my addr 172.21.15.77:0/3300743181 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/3300743181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c4126353b0 0x55c412637840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa6ffd640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0f940a2800 0x7f0f940a4cc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8101730 tx=0x7f0f98009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa67fc640 1 -- 172.21.15.77:0/3300743181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41262f850 msgr2=0x55c41262fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa67fc640 1 --2- 172.21.15.77:0/3300743181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41262f850 0x55c41262fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa67fc640 1 -- 172.21.15.77:0/3300743181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c4126353b0 msgr2=0x55c412637840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa67fc640 1 --2- 172.21.15.77:0/3300743181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c4126353b0 0x55c412637840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa67fc640 1 -- 172.21.15.77:0/3300743181 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4125f7500 con 0x55c412630240 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/3300743181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c4126353b0 0x55c412637840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.257+0000 7f0fa67fc640 1 --2- 172.21.15.77:0/3300743181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c412630240 0x55c412634d40 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f0f90009900 tx=0x7f0f90009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a6e78016b8d497 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.258+0000 7f0f9d7fa640 1 -- 172.21.15.77:0/3300743181 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9002cb80 con 0x55c412630240 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.258+0000 7f0f9d7fa640 1 -- 172.21.15.77:0/3300743181 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f90005040 con 0x55c412630240 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.258+0000 7f0f9d7fa640 1 -- 172.21.15.77:0/3300743181 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f900048e0 con 0x55c412630240 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.258+0000 7f0fb0180740 1 -- 172.21.15.77:0/3300743181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c412630240 msgr2=0x55c412634d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.258+0000 7f0fb0180740 1 --2- 172.21.15.77:0/3300743181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c412630240 0x55c412634d40 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f0f90009900 tx=0x7f0f90009560 comp rx=0 tx=0).stop 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.258+0000 7f0fb0180740 1 -- 172.21.15.77:0/3300743181 shutdown_connections 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.258+0000 7f0fb0180740 1 --2- 172.21.15.77:0/3300743181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c4126353b0 0x55c412637840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.258+0000 7f0fb0180740 1 --2- 172.21.15.77:0/3300743181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c412630240 0x55c412634d40 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.258+0000 7f0fb0180740 1 --2- 172.21.15.77:0/3300743181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41262f850 0x55c41262fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.258+0000 7f0fb0180740 1 -- 172.21.15.77:0/3300743181 >> 172.21.15.77:0/3300743181 conn(0x55c41262adc0 msgr2=0x55c41262b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.258+0000 7f0fb0180740 1 -- 172.21.15.77:0/3300743181 shutdown_connections 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.258+0000 7f0fb0180740 1 -- 172.21.15.77:0/3300743181 wait complete. 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fb0180740 1 -- start start 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fb0180740 1 Processor -- start 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fb0180740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c41262f850 0x55c412639d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fb0180740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c412630240 0x55c41263a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fb0180740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c4126353b0 0x55c4126412c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fb0180740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c412638300 con 0x55c41262f850 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fb0180740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55c412550230 con 0x55c4126353b0 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fb0180740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c412638480 con 0x55c412630240 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fb0180740 10 client.0 _ll_register_callbacks cb 0x55c41258c010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:26:26.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fa6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c41262f850 0x55c412639d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fa6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c41262f850 0x55c412639d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46828/0 (socket says 172.21.15.77:46828) 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fa6ffd640 1 -- 172.21.15.77:0/1051589078 learned_addr learned my addr 172.21.15.77:0/1051589078 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fa67fc640 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c412630240 0x55c41263a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c4126353b0 0x55c4126412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fa6ffd640 1 -- 172.21.15.77:0/1051589078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c412630240 msgr2=0x55c41263a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fa6ffd640 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c412630240 0x55c41263a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fa6ffd640 1 -- 172.21.15.77:0/1051589078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c4126353b0 msgr2=0x55c4126412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fa6ffd640 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c4126353b0 0x55c4126412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.259+0000 7f0fa6ffd640 1 -- 172.21.15.77:0/1051589078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c41255f100 con 0x55c41262f850 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0fa6ffd640 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c41262f850 0x55c412639d80 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f0f98004070 tx=0x7f0f98004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418d3cf5390647a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0f8e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0f8e7fc640 1 -- 172.21.15.77:0/1051589078 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9800f930 con 0x55c41262f850 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0fb0180740 1 -- 172.21.15.77:0/1051589078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f90005040 con 0x55c41262f850 2024-03-16T10:26:26.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0f8e7fc640 1 -- 172.21.15.77:0/1051589078 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f0f9800fad0 con 0x55c41262f850 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0f8e7fc640 1 -- 172.21.15.77:0/1051589078 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f0f9800fcf0 con 0x55c41262f850 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0fb0180740 10 client.17739 fetch_fsmap learned FSMap version 49 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0fb0180740 1 -- 172.21.15.77:0/1051589078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55c4122a83f0 con 0x55c41262f850 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0f8e7fc640 1 -- 172.21.15.77:0/1051589078 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f90005040 con 0x55c41262f850 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0f8e7fc640 1 -- 172.21.15.77:0/1051589078 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 49) ==== 8378+0+0 (secure 0 0 0) 0x7f0f9801b4d0 con 0x55c41262f850 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0fb0180740 10 client.17739 fetch_fsmap finished waiting for FSMap version 49 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0fb0180740 10 client.17739 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0fb0180740 20 client.17739 populate_metadata read hostname 'smithi077' 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0fb0180740 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55c412645c10 0x55c4126480b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0fb0180740 4 client.17739 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.260+0000 7f0fb0180740 1 -- 172.21.15.77:0/1051589078 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4125a19d0 con 0x55c412645c10 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.261+0000 7f0fa67fc640 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55c412645c10 0x55c4126480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.261+0000 7f0f8e7fc640 10 client.17739 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.261+0000 7f0fa67fc640 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55c412645c10 0x55c4126480b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a301fa1e21925b11 server_cookie=ec4d66bae9f52794 in_seq=0 out_seq=0 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.261+0000 7f0f8e7fc640 1 -- 172.21.15.77:0/1051589078 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c4125a19d0 con 0x55c412645c10 2024-03-16T10:26:26.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.261+0000 7f0f8e7fc640 10 client.17739 handle_command_reply: tid=0 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.261+0000 7f0fb0180740 1 client.17739 shutdown 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.261+0000 7f0fb0180740 1 -- 172.21.15.77:0/1051589078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c41262f850 msgr2=0x55c412639d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.261+0000 7f0fb0180740 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c41262f850 0x55c412639d80 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f0f98004070 tx=0x7f0f98004ab0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.261+0000 7f0fb0180740 1 -- 172.21.15.77:0/1051589078 shutdown_connections 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.262+0000 7f0fb0180740 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c4126353b0 0x55c4126412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.262+0000 7f0fb0180740 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55c412645c10 0x55c4126480b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.262+0000 7f0fb0180740 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c412630240 0x55c41263a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.262+0000 7f0fb0180740 1 --2- 172.21.15.77:0/1051589078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c41262f850 0x55c412639d80 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.262+0000 7f0f8e7fc640 0 client.17739 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.262+0000 7f0fb0180740 1 -- 172.21.15.77:0/1051589078 >> 172.21.15.77:0/1051589078 conn(0x55c41262adc0 msgr2=0x55c412629fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.262+0000 7f0fb0180740 1 -- 172.21.15.77:0/1051589078 shutdown_connections 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.262+0000 7f0fb0180740 1 -- 172.21.15.77:0/1051589078 wait complete. 2024-03-16T10:26:26.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.262+0000 7f0fb0180740 20 client.17739 trim_cache size 0 max 16384 2024-03-16T10:26:26.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fb0180740 1 Processor -- start 2024-03-16T10:26:26.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fb0180740 1 -- start start 2024-03-16T10:26:26.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fb0180740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c41262f850 0x55c41262e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fb0180740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c412630240 0x55c41264c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fb0180740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 0x55c41264ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fb0180740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa81a8b00 con 0x55c4126353b0 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fb0180740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa81aa700 con 0x55c41262f850 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fb0180740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa81a9230 con 0x55c412630240 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fa77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 0x55c41264ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fa77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 0x55c41264ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46832/0 (socket says 172.21.15.77:46832) 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fa77fe640 1 -- 172.21.15.77:0/1116968121 learned_addr learned my addr 172.21.15.77:0/1116968121 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fa6ffd640 1 --2- 172.21.15.77:0/1116968121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c41262f850 0x55c41262e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fa67fc640 1 --2- 172.21.15.77:0/1116968121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c412630240 0x55c41264c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fa77fe640 1 -- 172.21.15.77:0/1116968121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c412630240 msgr2=0x55c41264c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/1116968121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c412630240 0x55c41264c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fa77fe640 1 -- 172.21.15.77:0/1116968121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c41262f850 msgr2=0x55c41262e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/1116968121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c41262f850 0x55c41262e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fa77fe640 1 -- 172.21.15.77:0/1116968121 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c412639920 con 0x55c4126353b0 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fa6ffd640 1 --2- 172.21.15.77:0/1116968121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c41262f850 0x55c41262e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.275+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/1116968121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 0x55c41264ceb0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0097c60 tx=0x7f0fa0086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88aa6d71b7470f09 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.276+0000 7f0f9cff9640 1 -- 172.21.15.77:0/1116968121 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa00b4070 con 0x55c4126353b0 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.276+0000 7f0f9cff9640 1 -- 172.21.15.77:0/1116968121 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fa009f950 con 0x55c4126353b0 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.276+0000 7f0f9cff9640 1 -- 172.21.15.77:0/1116968121 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa00860b0 con 0x55c4126353b0 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.276+0000 7f0fb0180740 1 -- 172.21.15.77:0/1116968121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 msgr2=0x55c41264ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.276+0000 7f0fb0180740 1 --2- 172.21.15.77:0/1116968121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 0x55c41264ceb0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0097c60 tx=0x7f0fa0086040 comp rx=0 tx=0).stop 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.276+0000 7f0fb0180740 1 -- 172.21.15.77:0/1116968121 shutdown_connections 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.276+0000 7f0fb0180740 1 --2- 172.21.15.77:0/1116968121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 0x55c41264ceb0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.276+0000 7f0fb0180740 1 --2- 172.21.15.77:0/1116968121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c412630240 0x55c41264c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.276+0000 7f0fb0180740 1 --2- 172.21.15.77:0/1116968121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c41262f850 0x55c41262e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.276+0000 7f0fb0180740 1 -- 172.21.15.77:0/1116968121 >> 172.21.15.77:0/1116968121 conn(0x55c41262adc0 msgr2=0x55c41262e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.276+0000 7f0fb0180740 1 -- 172.21.15.77:0/1116968121 shutdown_connections 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.276+0000 7f0fb0180740 1 -- 172.21.15.77:0/1116968121 wait complete. 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fb0180740 1 -- start start 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fb0180740 1 Processor -- start 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fb0180740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41262f850 0x55c4126a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fb0180740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c412630240 0x55c4126a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fb0180740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 0x55c4126aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fb0180740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa81aa030 con 0x55c4126353b0 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fb0180740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa81a9930 con 0x55c412630240 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fb0180740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa81a6ae0 con 0x55c41262f850 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fb0180740 10 client.0 _ll_register_callbacks cb 0x7f0f98003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fa77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 0x55c4126aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fa6ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41262f850 0x55c4126a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fa77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 0x55c4126aeb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46846/0 (socket says 172.21.15.77:46846) 2024-03-16T10:26:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fa77fe640 1 -- 172.21.15.77:0/416999534 learned_addr learned my addr 172.21.15.77:0/416999534 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fa67fc640 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c412630240 0x55c4126a7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fa6ffd640 1 -- 172.21.15.77:0/416999534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c412630240 msgr2=0x55c4126a7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fa6ffd640 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c412630240 0x55c4126a7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fa6ffd640 1 -- 172.21.15.77:0/416999534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 msgr2=0x55c4126aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fa6ffd640 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 0x55c4126aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fa6ffd640 1 -- 172.21.15.77:0/416999534 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4123727d0 con 0x55c41262f850 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.277+0000 7f0fa77fe640 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 0x55c4126aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0fa6ffd640 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41262f850 0x55c4126a75f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0f98009db0 tx=0x7f0f98009570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db24bcea254e0c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0f8e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0f8e7fc640 1 -- 172.21.15.77:0/416999534 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f98034070 con 0x55c41262f850 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0fb0180740 1 -- 172.21.15.77:0/416999534 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fa009f950 con 0x55c41262f850 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0f8e7fc640 1 -- 172.21.15.77:0/416999534 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f0f98022040 con 0x55c41262f850 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0f8e7fc640 1 -- 172.21.15.77:0/416999534 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f0f9801b440 con 0x55c41262f850 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0fb0180740 10 client.27499 fetch_fsmap learned FSMap version 49 2024-03-16T10:26:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0fb0180740 1 -- 172.21.15.77:0/416999534 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55c4126b0300 con 0x55c41262f850 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0f8e7fc640 1 -- 172.21.15.77:0/416999534 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fa009f950 con 0x55c41262f850 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0f8e7fc640 1 -- 172.21.15.77:0/416999534 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 49) ==== 8378+0+0 (secure 0 0 0) 0x7f0f98030420 con 0x55c41262f850 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0fb0180740 10 client.27499 fetch_fsmap finished waiting for FSMap version 49 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0fb0180740 10 client.27499 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0fb0180740 20 client.27499 populate_metadata read hostname 'smithi077' 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0fb0180740 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55c4126b0cf0 0x55c4126b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0fb0180740 4 client.27499 mds_command: new command op to 14658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.278+0000 7f0fb0180740 1 -- 172.21.15.77:0/416999534 --> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c41259bd20 con 0x55c4126b0cf0 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fa67fc640 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55c4126b0cf0 0x55c4126b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0f8e7fc640 10 client.27499 ms_handle_connect on v2:172.21.15.77:6840/857863693 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fa67fc640 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55c4126b0cf0 0x55c4126b3110 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=427958c94f83cfe2 server_cookie=24ce11fe96c7181e in_seq=0 out_seq=0 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0f8e7fc640 1 -- 172.21.15.77:0/416999534 <== mds.4 v2:172.21.15.77:6840/857863693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c41259bd20 con 0x55c4126b0cf0 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0f8e7fc640 10 client.27499 handle_command_reply: tid=0 2024-03-16T10:26:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fb0180740 1 client.27499 shutdown 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fb0180740 1 -- 172.21.15.77:0/416999534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41262f850 msgr2=0x55c4126a75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fb0180740 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41262f850 0x55c4126a75f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0f98009db0 tx=0x7f0f98009570 comp rx=0 tx=0).stop 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fb0180740 1 -- 172.21.15.77:0/416999534 shutdown_connections 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fb0180740 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c4126353b0 0x55c4126aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fb0180740 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c412630240 0x55c4126a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fb0180740 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.77:6840/857863693,v1:172.21.15.77:6841/857863693] conn(0x55c4126b0cf0 0x55c4126b3110 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fb0180740 1 --2- 172.21.15.77:0/416999534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41262f850 0x55c4126a75f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0f8e7fc640 0 client.27499 ms_handle_reset on v2:172.21.15.77:6840/857863693 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fb0180740 1 -- 172.21.15.77:0/416999534 >> 172.21.15.77:0/416999534 conn(0x55c41262adc0 msgr2=0x55c41262e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fb0180740 1 -- 172.21.15.77:0/416999534 shutdown_connections 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fb0180740 1 -- 172.21.15.77:0/416999534 wait complete. 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.279+0000 7f0fb0180740 20 client.27499 trim_cache size 0 max 16384 2024-03-16T10:26:26.287 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:26:26.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.280+0000 7f0fad4b2640 1 -- 172.21.15.77:0/1337249951 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0f940a2800 msgr2=0x7f0f940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.280+0000 7f0fad4b2640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0f940a2800 0x7f0f940a4cc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8101730 tx=0x7f0f98009500 comp rx=0 tx=0).stop 2024-03-16T10:26:26.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.280+0000 7f0fad4b2640 1 -- 172.21.15.77:0/1337249951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fa8102290 msgr2=0x7f0fa819f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.280+0000 7f0fad4b2640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fa8102290 0x7f0fa819f0a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f0fa000e970 tx=0x7f0fa000ed70 comp rx=0 tx=0).stop 2024-03-16T10:26:26.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.280+0000 7f0fad4b2640 1 -- 172.21.15.77:0/1337249951 shutdown_connections 2024-03-16T10:26:26.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.280+0000 7f0fad4b2640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fa8102290 0x7f0fa819f0a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.280+0000 7f0fad4b2640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fa81018d0 0x7f0fa81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.280+0000 7f0fad4b2640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f0f940a2800 0x7f0f940a4cc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.280+0000 7f0fad4b2640 1 --2- 172.21.15.77:0/1337249951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fa81006e0 0x7f0fa8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.280+0000 7f0fad4b2640 1 -- 172.21.15.77:0/1337249951 >> 172.21.15.77:0/1337249951 conn(0x7f0fa80fbc70 msgr2=0x7f0fa80fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.280+0000 7f0fad4b2640 1 -- 172.21.15.77:0/1337249951 shutdown_connections 2024-03-16T10:26:26.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.280+0000 7f0fad4b2640 1 -- 172.21.15.77:0/1337249951 wait complete. 2024-03-16T10:26:26.292 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-16T10:26:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.394+0000 7f74a659e640 1 -- 172.21.15.77:0/679599861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a0102a00 msgr2=0x7f74a0102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.394+0000 7f74a659e640 1 --2- 172.21.15.77:0/679599861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a0102a00 0x7f74a0102e80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7490009900 tx=0x7f749002bc30 comp rx=0 tx=0).stop 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.394+0000 7f74a659e640 1 -- 172.21.15.77:0/679599861 shutdown_connections 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.394+0000 7f74a659e640 1 --2- 172.21.15.77:0/679599861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a01033c0 0x7f74a0109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.394+0000 7f74a659e640 1 --2- 172.21.15.77:0/679599861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a0102a00 0x7f74a0102e80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.394+0000 7f74a659e640 1 --2- 172.21.15.77:0/679599861 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74a0101810 0x7f74a0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.394+0000 7f74a659e640 1 -- 172.21.15.77:0/679599861 >> 172.21.15.77:0/679599861 conn(0x7f74a00fcd80 msgr2=0x7f74a00ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.394+0000 7f74a659e640 1 -- 172.21.15.77:0/679599861 shutdown_connections 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.394+0000 7f74a659e640 1 -- 172.21.15.77:0/679599861 wait complete. 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f74a659e640 1 Processor -- start 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f74a659e640 1 -- start start 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f74a659e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a0101810 0x7f74a0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f74a659e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a0102a00 0x7f74a0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f74a659e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74a01033c0 0x7f74a019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f74a659e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f74a010d470 con 0x7f74a0101810 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f74a659e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f74a010d2f0 con 0x7f74a01033c0 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f74a659e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f74a010d5f0 con 0x7f74a0102a00 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f749ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a0101810 0x7f74a0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f749ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a0101810 0x7f74a0197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46860/0 (socket says 172.21.15.77:46860) 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f749ffff640 1 -- 172.21.15.77:0/1434065593 learned_addr learned my addr 172.21.15.77:0/1434065593 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f749f7fe640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a0102a00 0x7f74a0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.395+0000 7f74a4b3b640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74a01033c0 0x7f74a019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.396+0000 7f749ffff640 1 -- 172.21.15.77:0/1434065593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a0102a00 msgr2=0x7f74a0198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.396+0000 7f749ffff640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a0102a00 0x7f74a0198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.396+0000 7f749ffff640 1 -- 172.21.15.77:0/1434065593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74a01033c0 msgr2=0x7f74a019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.396+0000 7f749ffff640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74a01033c0 0x7f74a019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.396+0000 7f749ffff640 1 -- 172.21.15.77:0/1434065593 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a019f730 con 0x7f74a0101810 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.396+0000 7f749ffff640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a0101810 0x7f74a0197b10 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f749400e9c0 tx=0x7f749400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.396+0000 7f749d7fa640 1 -- 172.21.15.77:0/1434065593 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749400cde0 con 0x7f74a0101810 2024-03-16T10:26:26.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.396+0000 7f74a659e640 1 -- 172.21.15.77:0/1434065593 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a019fa20 con 0x7f74a0101810 2024-03-16T10:26:26.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.396+0000 7f749d7fa640 1 -- 172.21.15.77:0/1434065593 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7494010910 con 0x7f74a0101810 2024-03-16T10:26:26.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.396+0000 7f749d7fa640 1 -- 172.21.15.77:0/1434065593 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494018a10 con 0x7f74a0101810 2024-03-16T10:26:26.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.396+0000 7f74a659e640 1 -- 172.21.15.77:0/1434065593 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a019ff90 con 0x7f74a0101810 2024-03-16T10:26:26.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.398+0000 7f749d7fa640 1 -- 172.21.15.77:0/1434065593 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f749401f020 con 0x7f74a0101810 2024-03-16T10:26:26.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.399+0000 7f749d7fa640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f74880a2800 0x7f74880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.399+0000 7f749f7fe640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f74880a2800 0x7f74880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.399+0000 7f749d7fa640 1 -- 172.21.15.77:0/1434065593 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f74940ceb90 con 0x7f74a0101810 2024-03-16T10:26:26.407 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.399+0000 7f749f7fe640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f74880a2800 0x7f74880a4cc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f749002c7b0 tx=0x7f7490005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.399+0000 7f74a9188740 1 Processor -- start 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.399+0000 7f74a9188740 1 -- start start 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.399+0000 7f74a9188740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed6310c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.399+0000 7f74a9188740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed6311240 0x55aed6315d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.399+0000 7f74a9188740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed63163b0 0x55aed6318840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.399+0000 7f74a9188740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55aed6275490 con 0x55aed6310850 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.399+0000 7f74a9188740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55aed621dc90 con 0x55aed63163b0 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.399+0000 7f74a9188740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55aed6271530 con 0x55aed6311240 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f749ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed6310c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f749ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed6310c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46868/0 (socket says 172.21.15.77:46868) 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f749ffff640 1 -- 172.21.15.77:0/2326457364 learned_addr learned my addr 172.21.15.77:0/2326457364 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f74a4b3b640 1 --2- 172.21.15.77:0/2326457364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed63163b0 0x55aed6318840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f749f7fe640 1 --2- 172.21.15.77:0/2326457364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed6311240 0x55aed6315d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.408 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f749ffff640 1 -- 172.21.15.77:0/2326457364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed6311240 msgr2=0x55aed6315d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f749ffff640 1 --2- 172.21.15.77:0/2326457364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed6311240 0x55aed6315d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f749ffff640 1 -- 172.21.15.77:0/2326457364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed63163b0 msgr2=0x55aed6318840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f749ffff640 1 --2- 172.21.15.77:0/2326457364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed63163b0 0x55aed6318840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f749ffff640 1 -- 172.21.15.77:0/2326457364 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aed62d8500 con 0x55aed6310850 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f749ffff640 1 --2- 172.21.15.77:0/2326457364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed6310c70 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f74940a3830 tx=0x7f749409afe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6bb1d5ab21f27ff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f74867fc640 1 -- 172.21.15.77:0/2326457364 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749409b5c0 con 0x55aed6310850 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f74867fc640 1 -- 172.21.15.77:0/2326457364 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f749409f710 con 0x55aed6310850 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.400+0000 7f74867fc640 1 -- 172.21.15.77:0/2326457364 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494097c30 con 0x55aed6310850 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.401+0000 7f74a9188740 1 -- 172.21.15.77:0/2326457364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 msgr2=0x55aed6310c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.401+0000 7f74a9188740 1 --2- 172.21.15.77:0/2326457364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed6310c70 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f74940a3830 tx=0x7f749409afe0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.401+0000 7f74a9188740 1 -- 172.21.15.77:0/2326457364 shutdown_connections 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.401+0000 7f74a9188740 1 --2- 172.21.15.77:0/2326457364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed63163b0 0x55aed6318840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.401+0000 7f74a9188740 1 --2- 172.21.15.77:0/2326457364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed6311240 0x55aed6315d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.401+0000 7f74a9188740 1 --2- 172.21.15.77:0/2326457364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed6310c70 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.409 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.401+0000 7f74a9188740 1 -- 172.21.15.77:0/2326457364 >> 172.21.15.77:0/2326457364 conn(0x55aed630bdc0 msgr2=0x55aed630c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.401+0000 7f74a9188740 1 -- 172.21.15.77:0/2326457364 shutdown_connections 2024-03-16T10:26:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.401+0000 7f74a9188740 1 -- 172.21.15.77:0/2326457364 wait complete. 2024-03-16T10:26:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.401+0000 7f74a9188740 1 -- start start 2024-03-16T10:26:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f74a9188740 1 Processor -- start 2024-03-16T10:26:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f74a9188740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed631ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f74a9188740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed6311240 0x55aed631b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f74a9188740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed63163b0 0x55aed63222b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f74a9188740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55aed6319360 con 0x55aed6310850 2024-03-16T10:26:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f74a9188740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55aed6231230 con 0x55aed6311240 2024-03-16T10:26:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f74a9188740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55aed63194e0 con 0x55aed63163b0 2024-03-16T10:26:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f749ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed631ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f74a9188740 10 client.0 _ll_register_callbacks cb 0x55aed626d010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f749ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed631ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46870/0 (socket says 172.21.15.77:46870) 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f749ffff640 1 -- 172.21.15.77:0/1164310821 learned_addr learned my addr 172.21.15.77:0/1164310821 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f749f7fe640 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed6311240 0x55aed631b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f74a4b3b640 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed63163b0 0x55aed63222b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f749ffff640 1 -- 172.21.15.77:0/1164310821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed63163b0 msgr2=0x55aed63222b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f749ffff640 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed63163b0 0x55aed63222b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f749ffff640 1 -- 172.21.15.77:0/1164310821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed6311240 msgr2=0x55aed631b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f749ffff640 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed6311240 0x55aed631b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f749ffff640 1 -- 172.21.15.77:0/1164310821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aed6240100 con 0x55aed6310850 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f749ffff640 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed631ad70 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f74940ceeb0 tx=0x7f7494018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f2d87886f9ee2d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.402+0000 7f747f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f747f7fe640 1 -- 172.21.15.77:0/1164310821 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74940a3d60 con 0x55aed6310850 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f747f7fe640 1 -- 172.21.15.77:0/1164310821 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f7494010ac0 con 0x55aed6310850 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f74a9188740 1 -- 172.21.15.77:0/1164310821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f749409f710 con 0x55aed6310850 2024-03-16T10:26:26.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f747f7fe640 1 -- 172.21.15.77:0/1164310821 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f74940a38d0 con 0x55aed6310850 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f74a9188740 10 client.17757 fetch_fsmap learned FSMap version 49 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f74a9188740 1 -- 172.21.15.77:0/1164310821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55aed5f893f0 con 0x55aed6310850 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f747f7fe640 1 -- 172.21.15.77:0/1164310821 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f749409f710 con 0x55aed6310850 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f747f7fe640 1 -- 172.21.15.77:0/1164310821 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 49) ==== 8378+0+0 (secure 0 0 0) 0x7f74940b4170 con 0x55aed6310850 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f74a9188740 10 client.17757 fetch_fsmap finished waiting for FSMap version 49 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f74a9188740 10 client.17757 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f74a9188740 20 client.17757 populate_metadata read hostname 'smithi077' 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f74a9188740 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55aed63267c0 0x55aed6328bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f74a9188740 4 client.17757 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f74a9188740 1 -- 172.21.15.77:0/1164310821 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aed62829d0 con 0x55aed63267c0 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.403+0000 7f749f7fe640 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55aed63267c0 0x55aed6328bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.404+0000 7f747f7fe640 10 client.17757 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.404+0000 7f749f7fe640 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55aed63267c0 0x55aed6328bb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=36bbea567b3cbbf2 server_cookie=4ab0ff0129a88c80 in_seq=0 out_seq=0 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.404+0000 7f747f7fe640 1 -- 172.21.15.77:0/1164310821 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aed62829d0 con 0x55aed63267c0 2024-03-16T10:26:26.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.404+0000 7f747f7fe640 10 client.17757 handle_command_reply: tid=0 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.404+0000 7f74a9188740 1 client.17757 shutdown 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.405+0000 7f74a9188740 1 -- 172.21.15.77:0/1164310821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 msgr2=0x55aed631ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.405+0000 7f74a9188740 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed631ad70 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f74940ceeb0 tx=0x7f7494018420 comp rx=0 tx=0).stop 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.405+0000 7f74a9188740 1 -- 172.21.15.77:0/1164310821 shutdown_connections 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.405+0000 7f74a9188740 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55aed63267c0 0x55aed6328bb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.405+0000 7f74a9188740 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed63163b0 0x55aed63222b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.405+0000 7f74a9188740 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed6311240 0x55aed631b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.405+0000 7f747f7fe640 0 client.17757 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.405+0000 7f74a9188740 1 --2- 172.21.15.77:0/1164310821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed631ad70 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.405+0000 7f74a9188740 1 -- 172.21.15.77:0/1164310821 >> 172.21.15.77:0/1164310821 conn(0x55aed630bdc0 msgr2=0x55aed630afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.405+0000 7f74a9188740 1 -- 172.21.15.77:0/1164310821 shutdown_connections 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.405+0000 7f74a9188740 1 -- 172.21.15.77:0/1164310821 wait complete. 2024-03-16T10:26:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.405+0000 7f74a9188740 20 client.17757 trim_cache size 0 max 16384 2024-03-16T10:26:26.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:25 smithi192 ceph-mon[36354]: pgmap v631: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T10:26:26.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f74a9188740 1 Processor -- start 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f74a9188740 1 -- start start 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f74a9188740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed630c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f74a9188740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed6311240 0x55aed6322270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f74a9188740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed63163b0 0x55aed631ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f74a9188740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f74a01a8c50 con 0x55aed6310850 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f74a9188740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f74a01aa850 con 0x55aed6311240 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f74a9188740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f74a01a9380 con 0x55aed63163b0 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f749ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed630c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f749ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed630c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46886/0 (socket says 172.21.15.77:46886) 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f749ffff640 1 -- 172.21.15.77:0/555859072 learned_addr learned my addr 172.21.15.77:0/555859072 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f74a4b3b640 1 --2- 172.21.15.77:0/555859072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed63163b0 0x55aed631ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f749f7fe640 1 --2- 172.21.15.77:0/555859072 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed6311240 0x55aed6322270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f749ffff640 1 -- 172.21.15.77:0/555859072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed63163b0 msgr2=0x55aed631ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.421 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f749ffff640 1 --2- 172.21.15.77:0/555859072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed63163b0 0x55aed631ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f749ffff640 1 -- 172.21.15.77:0/555859072 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed6311240 msgr2=0x55aed6322270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f749ffff640 1 --2- 172.21.15.77:0/555859072 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed6311240 0x55aed6322270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.418+0000 7f749ffff640 1 -- 172.21.15.77:0/555859072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aed60537d0 con 0x55aed6310850 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.419+0000 7f749ffff640 1 --2- 172.21.15.77:0/555859072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed630c1a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f7494097f40 tx=0x7f74940860b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77e80c50679757a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.419+0000 7f7485ffb640 1 -- 172.21.15.77:0/555859072 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749409f710 con 0x55aed6310850 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.419+0000 7f7485ffb640 1 -- 172.21.15.77:0/555859072 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7494084b50 con 0x55aed6310850 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.419+0000 7f7485ffb640 1 -- 172.21.15.77:0/555859072 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74940b4170 con 0x55aed6310850 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.419+0000 7f74a9188740 1 -- 172.21.15.77:0/555859072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 msgr2=0x55aed630c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.419+0000 7f74a9188740 1 --2- 172.21.15.77:0/555859072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed630c1a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f7494097f40 tx=0x7f74940860b0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.419+0000 7f74a9188740 1 -- 172.21.15.77:0/555859072 shutdown_connections 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.419+0000 7f74a9188740 1 --2- 172.21.15.77:0/555859072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed63163b0 0x55aed631ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.419+0000 7f74a9188740 1 --2- 172.21.15.77:0/555859072 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed6311240 0x55aed6322270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.419+0000 7f74a9188740 1 --2- 172.21.15.77:0/555859072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed630c1a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.419+0000 7f74a9188740 1 -- 172.21.15.77:0/555859072 >> 172.21.15.77:0/555859072 conn(0x55aed630bdc0 msgr2=0x55aed6319880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.420+0000 7f74a9188740 1 -- 172.21.15.77:0/555859072 shutdown_connections 2024-03-16T10:26:26.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.420+0000 7f74a9188740 1 -- 172.21.15.77:0/555859072 wait complete. 2024-03-16T10:26:26.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.420+0000 7f74a9188740 1 -- start start 2024-03-16T10:26:26.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.420+0000 7f74a9188740 1 Processor -- start 2024-03-16T10:26:26.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.420+0000 7f74a9188740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed631f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.420+0000 7f74a9188740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed6311240 0x55aed631f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.420+0000 7f74a9188740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed63163b0 0x55aed6320930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.420+0000 7f74a9188740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f74a01aa180 con 0x55aed6310850 2024-03-16T10:26:26.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.420+0000 7f74a9188740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f74a01a9a80 con 0x55aed63163b0 2024-03-16T10:26:26.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.420+0000 7f74a9188740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f74a01a6c30 con 0x55aed6311240 2024-03-16T10:26:26.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f74a9188740 10 client.0 _ll_register_callbacks cb 0x7f7498009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f749ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed631f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f749ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed631f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46894/0 (socket says 172.21.15.77:46894) 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f749ffff640 1 -- 172.21.15.77:0/1942087403 learned_addr learned my addr 172.21.15.77:0/1942087403 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f749f7fe640 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed6311240 0x55aed631f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f74a4b3b640 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed63163b0 0x55aed6320930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f749ffff640 1 -- 172.21.15.77:0/1942087403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed6311240 msgr2=0x55aed631f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f749ffff640 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed6311240 0x55aed631f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f749ffff640 1 -- 172.21.15.77:0/1942087403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed63163b0 msgr2=0x55aed6320930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f749ffff640 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed63163b0 0x55aed6320930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f749ffff640 1 -- 172.21.15.77:0/1942087403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aed6320f20 con 0x55aed6310850 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f749ffff640 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed631f3e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f749409b550 tx=0x7f74940b4590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06bd8398ed49edd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f747f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f747f7fe640 1 -- 172.21.15.77:0/1942087403 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74940a9060 con 0x55aed6310850 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f747f7fe640 1 -- 172.21.15.77:0/1942087403 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f7494010ac0 con 0x55aed6310850 2024-03-16T10:26:26.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f74a9188740 1 -- 172.21.15.77:0/1942087403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7494084b50 con 0x55aed6310850 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f747f7fe640 1 -- 172.21.15.77:0/1942087403 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f74940ccb40 con 0x55aed6310850 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f74a9188740 10 client.17763 fetch_fsmap learned FSMap version 49 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f74a9188740 1 -- 172.21.15.77:0/1942087403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55aed6320f20 con 0x55aed6310850 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.421+0000 7f747f7fe640 1 -- 172.21.15.77:0/1942087403 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7494084b50 con 0x55aed6310850 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f747f7fe640 1 -- 172.21.15.77:0/1942087403 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 49) ==== 8378+0+0 (secure 0 0 0) 0x7f749409aa10 con 0x55aed6310850 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f74a9188740 10 client.17763 fetch_fsmap finished waiting for FSMap version 49 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f74a9188740 10 client.17763 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f74a9188740 20 client.17763 populate_metadata read hostname 'smithi077' 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f74a9188740 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55aed638c9c0 0x55aed638edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f74a9188740 4 client.17763 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f74a9188740 1 -- 172.21.15.77:0/1942087403 --> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aed627cd20 con 0x55aed638c9c0 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f749f7fe640 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55aed638c9c0 0x55aed638edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f747f7fe640 10 client.17763 ms_handle_connect on v2:172.21.15.193:6836/3018320989 2024-03-16T10:26:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f749f7fe640 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55aed638c9c0 0x55aed638edb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8ab9b1319f5be6a6 server_cookie=5ef31ab3d8f13c06 in_seq=0 out_seq=0 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f747f7fe640 1 -- 172.21.15.77:0/1942087403 <== mds.3 v2:172.21.15.193:6836/3018320989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aed627cd20 con 0x55aed638c9c0 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f747f7fe640 10 client.17763 handle_command_reply: tid=0 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.422+0000 7f74a9188740 1 client.17763 shutdown 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a9188740 1 -- 172.21.15.77:0/1942087403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 msgr2=0x55aed631f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a9188740 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed631f3e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f749409b550 tx=0x7f74940b4590 comp rx=0 tx=0).stop 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a9188740 1 -- 172.21.15.77:0/1942087403 shutdown_connections 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a9188740 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.193:6836/3018320989,v1:172.21.15.193:6837/3018320989] conn(0x55aed638c9c0 0x55aed638edb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a9188740 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aed63163b0 0x55aed6320930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a9188740 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aed6311240 0x55aed631f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f747f7fe640 0 client.17763 ms_handle_reset on v2:172.21.15.193:6836/3018320989 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a9188740 1 --2- 172.21.15.77:0/1942087403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aed6310850 0x55aed631f3e0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a9188740 1 -- 172.21.15.77:0/1942087403 >> 172.21.15.77:0/1942087403 conn(0x55aed630bdc0 msgr2=0x55aed630c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a9188740 1 -- 172.21.15.77:0/1942087403 shutdown_connections 2024-03-16T10:26:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a9188740 1 -- 172.21.15.77:0/1942087403 wait complete. 2024-03-16T10:26:26.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a9188740 20 client.17763 trim_cache size 0 max 16384 2024-03-16T10:26:26.431 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:26:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a659e640 1 -- 172.21.15.77:0/1434065593 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f74880a2800 msgr2=0x7f74880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a659e640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f74880a2800 0x7f74880a4cc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f749002c7b0 tx=0x7f7490005fd0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a659e640 1 -- 172.21.15.77:0/1434065593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a0101810 msgr2=0x7f74a0197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a659e640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a0101810 0x7f74a0197b10 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f749400e9c0 tx=0x7f749400ee80 comp rx=0 tx=0).stop 2024-03-16T10:26:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a659e640 1 -- 172.21.15.77:0/1434065593 shutdown_connections 2024-03-16T10:26:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a659e640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74a01033c0 0x7f74a019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a659e640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f74880a2800 0x7f74880a4cc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a659e640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a0102a00 0x7f74a0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a659e640 1 --2- 172.21.15.77:0/1434065593 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74a0101810 0x7f74a0197b10 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.423+0000 7f74a659e640 1 -- 172.21.15.77:0/1434065593 >> 172.21.15.77:0/1434065593 conn(0x7f74a00fcd80 msgr2=0x7f74a0105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.424+0000 7f74a659e640 1 -- 172.21.15.77:0/1434065593 shutdown_connections 2024-03-16T10:26:26.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.424+0000 7f74a659e640 1 -- 172.21.15.77:0/1434065593 wait complete. 2024-03-16T10:26:26.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-16T10:26:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 -- 172.21.15.77:0/2650252748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2234102290 msgr2=0x7f2234108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 --2- 172.21.15.77:0/2650252748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2234102290 0x7f2234108b20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f222c009be0 tx=0x7f222c02bef0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 -- 172.21.15.77:0/2650252748 shutdown_connections 2024-03-16T10:26:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 --2- 172.21.15.77:0/2650252748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2234102290 0x7f2234108b20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 --2- 172.21.15.77:0/2650252748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22341018d0 0x7f2234101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 --2- 172.21.15.77:0/2650252748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22341006e0 0x7f2234100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 -- 172.21.15.77:0/2650252748 >> 172.21.15.77:0/2650252748 conn(0x7f22340fbc70 msgr2=0x7f22340fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 -- 172.21.15.77:0/2650252748 shutdown_connections 2024-03-16T10:26:26.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 -- 172.21.15.77:0/2650252748 wait complete. 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 Processor -- start 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 -- start start 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22341006e0 0x7f22340694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.540+0000 7f223b2bd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22341018d0 0x7f223419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f223b2bd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2234102290 0x7f223419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f223b2bd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f223410c340 con 0x7f22341006e0 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f223b2bd640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f223410c1c0 con 0x7f22341018d0 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f223b2bd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f223410c4c0 con 0x7f2234102290 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f2239059640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22341006e0 0x7f22340694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f2239059640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22341006e0 0x7f22340694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46914/0 (socket says 172.21.15.77:46914) 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f2239059640 1 -- 172.21.15.77:0/3952730162 learned_addr learned my addr 172.21.15.77:0/3952730162 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f223985a640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2234102290 0x7f223419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f2238858640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22341018d0 0x7f223419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f2239059640 1 -- 172.21.15.77:0/3952730162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2234102290 msgr2=0x7f223419ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f2239059640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2234102290 0x7f223419ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f2239059640 1 -- 172.21.15.77:0/3952730162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22341018d0 msgr2=0x7f223419c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f2239059640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22341018d0 0x7f223419c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f2239059640 1 -- 172.21.15.77:0/3952730162 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223419f6f0 con 0x7f22341006e0 2024-03-16T10:26:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f2239059640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22341006e0 0x7f22340694d0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f223000e9a0 tx=0x7f223000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f222a7fc640 1 -- 172.21.15.77:0/3952730162 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223000cd80 con 0x7f22341006e0 2024-03-16T10:26:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f223b2bd640 1 -- 172.21.15.77:0/3952730162 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223419f9e0 con 0x7f22341006e0 2024-03-16T10:26:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f222a7fc640 1 -- 172.21.15.77:0/3952730162 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2230010910 con 0x7f22341006e0 2024-03-16T10:26:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f222a7fc640 1 -- 172.21.15.77:0/3952730162 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2230018a40 con 0x7f22341006e0 2024-03-16T10:26:26.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.541+0000 7f223b2bd640 1 -- 172.21.15.77:0/3952730162 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f223419ff80 con 0x7f22341006e0 2024-03-16T10:26:26.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.543+0000 7f222a7fc640 1 -- 172.21.15.77:0/3952730162 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f2230010ac0 con 0x7f22341006e0 2024-03-16T10:26:26.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f223df83740 1 Processor -- start 2024-03-16T10:26:26.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f223df83740 1 -- start start 2024-03-16T10:26:26.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f222a7fc640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f22200a2800 0x7f22200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f223df83740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f223df83740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6b6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f223df83740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b73b0 0x5577fa6b9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f223df83740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5577fa616490 con 0x5577fa6b2240 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f223df83740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5577fa5bec90 con 0x5577fa6b73b0 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f223df83740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5577fa612530 con 0x5577fa6b1850 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f2238858640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6b6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f2239059640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f223985a640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f22200a2800 0x7f22200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f2238858640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6b6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46918/0 (socket says 172.21.15.77:46918) 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f2238858640 1 -- 172.21.15.77:0/935666947 learned_addr learned my addr 172.21.15.77:0/935666947 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.544+0000 7f222a7fc640 1 -- 172.21.15.77:0/3952730162 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f22300ced10 con 0x7f22341006e0 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.545+0000 7f223985a640 1 --2- 172.21.15.77:0/935666947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b73b0 0x5577fa6b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.545+0000 7f2239059640 1 -- 172.21.15.77:0/935666947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b73b0 msgr2=0x5577fa6b9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.545+0000 7f2239059640 1 --2- 172.21.15.77:0/935666947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b73b0 0x5577fa6b9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.545+0000 7f2239059640 1 -- 172.21.15.77:0/935666947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 msgr2=0x5577fa6b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.545+0000 7f2239059640 1 --2- 172.21.15.77:0/935666947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6b6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.545+0000 7f2239059640 1 -- 172.21.15.77:0/935666947 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577fa679500 con 0x5577fa6b1850 2024-03-16T10:26:26.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.545+0000 7f2238858640 1 --2- 172.21.15.77:0/935666947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6b6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:26.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.545+0000 7f223985a640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f22200a2800 0x7f22200a4cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f222c005f00 tx=0x7f222c0098d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.545+0000 7f2239059640 1 --2- 172.21.15.77:0/935666947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6b1c70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f22300cc8f0 tx=0x7f223009b280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b227efb66f387230 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.545+0000 7f221f7fe640 1 -- 172.21.15.77:0/935666947 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2230099d00 con 0x5577fa6b1850 2024-03-16T10:26:26.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.545+0000 7f221f7fe640 1 -- 172.21.15.77:0/935666947 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2230010e50 con 0x5577fa6b1850 2024-03-16T10:26:26.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.545+0000 7f221f7fe640 1 -- 172.21.15.77:0/935666947 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22300a3650 con 0x5577fa6b1850 2024-03-16T10:26:26.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.546+0000 7f223df83740 1 -- 172.21.15.77:0/935666947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 msgr2=0x5577fa6b1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.546+0000 7f223df83740 1 --2- 172.21.15.77:0/935666947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6b1c70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f22300cc8f0 tx=0x7f223009b280 comp rx=0 tx=0).stop 2024-03-16T10:26:26.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.546+0000 7f223df83740 1 -- 172.21.15.77:0/935666947 shutdown_connections 2024-03-16T10:26:26.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.546+0000 7f223df83740 1 --2- 172.21.15.77:0/935666947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b73b0 0x5577fa6b9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.546+0000 7f223df83740 1 --2- 172.21.15.77:0/935666947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6b6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.546+0000 7f223df83740 1 --2- 172.21.15.77:0/935666947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6b1c70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.546+0000 7f223df83740 1 -- 172.21.15.77:0/935666947 >> 172.21.15.77:0/935666947 conn(0x5577fa6acdc0 msgr2=0x5577fa6ad1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.546+0000 7f223df83740 1 -- 172.21.15.77:0/935666947 shutdown_connections 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.546+0000 7f223df83740 1 -- 172.21.15.77:0/935666947 wait complete. 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.546+0000 7f223df83740 1 -- start start 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.546+0000 7f223df83740 1 Processor -- start 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f223df83740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6bbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f223df83740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6bc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f223df83740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b73b0 0x5577fa6c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f223df83740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5577fa6ba300 con 0x5577fa6b2240 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f223df83740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5577fa5d2230 con 0x5577fa6b73b0 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f223df83740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5577fa6ba480 con 0x5577fa6b1850 2024-03-16T10:26:26.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f223df83740 10 client.0 _ll_register_callbacks cb 0x5577fa60e010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f2238858640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6bc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f2239059640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6bbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f2238858640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6bc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46928/0 (socket says 172.21.15.77:46928) 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f2238858640 1 -- 172.21.15.77:0/69422214 learned_addr learned my addr 172.21.15.77:0/69422214 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f223985a640 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b73b0 0x5577fa6c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f2238858640 1 -- 172.21.15.77:0/69422214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 msgr2=0x5577fa6bbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f2238858640 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6bbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f2238858640 1 -- 172.21.15.77:0/69422214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b73b0 msgr2=0x5577fa6c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f2238858640 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b73b0 0x5577fa6c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f2238858640 1 -- 172.21.15.77:0/69422214 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577fa5e1100 con 0x5577fa6b2240 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f2239059640 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6bbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f2238858640 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6bc280 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f2224012730 tx=0x7f2224012bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c659ae213ba5e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.547+0000 7f221cff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f221cff9640 1 -- 172.21.15.77:0/69422214 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224004050 con 0x5577fa6b2240 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f223df83740 1 -- 172.21.15.77:0/69422214 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2230010e50 con 0x5577fa6b2240 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f221cff9640 1 -- 172.21.15.77:0/69422214 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f22240041f0 con 0x5577fa6b2240 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f221cff9640 1 -- 172.21.15.77:0/69422214 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f22240043f0 con 0x5577fa6b2240 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f223df83740 10 client.17775 fetch_fsmap learned FSMap version 49 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f223df83740 1 -- 172.21.15.77:0/69422214 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5577fa32a3f0 con 0x5577fa6b2240 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f221cff9640 1 -- 172.21.15.77:0/69422214 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2230010e50 con 0x5577fa6b2240 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f221cff9640 1 -- 172.21.15.77:0/69422214 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 49) ==== 8378+0+0 (secure 0 0 0) 0x7f2224003bc0 con 0x5577fa6b2240 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f223df83740 10 client.17775 fetch_fsmap finished waiting for FSMap version 49 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f223df83740 10 client.17775 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f223df83740 20 client.17775 populate_metadata read hostname 'smithi077' 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f223df83740 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5577fa6c7bf0 0x5577fa6ca090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f223df83740 4 client.17775 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f223df83740 1 -- 172.21.15.77:0/69422214 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577fa6239d0 con 0x5577fa6c7bf0 2024-03-16T10:26:26.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.548+0000 7f2239059640 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5577fa6c7bf0 0x5577fa6ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.549+0000 7f2239059640 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5577fa6c7bf0 0x5577fa6ca090 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5cb96183aa7e21de server_cookie=a371fe801431c5d8 in_seq=0 out_seq=0 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.549+0000 7f221cff9640 10 client.17775 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f221cff9640 1 -- 172.21.15.77:0/69422214 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5577fa6239d0 con 0x5577fa6c7bf0 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f221cff9640 10 client.17775 handle_command_reply: tid=0 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f223df83740 1 client.17775 shutdown 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f223df83740 1 -- 172.21.15.77:0/69422214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 msgr2=0x5577fa6bc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f223df83740 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6bc280 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f2224012730 tx=0x7f2224012bf0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f223df83740 1 -- 172.21.15.77:0/69422214 shutdown_connections 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f223df83740 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5577fa6c7bf0 0x5577fa6ca090 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f223df83740 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b73b0 0x5577fa6c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f223df83740 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6bc280 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f223df83740 1 --2- 172.21.15.77:0/69422214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6bbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f223df83740 1 -- 172.21.15.77:0/69422214 >> 172.21.15.77:0/69422214 conn(0x5577fa6acdc0 msgr2=0x5577fa6abfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f221cff9640 0 client.17775 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:26:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f223df83740 1 -- 172.21.15.77:0/69422214 shutdown_connections 2024-03-16T10:26:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f223df83740 1 -- 172.21.15.77:0/69422214 wait complete. 2024-03-16T10:26:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.550+0000 7f223df83740 20 client.17775 trim_cache size 0 max 16384 2024-03-16T10:26:26.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f223df83740 1 Processor -- start 2024-03-16T10:26:26.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f223df83740 1 -- start start 2024-03-16T10:26:26.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f223df83740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b1850 0x5577fa6b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f223df83740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6c3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f223df83740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b73b0 0x5577fa6ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f223df83740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f22341a6a40 con 0x5577fa6b2240 2024-03-16T10:26:26.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f223df83740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f22341a8640 con 0x5577fa6b1850 2024-03-16T10:26:26.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f223df83740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f22341a7170 con 0x5577fa6b73b0 2024-03-16T10:26:26.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f2238858640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f2238858640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6c3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46944/0 (socket says 172.21.15.77:46944) 2024-03-16T10:26:26.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f2238858640 1 -- 172.21.15.77:0/1189906431 learned_addr learned my addr 172.21.15.77:0/1189906431 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.573 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f2239059640 1 --2- 172.21.15.77:0/1189906431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b1850 0x5577fa6b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.566+0000 7f223985a640 1 --2- 172.21.15.77:0/1189906431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b73b0 0x5577fa6ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.567+0000 7f2238858640 1 -- 172.21.15.77:0/1189906431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b73b0 msgr2=0x5577fa6ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.567+0000 7f2238858640 1 --2- 172.21.15.77:0/1189906431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b73b0 0x5577fa6ad1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.567+0000 7f2238858640 1 -- 172.21.15.77:0/1189906431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b1850 msgr2=0x5577fa6b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.567+0000 7f2238858640 1 --2- 172.21.15.77:0/1189906431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b1850 0x5577fa6b2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.567+0000 7f2238858640 1 -- 172.21.15.77:0/1189906431 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577fa3f47d0 con 0x5577fa6b2240 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.567+0000 7f2238858640 1 --2- 172.21.15.77:0/1189906431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6c3240 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f22240059f0 tx=0x7f22240108a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c71297620ee81f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.567+0000 7f221effd640 1 -- 172.21.15.77:0/1189906431 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224003fb0 con 0x5577fa6b2240 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.567+0000 7f221effd640 1 -- 172.21.15.77:0/1189906431 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2224004150 con 0x5577fa6b2240 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.567+0000 7f221effd640 1 -- 172.21.15.77:0/1189906431 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224005040 con 0x5577fa6b2240 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.568+0000 7f223df83740 1 -- 172.21.15.77:0/1189906431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 msgr2=0x5577fa6c3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.568+0000 7f223df83740 1 --2- 172.21.15.77:0/1189906431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6c3240 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f22240059f0 tx=0x7f22240108a0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.568+0000 7f223df83740 1 -- 172.21.15.77:0/1189906431 shutdown_connections 2024-03-16T10:26:26.574 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.568+0000 7f223df83740 1 --2- 172.21.15.77:0/1189906431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b73b0 0x5577fa6ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.568+0000 7f223df83740 1 --2- 172.21.15.77:0/1189906431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b2240 0x5577fa6c3240 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.568+0000 7f223df83740 1 --2- 172.21.15.77:0/1189906431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b1850 0x5577fa6b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.568+0000 7f223df83740 1 -- 172.21.15.77:0/1189906431 >> 172.21.15.77:0/1189906431 conn(0x5577fa6acdc0 msgr2=0x5577fa6ba850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.568+0000 7f223df83740 1 -- 172.21.15.77:0/1189906431 shutdown_connections 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.568+0000 7f223df83740 1 -- 172.21.15.77:0/1189906431 wait complete. 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.568+0000 7f223df83740 1 -- start start 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.568+0000 7f223df83740 1 Processor -- start 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223df83740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223df83740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b2240 0x5577fa6bfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223df83740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b73b0 0x5577fa6c0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223df83740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f22341a7f70 con 0x5577fa6b73b0 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223df83740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f22341a7870 con 0x5577fa6b2240 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223df83740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f22341a4a40 con 0x5577fa6b1850 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223df83740 10 client.0 _ll_register_callbacks cb 0x7f2224009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:26:26.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223985a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b73b0 0x5577fa6c0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f2238858640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b2240 0x5577fa6bfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223985a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b73b0 0x5577fa6c0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46954/0 (socket says 172.21.15.77:46954) 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f2239059640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223985a640 1 -- 172.21.15.77:0/3530603011 learned_addr learned my addr 172.21.15.77:0/3530603011 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f2238858640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b2240 0x5577fa6bfb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.77:43706/0 (socket says 172.21.15.77:43706) 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223985a640 1 -- 172.21.15.77:0/3530603011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 msgr2=0x5577fa6c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223985a640 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6c30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223985a640 1 -- 172.21.15.77:0/3530603011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b2240 msgr2=0x5577fa6bfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223985a640 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b2240 0x5577fa6bfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223985a640 1 -- 172.21.15.77:0/3530603011 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577fa72bf80 con 0x5577fa6b73b0 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f2238858640 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b2240 0x5577fa6bfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f2239059640 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6c30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:26.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223985a640 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b73b0 0x5577fa6c0090 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f222c036040 tx=0x7f222c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f93905ba39e86b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f221cff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f221cff9640 1 -- 172.21.15.77:0/3530603011 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c055070 con 0x5577fa6b73b0 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f221cff9640 1 -- 172.21.15.77:0/3530603011 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f222c03a040 con 0x5577fa6b73b0 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.569+0000 7f223df83740 1 -- 172.21.15.77:0/3530603011 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2224004150 con 0x5577fa6b73b0 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f221cff9640 1 -- 172.21.15.77:0/3530603011 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f222c004040 con 0x5577fa6b73b0 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f223df83740 10 client.17781 fetch_fsmap learned FSMap version 49 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f223df83740 1 -- 172.21.15.77:0/3530603011 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5577fa72d120 con 0x5577fa6b73b0 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f221cff9640 1 -- 172.21.15.77:0/3530603011 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2224004150 con 0x5577fa6b73b0 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f221cff9640 1 -- 172.21.15.77:0/3530603011 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 49) ==== 8378+0+0 (secure 0 0 0) 0x7f222c034a60 con 0x5577fa6b73b0 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f223df83740 10 client.17781 fetch_fsmap finished waiting for FSMap version 49 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f223df83740 10 client.17781 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f223df83740 20 client.17781 populate_metadata read hostname 'smithi077' 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f223df83740 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5577fa72db30 0x5577fa72ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f223df83740 4 client.17781 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:26:26.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f223df83740 1 -- 172.21.15.77:0/3530603011 --> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5577fa61dd20 con 0x5577fa72db30 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f2239059640 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5577fa72db30 0x5577fa72ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f2239059640 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5577fa72db30 0x5577fa72ff50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8b0ccd483ec188fc server_cookie=e7d767763e3b58f in_seq=0 out_seq=0 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.570+0000 7f221cff9640 10 client.17781 ms_handle_connect on v2:172.21.15.192:6836/440836568 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f221cff9640 1 -- 172.21.15.77:0/3530603011 <== mds.2 v2:172.21.15.192:6836/440836568 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5577fa61dd20 con 0x5577fa72db30 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f221cff9640 10 client.17781 handle_command_reply: tid=0 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223df83740 1 client.17781 shutdown 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223df83740 1 -- 172.21.15.77:0/3530603011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b73b0 msgr2=0x5577fa6c0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223df83740 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b73b0 0x5577fa6c0090 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f222c036040 tx=0x7f222c004a40 comp rx=0 tx=0).stop 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223df83740 1 -- 172.21.15.77:0/3530603011 shutdown_connections 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223df83740 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577fa6b73b0 0x5577fa6c0090 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223df83740 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.192:6836/440836568,v1:172.21.15.192:6837/440836568] conn(0x5577fa72db30 0x5577fa72ff50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223df83740 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5577fa6b2240 0x5577fa6bfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223df83740 1 --2- 172.21.15.77:0/3530603011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577fa6b1850 0x5577fa6c30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f221cff9640 0 client.17781 ms_handle_reset on v2:172.21.15.192:6836/440836568 2024-03-16T10:26:26.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223df83740 1 -- 172.21.15.77:0/3530603011 >> 172.21.15.77:0/3530603011 conn(0x5577fa6acdc0 msgr2=0x5577fa6b2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223df83740 1 -- 172.21.15.77:0/3530603011 shutdown_connections 2024-03-16T10:26:26.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223df83740 1 -- 172.21.15.77:0/3530603011 wait complete. 2024-03-16T10:26:26.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223df83740 20 client.17781 trim_cache size 0 max 16384 2024-03-16T10:26:26.579 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:26:26.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223b2bd640 1 -- 172.21.15.77:0/3952730162 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f22200a2800 msgr2=0x7f22200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223b2bd640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f22200a2800 0x7f22200a4cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f222c005f00 tx=0x7f222c0098d0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223b2bd640 1 -- 172.21.15.77:0/3952730162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22341006e0 msgr2=0x7f22340694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223b2bd640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22341006e0 0x7f22340694d0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f223000e9a0 tx=0x7f223000ee60 comp rx=0 tx=0).stop 2024-03-16T10:26:26.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223b2bd640 1 -- 172.21.15.77:0/3952730162 shutdown_connections 2024-03-16T10:26:26.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223b2bd640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f22200a2800 0x7f22200a4cc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223b2bd640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2234102290 0x7f223419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223b2bd640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22341018d0 0x7f223419c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223b2bd640 1 --2- 172.21.15.77:0/3952730162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22341006e0 0x7f22340694d0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.571+0000 7f223b2bd640 1 -- 172.21.15.77:0/3952730162 >> 172.21.15.77:0/3952730162 conn(0x7f22340fbc70 msgr2=0x7f2234104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.572+0000 7f223b2bd640 1 -- 172.21.15.77:0/3952730162 shutdown_connections 2024-03-16T10:26:26.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.572+0000 7f223b2bd640 1 -- 172.21.15.77:0/3952730162 wait complete. 2024-03-16T10:26:26.583 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-16T10:26:26.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.700+0000 7f850d7e6640 1 -- 172.21.15.77:0/2699964387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85081055d0 msgr2=0x7f850810be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.700+0000 7f850d7e6640 1 --2- 172.21.15.77:0/2699964387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85081055d0 0x7f850810be60 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009900 tx=0x7f84f802bc30 comp rx=0 tx=0).stop 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.700+0000 7f850d7e6640 1 -- 172.21.15.77:0/2699964387 shutdown_connections 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.700+0000 7f850d7e6640 1 --2- 172.21.15.77:0/2699964387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85081055d0 0x7f850810be60 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.700+0000 7f850d7e6640 1 --2- 172.21.15.77:0/2699964387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8508104c10 0x7f8508105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.700+0000 7f850d7e6640 1 --2- 172.21.15.77:0/2699964387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8508103a20 0x7f8508103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.700+0000 7f850d7e6640 1 -- 172.21.15.77:0/2699964387 >> 172.21.15.77:0/2699964387 conn(0x7f85080fef90 msgr2=0x7f85081013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.700+0000 7f850d7e6640 1 -- 172.21.15.77:0/2699964387 shutdown_connections 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f850d7e6640 1 -- 172.21.15.77:0/2699964387 wait complete. 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f850d7e6640 1 Processor -- start 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f850d7e6640 1 -- start start 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f850d7e6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8508103a20 0x7f8508199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f850d7e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8508104c10 0x7f850819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f850d7e6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85081055d0 0x7f85081a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f850d7e6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f850810f680 con 0x7f8508104c10 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f850d7e6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f850810f500 con 0x7f85081055d0 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f850d7e6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f850810f800 con 0x7f8508103a20 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f85067fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8508104c10 0x7f850819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f85067fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8508104c10 0x7f850819a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46980/0 (socket says 172.21.15.77:46980) 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f85067fc640 1 -- 172.21.15.77:0/2239910794 learned_addr learned my addr 172.21.15.77:0/2239910794 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.701+0000 7f85077fe640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85081055d0 0x7f85081a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f8506ffd640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8508103a20 0x7f8508199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f85067fc640 1 -- 172.21.15.77:0/2239910794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8508103a20 msgr2=0x7f8508199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f85067fc640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8508103a20 0x7f8508199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f85067fc640 1 -- 172.21.15.77:0/2239910794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85081055d0 msgr2=0x7f85081a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f85067fc640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85081055d0 0x7f85081a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f85067fc640 1 -- 172.21.15.77:0/2239910794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85081a18e0 con 0x7f8508104c10 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f85077fe640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85081055d0 0x7f85081a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f85067fc640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8508104c10 0x7f850819a1e0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009d50 tx=0x7f84fc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f84effff640 1 -- 172.21.15.77:0/2239910794 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc003b70 con 0x7f8508104c10 2024-03-16T10:26:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f84effff640 1 -- 172.21.15.77:0/2239910794 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84fc003d10 con 0x7f8508104c10 2024-03-16T10:26:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f850d7e6640 1 -- 172.21.15.77:0/2239910794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85081a1bd0 con 0x7f8508104c10 2024-03-16T10:26:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f84effff640 1 -- 172.21.15.77:0/2239910794 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc010a20 con 0x7f8508104c10 2024-03-16T10:26:26.706 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.702+0000 7f850d7e6640 1 -- 172.21.15.77:0/2239910794 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85081a2120 con 0x7f8508104c10 2024-03-16T10:26:26.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.704+0000 7f85103d2740 1 Processor -- start 2024-03-16T10:26:26.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85103d2740 1 -- start start 2024-03-16T10:26:26.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85103d2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a8f850 0x563f81a8fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85103d2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a90240 0x563f81a94d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85103d2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 0x563f81a97840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85103d2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563f819f4490 con 0x563f81a8f850 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85103d2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563f8199cc90 con 0x563f81a90240 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85103d2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563f819f0530 con 0x563f81a953b0 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85077fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a8f850 0x563f81a8fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85077fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a8f850 0x563f81a8fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46986/0 (socket says 172.21.15.77:46986) 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85077fe640 1 -- 172.21.15.77:0/3543044020 learned_addr learned my addr 172.21.15.77:0/3543044020 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f8506ffd640 1 --2- 172.21.15.77:0/3543044020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a90240 0x563f81a94d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f84effff640 1 -- 172.21.15.77:0/2239910794 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f84fc003ec0 con 0x7f8508104c10 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85077fe640 1 -- 172.21.15.77:0/3543044020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 msgr2=0x563f81a97840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85077fe640 1 --2- 172.21.15.77:0/3543044020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 0x563f81a97840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85077fe640 1 -- 172.21.15.77:0/3543044020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a90240 msgr2=0x563f81a94d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85077fe640 1 --2- 172.21.15.77:0/3543044020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a90240 0x563f81a94d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85077fe640 1 -- 172.21.15.77:0/3543044020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f81a57500 con 0x563f81a8f850 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f8506ffd640 1 --2- 172.21.15.77:0/3543044020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a90240 0x563f81a94d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f85077fe640 1 --2- 172.21.15.77:0/3543044020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a8f850 0x563f81a8fc70 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f84f802cfd0 tx=0x7f84f802cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf38e49b638644f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.705+0000 7f84ed7fa640 1 -- 172.21.15.77:0/3543044020 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8042070 con 0x563f81a8f850 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f84ed7fa640 1 -- 172.21.15.77:0/3543044020 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84f8004ba0 con 0x563f81a8f850 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f84ed7fa640 1 -- 172.21.15.77:0/3543044020 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8005300 con 0x563f81a8f850 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f84effff640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84f00a2800 0x7f84f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f8506ffd640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84f00a2800 0x7f84f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f84effff640 1 -- 172.21.15.77:0/2239910794 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f84fc0cf050 con 0x7f8508104c10 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f85103d2740 1 -- 172.21.15.77:0/3543044020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a8f850 msgr2=0x563f81a8fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f85103d2740 1 --2- 172.21.15.77:0/3543044020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a8f850 0x563f81a8fc70 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f84f802cfd0 tx=0x7f84f802cd30 comp rx=0 tx=0).stop 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f8506ffd640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84f00a2800 0x7f84f00a4cc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x563f81a90640 tx=0x7f8500005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f85103d2740 1 -- 172.21.15.77:0/3543044020 shutdown_connections 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f85103d2740 1 --2- 172.21.15.77:0/3543044020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 0x563f81a97840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f85103d2740 1 --2- 172.21.15.77:0/3543044020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a90240 0x563f81a94d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f85103d2740 1 --2- 172.21.15.77:0/3543044020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a8f850 0x563f81a8fc70 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f85103d2740 1 -- 172.21.15.77:0/3543044020 >> 172.21.15.77:0/3543044020 conn(0x563f81a8adc0 msgr2=0x563f81a8b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f85103d2740 1 -- 172.21.15.77:0/3543044020 shutdown_connections 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.706+0000 7f85103d2740 1 -- 172.21.15.77:0/3543044020 wait complete. 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85103d2740 1 -- start start 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85103d2740 1 Processor -- start 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85103d2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a99dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85103d2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 0x563f81a9a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85103d2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 0x563f81aa1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85103d2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563f81a98300 con 0x563f81a90240 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85103d2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563f819b0230 con 0x563f81a8f850 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85103d2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563f81a98480 con 0x563f81a953b0 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85103d2740 10 client.0 _ll_register_callbacks cb 0x563f819ec010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85067fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 0x563f81a9a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85067fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 0x563f81a9a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46996/0 (socket says 172.21.15.77:46996) 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85067fc640 1 -- 172.21.15.77:0/1609869325 learned_addr learned my addr 172.21.15.77:0/1609869325 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85077fe640 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 0x563f81aa1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f8506ffd640 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a99dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85067fc640 1 -- 172.21.15.77:0/1609869325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 msgr2=0x563f81aa1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85067fc640 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 0x563f81aa1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85067fc640 1 -- 172.21.15.77:0/1609869325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 msgr2=0x563f81a99dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85067fc640 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a99dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85067fc640 1 -- 172.21.15.77:0/1609869325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f819bf100 con 0x563f81a90240 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f8506ffd640 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a99dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f85067fc640 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 0x563f81a9a310 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f84fc098450 tx=0x7f84fc09b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30186536e2d91b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.707+0000 7f84eaff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f84eaff5640 1 -- 172.21.15.77:0/1609869325 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc0b3020 con 0x563f81a90240 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f84eaff5640 1 -- 172.21.15.77:0/1609869325 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f84fc0980f0 con 0x563f81a90240 2024-03-16T10:26:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f85103d2740 1 -- 172.21.15.77:0/1609869325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84f8004ba0 con 0x563f81a90240 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f84eaff5640 1 -- 172.21.15.77:0/1609869325 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f84fc0afe60 con 0x563f81a90240 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f85103d2740 10 client.17793 fetch_fsmap learned FSMap version 49 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f85103d2740 1 -- 172.21.15.77:0/1609869325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x563f817083f0 con 0x563f81a90240 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f84eaff5640 1 -- 172.21.15.77:0/1609869325 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84f8004ba0 con 0x563f81a90240 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f84eaff5640 1 -- 172.21.15.77:0/1609869325 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 49) ==== 8378+0+0 (secure 0 0 0) 0x7f84fc0a3d90 con 0x563f81a90240 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f85103d2740 10 client.17793 fetch_fsmap finished waiting for FSMap version 49 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f85103d2740 10 client.17793 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f85103d2740 20 client.17793 populate_metadata read hostname 'smithi077' 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f85103d2740 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x563f81aa5bf0 0x563f81aa8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f85103d2740 4 client.17793 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f85103d2740 1 -- 172.21.15.77:0/1609869325 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f81a019d0 con 0x563f81aa5bf0 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.708+0000 7f8506ffd640 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x563f81aa5bf0 0x563f81aa8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f84eaff5640 10 client.17793 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f8506ffd640 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x563f81aa5bf0 0x563f81aa8030 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef01a1556c0b2a83 server_cookie=b578d52dca06f0a2 in_seq=0 out_seq=0 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f84eaff5640 1 -- 172.21.15.77:0/1609869325 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563f81a019d0 con 0x563f81aa5bf0 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f84eaff5640 10 client.17793 handle_command_reply: tid=0 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f85103d2740 1 client.17793 shutdown 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f85103d2740 1 -- 172.21.15.77:0/1609869325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 msgr2=0x563f81a9a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f85103d2740 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 0x563f81a9a310 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f84fc098450 tx=0x7f84fc09b3e0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f85103d2740 1 -- 172.21.15.77:0/1609869325 shutdown_connections 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f85103d2740 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x563f81aa5bf0 0x563f81aa8030 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f85103d2740 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 0x563f81aa1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f85103d2740 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 0x563f81a9a310 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f84eaff5640 0 client.17793 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f85103d2740 1 --2- 172.21.15.77:0/1609869325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a99dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.709+0000 7f85103d2740 1 -- 172.21.15.77:0/1609869325 >> 172.21.15.77:0/1609869325 conn(0x563f81a8adc0 msgr2=0x563f81a89fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.710+0000 7f85103d2740 1 -- 172.21.15.77:0/1609869325 shutdown_connections 2024-03-16T10:26:26.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.710+0000 7f85103d2740 1 -- 172.21.15.77:0/1609869325 wait complete. 2024-03-16T10:26:26.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.710+0000 7f85103d2740 20 client.17793 trim_cache size 0 max 16384 2024-03-16T10:26:26.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85103d2740 1 Processor -- start 2024-03-16T10:26:26.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85103d2740 1 -- start start 2024-03-16T10:26:26.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85103d2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a8e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85103d2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 0x563f81aa12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85103d2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 0x563f81aac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85103d2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f85081aacf0 con 0x563f81a90240 2024-03-16T10:26:26.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85103d2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f85081ac8f0 con 0x563f81a8f850 2024-03-16T10:26:26.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85103d2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f85081ab420 con 0x563f81a953b0 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85067fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 0x563f81aa12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85067fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 0x563f81aa12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47000/0 (socket says 172.21.15.77:47000) 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85067fc640 1 -- 172.21.15.77:0/893572858 learned_addr learned my addr 172.21.15.77:0/893572858 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85077fe640 1 --2- 172.21.15.77:0/893572858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 0x563f81aac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f8506ffd640 1 --2- 172.21.15.77:0/893572858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a8e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85067fc640 1 -- 172.21.15.77:0/893572858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 msgr2=0x563f81aac8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85067fc640 1 --2- 172.21.15.77:0/893572858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 0x563f81aac8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85067fc640 1 -- 172.21.15.77:0/893572858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 msgr2=0x563f81a8e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85067fc640 1 --2- 172.21.15.77:0/893572858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a8e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85067fc640 1 -- 172.21.15.77:0/893572858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f817d27d0 con 0x563f81a90240 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f8506ffd640 1 --2- 172.21.15.77:0/893572858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a8e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f85067fc640 1 --2- 172.21.15.77:0/893572858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 0x563f81aa12d0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f84fc005ee0 tx=0x7f84fc09b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf81d8f16d8fafa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.723+0000 7f84ecff9640 1 -- 172.21.15.77:0/893572858 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc0b3070 con 0x563f81a90240 2024-03-16T10:26:26.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f84ecff9640 1 -- 172.21.15.77:0/893572858 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84fc009820 con 0x563f81a90240 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f84ecff9640 1 -- 172.21.15.77:0/893572858 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc0980f0 con 0x563f81a90240 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f85103d2740 1 -- 172.21.15.77:0/893572858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 msgr2=0x563f81aa12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f85103d2740 1 --2- 172.21.15.77:0/893572858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 0x563f81aa12d0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f84fc005ee0 tx=0x7f84fc09b8d0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f85103d2740 1 -- 172.21.15.77:0/893572858 shutdown_connections 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f85103d2740 1 --2- 172.21.15.77:0/893572858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a953b0 0x563f81aac8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f85103d2740 1 --2- 172.21.15.77:0/893572858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a90240 0x563f81aa12d0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f85103d2740 1 --2- 172.21.15.77:0/893572858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a8e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f85103d2740 1 -- 172.21.15.77:0/893572858 >> 172.21.15.77:0/893572858 conn(0x563f81a8adc0 msgr2=0x563f81a8e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f85103d2740 1 -- 172.21.15.77:0/893572858 shutdown_connections 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f85103d2740 1 -- 172.21.15.77:0/893572858 wait complete. 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f85103d2740 1 -- start start 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.724+0000 7f85103d2740 1 Processor -- start 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85103d2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a9e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85103d2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a90240 0x563f81a9eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85103d2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a953b0 0x563f81a9fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85103d2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f85081ac220 con 0x563f81a953b0 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85103d2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f85081abb20 con 0x563f81a8f850 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85103d2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f85081a8cd0 con 0x563f81a90240 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85103d2740 10 client.0 _ll_register_callbacks cb 0x7f84fc0048f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85077fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a953b0 0x563f81a9fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f8506ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a9e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85077fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a953b0 0x563f81a9fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47012/0 (socket says 172.21.15.77:47012) 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85077fe640 1 -- 172.21.15.77:0/2162539029 learned_addr learned my addr 172.21.15.77:0/2162539029 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85067fc640 1 --2- 172.21.15.77:0/2162539029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a90240 0x563f81a9eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85077fe640 1 -- 172.21.15.77:0/2162539029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a90240 msgr2=0x563f81a9eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85077fe640 1 --2- 172.21.15.77:0/2162539029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a90240 0x563f81a9eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85077fe640 1 -- 172.21.15.77:0/2162539029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 msgr2=0x563f81a9e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85077fe640 1 --2- 172.21.15.77:0/2162539029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a9e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85077fe640 1 -- 172.21.15.77:0/2162539029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f81aa04e0 con 0x563f81a953b0 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85077fe640 1 --2- 172.21.15.77:0/2162539029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a953b0 0x563f81a9fef0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f84f8000c00 tx=0x7f84f80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d2f543a27fcb6b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f84eaff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f84eaff5640 1 -- 172.21.15.77:0/2162539029 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8042040 con 0x563f81a953b0 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f84eaff5640 1 -- 172.21.15.77:0/2162539029 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f84f8004b70 con 0x563f81a953b0 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.725+0000 7f85103d2740 1 -- 172.21.15.77:0/2162539029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84fc009820 con 0x563f81a953b0 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.726+0000 7f84eaff5640 1 -- 172.21.15.77:0/2162539029 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f84f8057050 con 0x563f81a953b0 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.726+0000 7f85103d2740 10 client.17799 fetch_fsmap learned FSMap version 49 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.726+0000 7f85103d2740 1 -- 172.21.15.77:0/2162539029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x563f81b0bf30 con 0x563f81a953b0 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.726+0000 7f84eaff5640 1 -- 172.21.15.77:0/2162539029 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84fc009820 con 0x563f81a953b0 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.726+0000 7f84eaff5640 1 -- 172.21.15.77:0/2162539029 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 49) ==== 8378+0+0 (secure 0 0 0) 0x7f84f80036a0 con 0x563f81a953b0 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.726+0000 7f85103d2740 10 client.17799 fetch_fsmap finished waiting for FSMap version 49 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.726+0000 7f85103d2740 10 client.17799 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-16T10:26:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.726+0000 7f85103d2740 20 client.17799 populate_metadata read hostname 'smithi077' 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.726+0000 7f85103d2740 1 --2- 172.21.15.77:0/2162539029 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x563f81b0c430 0x563f81b0e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.726+0000 7f85103d2740 4 client.17799 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.726+0000 7f85103d2740 1 -- 172.21.15.77:0/2162539029 --> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563f819fbd20 con 0x563f81b0c430 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.726+0000 7f8506ffd640 1 --2- 172.21.15.77:0/2162539029 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x563f81b0c430 0x563f81b0e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f8506ffd640 1 --2- 172.21.15.77:0/2162539029 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x563f81b0c430 0x563f81b0e850 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d24b4f995395259 server_cookie=f287570fdcb02589 in_seq=0 out_seq=0 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f84eaff5640 10 client.17799 ms_handle_connect on v2:172.21.15.193:6838/3991082280 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f84eaff5640 1 -- 172.21.15.77:0/2162539029 <== mds.1 v2:172.21.15.193:6838/3991082280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563f819fbd20 con 0x563f81b0c430 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f84eaff5640 10 client.17799 handle_command_reply: tid=0 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f85103d2740 1 client.17799 shutdown 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f85103d2740 1 -- 172.21.15.77:0/2162539029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a953b0 msgr2=0x563f81a9fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f85103d2740 1 --2- 172.21.15.77:0/2162539029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a953b0 0x563f81a9fef0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f84f8000c00 tx=0x7f84f80048f0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f85103d2740 1 -- 172.21.15.77:0/2162539029 shutdown_connections 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f85103d2740 1 --2- 172.21.15.77:0/2162539029 >> [v2:172.21.15.193:6838/3991082280,v1:172.21.15.193:6839/3991082280] conn(0x563f81b0c430 0x563f81b0e850 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f85103d2740 1 --2- 172.21.15.77:0/2162539029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f81a953b0 0x563f81a9fef0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f85103d2740 1 --2- 172.21.15.77:0/2162539029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563f81a90240 0x563f81a9eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f85103d2740 1 --2- 172.21.15.77:0/2162539029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f81a8f850 0x563f81a9e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f84eaff5640 0 client.17799 ms_handle_reset on v2:172.21.15.193:6838/3991082280 2024-03-16T10:26:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f85103d2740 1 -- 172.21.15.77:0/2162539029 >> 172.21.15.77:0/2162539029 conn(0x563f81a8adc0 msgr2=0x563f81a8e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f85103d2740 1 -- 172.21.15.77:0/2162539029 shutdown_connections 2024-03-16T10:26:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f85103d2740 1 -- 172.21.15.77:0/2162539029 wait complete. 2024-03-16T10:26:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f85103d2740 20 client.17799 trim_cache size 0 max 16384 2024-03-16T10:26:26.735 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:26:26.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f850d7e6640 1 -- 172.21.15.77:0/2239910794 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84f00a2800 msgr2=0x7f84f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f850d7e6640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84f00a2800 0x7f84f00a4cc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x563f81a90640 tx=0x7f8500005f90 comp rx=0 tx=0).stop 2024-03-16T10:26:26.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f850d7e6640 1 -- 172.21.15.77:0/2239910794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8508104c10 msgr2=0x7f850819a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.727+0000 7f850d7e6640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8508104c10 0x7f850819a1e0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009d50 tx=0x7f84fc00c6a0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.728+0000 7f850d7e6640 1 -- 172.21.15.77:0/2239910794 shutdown_connections 2024-03-16T10:26:26.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.728+0000 7f850d7e6640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85081055d0 0x7f85081a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.728+0000 7f850d7e6640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8508104c10 0x7f850819a1e0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.728+0000 7f850d7e6640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84f00a2800 0x7f84f00a4cc0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.728+0000 7f850d7e6640 1 --2- 172.21.15.77:0/2239910794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8508103a20 0x7f8508199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.728+0000 7f850d7e6640 1 -- 172.21.15.77:0/2239910794 >> 172.21.15.77:0/2239910794 conn(0x7f85080fef90 msgr2=0x7f8508107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.728+0000 7f850d7e6640 1 -- 172.21.15.77:0/2239910794 shutdown_connections 2024-03-16T10:26:26.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.728+0000 7f850d7e6640 1 -- 172.21.15.77:0/2239910794 wait complete. 2024-03-16T10:26:26.740 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-16T10:26:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.852+0000 7fa763dd3640 1 -- 172.21.15.77:0/3175243459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c102290 msgr2=0x7fa75c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.852+0000 7fa763dd3640 1 --2- 172.21.15.77:0/3175243459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c102290 0x7fa75c108b20 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa754009ca0 tx=0x7fa75402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.852+0000 7fa763dd3640 1 -- 172.21.15.77:0/3175243459 shutdown_connections 2024-03-16T10:26:26.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.852+0000 7fa763dd3640 1 --2- 172.21.15.77:0/3175243459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c102290 0x7fa75c108b20 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.852+0000 7fa763dd3640 1 --2- 172.21.15.77:0/3175243459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa75c1018d0 0x7fa75c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.852+0000 7fa763dd3640 1 --2- 172.21.15.77:0/3175243459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa75c1006e0 0x7fa75c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.852+0000 7fa763dd3640 1 -- 172.21.15.77:0/3175243459 >> 172.21.15.77:0/3175243459 conn(0x7fa75c0fbc70 msgr2=0x7fa75c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.852+0000 7fa763dd3640 1 -- 172.21.15.77:0/3175243459 shutdown_connections 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.852+0000 7fa763dd3640 1 -- 172.21.15.77:0/3175243459 wait complete. 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa763dd3640 1 Processor -- start 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa763dd3640 1 -- start start 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa763dd3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1006e0 0x7fa75c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa763dd3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa75c1018d0 0x7fa75c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa763dd3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa75c102290 0x7fa75c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa763dd3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c10c340 con 0x7fa75c1006e0 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa763dd3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c10c1c0 con 0x7fa75c1018d0 2024-03-16T10:26:26.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa763dd3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c10c4c0 con 0x7fa75c102290 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa761b6f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1006e0 0x7fa75c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa761b6f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1006e0 0x7fa75c197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47028/0 (socket says 172.21.15.77:47028) 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa761b6f640 1 -- 172.21.15.77:0/2799779842 learned_addr learned my addr 172.21.15.77:0/2799779842 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa762370640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa75c102290 0x7fa75c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa76136e640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa75c1018d0 0x7fa75c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa761b6f640 1 -- 172.21.15.77:0/2799779842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa75c102290 msgr2=0x7fa75c19f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa761b6f640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa75c102290 0x7fa75c19f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa761b6f640 1 -- 172.21.15.77:0/2799779842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa75c1018d0 msgr2=0x7fa75c198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa761b6f640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa75c1018d0 0x7fa75c198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.853+0000 7fa761b6f640 1 -- 172.21.15.77:0/2799779842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75c19f720 con 0x7fa75c1006e0 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.854+0000 7fa761b6f640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1006e0 0x7fa75c197b00 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fa75800e9c0 tx=0x7fa75800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.854+0000 7fa752ffd640 1 -- 172.21.15.77:0/2799779842 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75800cde0 con 0x7fa75c1006e0 2024-03-16T10:26:26.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.854+0000 7fa752ffd640 1 -- 172.21.15.77:0/2799779842 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa758010910 con 0x7fa75c1006e0 2024-03-16T10:26:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.854+0000 7fa763dd3640 1 -- 172.21.15.77:0/2799779842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75c19fa10 con 0x7fa75c1006e0 2024-03-16T10:26:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.854+0000 7fa752ffd640 1 -- 172.21.15.77:0/2799779842 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758018a00 con 0x7fa75c1006e0 2024-03-16T10:26:26.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.854+0000 7fa763dd3640 1 -- 172.21.15.77:0/2799779842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa75c19ff60 con 0x7fa75c1006e0 2024-03-16T10:26:26.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.856+0000 7fa752ffd640 1 -- 172.21.15.77:0/2799779842 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fa75801f020 con 0x7fa75c1006e0 2024-03-16T10:26:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa7669b6740 1 Processor -- start 2024-03-16T10:26:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa7669b6740 1 -- start start 2024-03-16T10:26:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa7669b6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4a850 0x55d489c4ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa7669b6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c4fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa7669b6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c503b0 0x55d489c52840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa7669b6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d489baf490 con 0x55d489c4a850 2024-03-16T10:26:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa7669b6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d489b57c90 con 0x55d489c503b0 2024-03-16T10:26:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa7669b6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d489bab530 con 0x55d489c4b240 2024-03-16T10:26:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa752ffd640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa7480a2800 0x7fa7480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa761b6f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c4fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.865 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa761b6f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c4fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:36974/0 (socket says 172.21.15.77:36974) 2024-03-16T10:26:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa761b6f640 1 -- 172.21.15.77:0/1358190777 learned_addr learned my addr 172.21.15.77:0/1358190777 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa762370640 1 --2- 172.21.15.77:0/1358190777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4a850 0x55d489c4ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.857+0000 7fa76136e640 1 --2- 172.21.15.77:0/1358190777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c503b0 0x55d489c52840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.858+0000 7fa762370640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa7480a2800 0x7fa7480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.858+0000 7fa761b6f640 1 -- 172.21.15.77:0/1358190777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c503b0 msgr2=0x55d489c52840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.858+0000 7fa761b6f640 1 --2- 172.21.15.77:0/1358190777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c503b0 0x55d489c52840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.858+0000 7fa761b6f640 1 -- 172.21.15.77:0/1358190777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4a850 msgr2=0x55d489c4ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.858+0000 7fa761b6f640 1 --2- 172.21.15.77:0/1358190777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4a850 0x55d489c4ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.858+0000 7fa752ffd640 1 -- 172.21.15.77:0/2799779842 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fa7580cebb0 con 0x7fa75c1006e0 2024-03-16T10:26:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.858+0000 7fa761b6f640 1 -- 172.21.15.77:0/1358190777 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d489c12500 con 0x55d489c4b240 2024-03-16T10:26:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.858+0000 7fa761b6f640 1 --2- 172.21.15.77:0/1358190777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c4fd40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa7580cc680 tx=0x7fa75800efb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f192349b25324fbb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.858+0000 7fa762370640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa7480a2800 0x7fa7480a4cc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa75402c630 tx=0x7fa7540093c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.858+0000 7fa747fff640 1 -- 172.21.15.77:0/1358190777 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75809bce0 con 0x55d489c4b240 2024-03-16T10:26:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.858+0000 7fa747fff640 1 -- 172.21.15.77:0/1358190777 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa75809b5e0 con 0x55d489c4b240 2024-03-16T10:26:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.858+0000 7fa747fff640 1 -- 172.21.15.77:0/1358190777 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758097c50 con 0x55d489c4b240 2024-03-16T10:26:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.859+0000 7fa7669b6740 1 -- 172.21.15.77:0/1358190777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 msgr2=0x55d489c4fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.859+0000 7fa7669b6740 1 --2- 172.21.15.77:0/1358190777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c4fd40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa7580cc680 tx=0x7fa75800efb0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.859+0000 7fa7669b6740 1 -- 172.21.15.77:0/1358190777 shutdown_connections 2024-03-16T10:26:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.859+0000 7fa7669b6740 1 --2- 172.21.15.77:0/1358190777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c503b0 0x55d489c52840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.859+0000 7fa7669b6740 1 --2- 172.21.15.77:0/1358190777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c4fd40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.859+0000 7fa7669b6740 1 --2- 172.21.15.77:0/1358190777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4a850 0x55d489c4ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.859+0000 7fa7669b6740 1 -- 172.21.15.77:0/1358190777 >> 172.21.15.77:0/1358190777 conn(0x55d489c45dc0 msgr2=0x55d489c461c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.859+0000 7fa7669b6740 1 -- 172.21.15.77:0/1358190777 shutdown_connections 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.859+0000 7fa7669b6740 1 -- 172.21.15.77:0/1358190777 wait complete. 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.859+0000 7fa7669b6740 1 -- start start 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.859+0000 7fa7669b6740 1 Processor -- start 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa7669b6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4a850 0x55d489c54dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa7669b6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c55310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa7669b6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c503b0 0x55d489c5c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa7669b6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d489c53360 con 0x55d489c4a850 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa7669b6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55d489b6b230 con 0x55d489c503b0 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa7669b6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d489c534e0 con 0x55d489c4b240 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa7669b6740 10 client.0 _ll_register_callbacks cb 0x55d489ba7010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:26:26.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa761b6f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4a850 0x55d489c54dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa761b6f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4a850 0x55d489c54dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47044/0 (socket says 172.21.15.77:47044) 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa761b6f640 1 -- 172.21.15.77:0/1165387568 learned_addr learned my addr 172.21.15.77:0/1165387568 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa76136e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c55310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa762370640 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c503b0 0x55d489c5c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa76136e640 1 -- 172.21.15.77:0/1165387568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c503b0 msgr2=0x55d489c5c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa76136e640 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c503b0 0x55d489c5c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa76136e640 1 -- 172.21.15.77:0/1165387568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4a850 msgr2=0x55d489c54dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa76136e640 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4a850 0x55d489c54dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa76136e640 1 -- 172.21.15.77:0/1165387568 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d489b7a100 con 0x55d489c4b240 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa761b6f640 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4a850 0x55d489c54dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa76136e640 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c55310 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa75c199010 tx=0x7fa74c012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae7cda79d3e65570 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.860+0000 7fa7457fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7457fa640 1 -- 172.21.15.77:0/1165387568 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74c00f920 con 0x55d489c4b240 2024-03-16T10:26:26.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7669b6740 1 -- 172.21.15.77:0/1165387568 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa75809b5e0 con 0x55d489c4b240 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7457fa640 1 -- 172.21.15.77:0/1165387568 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fa74c00fac0 con 0x55d489c4b240 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7457fa640 1 -- 172.21.15.77:0/1165387568 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fa74c00fcc0 con 0x55d489c4b240 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7669b6740 10 client.27562 fetch_fsmap learned FSMap version 49 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7669b6740 1 -- 172.21.15.77:0/1165387568 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55d4898c33f0 con 0x55d489c4b240 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7457fa640 1 -- 172.21.15.77:0/1165387568 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa75809b5e0 con 0x55d489c4b240 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7457fa640 1 -- 172.21.15.77:0/1165387568 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 49) ==== 8378+0+0 (secure 0 0 0) 0x7fa74c025050 con 0x55d489c4b240 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7669b6740 10 client.27562 fetch_fsmap finished waiting for FSMap version 49 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7669b6740 10 client.27562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7669b6740 20 client.27562 populate_metadata read hostname 'smithi077' 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7669b6740 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d489c60c00 0x55d489c63080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7669b6740 4 client.27562 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa7669b6740 1 -- 172.21.15.77:0/1165387568 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d489bbc9d0 con 0x55d489c60c00 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.861+0000 7fa761b6f640 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d489c60c00 0x55d489c63080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.862+0000 7fa761b6f640 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d489c60c00 0x55d489c63080 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9b1b5a29d53b5c9 server_cookie=5ec6fffb3a86615 in_seq=0 out_seq=0 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.862+0000 7fa7457fa640 10 client.27562 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7457fa640 1 -- 172.21.15.77:0/1165387568 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d489bbc9d0 con 0x55d489c60c00 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7457fa640 10 client.27562 handle_command_reply: tid=0 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7669b6740 1 client.27562 shutdown 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7669b6740 1 -- 172.21.15.77:0/1165387568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 msgr2=0x55d489c55310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7669b6740 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c55310 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa75c199010 tx=0x7fa74c012b60 comp rx=0 tx=0).stop 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7669b6740 1 -- 172.21.15.77:0/1165387568 shutdown_connections 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7669b6740 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c503b0 0x55d489c5c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7669b6740 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c55310 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7669b6740 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d489c60c00 0x55d489c63080 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7669b6740 1 --2- 172.21.15.77:0/1165387568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4a850 0x55d489c54dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7669b6740 1 -- 172.21.15.77:0/1165387568 >> 172.21.15.77:0/1165387568 conn(0x55d489c45dc0 msgr2=0x55d489c44fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7457fa640 0 client.27562 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:26:26.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7669b6740 1 -- 172.21.15.77:0/1165387568 shutdown_connections 2024-03-16T10:26:26.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7669b6740 1 -- 172.21.15.77:0/1165387568 wait complete. 2024-03-16T10:26:26.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.863+0000 7fa7669b6740 20 client.27562 trim_cache size 0 max 16384 2024-03-16T10:26:26.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa7669b6740 1 Processor -- start 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa7669b6740 1 -- start start 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa7669b6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c4a850 0x55d489c540c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa7669b6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4b240 0x55d489c54600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa7669b6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c503b0 0x55d489c5c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa7669b6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1a8c10 con 0x55d489c4b240 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa7669b6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1aa810 con 0x55d489c4a850 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa7669b6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1a9340 con 0x55d489c503b0 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa76136e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4b240 0x55d489c54600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa76136e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4b240 0x55d489c54600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47048/0 (socket says 172.21.15.77:47048) 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa76136e640 1 -- 172.21.15.77:0/2672914870 learned_addr learned my addr 172.21.15.77:0/2672914870 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa761b6f640 1 --2- 172.21.15.77:0/2672914870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c4a850 0x55d489c540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.880+0000 7fa762370640 1 --2- 172.21.15.77:0/2672914870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c503b0 0x55d489c5c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa76136e640 1 -- 172.21.15.77:0/2672914870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c503b0 msgr2=0x55d489c5c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa76136e640 1 --2- 172.21.15.77:0/2672914870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c503b0 0x55d489c5c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa76136e640 1 -- 172.21.15.77:0/2672914870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c4a850 msgr2=0x55d489c540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa76136e640 1 --2- 172.21.15.77:0/2672914870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c4a850 0x55d489c540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa76136e640 1 -- 172.21.15.77:0/2672914870 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d48998d7d0 con 0x55d489c4b240 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa76136e640 1 --2- 172.21.15.77:0/2672914870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4b240 0x55d489c54600 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009460 tx=0x7fa74c009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c2a6523360bf04 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa7477fe640 1 -- 172.21.15.77:0/2672914870 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74c00f9a0 con 0x55d489c4b240 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa7477fe640 1 -- 172.21.15.77:0/2672914870 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa74c02b020 con 0x55d489c4b240 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa7477fe640 1 -- 172.21.15.77:0/2672914870 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74c0053d0 con 0x55d489c4b240 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa7669b6740 1 -- 172.21.15.77:0/2672914870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4b240 msgr2=0x55d489c54600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa7669b6740 1 --2- 172.21.15.77:0/2672914870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4b240 0x55d489c54600 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009460 tx=0x7fa74c009fd0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa7669b6740 1 -- 172.21.15.77:0/2672914870 shutdown_connections 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa7669b6740 1 --2- 172.21.15.77:0/2672914870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c503b0 0x55d489c5c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa7669b6740 1 --2- 172.21.15.77:0/2672914870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c4b240 0x55d489c54600 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.881+0000 7fa7669b6740 1 --2- 172.21.15.77:0/2672914870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c4a850 0x55d489c540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa7669b6740 1 -- 172.21.15.77:0/2672914870 >> 172.21.15.77:0/2672914870 conn(0x55d489c45dc0 msgr2=0x55d489c461a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa7669b6740 1 -- 172.21.15.77:0/2672914870 shutdown_connections 2024-03-16T10:26:26.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa7669b6740 1 -- 172.21.15.77:0/2672914870 wait complete. 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa7669b6740 1 -- start start 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa7669b6740 1 Processor -- start 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa7669b6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c4a850 0x55d489c5c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa7669b6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c58cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa7669b6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c503b0 0x55d489c59210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa7669b6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1aa140 con 0x55d489c503b0 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa7669b6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1a9a40 con 0x55d489c4a850 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa7669b6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1a6bf0 con 0x55d489c4b240 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa7669b6740 10 client.0 _ll_register_callbacks cb 0x7fa74c009370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa762370640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c503b0 0x55d489c59210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa76136e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c58cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa762370640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c503b0 0x55d489c59210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47062/0 (socket says 172.21.15.77:47062) 2024-03-16T10:26:26.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa762370640 1 -- 172.21.15.77:0/980455572 learned_addr learned my addr 172.21.15.77:0/980455572 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.882+0000 7fa761b6f640 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c4a850 0x55d489c5c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa76136e640 1 -- 172.21.15.77:0/980455572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c4a850 msgr2=0x55d489c5c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa76136e640 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c4a850 0x55d489c5c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa76136e640 1 -- 172.21.15.77:0/980455572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c503b0 msgr2=0x55d489c59210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa76136e640 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c503b0 0x55d489c59210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa76136e640 1 -- 172.21.15.77:0/980455572 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d489cc4fb0 con 0x55d489c4b240 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa762370640 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c503b0 0x55d489c59210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa76136e640 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c58cd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0126a0 tx=0x7fa74c01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e53cfaca781fcf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa7457fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa7457fa640 1 -- 172.21.15.77:0/980455572 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74c01cc70 con 0x55d489c4b240 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa7457fa640 1 -- 172.21.15.77:0/980455572 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fa74c02c040 con 0x55d489c4b240 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa7669b6740 1 -- 172.21.15.77:0/980455572 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa74c02b020 con 0x55d489c4b240 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa7457fa640 1 -- 172.21.15.77:0/980455572 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fa74c01ce10 con 0x55d489c4b240 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa7669b6740 10 client.27568 fetch_fsmap learned FSMap version 49 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa7669b6740 1 -- 172.21.15.77:0/980455572 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55d489cc61c0 con 0x55d489c4b240 2024-03-16T10:26:26.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa7457fa640 1 -- 172.21.15.77:0/980455572 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa74c02b020 con 0x55d489c4b240 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.883+0000 7fa7457fa640 1 -- 172.21.15.77:0/980455572 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 49) ==== 8378+0+0 (secure 0 0 0) 0x7fa74c025070 con 0x55d489c4b240 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 10 client.27568 fetch_fsmap finished waiting for FSMap version 49 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 10 client.27568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 20 client.27568 populate_metadata read hostname 'smithi077' 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d489cc6b00 0x55d489cc8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 4 client.27568 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 1 -- 172.21.15.77:0/980455572 --> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d489bb6d20 con 0x55d489cc6b00 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa761b6f640 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d489cc6b00 0x55d489cc8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa761b6f640 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d489cc6b00 0x55d489cc8f20 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1879c3d3e268720a server_cookie=3524c4062cf0b731 in_seq=0 out_seq=0 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7457fa640 10 client.27568 ms_handle_connect on v2:172.21.15.192:6838/3223311954 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7457fa640 1 -- 172.21.15.77:0/980455572 <== mds.0 v2:172.21.15.192:6838/3223311954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d489bb6d20 con 0x55d489cc6b00 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7457fa640 10 client.27568 handle_command_reply: tid=0 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 1 client.27568 shutdown 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 1 -- 172.21.15.77:0/980455572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 msgr2=0x55d489c58cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c58cd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0126a0 tx=0x7fa74c01c8e0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 1 -- 172.21.15.77:0/980455572 shutdown_connections 2024-03-16T10:26:26.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.192:6838/3223311954,v1:172.21.15.192:6839/3223311954] conn(0x55d489cc6b00 0x55d489cc8f20 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d489c503b0 0x55d489c59210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7457fa640 0 client.27568 ms_handle_reset on v2:172.21.15.192:6838/3223311954 2024-03-16T10:26:26.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d489c4b240 0x55d489c58cd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 1 --2- 172.21.15.77:0/980455572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d489c4a850 0x55d489c5c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.884+0000 7fa7669b6740 1 -- 172.21.15.77:0/980455572 >> 172.21.15.77:0/980455572 conn(0x55d489c45dc0 msgr2=0x55d489c4c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa7669b6740 1 -- 172.21.15.77:0/980455572 shutdown_connections 2024-03-16T10:26:26.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa7669b6740 1 -- 172.21.15.77:0/980455572 wait complete. 2024-03-16T10:26:26.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa7669b6740 20 client.27568 trim_cache size 0 max 16384 2024-03-16T10:26:26.893 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2024-03-16T10:26:26.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa763dd3640 1 -- 172.21.15.77:0/2799779842 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa7480a2800 msgr2=0x7fa7480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa763dd3640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa7480a2800 0x7fa7480a4cc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa75402c630 tx=0x7fa7540093c0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa763dd3640 1 -- 172.21.15.77:0/2799779842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1006e0 msgr2=0x7fa75c197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:26.894 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa763dd3640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1006e0 0x7fa75c197b00 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fa75800e9c0 tx=0x7fa75800ee80 comp rx=0 tx=0).stop 2024-03-16T10:26:26.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa763dd3640 1 -- 172.21.15.77:0/2799779842 shutdown_connections 2024-03-16T10:26:26.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa763dd3640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fa7480a2800 0x7fa7480a4cc0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa763dd3640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa75c102290 0x7fa75c19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa763dd3640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa75c1018d0 0x7fa75c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa763dd3640 1 --2- 172.21.15.77:0/2799779842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1006e0 0x7fa75c197b00 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:26.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa763dd3640 1 -- 172.21.15.77:0/2799779842 >> 172.21.15.77:0/2799779842 conn(0x7fa75c0fbc70 msgr2=0x7fa75c068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:26.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa763dd3640 1 -- 172.21.15.77:0/2799779842 shutdown_connections 2024-03-16T10:26:26.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:26.885+0000 7fa763dd3640 1 -- 172.21.15.77:0/2799779842 wait complete. 2024-03-16T10:26:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:26 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1102572036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:26:27.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:26 smithi077 ceph-mon[32400]: pgmap v632: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 11 op/s 2024-03-16T10:26:27.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:26 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1102572036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:26:27.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:26 smithi193 ceph-mon[36311]: pgmap v632: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 11 op/s 2024-03-16T10:26:27.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:26 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1102572036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:26:27.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:26 smithi192 ceph-mon[36354]: pgmap v632: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 11 op/s 2024-03-16T10:26:27.898 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-16T10:26:27.898 INFO:tasks.fwd_scrub:done joining 2024-03-16T10:26:27.898 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-16T10:26:27.917 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:26:28.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.020+0000 7f22713f1640 1 -- 172.21.15.77:0/3352900713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f226c104c30 msgr2=0x7f226c107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.020+0000 7f22713f1640 1 --2- 172.21.15.77:0/3352900713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f226c104c30 0x7f226c107020 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2254009900 tx=0x7f225402bc30 comp rx=0 tx=0).stop 2024-03-16T10:26:28.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 -- 172.21.15.77:0/3352900713 shutdown_connections 2024-03-16T10:26:28.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 --2- 172.21.15.77:0/3352900713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f226c107560 0x7f226c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 --2- 172.21.15.77:0/3352900713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f226c104c30 0x7f226c107020 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 --2- 172.21.15.77:0/3352900713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f226c102300 0x7f226c1046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 -- 172.21.15.77:0/3352900713 >> 172.21.15.77:0/3352900713 conn(0x7f226c0fbc70 msgr2=0x7f226c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:28.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 -- 172.21.15.77:0/3352900713 shutdown_connections 2024-03-16T10:26:28.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 -- 172.21.15.77:0/3352900713 wait complete. 2024-03-16T10:26:28.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 Processor -- start 2024-03-16T10:26:28.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 -- start start 2024-03-16T10:26:28.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f226c102300 0x7f226c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f226c104c30 0x7f226c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f226c107560 0x7f226c19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f226c10d450 con 0x7f226c102300 2024-03-16T10:26:28.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f226c10d2d0 con 0x7f226c107560 2024-03-16T10:26:28.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f22713f1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f226c10d5d0 con 0x7f226c104c30 2024-03-16T10:26:28.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f226affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f226c102300 0x7f226c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f226affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f226c102300 0x7f226c197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47084/0 (socket says 172.21.15.77:47084) 2024-03-16T10:26:28.030 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f226affd640 1 -- 172.21.15.77:0/2625649003 learned_addr learned my addr 172.21.15.77:0/2625649003 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f226b7fe640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f226c107560 0x7f226c19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.021+0000 7f2263fff640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f226c104c30 0x7f226c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.022+0000 7f226affd640 1 -- 172.21.15.77:0/2625649003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f226c104c30 msgr2=0x7f226c198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.022+0000 7f226affd640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f226c104c30 0x7f226c198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.022+0000 7f226affd640 1 -- 172.21.15.77:0/2625649003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f226c107560 msgr2=0x7f226c19f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.022+0000 7f226affd640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f226c107560 0x7f226c19f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.022+0000 7f226affd640 1 -- 172.21.15.77:0/2625649003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f226c19f770 con 0x7f226c102300 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.022+0000 7f2263fff640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f226c104c30 0x7f226c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.022+0000 7f226affd640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f226c102300 0x7f226c197b50 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f225c00ea60 tx=0x7f225c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.022+0000 7f2268ff9640 1 -- 172.21.15.77:0/2625649003 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225c00cde0 con 0x7f226c102300 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.022+0000 7f22713f1640 1 -- 172.21.15.77:0/2625649003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f226c19fa60 con 0x7f226c102300 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.022+0000 7f2268ff9640 1 -- 172.21.15.77:0/2625649003 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f225c0035f0 con 0x7f226c102300 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.022+0000 7f2268ff9640 1 -- 172.21.15.77:0/2625649003 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225c010670 con 0x7f226c102300 2024-03-16T10:26:28.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.022+0000 7f22713f1640 1 -- 172.21.15.77:0/2625649003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f226c19ffd0 con 0x7f226c102300 2024-03-16T10:26:28.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.023+0000 7f22713f1640 1 -- 172.21.15.77:0/2625649003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f226c0fd760 con 0x7f226c102300 2024-03-16T10:26:28.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.025+0000 7f2268ff9640 1 -- 172.21.15.77:0/2625649003 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f225c0037a0 con 0x7f226c102300 2024-03-16T10:26:28.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.025+0000 7f2268ff9640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f22580a2800 0x7f22580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.025+0000 7f2263fff640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f22580a2800 0x7f22580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.025+0000 7f2268ff9640 1 -- 172.21.15.77:0/2625649003 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f225c0cecf0 con 0x7f226c102300 2024-03-16T10:26:28.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.026+0000 7f2263fff640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f22580a2800 0x7f22580a4cc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2254009230 tx=0x7f2254005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:28.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.030+0000 7f2268ff9640 1 -- 172.21.15.77:0/2625649003 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f225c0d3070 con 0x7f226c102300 2024-03-16T10:26:28.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.211+0000 7f22713f1640 1 -- 172.21.15.77:0/2625649003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f226c02d460 con 0x7f226c102300 2024-03-16T10:26:28.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.212+0000 7f2268ff9640 1 -- 172.21.15.77:0/2625649003 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22496 (secure 0 0 0) 0x7f225c09be20 con 0x7f226c102300 2024-03-16T10:26:28.213 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:26:28.214 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:23:44.816264+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:26:28.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.214+0000 7f22713f1640 1 -- 172.21.15.77:0/2625649003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f22580a2800 msgr2=0x7f22580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.215 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.214+0000 7f22713f1640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f22580a2800 0x7f22580a4cc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2254009230 tx=0x7f2254005fd0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.214+0000 7f22713f1640 1 -- 172.21.15.77:0/2625649003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f226c102300 msgr2=0x7f226c197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.214+0000 7f22713f1640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f226c102300 0x7f226c197b50 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f225c00ea60 tx=0x7f225c00ef20 comp rx=0 tx=0).stop 2024-03-16T10:26:28.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.214+0000 7f22713f1640 1 -- 172.21.15.77:0/2625649003 shutdown_connections 2024-03-16T10:26:28.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.214+0000 7f22713f1640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f226c107560 0x7f226c19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.214+0000 7f22713f1640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f22580a2800 0x7f22580a4cc0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.214+0000 7f22713f1640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f226c104c30 0x7f226c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.214+0000 7f22713f1640 1 --2- 172.21.15.77:0/2625649003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f226c102300 0x7f226c197b50 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.214+0000 7f22713f1640 1 -- 172.21.15.77:0/2625649003 >> 172.21.15.77:0/2625649003 conn(0x7f226c0fbc70 msgr2=0x7f226c102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:28.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.214+0000 7f22713f1640 1 -- 172.21.15.77:0/2625649003 shutdown_connections 2024-03-16T10:26:28.216 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.214+0000 7f22713f1640 1 -- 172.21.15.77:0/2625649003 wait complete. 2024-03-16T10:26:28.229 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:10:22.965004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T10:15:00.264039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T10:15:01.536067+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T10:26:28.230 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T10:26:28.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.334+0000 7fb9d24a1640 1 -- 172.21.15.77:0/3156726794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9cc1018d0 msgr2=0x7fb9cc101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.334+0000 7fb9d24a1640 1 --2- 172.21.15.77:0/3156726794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9cc1018d0 0x7fb9cc101d50 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0009900 tx=0x7fb9c002bc30 comp rx=0 tx=0).stop 2024-03-16T10:26:28.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.334+0000 7fb9d24a1640 1 -- 172.21.15.77:0/3156726794 shutdown_connections 2024-03-16T10:26:28.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.334+0000 7fb9d24a1640 1 --2- 172.21.15.77:0/3156726794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9cc102290 0x7fb9cc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.334+0000 7fb9d24a1640 1 --2- 172.21.15.77:0/3156726794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9cc1018d0 0x7fb9cc101d50 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.334+0000 7fb9d24a1640 1 --2- 172.21.15.77:0/3156726794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9cc1006e0 0x7fb9cc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.334+0000 7fb9d24a1640 1 -- 172.21.15.77:0/3156726794 >> 172.21.15.77:0/3156726794 conn(0x7fb9cc0fbc70 msgr2=0x7fb9cc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:28.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.334+0000 7fb9d24a1640 1 -- 172.21.15.77:0/3156726794 shutdown_connections 2024-03-16T10:26:28.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.334+0000 7fb9d24a1640 1 -- 172.21.15.77:0/3156726794 wait complete. 2024-03-16T10:26:28.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9d24a1640 1 Processor -- start 2024-03-16T10:26:28.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9d24a1640 1 -- start start 2024-03-16T10:26:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9d24a1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9cc1006e0 0x7fb9cc196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9d24a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9cc1018d0 0x7fb9cc196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9d24a1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9cc102290 0x7fb9cc19e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9d24a1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc10c3b0 con 0x7fb9cc1018d0 2024-03-16T10:26:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9d24a1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc10c230 con 0x7fb9cc1006e0 2024-03-16T10:26:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9d24a1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc10c530 con 0x7fb9cc102290 2024-03-16T10:26:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9cb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9cc1018d0 0x7fb9cc196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9cb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9cc1018d0 0x7fb9cc196fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47096/0 (socket says 172.21.15.77:47096) 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9cb7fe640 1 -- 172.21.15.77:0/2040980783 learned_addr learned my addr 172.21.15.77:0/2040980783 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9cbfff640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9cc1006e0 0x7fb9cc196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9d0a3e640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9cc102290 0x7fb9cc19e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9cb7fe640 1 -- 172.21.15.77:0/2040980783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9cc102290 msgr2=0x7fb9cc19e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9cb7fe640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9cc102290 0x7fb9cc19e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9cb7fe640 1 -- 172.21.15.77:0/2040980783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9cc1006e0 msgr2=0x7fb9cc196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9cb7fe640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9cc1006e0 0x7fb9cc196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9cb7fe640 1 -- 172.21.15.77:0/2040980783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9cc19e690 con 0x7fb9cc1018d0 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.335+0000 7fb9cb7fe640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9cc1018d0 0x7fb9cc196fb0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0005ab0 tx=0x7fb9c002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.336+0000 7fb9c97fa640 1 -- 172.21.15.77:0/2040980783 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0004450 con 0x7fb9cc1018d0 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.336+0000 7fb9d24a1640 1 -- 172.21.15.77:0/2040980783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9cc068940 con 0x7fb9cc1018d0 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.336+0000 7fb9c97fa640 1 -- 172.21.15.77:0/2040980783 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9c00054e0 con 0x7fb9cc1018d0 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.336+0000 7fb9c97fa640 1 -- 172.21.15.77:0/2040980783 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c00476e0 con 0x7fb9cc1018d0 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.336+0000 7fb9d24a1640 1 -- 172.21.15.77:0/2040980783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9cc068e90 con 0x7fb9cc1018d0 2024-03-16T10:26:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.337+0000 7fb9d24a1640 1 -- 172.21.15.77:0/2040980783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9cc0685d0 con 0x7fb9cc1018d0 2024-03-16T10:26:28.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.338+0000 7fb9c97fa640 1 -- 172.21.15.77:0/2040980783 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fb9c00035f0 con 0x7fb9cc1018d0 2024-03-16T10:26:28.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.338+0000 7fb9c97fa640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb9b40a2800 0x7fb9b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.338+0000 7fb9cbfff640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb9b40a2800 0x7fb9b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.338+0000 7fb9c97fa640 1 -- 172.21.15.77:0/2040980783 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fb9c00f2a10 con 0x7fb9cc1018d0 2024-03-16T10:26:28.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.339+0000 7fb9cbfff640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb9b40a2800 0x7fb9b40a4cc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc101730 tx=0x7fb9bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:28.343 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.342+0000 7fb9c97fa640 1 -- 172.21.15.77:0/2040980783 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb9c0003e20 con 0x7fb9cc1018d0 2024-03-16T10:26:28.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:28 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2625649003' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:26:28.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.523+0000 7fb9d24a1640 1 -- 172.21.15.77:0/2040980783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb9cc1076f0 con 0x7fb9cc1018d0 2024-03-16T10:26:28.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.524+0000 7fb9c97fa640 1 -- 172.21.15.77:0/2040980783 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fb9c00bfb40 con 0x7fb9cc1018d0 2024-03-16T10:26:28.525 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T10:26:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.525+0000 7fb9d24a1640 1 -- 172.21.15.77:0/2040980783 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb9b40a2800 msgr2=0x7fb9b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.525+0000 7fb9d24a1640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb9b40a2800 0x7fb9b40a4cc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc101730 tx=0x7fb9bc005f90 comp rx=0 tx=0).stop 2024-03-16T10:26:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.525+0000 7fb9d24a1640 1 -- 172.21.15.77:0/2040980783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9cc1018d0 msgr2=0x7fb9cc196fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.525+0000 7fb9d24a1640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9cc1018d0 0x7fb9cc196fb0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0005ab0 tx=0x7fb9c002c6c0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.526+0000 7fb9d24a1640 1 -- 172.21.15.77:0/2040980783 shutdown_connections 2024-03-16T10:26:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.526+0000 7fb9d24a1640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb9b40a2800 0x7fb9b40a4cc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.526+0000 7fb9d24a1640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9cc102290 0x7fb9cc19e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.526+0000 7fb9d24a1640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9cc1018d0 0x7fb9cc196fb0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.526+0000 7fb9d24a1640 1 --2- 172.21.15.77:0/2040980783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9cc1006e0 0x7fb9cc196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.526+0000 7fb9d24a1640 1 -- 172.21.15.77:0/2040980783 >> 172.21.15.77:0/2040980783 conn(0x7fb9cc0fbc70 msgr2=0x7fb9cc104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.526+0000 7fb9d24a1640 1 -- 172.21.15.77:0/2040980783 shutdown_connections 2024-03-16T10:26:28.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.526+0000 7fb9d24a1640 1 -- 172.21.15.77:0/2040980783 wait complete. 2024-03-16T10:26:28.538 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T10:26:28.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.642+0000 7eff0d3ff640 1 -- 172.21.15.77:0/2007400568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff081021a0 msgr2=0x7eff08108a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.642+0000 7eff0d3ff640 1 --2- 172.21.15.77:0/2007400568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff081021a0 0x7eff08108a30 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7efef8009900 tx=0x7efef802bc30 comp rx=0 tx=0).stop 2024-03-16T10:26:28.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.642+0000 7eff0d3ff640 1 -- 172.21.15.77:0/2007400568 shutdown_connections 2024-03-16T10:26:28.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.642+0000 7eff0d3ff640 1 --2- 172.21.15.77:0/2007400568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff081021a0 0x7eff08108a30 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.642+0000 7eff0d3ff640 1 --2- 172.21.15.77:0/2007400568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff081017e0 0x7eff08101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.642+0000 7eff0d3ff640 1 --2- 172.21.15.77:0/2007400568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff081005f0 0x7eff08100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.642+0000 7eff0d3ff640 1 -- 172.21.15.77:0/2007400568 >> 172.21.15.77:0/2007400568 conn(0x7eff080fbb80 msgr2=0x7eff080fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:28.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.642+0000 7eff0d3ff640 1 -- 172.21.15.77:0/2007400568 shutdown_connections 2024-03-16T10:26:28.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.643+0000 7eff0d3ff640 1 -- 172.21.15.77:0/2007400568 wait complete. 2024-03-16T10:26:28.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.643+0000 7eff0d3ff640 1 Processor -- start 2024-03-16T10:26:28.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.643+0000 7eff0d3ff640 1 -- start start 2024-03-16T10:26:28.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.643+0000 7eff0d3ff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff081005f0 0x7eff081979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.643+0000 7eff0d3ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff081017e0 0x7eff08197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.643+0000 7eff0d3ff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff081021a0 0x7eff0819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.643+0000 7eff0d3ff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff0810c270 con 0x7eff081017e0 2024-03-16T10:26:28.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.643+0000 7eff0d3ff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7eff0810c0f0 con 0x7eff081021a0 2024-03-16T10:26:28.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.643+0000 7eff0d3ff640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff0810c3f0 con 0x7eff081005f0 2024-03-16T10:26:28.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.643+0000 7eff067fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff081017e0 0x7eff08197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7eff067fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff081017e0 0x7eff08197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47104/0 (socket says 172.21.15.77:47104) 2024-03-16T10:26:28.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7eff067fc640 1 -- 172.21.15.77:0/1436442045 learned_addr learned my addr 172.21.15.77:0/1436442045 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:28.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7eff077fe640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff081021a0 0x7eff0819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7eff06ffd640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff081005f0 0x7eff081979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7eff067fc640 1 -- 172.21.15.77:0/1436442045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff081005f0 msgr2=0x7eff081979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7eff067fc640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff081005f0 0x7eff081979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7eff067fc640 1 -- 172.21.15.77:0/1436442045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff081021a0 msgr2=0x7eff0819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7eff067fc640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff081021a0 0x7eff0819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7eff067fc640 1 -- 172.21.15.77:0/1436442045 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0819f590 con 0x7eff081017e0 2024-03-16T10:26:28.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7eff067fc640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff081017e0 0x7eff08197f00 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7efefc00ebc0 tx=0x7efefc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:28.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7efeeffff640 1 -- 172.21.15.77:0/1436442045 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc010630 con 0x7eff081017e0 2024-03-16T10:26:28.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7efeeffff640 1 -- 172.21.15.77:0/1436442045 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efefc010cb0 con 0x7eff081017e0 2024-03-16T10:26:28.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7eff0d3ff640 1 -- 172.21.15.77:0/1436442045 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff0819f880 con 0x7eff081017e0 2024-03-16T10:26:28.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7efeeffff640 1 -- 172.21.15.77:0/1436442045 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc018bd0 con 0x7eff081017e0 2024-03-16T10:26:28.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.644+0000 7eff0d3ff640 1 -- 172.21.15.77:0/1436442045 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff0819fdd0 con 0x7eff081017e0 2024-03-16T10:26:28.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.645+0000 7eff0d3ff640 1 -- 172.21.15.77:0/1436442045 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff081a0030 con 0x7eff081017e0 2024-03-16T10:26:28.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.646+0000 7efeeffff640 1 -- 172.21.15.77:0/1436442045 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7efefc0107d0 con 0x7eff081017e0 2024-03-16T10:26:28.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.647+0000 7efeeffff640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efef00a2800 0x7efef00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.647+0000 7eff06ffd640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efef00a2800 0x7efef00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.647+0000 7efeeffff640 1 -- 172.21.15.77:0/1436442045 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7efefc0cefb0 con 0x7eff081017e0 2024-03-16T10:26:28.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.647+0000 7eff06ffd640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efef00a2800 0x7efef00a4cc0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7eff08101640 tx=0x7eff00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:28.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.651+0000 7efeeffff640 1 -- 172.21.15.77:0/1436442045 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efefc09ba90 con 0x7eff081017e0 2024-03-16T10:26:28.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:28 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2625649003' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:26:28.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:28 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2625649003' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:26:28.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.831+0000 7eff0d3ff640 1 -- 172.21.15.77:0/1436442045 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7eff081a0200 con 0x7eff081017e0 2024-03-16T10:26:28.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.832+0000 7efeeffff640 1 -- 172.21.15.77:0/1436442045 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7efefc09fb30 con 0x7eff081017e0 2024-03-16T10:26:28.833 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T10:26:28.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.833+0000 7eff0d3ff640 1 -- 172.21.15.77:0/1436442045 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efef00a2800 msgr2=0x7efef00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.833+0000 7eff0d3ff640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efef00a2800 0x7efef00a4cc0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7eff08101640 tx=0x7eff00005f90 comp rx=0 tx=0).stop 2024-03-16T10:26:28.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.833+0000 7eff0d3ff640 1 -- 172.21.15.77:0/1436442045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff081017e0 msgr2=0x7eff08197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.833+0000 7eff0d3ff640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff081017e0 0x7eff08197f00 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7efefc00ebc0 tx=0x7efefc00c6a0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.834+0000 7eff0d3ff640 1 -- 172.21.15.77:0/1436442045 shutdown_connections 2024-03-16T10:26:28.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.834+0000 7eff0d3ff640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff081021a0 0x7eff0819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.834+0000 7eff0d3ff640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff081017e0 0x7eff08197f00 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.834+0000 7eff0d3ff640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efef00a2800 0x7efef00a4cc0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.834+0000 7eff0d3ff640 1 --2- 172.21.15.77:0/1436442045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff081005f0 0x7eff081979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.834+0000 7eff0d3ff640 1 -- 172.21.15.77:0/1436442045 >> 172.21.15.77:0/1436442045 conn(0x7eff080fbb80 msgr2=0x7eff080fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:28.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.834+0000 7eff0d3ff640 1 -- 172.21.15.77:0/1436442045 shutdown_connections 2024-03-16T10:26:28.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.834+0000 7eff0d3ff640 1 -- 172.21.15.77:0/1436442045 wait complete. 2024-03-16T10:26:28.846 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T10:26:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.952+0000 7efc00cea640 1 -- 172.21.15.77:0/724133597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efbfc102a00 msgr2=0x7efbfc102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.952+0000 7efc00cea640 1 --2- 172.21.15.77:0/724133597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efbfc102a00 0x7efbfc102e80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7efbe4009900 tx=0x7efbe402bc30 comp rx=0 tx=0).stop 2024-03-16T10:26:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.952+0000 7efc00cea640 1 -- 172.21.15.77:0/724133597 shutdown_connections 2024-03-16T10:26:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.952+0000 7efc00cea640 1 --2- 172.21.15.77:0/724133597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efbfc1033c0 0x7efbfc109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.952+0000 7efc00cea640 1 --2- 172.21.15.77:0/724133597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efbfc102a00 0x7efbfc102e80 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.952+0000 7efc00cea640 1 --2- 172.21.15.77:0/724133597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbfc101810 0x7efbfc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.952+0000 7efc00cea640 1 -- 172.21.15.77:0/724133597 >> 172.21.15.77:0/724133597 conn(0x7efbfc0fcd80 msgr2=0x7efbfc0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.952+0000 7efc00cea640 1 -- 172.21.15.77:0/724133597 shutdown_connections 2024-03-16T10:26:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.952+0000 7efc00cea640 1 -- 172.21.15.77:0/724133597 wait complete. 2024-03-16T10:26:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efc00cea640 1 Processor -- start 2024-03-16T10:26:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efc00cea640 1 -- start start 2024-03-16T10:26:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efc00cea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbfc101810 0x7efbfc075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efc00cea640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efbfc102a00 0x7efbfc073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efc00cea640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efbfc1033c0 0x7efbfc074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efc00cea640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efbfc10d490 con 0x7efbfc1033c0 2024-03-16T10:26:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efc00cea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7efbfc10d310 con 0x7efbfc101810 2024-03-16T10:26:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efc00cea640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efbfc10d610 con 0x7efbfc102a00 2024-03-16T10:26:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efbfad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efbfc1033c0 0x7efbfc074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efbf3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efbfc102a00 0x7efbfc073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efbfad9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efbfc1033c0 0x7efbfc074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47132/0 (socket says 172.21.15.77:47132) 2024-03-16T10:26:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efbf3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efbfc102a00 0x7efbfc073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:37076/0 (socket says 172.21.15.77:37076) 2024-03-16T10:26:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efbfa59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbfc101810 0x7efbfc075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efbfad9d640 1 -- 172.21.15.77:0/681568637 learned_addr learned my addr 172.21.15.77:0/681568637 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efbfad9d640 1 -- 172.21.15.77:0/681568637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efbfc102a00 msgr2=0x7efbfc073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efbfad9d640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efbfc102a00 0x7efbfc073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efbfad9d640 1 -- 172.21.15.77:0/681568637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbfc101810 msgr2=0x7efbfc075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efbfad9d640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbfc101810 0x7efbfc075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efbfad9d640 1 -- 172.21.15.77:0/681568637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbfc0749a0 con 0x7efbfc1033c0 2024-03-16T10:26:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.953+0000 7efbf3fff640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efbfc102a00 0x7efbfc073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:26:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.954+0000 7efbfad9d640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efbfc1033c0 0x7efbfc074110 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7efbf400eb10 tx=0x7efbf400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.954+0000 7efbf37fe640 1 -- 172.21.15.77:0/681568637 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf4004b20 con 0x7efbfc1033c0 2024-03-16T10:26:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.954+0000 7efc00cea640 1 -- 172.21.15.77:0/681568637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efbfc1a1c20 con 0x7efbfc1033c0 2024-03-16T10:26:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.954+0000 7efbf37fe640 1 -- 172.21.15.77:0/681568637 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efbf4004cc0 con 0x7efbfc1033c0 2024-03-16T10:26:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.954+0000 7efbf37fe640 1 -- 172.21.15.77:0/681568637 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf4010760 con 0x7efbfc1033c0 2024-03-16T10:26:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.954+0000 7efc00cea640 1 -- 172.21.15.77:0/681568637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efbfc1a2140 con 0x7efbfc1033c0 2024-03-16T10:26:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.955+0000 7efc00cea640 1 -- 172.21.15.77:0/681568637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efbfc0685d0 con 0x7efbfc1033c0 2024-03-16T10:26:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.956+0000 7efbf37fe640 1 -- 172.21.15.77:0/681568637 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7efbf4005020 con 0x7efbfc1033c0 2024-03-16T10:26:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.956+0000 7efbf37fe640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efbe80a2800 0x7efbe80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.956+0000 7efbfa59c640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efbe80a2800 0x7efbe80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.956+0000 7efbf37fe640 1 -- 172.21.15.77:0/681568637 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7efbf40cf0d0 con 0x7efbfc1033c0 2024-03-16T10:26:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.957+0000 7efbfa59c640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efbe80a2800 0x7efbe80a4cc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efbfc102860 tx=0x7efbec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:28.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:28.965+0000 7efbf37fe640 1 -- 172.21.15.77:0/681568637 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efbf409c200 con 0x7efbfc1033c0 2024-03-16T10:26:29.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.159+0000 7efc00cea640 1 -- 172.21.15.77:0/681568637 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efbfc02d1b0 con 0x7efbfc1033c0 2024-03-16T10:26:29.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.159+0000 7efbf37fe640 1 -- 172.21.15.77:0/681568637 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7efbf409bb00 con 0x7efbfc1033c0 2024-03-16T10:26:29.161 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T10:26:29.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.161+0000 7efc00cea640 1 -- 172.21.15.77:0/681568637 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efbe80a2800 msgr2=0x7efbe80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:29.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.161+0000 7efc00cea640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efbe80a2800 0x7efbe80a4cc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efbfc102860 tx=0x7efbec005f90 comp rx=0 tx=0).stop 2024-03-16T10:26:29.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.161+0000 7efc00cea640 1 -- 172.21.15.77:0/681568637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efbfc1033c0 msgr2=0x7efbfc074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:29.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.161+0000 7efc00cea640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efbfc1033c0 0x7efbfc074110 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7efbf400eb10 tx=0x7efbf400c2d0 comp rx=0 tx=0).stop 2024-03-16T10:26:29.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.161+0000 7efc00cea640 1 -- 172.21.15.77:0/681568637 shutdown_connections 2024-03-16T10:26:29.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.161+0000 7efc00cea640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efbfc1033c0 0x7efbfc074110 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:29.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.161+0000 7efc00cea640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7efbe80a2800 0x7efbe80a4cc0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:29.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.161+0000 7efc00cea640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efbfc102a00 0x7efbfc073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:29.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.161+0000 7efc00cea640 1 --2- 172.21.15.77:0/681568637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efbfc101810 0x7efbfc075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:29.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.161+0000 7efc00cea640 1 -- 172.21.15.77:0/681568637 >> 172.21.15.77:0/681568637 conn(0x7efbfc0fcd80 msgr2=0x7efbfc105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:29.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.162+0000 7efc00cea640 1 -- 172.21.15.77:0/681568637 shutdown_connections 2024-03-16T10:26:29.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.162+0000 7efc00cea640 1 -- 172.21.15.77:0/681568637 wait complete. 2024-03-16T10:26:29.174 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-16T10:26:29.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 -- 172.21.15.77:0/278918452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f857c104bc0 msgr2=0x7f857c106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:29.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 --2- 172.21.15.77:0/278918452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f857c104bc0 0x7f857c106fb0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f856c009900 tx=0x7f856c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:26:29.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 -- 172.21.15.77:0/278918452 shutdown_connections 2024-03-16T10:26:29.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 --2- 172.21.15.77:0/278918452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f857c1074f0 0x7f857c109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:29.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 --2- 172.21.15.77:0/278918452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f857c104bc0 0x7f857c106fb0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:29.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 --2- 172.21.15.77:0/278918452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f857c102290 0x7f857c104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:29.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 -- 172.21.15.77:0/278918452 >> 172.21.15.77:0/278918452 conn(0x7f857c0fbcc0 msgr2=0x7f857c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:29.297 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 -- 172.21.15.77:0/278918452 shutdown_connections 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 -- 172.21.15.77:0/278918452 wait complete. 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 Processor -- start 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 -- start start 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f857c102290 0x7f857c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f857c104bc0 0x7f857c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f857c1074f0 0x7f857c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f857c10d4a0 con 0x7f857c102290 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f857c10d320 con 0x7f857c1074f0 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.290+0000 7f8584159640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f857c10d620 con 0x7f857c104bc0 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8581ef5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f857c102290 0x7f857c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8581ef5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f857c102290 0x7f857c197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47150/0 (socket says 172.21.15.77:47150) 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8581ef5640 1 -- 172.21.15.77:0/2649021351 learned_addr learned my addr 172.21.15.77:0/2649021351 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:26:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f85816f4640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f857c104bc0 0x7f857c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f85826f6640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f857c1074f0 0x7f857c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8581ef5640 1 -- 172.21.15.77:0/2649021351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f857c104bc0 msgr2=0x7f857c1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8581ef5640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f857c104bc0 0x7f857c1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8581ef5640 1 -- 172.21.15.77:0/2649021351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f857c1074f0 msgr2=0x7f857c19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8581ef5640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f857c1074f0 0x7f857c19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8581ef5640 1 -- 172.21.15.77:0/2649021351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857c19f790 con 0x7f857c102290 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8581ef5640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f857c102290 0x7f857c197b70 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f857400e9f0 tx=0x7f857400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8572ffd640 1 -- 172.21.15.77:0/2649021351 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857400cd80 con 0x7f857c102290 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8572ffd640 1 -- 172.21.15.77:0/2649021351 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8574005500 con 0x7f857c102290 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8584159640 1 -- 172.21.15.77:0/2649021351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f857c19fa80 con 0x7f857c102290 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.291+0000 7f8572ffd640 1 -- 172.21.15.77:0/2649021351 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8574010690 con 0x7f857c102290 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.292+0000 7f8584159640 1 -- 172.21.15.77:0/2649021351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f857c19ff50 con 0x7f857c102290 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.293+0000 7f8584159640 1 -- 172.21.15.77:0/2649021351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f857c0fd7b0 con 0x7f857c102290 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.293+0000 7f8572ffd640 1 -- 172.21.15.77:0/2649021351 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f85740046c0 con 0x7f857c102290 2024-03-16T10:26:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.294+0000 7f8572ffd640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f85680a2800 0x7f85680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:26:29.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.294+0000 7f85816f4640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f85680a2800 0x7f85680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:26:29.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.294+0000 7f8572ffd640 1 -- 172.21.15.77:0/2649021351 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f85740cee20 con 0x7f857c102290 2024-03-16T10:26:29.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.294+0000 7f85816f4640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f85680a2800 0x7f85680a4cc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f856c000c00 tx=0x7f856c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:26:29.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.299+0000 7f8572ffd640 1 -- 172.21.15.77:0/2649021351 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f857409bf50 con 0x7f857c102290 2024-03-16T10:26:29.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:29 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2040980783' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:26:29.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:29 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1436442045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:26:29.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:29 smithi077 ceph-mon[32400]: pgmap v633: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 op/s 2024-03-16T10:26:29.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:29 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/681568637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:26:29.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.529+0000 7f8584159640 1 -- 172.21.15.77:0/2649021351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f857c02d330 con 0x7f857c102290 2024-03-16T10:26:29.539 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.538+0000 7f8572ffd640 1 -- 172.21.15.77:0/2649021351 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+739831 (secure 0 0 0) 0x7f857409b850 con 0x7f857c102290 2024-03-16T10:26:29.548 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: "epoch": 20, 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: "active_gid": 14152, 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: "active_name": "x", 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: "active_addrs": { 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: "addrvec": [ 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v2", 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "172.21.15.77:6800", 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 1427985582 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.549 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v1", 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "172.21.15.77:6801", 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 1427985582 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: "active_addr": "172.21.15.77:6801/1427985582", 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: "active_change": "2024-03-16T10:07:58.634953+0000", 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: "active_mgr_features": 4540701547738038271, 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: "available": true, 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: "standbys": [ 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: "gid": 24106, 2024-03-16T10:26:29.550 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "z", 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr_features": 4540701547738038271, 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "available_modules": [ 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "alerts", 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.551 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.554 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.555 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.556 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_destination": { 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_destination", 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Email address to send alerts to", 2024-03-16T10:26:29.557 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_from_name": { 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_from_name", 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "Ceph", 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Email From: name", 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.558 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_host": { 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_host", 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP server", 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.559 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_password": { 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_password", 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Password to authenticate with", 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.560 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_port": { 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_port", 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "465", 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP port", 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.561 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_sender": { 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_sender", 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP envelope sender", 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.562 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_ssl": { 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_ssl", 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.563 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_user": { 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_user", 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "User to authenticate as", 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.564 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "balancer", 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "active": { 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "active", 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T10:26:29.566 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.566 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.566 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.566 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.566 INFO:teuthology.orchestra.run.smithi077.stdout: "begin_time": { 2024-03-16T10:26:29.566 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "begin_time", 2024-03-16T10:26:29.566 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.566 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.566 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.566 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0000", 2024-03-16T10:26:29.566 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.566 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "begin_weekday": { 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "begin_weekday", 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0", 2024-03-16T10:26:29.567 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "6", 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_max_iterations": { 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_max_iterations", 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "25", 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "250", 2024-03-16T10:26:29.568 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_metrics": { 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_metrics", 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.569 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_step": { 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_step", 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0.5", 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0.001", 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "0.999", 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.570 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "aggressiveness of optimization", 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "end_time": { 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "end_time", 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "2359", 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T10:26:29.571 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "end_weekday": { 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "end_weekday", 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0", 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "6", 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T10:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.573 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.574 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.575 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.576 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.577 INFO:teuthology.orchestra.run.smithi077.stdout: "min_score": { 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "min_score", 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.578 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": { 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mode", 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "upmap", 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "crush-compat", 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "none", 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "read", 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap", 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap-read" 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Balancer mode", 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "pool_ids": { 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pool_ids", 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.580 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.581 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap_max_deviation": { 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "upmap_max_deviation", 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T10:26:29.582 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap_max_optimizations": { 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "upmap_max_optimizations", 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.583 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephadm", 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_down_multiplier": { 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_down_multiplier", 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3.0", 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.584 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.585 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.585 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T10:26:29.585 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.585 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.585 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.585 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.585 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_refresh_rate": { 2024-03-16T10:26:29.585 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_refresh_rate", 2024-03-16T10:26:29.585 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.585 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.585 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "20", 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_starting_port": { 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_starting_port", 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.586 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.587 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "4721", 2024-03-16T10:26:29.587 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.587 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.587 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.587 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T10:26:29.587 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.587 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.587 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.587 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.587 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_ptrace": { 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_ptrace", 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "autotune_interval": { 2024-03-16T10:26:29.588 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "autotune_interval", 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "autotune_memory_target_ratio": { 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T10:26:29.589 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0.7", 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm_log_destination": { 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephadm_log_destination", 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.590 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.591 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.591 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.591 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.591 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.591 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.591 INFO:teuthology.orchestra.run.smithi077.stdout: "file", 2024-03-16T10:26:29.591 INFO:teuthology.orchestra.run.smithi077.stdout: "file,syslog", 2024-03-16T10:26:29.591 INFO:teuthology.orchestra.run.smithi077.stdout: "syslog" 2024-03-16T10:26:29.591 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.591 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T10:26:29.591 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.591 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "cgroups_split": { 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cgroups_split", 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "config_checks_enabled": { 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_checks_enabled", 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.593 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.594 INFO:teuthology.orchestra.run.smithi077.stdout: "config_dashboard": { 2024-03-16T10:26:29.594 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_dashboard", 2024-03-16T10:26:29.594 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.594 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.594 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.594 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.594 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.594 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.594 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.594 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T10:26:29.594 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.594 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_alertmanager": { 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_alertmanager", 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.595 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_base": { 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_base", 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Container image name, without the tag", 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.596 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_elasticsearch": { 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_elasticsearch", 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "elasticsearch container image", 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_grafana": { 2024-03-16T10:26:29.597 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_grafana", 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_haproxy": { 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_haproxy", 2024-03-16T10:26:29.598 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "HAproxy container image", 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_agent": { 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_agent", 2024-03-16T10:26:29.599 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger agent container image", 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_collector": { 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_collector", 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.600 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger collector container image", 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_query": { 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_query", 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.601 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger query container image", 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_keepalived": { 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_keepalived", 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.602 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Keepalived container image", 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_loki": { 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_loki", 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T10:26:29.603 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Loki container image", 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_node_exporter": { 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_node_exporter", 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_nvmeof": { 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_nvmeof", 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.605 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Nvme-of container image", 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_prometheus": { 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_prometheus", 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.606 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_promtail": { 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_promtail", 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Promtail container image", 2024-03-16T10:26:29.607 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_snmp_gateway": { 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_snmp_gateway", 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SNMP Gateway container image", 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "container_init": { 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_init", 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.609 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: "daemon_cache_timeout": { 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "daemon_cache_timeout", 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.610 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "default_cephadm_command_timeout": { 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "900", 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.611 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "default_registry": { 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default_registry", 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io", 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.612 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "device_cache_timeout": { 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_cache_timeout", 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1800", 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache device inventory", 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.613 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "device_enhanced_scan": { 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_enhanced_scan", 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.614 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "facts_cache_timeout": { 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "facts_cache_timeout", 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache host facts data", 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.615 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "host_check_interval": { 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "host_check_interval", 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to perform a host check", 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.616 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "hw_monitoring": { 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "hw_monitoring", 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.617 INFO:teuthology.orchestra.run.smithi077.stdout: "inventory_list_all": { 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "inventory_list_all", 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.618 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.619 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "log_refresh_metadata": { 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_refresh_metadata", 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.620 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T10:26:29.621 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.622 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.623 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.624 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.625 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "*", 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: "max_count_per_host": { 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_count_per_host", 2024-03-16T10:26:29.626 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "max number of daemons per service per host", 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "max_osd_draining_count": { 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_osd_draining_count", 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.627 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "migration_current": { 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "migration_current", 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.628 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "internal - do not modify", 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": { 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mode", 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "root", 2024-03-16T10:26:29.629 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.630 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.630 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.630 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm-package", 2024-03-16T10:26:29.630 INFO:teuthology.orchestra.run.smithi077.stdout: "root" 2024-03-16T10:26:29.630 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.630 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T10:26:29.630 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.630 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.630 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.630 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.630 INFO:teuthology.orchestra.run.smithi077.stdout: "oob_default_addr": { 2024-03-16T10:26:29.630 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "oob_default_addr", 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "169.254.1.1", 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "prometheus_alerts_path": { 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prometheus_alerts_path", 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_insecure": { 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_insecure", 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.632 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_password": { 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_password", 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.633 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T10:26:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_url": { 2024-03-16T10:26:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_url", 2024-03-16T10:26:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_username": { 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_username", 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "secure_monitoring_stack": { 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "secure_monitoring_stack", 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.636 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "service_discovery_port": { 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "service_discovery_port", 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8765", 2024-03-16T10:26:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "cephadm service discovery port", 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "ssh_config_file": { 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssh_config_file", 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "use_agent": { 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "use_agent", 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.639 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "use_repo_digest": { 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "use_repo_digest", 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.640 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_failed_host_check": { 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_failed_host_check", 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.641 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_stray_daemons": { 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_stray_daemons", 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.642 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_stray_hosts": { 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_stray_hosts", 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.643 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crash", 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.644 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.645 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.646 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.647 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.648 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "retain_interval": { 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "retain_interval", 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "31536000", 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.649 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_recent_interval": { 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_recent_interval", 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1209600", 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.650 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "dashboard", 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.651 INFO:teuthology.orchestra.run.smithi077.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T10:26:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T10:26:29.653 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "AUDIT_API_ENABLED": { 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.654 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.655 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.656 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.657 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.658 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.659 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.660 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.661 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.662 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.664 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.666 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_URL": { 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_URL", 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.667 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.668 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.669 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.670 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T10:26:29.671 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T10:26:29.672 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "{}", 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T10:26:29.673 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T10:26:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T10:26:29.675 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.676 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.677 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.678 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.679 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.680 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.681 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.683 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.684 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8", 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.685 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "45", 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.686 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.687 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.688 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.689 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.690 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.691 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T10:26:29.692 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T10:26:29.693 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "cross_origin_url": { 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cross_origin_url", 2024-03-16T10:26:29.694 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "crt_file": { 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crt_file", 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.695 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "debug": { 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "debug", 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.696 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable/disable debug options", 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "jwt_token_ttl": { 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "jwt_token_ttl", 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "28800", 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "key_file": { 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "key_file", 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.698 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.699 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.700 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.701 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.702 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.703 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "motd": { 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "motd", 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.704 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "The message of the day", 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect_resolve_ip_addr": { 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.705 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "::", 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.706 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8080", 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl": { 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl", 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.708 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl_server_port": { 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl_server_port", 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8443", 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.709 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_behaviour": { 2024-03-16T10:26:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_behaviour", 2024-03-16T10:26:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "redirect", 2024-03-16T10:26:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.710 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect" 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_error_status_code": { 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_error_status_code", 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "500", 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "400", 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "599", 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.711 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "url_prefix": { 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "url_prefix", 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.712 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "devicehealth", 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "enable_monitoring": { 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enable_monitoring", 2024-03-16T10:26:29.713 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "monitor device health metrics", 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.714 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.715 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.716 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.717 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.718 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "mark_out_threshold": { 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mark_out_threshold", 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "2419200", 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.719 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "pool_name": { 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pool_name", 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "device_health_metrics", 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.720 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "retention_period": { 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "retention_period", 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15552000", 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.721 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long to retain device health metrics", 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "scrape_frequency": { 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "scrape_frequency", 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "86400", 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.722 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "self_heal": { 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "self_heal", 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.723 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.724 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_threshold": { 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_threshold", 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "7257600", 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.725 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "diskprediction_local", 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.726 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.727 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.728 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.729 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.730 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "predict_interval": { 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "predict_interval", 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "86400", 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.731 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "predictor_model": { 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "predictor_model", 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "prophetstor", 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.732 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.733 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "influx", 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": false, 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "influxdb python module not found", 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "batch_size": { 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "batch_size", 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.734 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5000", 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "database": { 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "database", 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "ceph", 2024-03-16T10:26:29.735 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": { 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "hostname", 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.736 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB server hostname", 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "30", 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "5", 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T10:26:29.737 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.738 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.739 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.740 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.741 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.742 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "password": { 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "password", 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "password of InfluxDB server user", 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "port": { 2024-03-16T10:26:29.743 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "port", 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8086", 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB server port", 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl": { 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl", 2024-03-16T10:26:29.744 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "false", 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "threads": { 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "threads", 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.745 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "32", 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "username": { 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "username", 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.746 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "username of InfluxDB server user", 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "verify_ssl": { 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "verify_ssl", 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.747 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "true", 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "insights", 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.749 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.750 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.751 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.752 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "iostat", 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.754 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.755 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.756 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.757 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.758 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "k8sevents", 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_event_retention_days": { 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ceph_event_retention_days", 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "7", 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T10:26:29.759 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "config_check_secs": { 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_check_secs", 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "10", 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T10:26:29.760 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.761 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.763 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.764 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.765 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "localpool", 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "failure_domain": { 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "failure_domain", 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "host", 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.766 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "failure domain for any created local pool", 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.767 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.769 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.770 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.771 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "min_size": { 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "min_size", 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default min_size for any created local pool", 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.772 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "num_rep": { 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "num_rep", 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3", 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default replica count for any created local pool", 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.773 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_num": { 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pg_num", 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "128", 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.774 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "prefix": { 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prefix", 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "name prefix for any created local pool", 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "subtree": { 2024-03-16T10:26:29.775 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "subtree", 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "rack", 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.776 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mds_autoscaler", 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.777 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.778 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.779 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.780 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.781 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mirroring", 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.782 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.783 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.784 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.785 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.786 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "nfs", 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.787 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.788 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.789 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.790 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.791 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "orchestrator", 2024-03-16T10:26:29.792 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "fail_fs": { 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "fail_fs", 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.793 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.794 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.795 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.796 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.797 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator": { 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "orchestrator", 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm", 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "rook", 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "test_orchestrator" 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Orchestrator backend", 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.799 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "osd_perf_query", 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.800 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.801 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.802 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.803 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.804 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "osd_support", 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.805 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.806 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.807 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.808 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.809 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.810 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pg_autoscaler", 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.811 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.813 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.814 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.815 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.816 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "threshold": { 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "threshold", 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3.0", 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1.0", 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "scaling threshold", 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.817 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "progress", 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_pg_recovery_event": { 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_pg_recovery_event", 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.818 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": { 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enabled", 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.819 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.820 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.821 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.823 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.824 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "max_completed_events": { 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_completed_events", 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "50", 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "number of past completed events to remember", 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.825 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long the module is going to sleep", 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.826 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prometheus", 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: "cache": { 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cache", 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.827 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "exclude_perf_counters": { 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "exclude_perf_counters", 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.829 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.830 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.831 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.832 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.833 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_stats_pools": { 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_stats_pools", 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.834 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "300", 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.835 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "scrape_interval": { 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "scrape_interval", 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15.0", 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.836 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "::", 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.837 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "9283", 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.838 INFO:teuthology.orchestra.run.smithi077.stdout: "stale_cache_strategy": { 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "stale_cache_strategy", 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "log", 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.839 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_behaviour": { 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_behaviour", 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "default", 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "default", 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "error" 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.840 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_error_status_code": { 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_error_status_code", 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "500", 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "400", 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "599", 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.841 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_support", 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.842 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.843 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.844 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.845 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.846 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "max_concurrent_snap_create": { 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_concurrent_snap_create", 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.847 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "mirror_snapshot_schedule": { 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.848 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "trash_purge_schedule": { 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "trash_purge_schedule", 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.849 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "restful", 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: "enable_auth": { 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enable_auth", 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.850 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "key_file": { 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "key_file", 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.851 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.852 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.852 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.852 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.852 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.852 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.852 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.852 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.852 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.852 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.852 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.852 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.852 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.853 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.854 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.855 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.856 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.857 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.858 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rgw", 2024-03-16T10:26:29.859 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.860 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.861 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.862 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rook", 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.869 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "storage_class": { 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "storage_class", 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "local", 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.870 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "selftest", 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.871 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.872 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.873 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.874 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.875 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "roption1": { 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "roption1", 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.876 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "roption2": { 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "roption2", 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "xyz", 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.877 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption1": { 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption1", 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.878 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption2": { 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption2", 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.879 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption3": { 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption3", 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.880 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption4": { 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption4", 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.881 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption5": { 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption5", 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.882 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption6": { 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption6", 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.883 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption7": { 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption7", 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "42", 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.884 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "testkey": { 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testkey", 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.885 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: "testlkey": { 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testlkey", 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "testnewline": { 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testnewline", 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.887 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snap_schedule", 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_m_granularity": { 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_m_granularity", 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.888 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "dump_on_update": { 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "dump_on_update", 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.889 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "dump database to debug log on update", 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.890 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.891 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.892 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.893 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.894 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "stats", 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.895 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.896 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.897 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.898 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.899 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "status", 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.900 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.901 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.902 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.903 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.904 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "telegraf", 2024-03-16T10:26:29.905 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "address": { 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "address", 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.906 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15", 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.907 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.908 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.910 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.911 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.912 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "telemetry", 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_basic": { 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_basic", 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.913 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_crash": { 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_crash", 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.914 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_device": { 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_device", 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.915 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_ident": { 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_ident", 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.916 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_perf": { 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_perf", 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T10:26:29.917 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: "contact": { 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "contact", 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.918 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "description": { 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "description", 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.919 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "device_url": { 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_url", 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.920 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": { 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enabled", 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.921 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "24", 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "8", 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.922 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "last_opt_revision": { 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "last_opt_revision", 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1", 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.923 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "leaderboard": { 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "leaderboard", 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.924 INFO:teuthology.orchestra.run.smithi077.stdout: "leaderboard_description": { 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "leaderboard_description", 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.925 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.926 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.927 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.928 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.929 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "organization": { 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "organization", 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.930 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "proxy": { 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "proxy", 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.931 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "url": { 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "url", 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.932 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "test_orchestrator", 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.933 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.934 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.936 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.937 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.938 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "volumes", 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.939 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.940 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.941 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.942 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "max_concurrent_clones": { 2024-03-16T10:26:29.943 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_concurrent_clones", 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "4", 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "periodic_async_work": { 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "periodic_async_work", 2024-03-16T10:26:29.944 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Periodically check for async work", 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot_clone_delay": { 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snapshot_clone_delay", 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.945 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot_clone_no_wait": { 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.946 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix", 2024-03-16T10:26:29.947 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "discovery_interval": { 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "discovery_interval", 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "100", 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.948 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "identifier": { 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "identifier", 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.949 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.950 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.952 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.953 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.954 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_host": { 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_host", 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.955 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_port": { 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_port", 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10051", 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.956 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_sender": { 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_sender", 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.957 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: "gid": 24107, 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "y", 2024-03-16T10:26:29.958 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr_features": 4540701547738038271, 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "available_modules": [ 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "alerts", 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.959 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.960 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.961 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.962 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.963 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.964 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_destination": { 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_destination", 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Email address to send alerts to", 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.965 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_from_name": { 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_from_name", 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "Ceph", 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Email From: name", 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.966 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_host": { 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_host", 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP server", 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_password": { 2024-03-16T10:26:29.967 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_password", 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Password to authenticate with", 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_port": { 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_port", 2024-03-16T10:26:29.968 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "465", 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP port", 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_sender": { 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_sender", 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.969 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP envelope sender", 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_ssl": { 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_ssl", 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.970 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_user": { 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_user", 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.971 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "User to authenticate as", 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "balancer", 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.972 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "active": { 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "active", 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.973 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "begin_time": { 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "begin_time", 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0000", 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.974 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "begin_weekday": { 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "begin_weekday", 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0", 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "6", 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.975 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_max_iterations": { 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_max_iterations", 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "25", 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "250", 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.976 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_metrics": { 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_metrics", 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_step": { 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_step", 2024-03-16T10:26:29.977 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0.5", 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0.001", 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "0.999", 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "aggressiveness of optimization", 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "end_time": { 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "end_time", 2024-03-16T10:26:29.978 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "2359", 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "end_weekday": { 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "end_weekday", 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0", 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "6", 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.980 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.981 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:29.982 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.983 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.984 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "min_score": { 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "min_score", 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T10:26:29.985 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": { 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mode", 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "upmap", 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "crush-compat", 2024-03-16T10:26:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "none", 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "read", 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap", 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap-read" 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Balancer mode", 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "pool_ids": { 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pool_ids", 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.987 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.988 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap_max_deviation": { 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "upmap_max_deviation", 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T10:26:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap_max_optimizations": { 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "upmap_max_optimizations", 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:29.990 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephadm", 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:29.991 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_down_multiplier": { 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_down_multiplier", 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3.0", 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.992 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_refresh_rate": { 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_refresh_rate", 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "20", 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_starting_port": { 2024-03-16T10:26:29.993 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_starting_port", 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "4721", 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_ptrace": { 2024-03-16T10:26:29.994 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_ptrace", 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "autotune_interval": { 2024-03-16T10:26:29.995 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "autotune_interval", 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "autotune_memory_target_ratio": { 2024-03-16T10:26:29.996 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0.7", 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm_log_destination": { 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephadm_log_destination", 2024-03-16T10:26:29.997 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "file", 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "file,syslog", 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "syslog" 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.998 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "cgroups_split": { 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cgroups_split", 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:29.999 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "config_checks_enabled": { 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_checks_enabled", 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.000 INFO:teuthology.orchestra.run.smithi077.stdout: "config_dashboard": { 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_dashboard", 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_alertmanager": { 2024-03-16T10:26:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_alertmanager", 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_base": { 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_base", 2024-03-16T10:26:30.002 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Container image name, without the tag", 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_elasticsearch": { 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_elasticsearch", 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.003 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "elasticsearch container image", 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_grafana": { 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_grafana", 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.004 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_haproxy": { 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_haproxy", 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.005 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "HAproxy container image", 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_agent": { 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_agent", 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.006 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger agent container image", 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_collector": { 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_collector", 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.007 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger collector container image", 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_query": { 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_query", 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger query container image", 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.008 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_keepalived": { 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_keepalived", 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Keepalived container image", 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.009 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_loki": { 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_loki", 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Loki container image", 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.010 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_node_exporter": { 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_node_exporter", 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.011 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_nvmeof": { 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_nvmeof", 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Nvme-of container image", 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_prometheus": { 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_prometheus", 2024-03-16T10:26:30.012 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_promtail": { 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_promtail", 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Promtail container image", 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_snmp_gateway": { 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_snmp_gateway", 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.014 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SNMP Gateway container image", 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "container_init": { 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_init", 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.015 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "daemon_cache_timeout": { 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "daemon_cache_timeout", 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.016 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "default_cephadm_command_timeout": { 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "900", 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.018 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.018 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T10:26:30.018 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.018 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.018 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.018 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.018 INFO:teuthology.orchestra.run.smithi077.stdout: "default_registry": { 2024-03-16T10:26:30.018 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default_registry", 2024-03-16T10:26:30.018 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.018 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.018 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.018 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io", 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "device_cache_timeout": { 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_cache_timeout", 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1800", 2024-03-16T10:26:30.019 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache device inventory", 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "device_enhanced_scan": { 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_enhanced_scan", 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.020 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "facts_cache_timeout": { 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "facts_cache_timeout", 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.021 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache host facts data", 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "host_check_interval": { 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "host_check_interval", 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.022 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to perform a host check", 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "hw_monitoring": { 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "hw_monitoring", 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "inventory_list_all": { 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "inventory_list_all", 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.024 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.025 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: "log_refresh_metadata": { 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_refresh_metadata", 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.026 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.027 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.028 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.029 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.030 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.031 INFO:teuthology.orchestra.run.smithi077.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "*", 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "max_count_per_host": { 2024-03-16T10:26:30.032 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_count_per_host", 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "max number of daemons per service per host", 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "max_osd_draining_count": { 2024-03-16T10:26:30.033 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_osd_draining_count", 2024-03-16T10:26:30.034 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.034 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.034 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.034 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.034 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.034 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.034 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.034 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T10:26:30.034 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.034 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.034 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.034 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "migration_current": { 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "migration_current", 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "internal - do not modify", 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.035 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": { 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mode", 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "root", 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm-package", 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "root" 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.036 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "oob_default_addr": { 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "oob_default_addr", 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "169.254.1.1", 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.037 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: "prometheus_alerts_path": { 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prometheus_alerts_path", 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.038 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_insecure": { 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_insecure", 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.039 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_password": { 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_password", 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.040 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_url": { 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_url", 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.041 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_username": { 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_username", 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.042 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "secure_monitoring_stack": { 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "secure_monitoring_stack", 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.043 INFO:teuthology.orchestra.run.smithi077.stdout: "service_discovery_port": { 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "service_discovery_port", 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8765", 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "cephadm service discovery port", 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "ssh_config_file": { 2024-03-16T10:26:30.044 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssh_config_file", 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "use_agent": { 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "use_agent", 2024-03-16T10:26:30.045 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "use_repo_digest": { 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "use_repo_digest", 2024-03-16T10:26:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_failed_host_check": { 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_failed_host_check", 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.047 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_stray_daemons": { 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_stray_daemons", 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.048 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_stray_hosts": { 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_stray_hosts", 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.049 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crash", 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.050 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.051 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.052 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.053 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.054 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "retain_interval": { 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "retain_interval", 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.055 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "31536000", 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_recent_interval": { 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_recent_interval", 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.056 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1209600", 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "dashboard", 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T10:26:30.057 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T10:26:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.059 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "AUDIT_API_ENABLED": { 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.060 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.061 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.062 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.063 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.064 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.065 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.066 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.067 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T10:26:30.068 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.069 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.070 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.071 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_URL": { 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_URL", 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.072 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.073 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.074 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.075 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.076 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.077 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "{}", 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.078 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.079 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.080 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.081 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.082 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.083 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.084 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.086 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.087 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T10:26:30.088 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T10:26:30.089 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T10:26:30.090 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8", 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T10:26:30.091 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "45", 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.092 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.093 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.094 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.095 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.096 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.097 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.098 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.099 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "cross_origin_url": { 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cross_origin_url", 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.100 INFO:teuthology.orchestra.run.smithi077.stdout: "crt_file": { 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crt_file", 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "debug": { 2024-03-16T10:26:30.101 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "debug", 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable/disable debug options", 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "jwt_token_ttl": { 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "jwt_token_ttl", 2024-03-16T10:26:30.102 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "28800", 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "key_file": { 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "key_file", 2024-03-16T10:26:30.103 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.104 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.105 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.106 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.107 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.108 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "motd": { 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "motd", 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.109 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "The message of the day", 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect_resolve_ip_addr": { 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.110 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "::", 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.111 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8080", 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.112 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl": { 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl", 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.113 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl_server_port": { 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl_server_port", 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8443", 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.114 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_behaviour": { 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_behaviour", 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "redirect", 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect" 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.115 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_error_status_code": { 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_error_status_code", 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "500", 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "400", 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "599", 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.116 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "url_prefix": { 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "url_prefix", 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.117 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "devicehealth", 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: "enable_monitoring": { 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enable_monitoring", 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.119 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.119 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.119 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.119 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "monitor device health metrics", 2024-03-16T10:26:30.119 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.122 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.123 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.124 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.125 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "mark_out_threshold": { 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mark_out_threshold", 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "2419200", 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.126 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "pool_name": { 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pool_name", 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "device_health_metrics", 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.127 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "retention_period": { 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "retention_period", 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15552000", 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long to retain device health metrics", 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.128 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "scrape_frequency": { 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "scrape_frequency", 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "86400", 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.129 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "self_heal": { 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "self_heal", 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.130 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.131 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_threshold": { 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_threshold", 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "7257600", 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.132 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "diskprediction_local", 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.133 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.134 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.135 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.136 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.137 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "predict_interval": { 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "predict_interval", 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "86400", 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.138 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "predictor_model": { 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "predictor_model", 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "prophetstor", 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.139 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.140 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "influx", 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": false, 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "influxdb python module not found", 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: "batch_size": { 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "batch_size", 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5000", 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "database": { 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "database", 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "ceph", 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": { 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "hostname", 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.143 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB server hostname", 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "30", 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "5", 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T10:26:30.144 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.145 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.146 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.147 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.148 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "password": { 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "password", 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.149 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "password of InfluxDB server user", 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "port": { 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "port", 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.150 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8086", 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB server port", 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl": { 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl", 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "false", 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.151 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "threads": { 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "threads", 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "32", 2024-03-16T10:26:30.152 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "username": { 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "username", 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "username of InfluxDB server user", 2024-03-16T10:26:30.153 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "verify_ssl": { 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "verify_ssl", 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "true", 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T10:26:30.154 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "insights", 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.155 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.156 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.157 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.158 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.159 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "iostat", 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.160 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.161 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.162 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.163 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.164 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "k8sevents", 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_event_retention_days": { 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ceph_event_retention_days", 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "7", 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.166 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "config_check_secs": { 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_check_secs", 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "10", 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.167 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.168 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.170 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.171 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.172 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "localpool", 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "failure_domain": { 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "failure_domain", 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "host", 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "failure domain for any created local pool", 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.173 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.174 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.175 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.176 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.177 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "min_size": { 2024-03-16T10:26:30.178 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "min_size", 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default min_size for any created local pool", 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "num_rep": { 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "num_rep", 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.179 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3", 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default replica count for any created local pool", 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_num": { 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pg_num", 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "128", 2024-03-16T10:26:30.181 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.181 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.181 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.181 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T10:26:30.181 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.181 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.181 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.181 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.181 INFO:teuthology.orchestra.run.smithi077.stdout: "prefix": { 2024-03-16T10:26:30.181 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prefix", 2024-03-16T10:26:30.181 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.181 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "name prefix for any created local pool", 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "subtree": { 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "subtree", 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.182 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "rack", 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mds_autoscaler", 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.183 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.184 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.185 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.186 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.187 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mirroring", 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.188 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.189 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.190 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.191 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.192 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.193 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "nfs", 2024-03-16T10:26:30.194 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.194 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.194 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.194 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.194 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.194 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.194 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.194 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.194 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.194 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.194 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.194 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.195 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.196 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.197 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "orchestrator", 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "fail_fs": { 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "fail_fs", 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.199 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.200 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.201 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.202 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.203 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.204 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator": { 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "orchestrator", 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm", 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "rook", 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "test_orchestrator" 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Orchestrator backend", 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.205 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "osd_perf_query", 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.206 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.207 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.208 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.209 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.210 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "osd_support", 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.211 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.212 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.213 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pg_autoscaler", 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.219 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.220 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.221 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "threshold": { 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "threshold", 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3.0", 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1.0", 2024-03-16T10:26:30.222 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "scaling threshold", 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "progress", 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_pg_recovery_event": { 2024-03-16T10:26:30.223 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_pg_recovery_event", 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": { 2024-03-16T10:26:30.224 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enabled", 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.225 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.227 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.228 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.229 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "max_completed_events": { 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_completed_events", 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.230 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "50", 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "number of past completed events to remember", 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T10:26:30.231 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long the module is going to sleep", 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prometheus", 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.232 INFO:teuthology.orchestra.run.smithi077.stdout: "cache": { 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cache", 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "exclude_perf_counters": { 2024-03-16T10:26:30.233 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "exclude_perf_counters", 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.234 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.235 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.236 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.237 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.238 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_stats_pools": { 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_stats_pools", 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.239 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.240 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "300", 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "scrape_interval": { 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "scrape_interval", 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15.0", 2024-03-16T10:26:30.241 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "::", 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.242 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "9283", 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.243 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "stale_cache_strategy": { 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "stale_cache_strategy", 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "log", 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.244 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_behaviour": { 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_behaviour", 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "default", 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "default", 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "error" 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.245 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_error_status_code": { 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_error_status_code", 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "500", 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "400", 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "599", 2024-03-16T10:26:30.246 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_support", 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.247 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.248 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.249 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.250 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.251 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "max_concurrent_snap_create": { 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_concurrent_snap_create", 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.252 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "mirror_snapshot_schedule": { 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.253 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "trash_purge_schedule": { 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "trash_purge_schedule", 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.254 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "restful", 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.255 INFO:teuthology.orchestra.run.smithi077.stdout: "enable_auth": { 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enable_auth", 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "key_file": { 2024-03-16T10:26:30.256 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "key_file", 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.257 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.258 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.259 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.260 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.261 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.262 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.263 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rgw", 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.264 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.265 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.267 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.268 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rook", 2024-03-16T10:26:30.269 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.270 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.271 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.272 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.272 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.272 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.272 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.272 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.272 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.272 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.272 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.272 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.272 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.272 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.272 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.274 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "storage_class": { 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "storage_class", 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "local", 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.275 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "selftest", 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.276 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.277 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.278 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.279 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.280 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "roption1": { 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "roption1", 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.281 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "roption2": { 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "roption2", 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "xyz", 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.282 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption1": { 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption1", 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.283 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption2": { 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption2", 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.284 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption3": { 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption3", 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.285 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption4": { 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption4", 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.286 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption5": { 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption5", 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.287 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption6": { 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption6", 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.288 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption7": { 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption7", 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "42", 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.289 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "testkey": { 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testkey", 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.290 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "testlkey": { 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testlkey", 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: "testnewline": { 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testnewline", 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.292 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snap_schedule", 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_m_granularity": { 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_m_granularity", 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.293 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "dump_on_update": { 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "dump_on_update", 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "dump database to debug log on update", 2024-03-16T10:26:30.294 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.295 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.296 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.297 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.298 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "stats", 2024-03-16T10:26:30.299 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.300 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.301 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.302 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.302 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.302 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.302 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.302 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.302 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.303 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.303 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.303 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.303 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.304 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.305 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.306 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.306 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.306 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.306 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.306 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.306 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "status", 2024-03-16T10:26:30.306 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.306 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.306 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.306 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.306 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.306 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.307 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.308 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.309 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.309 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.309 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.309 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.309 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.309 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.309 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.309 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.309 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.309 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.309 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.310 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.310 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.310 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.310 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.310 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.310 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.310 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.310 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.310 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.310 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.310 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "telegraf", 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.312 INFO:teuthology.orchestra.run.smithi077.stdout: "address": { 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "address", 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:30.313 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:30.314 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.314 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.314 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.314 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15", 2024-03-16T10:26:30.314 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.314 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.314 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.314 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.314 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.314 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.314 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.314 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.315 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.317 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.318 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "telemetry", 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_basic": { 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_basic", 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.320 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_crash": { 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_crash", 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T10:26:30.321 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_device": { 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_device", 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.322 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_ident": { 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_ident", 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.323 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_perf": { 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_perf", 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.324 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: "contact": { 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "contact", 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.325 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.326 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.326 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.326 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.326 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.326 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.326 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.326 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.326 INFO:teuthology.orchestra.run.smithi077.stdout: "description": { 2024-03-16T10:26:30.326 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "description", 2024-03-16T10:26:30.326 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.326 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.326 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "device_url": { 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_url", 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.327 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": { 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enabled", 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.328 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "24", 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "8", 2024-03-16T10:26:30.329 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: "last_opt_revision": { 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "last_opt_revision", 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1", 2024-03-16T10:26:30.330 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "leaderboard": { 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "leaderboard", 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.331 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "leaderboard_description": { 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "leaderboard_description", 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.332 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.333 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.334 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.335 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.336 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.337 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "organization": { 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "organization", 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.338 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: "proxy": { 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "proxy", 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.339 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "url": { 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "url", 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.340 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "test_orchestrator", 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.341 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.342 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.343 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.344 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.345 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.346 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "volumes", 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.347 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.348 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.349 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.350 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.351 INFO:teuthology.orchestra.run.smithi077.stdout: "max_concurrent_clones": { 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_concurrent_clones", 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "4", 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "periodic_async_work": { 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "periodic_async_work", 2024-03-16T10:26:30.352 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Periodically check for async work", 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot_clone_delay": { 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snapshot_clone_delay", 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.353 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot_clone_no_wait": { 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T10:26:30.354 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.355 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix", 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "discovery_interval": { 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "discovery_interval", 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "100", 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.356 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "identifier": { 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "identifier", 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.357 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.358 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.359 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.360 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.361 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.362 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.363 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_host": { 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_host", 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.364 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_port": { 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_port", 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10051", 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.365 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_sender": { 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_sender", 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.366 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: "modules": [ 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm", 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: "dashboard", 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: "iostat", 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: "nfs", 2024-03-16T10:26:30.367 INFO:teuthology.orchestra.run.smithi077.stdout: "restful" 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "available_modules": [ 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "alerts", 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.368 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.369 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.370 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.371 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.371 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.371 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.371 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.371 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.371 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.371 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.371 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.371 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.371 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.371 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.371 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.372 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.372 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.372 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.372 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.372 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.372 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.372 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.372 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.372 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.372 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.372 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.372 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.373 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_destination": { 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_destination", 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.374 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Email address to send alerts to", 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_from_name": { 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_from_name", 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "Ceph", 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.375 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Email From: name", 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_host": { 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_host", 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.376 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP server", 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_password": { 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_password", 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Password to authenticate with", 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.377 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_port": { 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_port", 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "465", 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP port", 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.378 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_sender": { 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_sender", 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SMTP envelope sender", 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.379 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_ssl": { 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_ssl", 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.380 INFO:teuthology.orchestra.run.smithi077.stdout: "smtp_user": { 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "smtp_user", 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "User to authenticate as", 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.381 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "balancer", 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "active": { 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "active", 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.382 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.383 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.383 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.383 INFO:teuthology.orchestra.run.smithi077.stdout: "begin_time": { 2024-03-16T10:26:30.383 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "begin_time", 2024-03-16T10:26:30.383 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.383 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.383 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.383 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0000", 2024-03-16T10:26:30.383 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.383 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.383 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.383 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "begin_weekday": { 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "begin_weekday", 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0", 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "6", 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.384 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_max_iterations": { 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_max_iterations", 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "25", 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "250", 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.385 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_metrics": { 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_metrics", 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.386 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "crush_compat_step": { 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crush_compat_step", 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0.5", 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0.001", 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "0.999", 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.387 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "aggressiveness of optimization", 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "end_time": { 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "end_time", 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "2359", 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T10:26:30.388 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: "end_weekday": { 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "end_weekday", 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "0", 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "6", 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.389 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.390 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.391 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.394 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "min_score": { 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "min_score", 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.395 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": { 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mode", 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "upmap", 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "crush-compat", 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "none", 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "read", 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap", 2024-03-16T10:26:30.396 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap-read" 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Balancer mode", 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: "pool_ids": { 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pool_ids", 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.397 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:30.398 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap_max_deviation": { 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "upmap_max_deviation", 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T10:26:30.399 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "upmap_max_optimizations": { 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "upmap_max_optimizations", 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.400 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephadm", 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_down_multiplier": { 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_down_multiplier", 2024-03-16T10:26:30.401 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3.0", 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_refresh_rate": { 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_refresh_rate", 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.402 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.403 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "20", 2024-03-16T10:26:30.403 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.403 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.403 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.403 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T10:26:30.403 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.403 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.403 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.403 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.403 INFO:teuthology.orchestra.run.smithi077.stdout: "agent_starting_port": { 2024-03-16T10:26:30.403 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "agent_starting_port", 2024-03-16T10:26:30.403 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "4721", 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_ptrace": { 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_ptrace", 2024-03-16T10:26:30.404 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "autotune_interval": { 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "autotune_interval", 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.405 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: "autotune_memory_target_ratio": { 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0.7", 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm_log_destination": { 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cephadm_log_destination", 2024-03-16T10:26:30.407 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "file", 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "file,syslog", 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "syslog" 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.408 INFO:teuthology.orchestra.run.smithi077.stdout: "cgroups_split": { 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cgroups_split", 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.409 INFO:teuthology.orchestra.run.smithi077.stdout: "config_checks_enabled": { 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_checks_enabled", 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.410 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "config_dashboard": { 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_dashboard", 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_alertmanager": { 2024-03-16T10:26:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_alertmanager", 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_base": { 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_base", 2024-03-16T10:26:30.412 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Container image name, without the tag", 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_elasticsearch": { 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_elasticsearch", 2024-03-16T10:26:30.413 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "elasticsearch container image", 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_grafana": { 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_grafana", 2024-03-16T10:26:30.414 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_haproxy": { 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_haproxy", 2024-03-16T10:26:30.415 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "HAproxy container image", 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_agent": { 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_agent", 2024-03-16T10:26:30.416 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger agent container image", 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_collector": { 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_collector", 2024-03-16T10:26:30.417 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger collector container image", 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_jaeger_query": { 2024-03-16T10:26:30.418 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_jaeger_query", 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Jaeger query container image", 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_keepalived": { 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_keepalived", 2024-03-16T10:26:30.419 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Keepalived container image", 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_loki": { 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_loki", 2024-03-16T10:26:30.420 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Loki container image", 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_node_exporter": { 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_node_exporter", 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.421 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_nvmeof": { 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_nvmeof", 2024-03-16T10:26:30.422 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Nvme-of container image", 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_prometheus": { 2024-03-16T10:26:30.423 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_prometheus", 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Prometheus container image", 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_promtail": { 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_promtail", 2024-03-16T10:26:30.424 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Promtail container image", 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: "container_image_snmp_gateway": { 2024-03-16T10:26:30.425 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_image_snmp_gateway", 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "SNMP Gateway container image", 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "container_init": { 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "container_init", 2024-03-16T10:26:30.426 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "daemon_cache_timeout": { 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "daemon_cache_timeout", 2024-03-16T10:26:30.427 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: "default_cephadm_command_timeout": { 2024-03-16T10:26:30.428 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "900", 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "default_registry": { 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default_registry", 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.429 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "docker.io", 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "device_cache_timeout": { 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_cache_timeout", 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.430 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1800", 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache device inventory", 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "device_enhanced_scan": { 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_enhanced_scan", 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.431 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "facts_cache_timeout": { 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "facts_cache_timeout", 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:30.432 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "seconds to cache host facts data", 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "host_check_interval": { 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "host_check_interval", 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.433 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to perform a host check", 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "hw_monitoring": { 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "hw_monitoring", 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.434 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "inventory_list_all": { 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "inventory_list_all", 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.435 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.436 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "log_refresh_metadata": { 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_refresh_metadata", 2024-03-16T10:26:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.438 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.439 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.440 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T10:26:30.441 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.442 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.442 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.442 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.442 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.442 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.442 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.442 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T10:26:30.442 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.442 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.442 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.442 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "*", 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.443 INFO:teuthology.orchestra.run.smithi077.stdout: "max_count_per_host": { 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_count_per_host", 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "max number of daemons per service per host", 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: "max_osd_draining_count": { 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_osd_draining_count", 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "migration_current": { 2024-03-16T10:26:30.445 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "migration_current", 2024-03-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "internal - do not modify", 2024-03-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.446 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "mode": { 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mode", 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "root", 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm-package", 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "root" 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T10:26:30.447 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "oob_default_addr": { 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "oob_default_addr", 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "169.254.1.1", 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.448 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: "prometheus_alerts_path": { 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prometheus_alerts_path", 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T10:26:30.449 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_insecure": { 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_insecure", 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T10:26:30.450 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_password": { 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_password", 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.451 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_url": { 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_url", 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T10:26:30.452 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "registry_username": { 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "registry_username", 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T10:26:30.453 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.454 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.454 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.454 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.454 INFO:teuthology.orchestra.run.smithi077.stdout: "secure_monitoring_stack": { 2024-03-16T10:26:30.454 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "secure_monitoring_stack", 2024-03-16T10:26:30.454 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.454 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.454 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.454 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.454 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.454 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "service_discovery_port": { 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "service_discovery_port", 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8765", 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.455 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "cephadm service discovery port", 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: "ssh_config_file": { 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssh_config_file", 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.456 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "use_agent": { 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "use_agent", 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.457 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: "use_repo_digest": { 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "use_repo_digest", 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.458 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_failed_host_check": { 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_failed_host_check", 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.459 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_stray_daemons": { 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_stray_daemons", 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.460 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_on_stray_hosts": { 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_on_stray_hosts", 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.461 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.462 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.462 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.462 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.462 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T10:26:30.462 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.462 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.462 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.462 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.462 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.462 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.462 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.462 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crash", 2024-03-16T10:26:30.463 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.463 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.463 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.463 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.463 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.463 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.463 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.463 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.463 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.463 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.463 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.464 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.465 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.466 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.466 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.466 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.466 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.466 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.466 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.466 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.466 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.466 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.466 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.466 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.466 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.467 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.467 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.467 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.467 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.467 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.467 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.467 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.467 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.467 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.467 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.467 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.467 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: "retain_interval": { 2024-03-16T10:26:30.468 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "retain_interval", 2024-03-16T10:26:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "31536000", 2024-03-16T10:26:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T10:26:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_recent_interval": { 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_recent_interval", 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1209600", 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "dashboard", 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.471 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.472 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.473 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.474 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.474 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.474 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.474 INFO:teuthology.orchestra.run.smithi077.stdout: "AUDIT_API_ENABLED": { 2024-03-16T10:26:30.474 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T10:26:30.474 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.474 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.474 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.474 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.474 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.474 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.475 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.476 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.477 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.478 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T10:26:30.479 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.480 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.480 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.480 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.480 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.480 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.480 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.480 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.480 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.480 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.480 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.480 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.480 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T10:26:30.481 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.482 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.483 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.484 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.485 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.486 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.486 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.486 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.486 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.486 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.486 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.486 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.486 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.486 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.486 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_URL": { 2024-03-16T10:26:30.486 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_URL", 2024-03-16T10:26:30.486 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.487 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.488 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.489 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.490 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.491 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.492 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "{}", 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.493 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.494 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T10:26:30.496 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.497 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.497 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.497 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.497 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.497 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.497 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.497 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.497 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.497 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.497 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.497 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.497 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T10:26:30.498 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T10:26:30.498 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.498 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.498 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.498 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.498 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.498 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.498 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.498 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.498 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.498 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.498 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.499 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.500 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.501 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.502 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T10:26:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T10:26:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T10:26:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.504 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.505 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8", 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.506 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "45", 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.507 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.508 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "admin", 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.509 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.510 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.511 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.512 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.513 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.514 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.515 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "cross_origin_url": { 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cross_origin_url", 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.516 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "crt_file": { 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "crt_file", 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.517 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "debug": { 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "debug", 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.518 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Enable/disable debug options", 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "jwt_token_ttl": { 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "jwt_token_ttl", 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "28800", 2024-03-16T10:26:30.519 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "key_file": { 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "key_file", 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.520 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.521 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.522 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.522 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.522 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.522 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.522 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.522 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.522 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.522 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.522 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.522 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.522 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.522 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.523 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.524 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.525 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.525 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.525 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.525 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.525 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.525 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.525 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.525 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.525 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.525 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.525 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.525 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: "motd": { 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "motd", 2024-03-16T10:26:30.526 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "The message of the day", 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect_resolve_ip_addr": { 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T10:26:30.527 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.528 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "::", 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.529 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8080", 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl": { 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl", 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.530 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl_server_port": { 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl_server_port", 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.531 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.533 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.533 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8443", 2024-03-16T10:26:30.533 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.533 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.533 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.533 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.533 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.533 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.533 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.533 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_behaviour": { 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_behaviour", 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "redirect", 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect" 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.534 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_error_status_code": { 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_error_status_code", 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "500", 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "400", 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "599", 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.535 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "url_prefix": { 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "url_prefix", 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.536 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "devicehealth", 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: "enable_monitoring": { 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enable_monitoring", 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.537 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "monitor device health metrics", 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.538 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.539 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.540 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.541 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.542 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "mark_out_threshold": { 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mark_out_threshold", 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "2419200", 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.543 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "pool_name": { 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pool_name", 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "device_health_metrics", 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T10:26:30.544 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "retention_period": { 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "retention_period", 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15552000", 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long to retain device health metrics", 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.545 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "scrape_frequency": { 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "scrape_frequency", 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "86400", 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.546 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "self_heal": { 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "self_heal", 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:30.547 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "warn_threshold": { 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "warn_threshold", 2024-03-16T10:26:30.548 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "7257600", 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "diskprediction_local", 2024-03-16T10:26:30.549 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.550 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.551 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.552 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.552 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.552 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.552 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.552 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.552 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.552 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.552 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.552 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.552 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.552 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.552 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.553 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.554 INFO:teuthology.orchestra.run.smithi077.stdout: "predict_interval": { 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "predict_interval", 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "86400", 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "predictor_model": { 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "predictor_model", 2024-03-16T10:26:30.555 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.556 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.556 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.556 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "prophetstor", 2024-03-16T10:26:30.556 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.556 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.556 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.556 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.556 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.556 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.556 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "600", 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.557 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "influx", 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": false, 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "influxdb python module not found", 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "batch_size": { 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "batch_size", 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5000", 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.558 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "database": { 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "database", 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "ceph", 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.559 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "hostname": { 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "hostname", 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB server hostname", 2024-03-16T10:26:30.560 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "30", 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "5", 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.561 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.562 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.563 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.564 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.565 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "password": { 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "password", 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.566 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "password of InfluxDB server user", 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "port": { 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "port", 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "8086", 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.567 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "InfluxDB server port", 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "ssl": { 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ssl", 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "false", 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T10:26:30.568 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "threads": { 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "threads", 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "32", 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.569 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "username": { 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "username", 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "username of InfluxDB server user", 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.570 INFO:teuthology.orchestra.run.smithi077.stdout: "verify_ssl": { 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "verify_ssl", 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "true", 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.571 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "insights", 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.572 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.573 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.574 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.575 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.576 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "iostat", 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.577 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.578 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.579 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.581 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "k8sevents", 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_event_retention_days": { 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "ceph_event_retention_days", 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.582 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "7", 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "config_check_secs": { 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "config_check_secs", 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.583 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "10", 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.584 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.585 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.586 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.587 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.588 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "localpool", 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.589 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "failure_domain": { 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "failure_domain", 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "host", 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "failure domain for any created local pool", 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.590 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.591 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.592 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.594 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "min_size": { 2024-03-16T10:26:30.595 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "min_size", 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default min_size for any created local pool", 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.596 INFO:teuthology.orchestra.run.smithi077.stdout: "num_rep": { 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "num_rep", 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3", 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default replica count for any created local pool", 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_num": { 2024-03-16T10:26:30.597 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pg_num", 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "128", 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.598 INFO:teuthology.orchestra.run.smithi077.stdout: "prefix": { 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prefix", 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "name prefix for any created local pool", 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "subtree": { 2024-03-16T10:26:30.599 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "subtree", 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "rack", 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.600 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mds_autoscaler", 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.601 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.602 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.603 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.604 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.605 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mirroring", 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.606 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.607 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.608 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.609 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.610 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "nfs", 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.611 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.612 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.613 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.614 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.615 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "orchestrator", 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "fail_fs": { 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "fail_fs", 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.616 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.617 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.618 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.619 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.620 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator": { 2024-03-16T10:26:30.621 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "orchestrator", 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "cephadm", 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "rook", 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "test_orchestrator" 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Orchestrator backend", 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.622 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "osd_perf_query", 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.624 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.625 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.626 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.627 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "osd_support", 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.628 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.629 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.630 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.631 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.632 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "pg_autoscaler", 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.633 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.634 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.636 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.637 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.638 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "threshold": { 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "threshold", 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "3.0", 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1.0", 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "scaling threshold", 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.639 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "progress", 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_pg_recovery_event": { 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_pg_recovery_event", 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T10:26:30.640 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": { 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enabled", 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.641 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.642 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.643 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.644 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.645 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "max_completed_events": { 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_completed_events", 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "50", 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.646 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "number of past completed events to remember", 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "sleep_interval": { 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "sleep_interval", 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "5", 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "how long the module is going to sleep", 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.647 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "prometheus", 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: "cache": { 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "cache", 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.648 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "exclude_perf_counters": { 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "exclude_perf_counters", 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.649 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.650 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.651 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.652 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.653 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.654 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_stats_pools": { 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_stats_pools", 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.655 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "300", 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.656 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "scrape_interval": { 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "scrape_interval", 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15.0", 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.657 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "::", 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.658 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.659 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.659 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T10:26:30.659 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T10:26:30.659 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.659 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.659 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.659 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "9283", 2024-03-16T10:26:30.659 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.659 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.659 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.659 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "stale_cache_strategy": { 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "stale_cache_strategy", 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "log", 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.660 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_behaviour": { 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_behaviour", 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "default", 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "default", 2024-03-16T10:26:30.661 INFO:teuthology.orchestra.run.smithi077.stdout: "error" 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: "standby_error_status_code": { 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "standby_error_status_code", 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "500", 2024-03-16T10:26:30.662 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "400", 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "599", 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_support", 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.663 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.664 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.665 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.666 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.667 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "max_concurrent_snap_create": { 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_concurrent_snap_create", 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10", 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.668 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: "mirror_snapshot_schedule": { 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "trash_purge_schedule": { 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "trash_purge_schedule", 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.670 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "restful", 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: "enable_auth": { 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enable_auth", 2024-03-16T10:26:30.671 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "key_file": { 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "key_file", 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.673 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.674 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.675 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.676 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.677 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "server_addr": { 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_addr", 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.678 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "server_port": { 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "server_port", 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.679 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rgw", 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.680 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.681 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.682 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.683 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.684 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rook", 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.685 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.686 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.687 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.689 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: "storage_class": { 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "storage_class", 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "local", 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.691 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "selftest", 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.692 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.693 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.694 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.695 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.696 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "roption1": { 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "roption1", 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.697 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "roption2": { 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "roption2", 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "xyz", 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.698 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption1": { 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption1", 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.699 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption2": { 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption2", 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.700 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption3": { 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption3", 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "float", 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.701 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption4": { 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption4", 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.702 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption5": { 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption5", 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.703 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption6": { 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption6", 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.704 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: "rwoption7": { 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rwoption7", 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.705 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "1", 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "42", 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: "testkey": { 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testkey", 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.706 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.707 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.707 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.707 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.707 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.707 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.707 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.707 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.707 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.707 INFO:teuthology.orchestra.run.smithi077.stdout: "testlkey": { 2024-03-16T10:26:30.707 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testlkey", 2024-03-16T10:26:30.707 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.707 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "testnewline": { 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "testnewline", 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.708 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snap_schedule", 2024-03-16T10:26:30.709 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "allow_m_granularity": { 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "allow_m_granularity", 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.710 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "dump_on_update": { 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "dump_on_update", 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "dump database to debug log on update", 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.711 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.712 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.713 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.714 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.715 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.716 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "stats", 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.717 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.718 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.719 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "status", 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.724 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.725 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "telegraf", 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: "address": { 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "address", 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:30.728 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "15", 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.729 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.730 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.731 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.732 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.733 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "telemetry", 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.734 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_basic": { 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_basic", 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_crash": { 2024-03-16T10:26:30.735 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_crash", 2024-03-16T10:26:30.737 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.737 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.737 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.737 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.737 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.737 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.737 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.737 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T10:26:30.737 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.737 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.737 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.737 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_device": { 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_device", 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.738 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_ident": { 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_ident", 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "channel_perf": { 2024-03-16T10:26:30.739 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "channel_perf", 2024-03-16T10:26:30.740 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.740 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.740 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.740 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.740 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.740 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.740 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.740 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T10:26:30.740 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.740 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.740 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.740 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "contact": { 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "contact", 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "description": { 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "description", 2024-03-16T10:26:30.741 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "device_url": { 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "device_url", 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.742 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": { 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "enabled", 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.743 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "24", 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "8", 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "last_opt_revision": { 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "last_opt_revision", 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.745 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "1", 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "leaderboard": { 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "leaderboard", 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.746 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "leaderboard_description": { 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "leaderboard_description", 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.747 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.750 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.751 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.752 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: "organization": { 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "organization", 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.753 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.754 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.754 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.754 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.754 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.754 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.754 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.754 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.754 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.754 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.754 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.754 INFO:teuthology.orchestra.run.smithi077.stdout: "proxy": { 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "proxy", 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.755 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: "url": { 2024-03-16T10:26:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "url", 2024-03-16T10:26:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T10:26:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.756 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "test_orchestrator", 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.757 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.758 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.759 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.760 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.761 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "volumes", 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.762 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.763 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.764 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.765 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.766 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "max_concurrent_clones": { 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "max_concurrent_clones", 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "4", 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.767 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "periodic_async_work": { 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "periodic_async_work", 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.768 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Periodically check for async work", 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot_clone_delay": { 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snapshot_clone_delay", 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "0", 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.769 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "snapshot_clone_no_wait": { 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "True", 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.770 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix", 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: "can_run": true, 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: "error_string": "", 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: "module_options": { 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: "discovery_interval": { 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "discovery_interval", 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "uint", 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "100", 2024-03-16T10:26:30.771 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "identifier": { 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "identifier", 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.772 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: "interval": { 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "interval", 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "secs", 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "60", 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.773 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.774 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.774 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.774 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.774 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.774 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.774 INFO:teuthology.orchestra.run.smithi077.stdout: "log_level": { 2024-03-16T10:26:30.774 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_level", 2024-03-16T10:26:30.774 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.774 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.774 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.774 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.774 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.775 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster": { 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster", 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.776 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_cluster_level": { 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_cluster_level", 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "info", 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [ 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "", 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "critical", 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "debug", 2024-03-16T10:26:30.777 INFO:teuthology.orchestra.run.smithi077.stdout: "error", 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: "info", 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: "warning" 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: "log_to_file": { 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "log_to_file", 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "bool", 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 1, 2024-03-16T10:26:30.778 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "False", 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_host": { 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_host", 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.779 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "", 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_port": { 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_port", 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "int", 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.780 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "10051", 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "zabbix_sender": { 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "zabbix_sender", 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "str", 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "level": "advanced", 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "flags": 0, 2024-03-16T10:26:30.781 INFO:teuthology.orchestra.run.smithi077.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: "min": "", 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: "max": "", 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: "enum_allowed": [], 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: "desc": "", 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: "long_desc": "", 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: "see_also": [] 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: "services": { 2024-03-16T10:26:30.782 INFO:teuthology.orchestra.run.smithi077.stdout: "dashboard": "https://172.21.15.77:8443/" 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: "always_on_modules": { 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: "octopus": [ 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: "balancer", 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: "crash", 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: "devicehealth", 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator", 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_autoscaler", 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: "progress", 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_support", 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: "status", 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: "telemetry", 2024-03-16T10:26:30.783 INFO:teuthology.orchestra.run.smithi077.stdout: "volumes" 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "pacific": [ 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "balancer", 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "crash", 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "devicehealth", 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator", 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_autoscaler", 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "progress", 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_support", 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "status", 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "telemetry", 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "volumes" 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "quincy": [ 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "balancer", 2024-03-16T10:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: "crash", 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "devicehealth", 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator", 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_autoscaler", 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "progress", 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_support", 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "status", 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "telemetry", 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "volumes" 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "reef": [ 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "balancer", 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "crash", 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "devicehealth", 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator", 2024-03-16T10:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_autoscaler", 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "progress", 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_support", 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "status", 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "telemetry", 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "volumes" 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "squid": [ 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "balancer", 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "crash", 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "devicehealth", 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "orchestrator", 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "pg_autoscaler", 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "progress", 2024-03-16T10:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: "rbd_support", 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: "status", 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: "telemetry", 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: "volumes" 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: "last_failure_osd_epoch": 3, 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: "active_clients": [ 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "devicehealth", 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: "addrvec": [ 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v2", 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "172.21.15.77:0", 2024-03-16T10:26:30.787 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3759057583 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "libcephsqlite", 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: "addrvec": [ 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v2", 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "172.21.15.77:0", 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 2271041119 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.788 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "rbd_support", 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: "addrvec": [ 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v2", 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "172.21.15.77:0", 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 1908073181 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "volumes", 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: "addrvec": [ 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: "type": "v2", 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: "addr": "172.21.15.77:0", 2024-03-16T10:26:30.789 INFO:teuthology.orchestra.run.smithi077.stdout: "nonce": 3031899593 2024-03-16T10:26:30.790 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.790 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T10:26:30.790 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:30.790 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-03-16T10:26:30.790 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:26:30.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:29 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2040980783' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:26:30.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:29 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1436442045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:26:30.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:29 smithi193 ceph-mon[36311]: pgmap v633: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 op/s 2024-03-16T10:26:30.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:29 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/681568637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:26:30.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:30 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2649021351' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T10:26:30.792 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:29 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2040980783' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:26:30.792 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:29 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1436442045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:26:30.792 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:29 smithi192 ceph-mon[36354]: pgmap v633: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 op/s 2024-03-16T10:26:30.792 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:29 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/681568637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:26:30.792 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:30 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2649021351' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T10:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.543+0000 7f8584159640 1 -- 172.21.15.77:0/2649021351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f85680a2800 msgr2=0x7f85680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.543+0000 7f8584159640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f85680a2800 0x7f85680a4cc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f856c000c00 tx=0x7f856c005fd0 comp rx=0 tx=0).stop 2024-03-16T10:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.543+0000 7f8584159640 1 -- 172.21.15.77:0/2649021351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f857c102290 msgr2=0x7f857c197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.543+0000 7f8584159640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f857c102290 0x7f857c197b70 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f857400e9f0 tx=0x7f857400edf0 comp rx=0 tx=0).stop 2024-03-16T10:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.543+0000 7f8584159640 1 -- 172.21.15.77:0/2649021351 shutdown_connections 2024-03-16T10:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.543+0000 7f8584159640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f857c1074f0 0x7f857c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.543+0000 7f8584159640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f85680a2800 0x7f85680a4cc0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.543+0000 7f8584159640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f857c104bc0 0x7f857c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.543+0000 7f8584159640 1 --2- 172.21.15.77:0/2649021351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f857c102290 0x7f857c197b70 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.543+0000 7f8584159640 1 -- 172.21.15.77:0/2649021351 >> 172.21.15.77:0/2649021351 conn(0x7f857c0fbcc0 msgr2=0x7f857c102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.544+0000 7f8584159640 1 -- 172.21.15.77:0/2649021351 shutdown_connections 2024-03-16T10:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:26:29.544+0000 7f8584159640 1 -- 172.21.15.77:0/2649021351 wait complete. 2024-03-16T10:26:30.799 DEBUG:tasks.check_counter:Getting stats from a 2024-03-16T10:26:30.800 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-16T10:26:30.800 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:30 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2649021351' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T10:26:31.039 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:26:31.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:31 smithi077 ceph-mon[32400]: pgmap v634: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-16T10:26:31.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:31 smithi192 ceph-mon[36354]: pgmap v634: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-16T10:26:31.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:31 smithi193 ceph-mon[36311]: pgmap v634: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-16T10:26:32.883 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:26:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T10:26:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 2147, 2024-03-16T10:26:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 2020, 2024-03-16T10:26:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 820995, 2024-03-16T10:26:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 5113856, 2024-03-16T10:26:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 20, 2024-03-16T10:26:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 5, 2024-03-16T10:26:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.192613079, 2024-03-16T10:26:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.071968186, 2024-03-16T10:26:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.097448360, 2024-03-16T10:26:32.883 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.029940311, 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 2018, 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.192104094, 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000095195 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 2581, 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000572388, 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000221 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.884 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 10791, 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 13449, 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 3343487, 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 7946902, 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 16, 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 6, 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 1.449766121, 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 1.100273422, 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.766216203, 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.011887355, 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 13448, 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 4.697707332, 2024-03-16T10:26:32.885 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000349323 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 10702, 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.004076564, 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000380 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 335296 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 13153, 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 9317, 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 6782161, 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 2748685, 2024-03-16T10:26:32.886 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 18, 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 6, 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.865182546, 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.315468514, 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.697813636, 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.024678717, 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 9316, 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 1.141179840, 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000122496 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 20380, 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.003365772, 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000165 2024-03-16T10:26:32.887 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 934608, 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 127280 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 1, 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-MDSRank": { 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1044, 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 2.407166232, 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.002305714 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.888 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-PurgeQueue": { 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1114, 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.085266302, 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000076540 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "request": 6283, 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "reply": 6207, 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "reply_latency": { 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 6207, 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 167.010100821, 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.026906734 2024-03-16T10:26:32.889 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_reply": 0, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "forward": 76, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_complete": 0, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_keys": 0, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_commit": 29, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_split": 2, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_merge": 0, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes": 25, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_top": 6, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_bottom": 9, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pin_tail": 10, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pinned": 13, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_expired": 382, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_with_caps": 1, 2024-03-16T10:26:32.890 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": 1, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "subtrees": 4, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse": 14952, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_hit": 12803, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_forward": 76, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_discover": 196, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_dir_fetch": 0, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_remote_ino": 0, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_lock": 81, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "load_cent": 0, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "q": 0, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "exported": 17, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "exported_inodes": 4, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "imported": 3, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "imported_inodes": 199, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_dir_fetch": 0, 2024-03-16T10:26:32.891 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_backtrace_fetch": 868, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_peer_discover": 0, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rfiles": 736, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rbytes": 15642193, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rsnaps": 0, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_fetch": 868, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_set_tag": 0, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_inotable_repaired": 0, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_inodes": 0, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_file_inodes": 868, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_inode_file_caps": 6, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_revoke": 522, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_grant": 1091, 2024-03-16T10:26:32.892 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps": 1099, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps_dirty": 577, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_cap_release": 29, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "process_request_cap_release": 2456 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_cache": { 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update": 185, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update_receipt": 3173, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_try_discover": 1826, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_send_discover": 2100, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_handle_discover": 171, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays": 0, 2024-03-16T10:26:32.893 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_delayed": 0, 2024-03-16T10:26:32.894 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_enqueuing": 0, 2024-03-16T10:26:32.894 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_created": 580, 2024-03-16T10:26:32.894 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_enqueued": 580, 2024-03-16T10:26:32.894 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_reintegrated": 0, 2024-03-16T10:26:32.894 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_migrated": 0, 2024-03-16T10:26:32.894 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_processing": 0, 2024-03-16T10:26:32.894 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_enqueued": 0, 2024-03-16T10:26:32.894 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_prioritized": 0, 2024-03-16T10:26:32.894 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_started": 0, 2024-03-16T10:26:32.894 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_completed": 0, 2024-03-16T10:26:32.894 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_quiesce_path": 0, 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_quiesce_inode": 0, 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_exportdir": 17, 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_flush": 0, 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragmentdir": 2, 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragstats": 0, 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_inodestats": 0 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_log": { 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "evlrg": 0, 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "evadd": 3098, 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "evex": 0, 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "evtrm": 0, 2024-03-16T10:26:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: "ev": 3098, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "evexg": 0, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "evexd": 0, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "segadd": 5, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "segex": 0, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "segtrm": 0, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "seg": 5, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "segmjr": 2, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "segexg": 0, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "segexd": 0, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "expos": 4194304, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "wrpos": 8499233, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "rdpos": 0, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "jlat": { 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 147, 2024-03-16T10:26:32.896 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 5.433752056, 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.036964299 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: "replayed": 0 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_mem": { 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: "ino": 30, 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: "ino+": 801, 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: "ino-": 771, 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: "dir": 29, 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: "dir+": 1599, 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: "dir-": 1570, 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: "dn": 25, 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: "dn+": 1877, 2024-03-16T10:26:32.897 INFO:teuthology.orchestra.run.smithi077.stdout: "dn-": 1852, 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "cap": 1, 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "cap+": 777, 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "cap-": 776, 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "rss": 66212, 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "heap": 223612 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_server": { 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_client_request": 7845, 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_server_request": 46, 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_request": 6283, 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_session": 12, 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_peer_request": 1489, 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "req_create_latency": { 2024-03-16T10:26:32.898 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 481, 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 3.532874998, 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.007344854 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getattr_latency": { 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getfilelock_latency": { 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: "req_link_latency": { 2024-03-16T10:26:32.899 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookup_latency": { 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 4322, 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 156.494419720, 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.036208796 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookuphash_latency": { 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupino_latency": { 2024-03-16T10:26:32.900 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1, 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000145763, 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000145763 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupname_latency": { 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupparent_latency": { 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupsnap_latency": { 2024-03-16T10:26:32.901 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.902 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.902 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.902 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.902 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lssnap_latency": { 2024-03-16T10:26:32.902 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.902 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mkdir_latency": { 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mknod_latency": { 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mksnap_latency": { 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.903 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: "req_open_latency": { 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 493, 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 1.928251548, 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.003911260 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: "req_readdir_latency": { 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 17, 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.014329827, 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000842931 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rename_latency": { 2024-03-16T10:26:32.904 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 483, 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 4.192909907, 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.008680972 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "req_renamesnap_latency": { 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "req_snapdiff_latency": { 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmdir_latency": { 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmsnap_latency": { 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmxattr_latency": { 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.906 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setattr_latency": { 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setdirlayout_latency": { 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setfilelock_latency": { 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.907 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setlayout_latency": { 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setxattr_latency": { 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: "req_symlink_latency": { 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.908 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: "req_unlink_latency": { 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 410, 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.847169058, 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.002066265 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_revoke_eviction": 0, 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_acquisition_throttle": 0, 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getvxattr_latency": { 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.909 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_sessions": { 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: "session_count": 1, 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: "session_add": 2, 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: "session_remove": 1, 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_open": 1, 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_stale": 0, 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: "total_load": 1298, 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: "average_load": 1298, 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: "avg_session_uptime": 565, 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: "mdthresh_evicted": 0 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-03-16T10:26:32.910 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T10:26:32.911 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T10:26:32.912 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 309937, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 134, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 704, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 8, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-03-16T10:26:32.913 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 7344, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 155, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 122309, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 465, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-03-16T10:26:32.914 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: "objecter": { 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: "op_active": 0, 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: "op_laggy": 0, 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send": 1854, 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send_bytes": 4777957, 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: "op_resend": 33, 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: "op_reply": 1854, 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: "op_latency": { 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1854, 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 10.254781692, 2024-03-16T10:26:32.915 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.005531165 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: "op_inflight": 0, 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: "oplen_avg": { 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1854, 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 1992 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: "op": 1854, 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: "op_r": 904, 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: "op_w": 950, 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: "op_rmw": 0, 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: "op_pg": 0, 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_stat": 0, 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_create": 10, 2024-03-16T10:26:32.916 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_read": 36, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_write": 183, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writefull": 49, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writesame": 0, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_append": 0, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_zero": 0, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_truncate": 2, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_delete": 593, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_mapext": 0, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_sparse_read": 0, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_clonerange": 0, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_getxattr": 868, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_setxattr": 20, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_cmpxattr": 0, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_rmxattr": 0, 2024-03-16T10:26:32.917 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_resetxattrs": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_call": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_watch": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_notify": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls_filter": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_other": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_active": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_send": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_resend": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_ping": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_active": 0, 2024-03-16T10:26:32.918 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_send": 0, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_resend": 0, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_active": 0, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_send": 0, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_resend": 0, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_active": 0, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_send": 0, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_resend": 0, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "command_active": 0, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "command_send": 0, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "command_resend": 0, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "map_epoch": 88, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "map_full": 0, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "map_inc": 3, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_sessions": 12, 2024-03-16T10:26:32.919 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_open": 12, 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_close": 0, 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_laggy": 0, 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_wr": 191, 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_rd": 0, 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_del": 40 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: "oft": { 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_objs": 1, 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_kv_pairs": 0, 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_updates": 0, 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_removes": 0 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: "osd-slow-ops": { 2024-03-16T10:26:32.920 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_ops_count": 0 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "purge_queue": { 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops": 0, 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops_high_water": 6, 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing": 0, 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_high_water": 6, 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed_ops": 580, 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed": 580, 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_item_in_journal": 0 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T10:26:32.921 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 26087, 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 8743208, 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 26087, 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 26087, 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 8743208, 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_bytes": { 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 1854, 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 4836537, 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 1854, 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 4836537, 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T10:26:32.923 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_ops": { 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 1024, 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 1854, 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 1854, 2024-03-16T10:26:32.924 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 1854, 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 1854, 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle": { 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 580, 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 58580, 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:32.925 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 580, 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 36, 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 58580, 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle-0x55908ff6e500": { 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2024-03-16T10:26:32.926 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 3098, 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 4304929, 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 3098, 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 147, 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 4304929, 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:32.927 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:26:33.641 INFO:tasks.check_counter:Daemon mds.a mds.exported=17 2024-03-16T10:26:33.642 INFO:tasks.check_counter:Daemon mds.a mds.imported=3 2024-03-16T10:26:33.642 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=185 2024-03-16T10:26:33.642 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=3173 2024-03-16T10:26:33.642 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=2 2024-03-16T10:26:33.642 DEBUG:tasks.check_counter:Getting stats from d 2024-03-16T10:26:33.642 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-16T10:26:33.894 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:26:34.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:33 smithi077 ceph-mon[32400]: pgmap v635: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 13 op/s 2024-03-16T10:26:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:33 smithi193 ceph-mon[36311]: pgmap v635: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 13 op/s 2024-03-16T10:26:34.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:33 smithi192 ceph-mon[36354]: pgmap v635: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 13 op/s 2024-03-16T10:26:35.742 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:26:35.742 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 4, 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 4, 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 3072, 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 448, 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.001305310, 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000034914, 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000462698, 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 3, 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000680463, 2024-03-16T10:26:35.743 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000226821 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 200, 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 195, 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 745808, 2024-03-16T10:26:35.744 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 119584, 2024-03-16T10:26:35.745 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2024-03-16T10:26:35.745 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2024-03-16T10:26:35.745 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.029125431, 2024-03-16T10:26:35.745 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.010370658, 2024-03-16T10:26:35.745 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.018631835, 2024-03-16T10:26:35.745 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.003904845, 2024-03-16T10:26:35.745 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:35.745 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 194, 2024-03-16T10:26:35.745 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.007433594, 2024-03-16T10:26:35.746 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000038317 2024-03-16T10:26:35.746 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:35.746 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:35.746 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:35.746 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:35.746 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:35.746 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:35.746 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 745808, 2024-03-16T10:26:35.746 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 119584 2024-03-16T10:26:35.746 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:35.746 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 1, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 148, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 160, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 103808, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 3, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 2, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.010251184, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.007874636, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.001338013, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 146, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.005202934, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000035636 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:35.747 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 103808 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-03-16T10:26:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-03-16T10:26:35.749 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-03-16T10:26:35.750 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 149007, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 30, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2024-03-16T10:26:35.751 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T10:26:35.752 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 201, 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 717586, 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 201, 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 201, 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 717586, 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:35.753 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:26:35.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:35 smithi077 ceph-mon[32400]: pgmap v636: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T10:26:35.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:35 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:26:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:35 smithi193 ceph-mon[36311]: pgmap v636: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T10:26:36.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:35 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:26:36.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:35 smithi192 ceph-mon[36354]: pgmap v636: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T10:26:36.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:35 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T10:26:36.439 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-16T10:26:36.439 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-16T10:26:36.439 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-16T10:26:36.439 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-16T10:26:36.439 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-16T10:26:36.439 DEBUG:tasks.check_counter:Getting stats from g 2024-03-16T10:26:36.439 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-16T10:26:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:26:37.111 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:36 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:26:37.111 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:36 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:26:37.111 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:36 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:26:37.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:36 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:26:37.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:36 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:26:37.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:36 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:26:37.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:36 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T10:26:37.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:36 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T10:26:37.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:36 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' 2024-03-16T10:26:38.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:37 smithi077 ceph-mon[32400]: pgmap v637: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 11 op/s 2024-03-16T10:26:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:37 smithi193 ceph-mon[36311]: pgmap v637: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 11 op/s 2024-03-16T10:26:38.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:37 smithi192 ceph-mon[36354]: pgmap v637: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 11 op/s 2024-03-16T10:26:38.504 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:26:38.546 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T10:26:38.546 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 687, 2024-03-16T10:26:38.546 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 689, 2024-03-16T10:26:38.546 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 230777, 2024-03-16T10:26:38.546 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 218467, 2024-03-16T10:26:38.546 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 6, 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 2, 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.068268852, 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.025532451, 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.016180260, 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.020207374, 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 687, 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.029511785, 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000042957 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.547 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 1338, 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 1482, 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 4646215, 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 760791, 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 5, 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 4, 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.091992799, 2024-03-16T10:26:38.548 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.030227739, 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.025012305, 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.029433598, 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1480, 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.066544689, 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000044962 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.549 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T10:26:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 335568 2024-03-16T10:26:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T10:26:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 234, 2024-03-16T10:26:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 204, 2024-03-16T10:26:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 920116, 2024-03-16T10:26:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 120101, 2024-03-16T10:26:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2024-03-16T10:26:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 3, 2024-03-16T10:26:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.029631216, 2024-03-16T10:26:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.010337741, 2024-03-16T10:26:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.019463133, 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.003866442, 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 204, 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.011263432, 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000055212 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 918768, 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 119136 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 1, 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-MDSRank": { 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 3390, 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.082440892, 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000024318 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-PurgeQueue": { 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2024-03-16T10:26:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: "request": 0, 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: "reply": 0, 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: "reply_latency": { 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.553 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_reply": 0, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "forward": 0, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_complete": 0, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_keys": 0, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_commit": 0, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_split": 0, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_merge": 0, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes": 583, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_top": 0, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_bottom": 583, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pin_tail": 0, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pinned": 583, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_expired": 0, 2024-03-16T10:26:38.554 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_with_caps": 0, 2024-03-16T10:26:38.555 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": 0, 2024-03-16T10:26:38.555 INFO:teuthology.orchestra.run.smithi077.stdout: "subtrees": 1, 2024-03-16T10:26:38.555 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse": 0, 2024-03-16T10:26:38.555 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_hit": 0, 2024-03-16T10:26:38.555 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_forward": 0, 2024-03-16T10:26:38.555 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_discover": 0, 2024-03-16T10:26:38.555 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_dir_fetch": 0, 2024-03-16T10:26:38.555 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_remote_ino": 0, 2024-03-16T10:26:38.555 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_lock": 0, 2024-03-16T10:26:38.555 INFO:teuthology.orchestra.run.smithi077.stdout: "load_cent": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "q": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "exported": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "exported_inodes": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "imported": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "imported_inodes": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_dir_fetch": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_backtrace_fetch": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_peer_discover": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rfiles": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rbytes": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rsnaps": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_set_tag": 0, 2024-03-16T10:26:38.556 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_inotable_repaired": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_inodes": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_file_inodes": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_inode_file_caps": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_grant": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps_dirty": 0, 2024-03-16T10:26:38.557 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_cap_release": 0, 2024-03-16T10:26:38.558 INFO:teuthology.orchestra.run.smithi077.stdout: "process_request_cap_release": 0 2024-03-16T10:26:38.558 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.558 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_cache": { 2024-03-16T10:26:38.558 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update": 0, 2024-03-16T10:26:38.558 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update_receipt": 0, 2024-03-16T10:26:38.558 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_try_discover": 0, 2024-03-16T10:26:38.558 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_send_discover": 0, 2024-03-16T10:26:38.558 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_handle_discover": 0, 2024-03-16T10:26:38.558 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays": 0, 2024-03-16T10:26:38.558 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_delayed": 0, 2024-03-16T10:26:38.558 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_enqueuing": 0, 2024-03-16T10:26:38.559 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_created": 0, 2024-03-16T10:26:38.559 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_enqueued": 0, 2024-03-16T10:26:38.559 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_reintegrated": 0, 2024-03-16T10:26:38.559 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_migrated": 0, 2024-03-16T10:26:38.559 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_processing": 0, 2024-03-16T10:26:38.559 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_enqueued": 0, 2024-03-16T10:26:38.559 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_prioritized": 0, 2024-03-16T10:26:38.559 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_started": 0, 2024-03-16T10:26:38.559 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_completed": 0, 2024-03-16T10:26:38.559 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_quiesce_path": 0, 2024-03-16T10:26:38.559 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_quiesce_inode": 0, 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_exportdir": 0, 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_flush": 0, 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragmentdir": 0, 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragstats": 0, 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_inodestats": 0 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_log": { 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: "evlrg": 0, 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: "evadd": 0, 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: "evex": 0, 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: "evtrm": 0, 2024-03-16T10:26:38.560 INFO:teuthology.orchestra.run.smithi077.stdout: "ev": 3098, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "evexg": 0, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "evexd": 0, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "segadd": 0, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "segex": 0, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "segtrm": 0, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "seg": 5, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "segmjr": 2, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "segexg": 0, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "segexd": 0, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "expos": 4194304, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "wrpos": 8499233, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "rdpos": 8498095, 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "jlat": { 2024-03-16T10:26:38.561 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.562 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.562 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.562 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.562 INFO:teuthology.orchestra.run.smithi077.stdout: "replayed": 3098 2024-03-16T10:26:38.562 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.562 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_mem": { 2024-03-16T10:26:38.562 INFO:teuthology.orchestra.run.smithi077.stdout: "ino": 6, 2024-03-16T10:26:38.562 INFO:teuthology.orchestra.run.smithi077.stdout: "ino+": 604, 2024-03-16T10:26:38.562 INFO:teuthology.orchestra.run.smithi077.stdout: "ino-": 598, 2024-03-16T10:26:38.562 INFO:teuthology.orchestra.run.smithi077.stdout: "dir": 4, 2024-03-16T10:26:38.562 INFO:teuthology.orchestra.run.smithi077.stdout: "dir+": 49, 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: "dir-": 45, 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: "dn": 583, 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: "dn+": 1422, 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: "dn-": 839, 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: "cap": 0, 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: "cap+": 0, 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: "cap-": 0, 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: "rss": 53740, 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: "heap": 215420 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_server": { 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_client_request": 0, 2024-03-16T10:26:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_server_request": 0, 2024-03-16T10:26:38.564 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_request": 0, 2024-03-16T10:26:38.564 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_session": 0, 2024-03-16T10:26:38.564 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_peer_request": 0, 2024-03-16T10:26:38.564 INFO:teuthology.orchestra.run.smithi077.stdout: "req_create_latency": { 2024-03-16T10:26:38.564 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.564 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.564 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.564 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.564 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getattr_latency": { 2024-03-16T10:26:38.564 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.564 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.564 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getfilelock_latency": { 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: "req_link_latency": { 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookup_latency": { 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.565 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookuphash_latency": { 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupino_latency": { 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupname_latency": { 2024-03-16T10:26:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupparent_latency": { 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupsnap_latency": { 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lssnap_latency": { 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.567 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.568 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.568 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.568 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mkdir_latency": { 2024-03-16T10:26:38.568 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.568 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.568 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mknod_latency": { 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mksnap_latency": { 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "req_open_latency": { 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.569 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "req_readdir_latency": { 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rename_latency": { 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "req_renamesnap_latency": { 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.570 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "req_snapdiff_latency": { 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmdir_latency": { 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmsnap_latency": { 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.571 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmxattr_latency": { 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setattr_latency": { 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setdirlayout_latency": { 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.572 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setfilelock_latency": { 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setlayout_latency": { 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setxattr_latency": { 2024-03-16T10:26:38.573 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "req_symlink_latency": { 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "req_unlink_latency": { 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.574 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_revoke_eviction": 0, 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_acquisition_throttle": 0, 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getvxattr_latency": { 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_sessions": { 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "session_count": 1, 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "session_add": 2, 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "session_remove": 1, 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_open": 2, 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_stale": 0, 2024-03-16T10:26:38.575 INFO:teuthology.orchestra.run.smithi077.stdout: "total_load": 0, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "average_load": 0, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "avg_session_uptime": 2742, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "mdthresh_evicted": 0 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T10:26:38.576 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-03-16T10:26:38.577 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 197481, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 55, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-03-16T10:26:38.578 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 7344, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 155, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 429400, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2383, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-03-16T10:26:38.579 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "objecter": { 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "op_active": 0, 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "op_laggy": 0, 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send": 2027, 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send_bytes": 21, 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "op_resend": 3, 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "op_reply": 2027, 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "op_latency": { 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 2027, 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 4.199909363, 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.002071982 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "op_inflight": 0, 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "oplen_avg": { 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 2027, 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 2028 2024-03-16T10:26:38.580 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "op": 2027, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "op_r": 2027, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "op_w": 0, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "op_rmw": 0, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "op_pg": 0, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_stat": 1259, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_create": 0, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_read": 766, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_write": 0, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writefull": 0, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writesame": 0, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_append": 0, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_zero": 0, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_truncate": 0, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_delete": 0, 2024-03-16T10:26:38.581 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_mapext": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_sparse_read": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_clonerange": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_getxattr": 1, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_setxattr": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_cmpxattr": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_rmxattr": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_resetxattrs": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_call": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_watch": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_notify": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls_filter": 0, 2024-03-16T10:26:38.582 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_other": 0, 2024-03-16T10:26:38.583 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_active": 0, 2024-03-16T10:26:38.583 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_send": 0, 2024-03-16T10:26:38.583 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_resend": 0, 2024-03-16T10:26:38.583 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_ping": 0, 2024-03-16T10:26:38.583 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_active": 0, 2024-03-16T10:26:38.583 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_send": 0, 2024-03-16T10:26:38.583 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_resend": 0, 2024-03-16T10:26:38.583 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_active": 0, 2024-03-16T10:26:38.583 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_send": 0, 2024-03-16T10:26:38.583 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_resend": 0, 2024-03-16T10:26:38.583 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_active": 0, 2024-03-16T10:26:38.583 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_send": 0, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_resend": 0, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "command_active": 0, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "command_send": 0, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "command_resend": 0, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "map_epoch": 88, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "map_full": 0, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "map_inc": 3, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_sessions": 6, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_open": 6, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_close": 0, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_laggy": 0, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_wr": 0, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_rd": 2, 2024-03-16T10:26:38.584 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_del": 0 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: "oft": { 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_objs": 0, 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_kv_pairs": 0, 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_updates": 0, 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_removes": 0 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: "osd-slow-ops": { 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_ops_count": 0 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: "purge_queue": { 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops": 0, 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing": 0, 2024-03-16T10:26:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_high_water": 0, 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed_ops": 0, 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed": 0, 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_item_in_journal": 0 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 2255, 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 5579535, 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 2255, 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-16T10:26:38.586 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 2255, 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 5579535, 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_bytes": { 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-16T10:26:38.587 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 2027, 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 4304934, 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 2027, 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 4304934, 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_ops": { 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 1024, 2024-03-16T10:26:38.588 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 2027, 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 2027, 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 2027, 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 2027, 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:38.589 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle": { 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 0, 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 0, 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T10:26:38.590 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle-0x55cf06c906e0": { 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-16T10:26:38.591 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 0, 2024-03-16T10:26:38.592 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 0, 2024-03-16T10:26:38.592 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T10:26:38.592 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:38.592 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:38.592 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:38.592 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:38.592 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:38.592 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:26:39.268 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-16T10:26:39.269 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-16T10:26:39.269 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-03-16T10:26:39.269 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-03-16T10:26:39.269 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-03-16T10:26:39.269 DEBUG:tasks.check_counter:Getting stats from j 2024-03-16T10:26:39.269 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-16T10:26:39.526 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T10:26:40.217 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:39 smithi192 ceph-mon[36354]: pgmap v638: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 op/s 2024-03-16T10:26:40.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:39 smithi077 ceph-mon[32400]: pgmap v638: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 op/s 2024-03-16T10:26:40.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:39 smithi193 ceph-mon[36311]: pgmap v638: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 op/s 2024-03-16T10:26:41.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:41 smithi077 ceph-mon[32400]: pgmap v639: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 12 op/s 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 4, 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 4, 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 3072, 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 448, 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.001469375, 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000032127, 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000574753, 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T10:26:41.245 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:41.246 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 3, 2024-03-16T10:26:41.246 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000779256, 2024-03-16T10:26:41.246 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000259752 2024-03-16T10:26:41.246 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:41.246 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:41.246 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:41.246 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:41.246 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:41.246 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:41.246 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T10:26:41.246 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T10:26:41.246 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:41.247 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T10:26:41.249 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 202, 2024-03-16T10:26:41.249 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 197, 2024-03-16T10:26:41.249 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 746352, 2024-03-16T10:26:41.249 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 120864, 2024-03-16T10:26:41.249 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2024-03-16T10:26:41.249 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2024-03-16T10:26:41.249 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.028439178, 2024-03-16T10:26:41.249 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.010055576, 2024-03-16T10:26:41.249 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.017748908, 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.003896264, 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 196, 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.009391219, 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000047914 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 746352, 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 120864 2024-03-16T10:26:41.250 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 1, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 150, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 160, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 104160, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 3, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 2, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.010324512, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.007721734, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.001161350, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 148, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.006465659, 2024-03-16T10:26:41.251 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000043686 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 104160 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-03-16T10:26:41.252 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-03-16T10:26:41.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-03-16T10:26:41.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 148945, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 29, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-03-16T10:26:41.255 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-03-16T10:26:41.256 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 203, 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 717846, 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 203, 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 203, 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 717846, 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-03-16T10:26:41.257 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-03-16T10:26:41.258 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-03-16T10:26:41.258 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-03-16T10:26:41.258 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:41.258 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-03-16T10:26:41.258 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-03-16T10:26:41.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:41 smithi192 ceph-mon[36354]: pgmap v639: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 12 op/s 2024-03-16T10:26:41.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:41 smithi193 ceph-mon[36311]: pgmap v639: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 12 op/s 2024-03-16T10:26:41.951 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-16T10:26:41.951 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-16T10:26:41.951 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-16T10:26:41.951 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-16T10:26:41.951 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-16T10:26:41.951 DEBUG:tasks.check_counter:Getting stats from b 2024-03-16T10:26:41.951 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-16T10:26:42.239 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:26:44.174 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-16T10:26:44.174 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T10:26:44.174 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 17589, 2024-03-16T10:26:44.174 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 14996, 2024-03-16T10:26:44.174 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 8719330, 2024-03-16T10:26:44.174 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 8844739, 2024-03-16T10:26:44.174 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 39, 2024-03-16T10:26:44.174 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 4, 2024-03-16T10:26:44.174 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 1.102335016, 2024-03-16T10:26:44.174 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.464295627, 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.714758195, 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.064290189, 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 14998, 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.932163010, 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000062152 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 28225, 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.004528559, 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000160 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 963904, 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 141024 2024-03-16T10:26:44.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 3662, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 3898, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 1622992, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 3212327, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 11, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 7, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.250812423, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.087658684, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.086188479, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.067445118, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 3894, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 1.521749846, 2024-03-16T10:26:44.176 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000390793 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2212, 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000375194, 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000169 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 344128 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 34380, 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 38244, 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 10519725, 2024-03-16T10:26:44.177 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 27961162, 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 161, 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 7, 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 1.497178942, 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.695647904, 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 1.110081198, 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.095096954, 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 38242, 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 3.575543794, 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000093497 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:44.178 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 34145, 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.007968140, 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000233 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 95488, 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 34352 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: "cct": { 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: "total_workers": 1, 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: "unhealthy_workers": 0 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-MDSRank": { 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 3015, 2024-03-16T10:26:44.179 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 3.776940668, 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001252716 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-PurgeQueue": { 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2042, 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.169704487, 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000083106 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: "mds": { 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: "request": 20254, 2024-03-16T10:26:44.180 INFO:teuthology.orchestra.run.smithi192.stdout: "reply": 18563, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "reply_latency": { 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 18563, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 924.039797086, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.049778580 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_reply": 0, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "forward": 1691, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_complete": 9, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_keys": 3, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_commit": 626, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_split": 3, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_merge": 0, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes": 31, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_top": 10, 2024-03-16T10:26:44.181 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_bottom": 11, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pin_tail": 10, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pinned": 16, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_expired": 538, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_with_caps": 1, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "caps": 1, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "subtrees": 3, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse": 40592, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_hit": 30529, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_forward": 1691, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_discover": 191, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_dir_fetch": 0, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_remote_ino": 0, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_lock": 232, 2024-03-16T10:26:44.182 INFO:teuthology.orchestra.run.smithi192.stdout: "load_cent": 0, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "q": 0, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "exported": 28, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "exported_inodes": 94, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "imported": 4, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "imported_inodes": 568, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_dir_fetch": 0, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_backtrace_fetch": 2146, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_peer_discover": 0, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rfiles": 4, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rbytes": 234, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rsnaps": 0, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_fetch": 2146, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_set_tag": 0, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T10:26:44.183 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_inotable_repaired": 0, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_inodes": 88, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_base_inodes": 11, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dirfrag_rstats": 77, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_file_inodes": 2058, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_inode_file_caps": 51, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_revoke": 1295, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_grant": 2292, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps": 2443, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps_dirty": 1018, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_cap_release": 104, 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: "process_request_cap_release": 3553 2024-03-16T10:26:44.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_cache": { 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update": 576, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update_receipt": 2765, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_try_discover": 1550, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_send_discover": 1744, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_handle_discover": 484, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays": 0, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_delayed": 0, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_enqueuing": 0, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_created": 1023, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_enqueued": 1023, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_reintegrated": 0, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_migrated": 0, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_processing": 0, 2024-03-16T10:26:44.185 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_enqueued": 0, 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_prioritized": 0, 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_started": 0, 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_completed": 0, 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_quiesce_path": 0, 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_quiesce_inode": 0, 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_enqueue_scrub": 11, 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_exportdir": 28, 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_flush": 0, 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragmentdir": 3, 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragstats": 0, 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_inodestats": 0 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.186 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_log": { 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "evlrg": 0, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "evadd": 5792, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "evex": 5782, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "evtrm": 5782, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "ev": 10, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "evexg": 0, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "evexd": 2, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "segadd": 42, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "segex": 41, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "segtrm": 41, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "seg": 1, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "segmjr": 1, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "segexg": 0, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "segexd": 1, 2024-03-16T10:26:44.187 INFO:teuthology.orchestra.run.smithi192.stdout: "expos": 13628367, 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "wrpos": 13655081, 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "rdpos": 0, 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "jlat": { 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 445, 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 3.464984997, 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.007786483 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "replayed": 0 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_mem": { 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "ino": 34, 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "ino+": 1354, 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "ino-": 1320, 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "dir": 29, 2024-03-16T10:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout: "dir+": 1521, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "dir-": 1492, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "dn": 31, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "dn+": 2780, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "dn-": 2749, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "cap": 1, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "cap+": 1687, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "cap-": 1686, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "rss": 69040, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "heap": 231804 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_server": { 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_client_request": 23039, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_server_request": 39, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_request": 20254, 2024-03-16T10:26:44.189 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_session": 22, 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_peer_request": 1946, 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: "req_create_latency": { 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 805, 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 7.638848848, 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.009489253 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getattr_latency": { 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1034, 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 31.029374162, 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.030009065 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getfilelock_latency": { 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.190 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: "req_link_latency": { 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookup_latency": { 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 14701, 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 874.018375909, 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.059452987 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookuphash_latency": { 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.191 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupino_latency": { 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupname_latency": { 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupparent_latency": { 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupsnap_latency": { 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lssnap_latency": { 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mkdir_latency": { 2024-03-16T10:26:44.193 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 15, 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.016260366, 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001084024 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mknod_latency": { 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mksnap_latency": { 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.194 INFO:teuthology.orchestra.run.smithi192.stdout: "req_open_latency": { 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 603, 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 1.546164906, 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.002564120 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: "req_readdir_latency": { 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 50, 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.303167585, 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.006063351 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rename_latency": { 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 627, 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 8.026217700, 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.012800985 2024-03-16T10:26:44.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: "req_renamesnap_latency": { 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: "req_snapdiff_latency": { 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmdir_latency": { 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 4, 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.086906631, 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.021726657 2024-03-16T10:26:44.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmsnap_latency": { 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmxattr_latency": { 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setattr_latency": { 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1, 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.001437081, 2024-03-16T10:26:44.197 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001437081 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setdirlayout_latency": { 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setfilelock_latency": { 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setlayout_latency": { 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.198 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setxattr_latency": { 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 8, 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.020057183, 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.002507147 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: "req_symlink_latency": { 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1, 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000643040, 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000643040 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: "req_unlink_latency": { 2024-03-16T10:26:44.199 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 714, 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 1.352343675, 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001894038 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_revoke_eviction": 0, 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_acquisition_throttle": 0, 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getvxattr_latency": { 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_sessions": { 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: "session_count": 1, 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: "session_add": 3, 2024-03-16T10:26:44.200 INFO:teuthology.orchestra.run.smithi192.stdout: "session_remove": 2, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_open": 1, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_stale": 0, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "total_load": 1071, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "average_load": 1071, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "avg_session_uptime": 578, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "mdthresh_evicted": 0 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "mempool": { 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_bytes": 0, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_items": 0, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_items": 0, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_items": 0, 2024-03-16T10:26:44.201 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_items": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_bytes": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_items": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_bytes": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_items": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T10:26:44.202 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_items": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_bytes": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_items": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_bytes": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_items": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_bytes": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_items": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_items": 0, 2024-03-16T10:26:44.203 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_items": 0, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_bytes": 314317, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_items": 127, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_bytes": 176, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_items": 2, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_bytes": 0, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_items": 0, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_bytes": 0, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_items": 0, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_bytes": 0, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_items": 0, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_bytes": 7344, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_items": 155, 2024-03-16T10:26:44.204 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_items": 0, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_bytes": 0, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_items": 0, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_bytes": 139178, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_items": 770, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_bytes": 0, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_items": 0, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_bytes": 0, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_items": 0 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "objecter": { 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "op_active": 0, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "op_laggy": 0, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send": 6351, 2024-03-16T10:26:44.205 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send_bytes": 11810795, 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: "op_resend": 27, 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: "op_reply": 6351, 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: "op_latency": { 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 6351, 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 111.481862953, 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.017553434 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: "op_inflight": 0, 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: "oplen_avg": { 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 6351, 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 9654 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: "op": 6351, 2024-03-16T10:26:44.206 INFO:teuthology.orchestra.run.smithi192.stdout: "op_r": 2266, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "op_w": 4085, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "op_rmw": 0, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "op_pg": 0, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_stat": 580, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_create": 955, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_read": 97, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_write": 531, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writefull": 207, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writesame": 0, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_append": 0, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_zero": 0, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_truncate": 3, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_delete": 1539, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_mapext": 0, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_sparse_read": 0, 2024-03-16T10:26:44.207 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_clonerange": 0, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_getxattr": 2169, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_setxattr": 1910, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_cmpxattr": 0, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_rmxattr": 0, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_resetxattrs": 0, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_call": 0, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_watch": 0, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_notify": 0, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls": 0, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls_filter": 0, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_other": 0, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_active": 0, 2024-03-16T10:26:44.208 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_send": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_resend": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_ping": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_active": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_send": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_resend": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_active": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_send": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_resend": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_active": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_send": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_resend": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "command_active": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "command_send": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "command_resend": 0, 2024-03-16T10:26:44.209 INFO:teuthology.orchestra.run.smithi192.stdout: "map_epoch": 88, 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "map_full": 0, 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "map_inc": 4, 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_sessions": 12, 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_open": 12, 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_close": 0, 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_laggy": 0, 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_wr": 1428, 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_rd": 24, 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_del": 211 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "oft": { 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_objs": 1, 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_kv_pairs": 0, 2024-03-16T10:26:44.210 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_updates": 0, 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_removes": 0 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "osd-slow-ops": { 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_ops_count": 0 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "purge_queue": { 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops": 0, 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops_high_water": 6, 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing": 0, 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_high_water": 6, 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed_ops": 1023, 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed": 1023, 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_item_in_journal": 0 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T10:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 55627, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 16321716, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 55627, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 55627, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 16321716, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.212 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_bytes": { 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 6351, 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 11913962, 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 6351, 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 11913962, 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:44.213 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_ops": { 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 1024, 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 6351, 2024-03-16T10:26:44.214 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 6351, 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 6351, 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 6351, 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle": { 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 1023, 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 103323, 2024-03-16T10:26:44.215 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 1023, 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 86, 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 103323, 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle-0x5639db406500": { 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:44.216 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 5792, 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 9460777, 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 5792, 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 445, 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 9460777, 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:44.217 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-16T10:26:44.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:43 smithi077 ceph-mon[32400]: pgmap v640: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 13 op/s 2024-03-16T10:26:44.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:43 smithi193 ceph-mon[36311]: pgmap v640: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 13 op/s 2024-03-16T10:26:44.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:43 smithi192 ceph-mon[36354]: pgmap v640: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 13 op/s 2024-03-16T10:26:45.006 INFO:tasks.check_counter:Daemon mds.b mds.exported=28 2024-03-16T10:26:45.006 INFO:tasks.check_counter:Daemon mds.b mds.imported=4 2024-03-16T10:26:45.006 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=576 2024-03-16T10:26:45.006 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=2765 2024-03-16T10:26:45.006 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=3 2024-03-16T10:26:45.006 DEBUG:tasks.check_counter:Getting stats from e 2024-03-16T10:26:45.007 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-16T10:26:45.392 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:26:46.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:45 smithi192 ceph-mon[36354]: pgmap v641: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T10:26:46.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:45 smithi077 ceph-mon[32400]: pgmap v641: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T10:26:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:45 smithi193 ceph-mon[36311]: pgmap v641: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T10:26:47.192 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:47 smithi192 ceph-mon[36354]: pgmap v642: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 11 op/s 2024-03-16T10:26:47.193 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-16T10:26:47.193 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T10:26:47.193 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 2066, 2024-03-16T10:26:47.193 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 2066, 2024-03-16T10:26:47.193 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 5159457, 2024-03-16T10:26:47.193 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 656171, 2024-03-16T10:26:47.193 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 6, 2024-03-16T10:26:47.193 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 2, 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.123098018, 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.036139684, 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.034647538, 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.043879104, 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2065, 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.040732335, 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000019725 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.194 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 238, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 207, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 936332, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 122922, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 3, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 2, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.028321514, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.008886762, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.020310524, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.003800978, 2024-03-16T10:26:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 206, 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.006515741, 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000031629 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 935776, 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 122224 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.196 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 2, 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 151, 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 428, 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 339998, 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 4, 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 3, 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.007239159, 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.004800664, 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.001400956, 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.000020436, 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 149, 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.005089204, 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000034155 2024-03-16T10:26:47.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 339680 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: "cct": { 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: "total_workers": 1, 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: "unhealthy_workers": 0 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-MDSRank": { 2024-03-16T10:26:47.198 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 3533, 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.070651112, 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000019997 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-PurgeQueue": { 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: "mds": { 2024-03-16T10:26:47.199 INFO:teuthology.orchestra.run.smithi192.stdout: "request": 0, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "reply": 0, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "reply_latency": { 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_reply": 0, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "forward": 0, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_complete": 0, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_keys": 0, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_commit": 0, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_split": 0, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_merge": 0, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes": 602, 2024-03-16T10:26:47.200 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_top": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_bottom": 602, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pin_tail": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pinned": 602, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_expired": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_with_caps": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "caps": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "subtrees": 1, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_hit": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_forward": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_discover": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_dir_fetch": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_remote_ino": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_lock": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "load_cent": 0, 2024-03-16T10:26:47.201 INFO:teuthology.orchestra.run.smithi192.stdout: "q": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "exported": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "exported_inodes": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "imported": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "imported_inodes": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_dir_fetch": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_backtrace_fetch": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_peer_discover": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rfiles": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rbytes": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rsnaps": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_set_tag": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T10:26:47.202 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_inotable_repaired": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_inodes": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_file_inodes": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_inode_file_caps": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_grant": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps_dirty": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_cap_release": 0, 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: "process_request_cap_release": 0 2024-03-16T10:26:47.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_cache": { 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update_receipt": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_try_discover": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_send_discover": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_handle_discover": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_delayed": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_enqueuing": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_created": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_enqueued": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_reintegrated": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_migrated": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_processing": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_enqueued": 0, 2024-03-16T10:26:47.204 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_prioritized": 0, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_started": 0, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_completed": 0, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_quiesce_path": 0, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_quiesce_inode": 0, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_exportdir": 0, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_flush": 0, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragmentdir": 0, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragstats": 0, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_inodestats": 0 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_log": { 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "evlrg": 0, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "evadd": 0, 2024-03-16T10:26:47.205 INFO:teuthology.orchestra.run.smithi192.stdout: "evex": 0, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "evtrm": 0, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "ev": 3156, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "evexg": 0, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "evexd": 0, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "segadd": 0, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "segex": 0, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "segtrm": 0, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "seg": 5, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "segmjr": 2, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "segexg": 0, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "segexd": 0, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "expos": 4194304, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "wrpos": 8767572, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "rdpos": 8766434, 2024-03-16T10:26:47.206 INFO:teuthology.orchestra.run.smithi192.stdout: "jlat": { 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "replayed": 3156 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_mem": { 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "ino": 7, 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "ino+": 623, 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "ino-": 616, 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "dir": 5, 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "dir+": 49, 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "dir-": 44, 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "dn": 602, 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "dn+": 1412, 2024-03-16T10:26:47.207 INFO:teuthology.orchestra.run.smithi192.stdout: "dn-": 810, 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "cap": 0, 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "cap+": 0, 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "cap-": 0, 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "rss": 53432, 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "heap": 215420 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_server": { 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_client_request": 0, 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_server_request": 0, 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_request": 0, 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_session": 0, 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_peer_request": 0, 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "req_create_latency": { 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.208 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getattr_latency": { 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getfilelock_latency": { 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "req_link_latency": { 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookup_latency": { 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookuphash_latency": { 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupino_latency": { 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupname_latency": { 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupparent_latency": { 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupsnap_latency": { 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lssnap_latency": { 2024-03-16T10:26:47.211 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.212 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.212 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.212 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mkdir_latency": { 2024-03-16T10:26:47.212 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.212 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.212 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mknod_latency": { 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mksnap_latency": { 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: "req_open_latency": { 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.213 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: "req_readdir_latency": { 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rename_latency": { 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: "req_renamesnap_latency": { 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.214 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: "req_snapdiff_latency": { 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmdir_latency": { 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmsnap_latency": { 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.215 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmxattr_latency": { 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setattr_latency": { 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setdirlayout_latency": { 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.216 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setfilelock_latency": { 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setlayout_latency": { 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setxattr_latency": { 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.217 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: "req_symlink_latency": { 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: "req_unlink_latency": { 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_revoke_eviction": 0, 2024-03-16T10:26:47.218 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_acquisition_throttle": 0, 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getvxattr_latency": { 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_sessions": { 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: "session_count": 1, 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: "session_add": 2, 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: "session_remove": 1, 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_open": 2, 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_stale": 0, 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: "total_load": 0, 2024-03-16T10:26:47.219 INFO:teuthology.orchestra.run.smithi192.stdout: "average_load": 0, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "avg_session_uptime": 3203, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "mdthresh_evicted": 0 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "mempool": { 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_bytes": 0, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_items": 0, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_items": 0, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_items": 0, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T10:26:47.220 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_items": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_bytes": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_items": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_bytes": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_items": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_items": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_bytes": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_items": 0, 2024-03-16T10:26:47.221 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_bytes": 0, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_items": 0, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_bytes": 0, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_items": 0, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_items": 0, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_items": 0, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_bytes": 181383, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_items": 48, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_bytes": 0, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_items": 0, 2024-03-16T10:26:47.222 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_bytes": 0, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_items": 0, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_bytes": 0, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_items": 0, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_bytes": 0, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_items": 0, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_bytes": 7344, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_items": 155, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_items": 0, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_bytes": 0, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_items": 0, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_bytes": 446264, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_items": 2470, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_bytes": 0, 2024-03-16T10:26:47.223 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_items": 0, 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_bytes": 0, 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_items": 0 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "objecter": { 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "op_active": 0, 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "op_laggy": 0, 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send": 2065, 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send_bytes": 21, 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "op_resend": 3, 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "op_reply": 2065, 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "op_latency": { 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2065, 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 3.157939377, 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001529268 2024-03-16T10:26:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "op_inflight": 0, 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "oplen_avg": { 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2065, 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 2066 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "op": 2065, 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "op_r": 2065, 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "op_w": 0, 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "op_rmw": 0, 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "op_pg": 0, 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_stat": 1278, 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_create": 0, 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_read": 785, 2024-03-16T10:26:47.225 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_write": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writefull": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writesame": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_append": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_zero": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_truncate": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_delete": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_mapext": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_sparse_read": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_clonerange": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_getxattr": 1, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_setxattr": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_cmpxattr": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_rmxattr": 0, 2024-03-16T10:26:47.226 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_resetxattrs": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_call": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_watch": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_notify": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls_filter": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_other": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_active": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_send": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_resend": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_ping": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_active": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_send": 0, 2024-03-16T10:26:47.227 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_resend": 0, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_active": 0, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_send": 0, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_resend": 0, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_active": 0, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_send": 0, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_resend": 0, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "command_active": 0, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "command_send": 0, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "command_resend": 0, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "map_epoch": 88, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "map_full": 0, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "map_inc": 3, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_sessions": 4, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_open": 4, 2024-03-16T10:26:47.228 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_close": 0, 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_laggy": 0, 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_wr": 0, 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_rd": 2, 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_del": 0 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: "oft": { 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_objs": 0, 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_kv_pairs": 0, 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_updates": 0, 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_removes": 0 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: "osd-slow-ops": { 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_ops_count": 0 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.229 INFO:teuthology.orchestra.run.smithi192.stdout: "purge_queue": { 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops": 0, 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing": 0, 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_high_water": 0, 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed_ops": 0, 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed": 0, 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_item_in_journal": 0 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 2302, 2024-03-16T10:26:47.230 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 5873988, 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 2302, 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2302, 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 5873988, 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_bytes": { 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:47.231 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 2065, 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 4573273, 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2065, 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 4573273, 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_ops": { 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 1024, 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 2065, 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 2065, 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2065, 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 2065, 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle": { 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 0, 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 0, 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle-0x55bd5f1446e0": { 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-16T10:26:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 0, 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 0, 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:47.236 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-16T10:26:47.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:47 smithi077 ceph-mon[32400]: pgmap v642: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 11 op/s 2024-03-16T10:26:47.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:47 smithi193 ceph-mon[36311]: pgmap v642: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 11 op/s 2024-03-16T10:26:47.934 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-16T10:26:47.934 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-16T10:26:47.934 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-16T10:26:47.934 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-16T10:26:47.934 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-16T10:26:47.934 DEBUG:tasks.check_counter:Getting stats from h 2024-03-16T10:26:47.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-16T10:26:48.190 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:26:49.957 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-16T10:26:49.957 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T10:26:49.957 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 65233, 2024-03-16T10:26:49.957 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 71716, 2024-03-16T10:26:49.957 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 19906985, 2024-03-16T10:26:49.957 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 205697783, 2024-03-16T10:26:49.957 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 10, 2024-03-16T10:26:49.957 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 4, 2024-03-16T10:26:49.957 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 5.383748666, 2024-03-16T10:26:49.957 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 2.360747752, 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 1.594024544, 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 1.299510954, 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 71718, 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 14.723729817, 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000205300 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 46311, 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.007690192, 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000166 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T10:26:49.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 81858, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 85274, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 26479622, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 368300370, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 8, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 7, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 6.417027641, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 2.401955992, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 1.547751829, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 2.126011411, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 85274, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 19.617421928, 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000230051 2024-03-16T10:26:49.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 21302, 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.004702853, 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000220 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 954816, 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 135904 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 364506, 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 567342, 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 114493199, 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 373361328, 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 35, 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 11, 2024-03-16T10:26:49.960 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 18.714938820, 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 9.791964924, 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 39.560672408, 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 1.455454129, 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 567336, 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 33.991300099, 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000059913 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 480281, 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.088785060, 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000184 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 343008 2024-03-16T10:26:49.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: "cct": { 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: "total_workers": 1, 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: "unhealthy_workers": 0 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-MDSRank": { 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 33120, 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 352.818737158, 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.010652739 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-PurgeQueue": { 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-16T10:26:49.962 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 83508, 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 47.046110135, 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000563372 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "mds": { 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "request": 191507, 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "reply": 188238, 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "reply_latency": { 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 188238, 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 6073.030864677, 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.032262512 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_reply": 0, 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "forward": 3269, 2024-03-16T10:26:49.963 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_complete": 1, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_keys": 0, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_commit": 2282, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_split": 164, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_merge": 22, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes": 1703, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_top": 1166, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_bottom": 524, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pin_tail": 13, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pinned": 37, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_expired": 2607, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_with_caps": 0, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "caps": 0, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "subtrees": 7, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse": 297461, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_hit": 275241, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_forward": 3245, 2024-03-16T10:26:49.964 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_discover": 52, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_dir_fetch": 0, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_remote_ino": 0, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_lock": 1745, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "load_cent": 0, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "q": 0, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "exported": 14, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "exported_inodes": 1506, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "imported": 80, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "imported_inodes": 178, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_dir_fetch": 0, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_backtrace_fetch": 90058, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_peer_discover": 0, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rfiles": 736, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rbytes": 15642193, 2024-03-16T10:26:49.965 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rsnaps": 0, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_fetch": 90058, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_set_tag": 0, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_inotable_repaired": 0, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_inodes": 643, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dirfrag_rstats": 643, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_file_inodes": 89415, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_inode_file_caps": 26, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_revoke": 30830, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_grant": 102059, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_trunc": 7, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T10:26:49.966 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps": 99842, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps_dirty": 48449, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_cap_release": 6025, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "process_request_cap_release": 100929 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_cache": { 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update": 4427, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update_receipt": 16, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_try_discover": 10, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_send_discover": 39, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_handle_discover": 7413, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays": 4, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_delayed": 0, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_enqueuing": 0, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_created": 47030, 2024-03-16T10:26:49.967 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_enqueued": 47026, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_reintegrated": 0, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_migrated": 0, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_processing": 0, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_enqueued": 0, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_prioritized": 0, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_started": 0, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_completed": 0, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_quiesce_path": 0, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_quiesce_inode": 0, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_exportdir": 14, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_flush": 0, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragmentdir": 187, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragstats": 0, 2024-03-16T10:26:49.968 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_inodestats": 0 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_log": { 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "evlrg": 0, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "evadd": 255197, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "evex": 155198, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "evtrm": 147481, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "ev": 107716, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "evexg": 0, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "evexd": 7717, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "segadd": 329, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "segex": 201, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "segtrm": 192, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "seg": 137, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "segmjr": 9, 2024-03-16T10:26:49.969 INFO:teuthology.orchestra.run.smithi192.stdout: "segexg": 0, 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: "segexd": 9, 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: "expos": 363507248, 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: "wrpos": 575932070, 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: "rdpos": 0, 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: "jlat": { 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 3576, 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 689.120374717, 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.192707039 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: "replayed": 0 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_mem": { 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: "ino": 36, 2024-03-16T10:26:49.970 INFO:teuthology.orchestra.run.smithi192.stdout: "ino+": 47793, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "ino-": 47757, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "dir": 38, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "dir+": 1630, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "dir-": 1592, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "dn": 1703, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "dn+": 107580, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "dn-": 105877, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "cap": 0, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "cap+": 49201, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "cap-": 49201, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "rss": 156864, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "heap": 223612 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_server": { 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_client_request": 199277, 2024-03-16T10:26:49.971 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_server_request": 5834, 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_request": 191507, 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_session": 20, 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_peer_request": 11919, 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "req_create_latency": { 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 47486, 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 983.829813594, 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.020718313 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getattr_latency": { 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 5659, 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 1479.531750242, 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.261447561 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getfilelock_latency": { 2024-03-16T10:26:49.972 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "req_link_latency": { 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookup_latency": { 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 49954, 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 2120.857530021, 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.042456210 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookuphash_latency": { 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.973 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupino_latency": { 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupname_latency": { 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupparent_latency": { 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.975 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupsnap_latency": { 2024-03-16T10:26:49.975 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.975 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.975 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.975 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lssnap_latency": { 2024-03-16T10:26:49.975 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mkdir_latency": { 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 179, 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 5.747613436, 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.032109572 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mknod_latency": { 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mksnap_latency": { 2024-03-16T10:26:49.976 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: "req_open_latency": { 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 8402, 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 176.079404275, 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.020956844 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: "req_readdir_latency": { 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1243, 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 11.502497140, 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.009253819 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rename_latency": { 2024-03-16T10:26:49.977 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 47282, 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 1212.820642592, 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.025650789 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "req_renamesnap_latency": { 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "req_snapdiff_latency": { 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmdir_latency": { 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 184, 2024-03-16T10:26:49.978 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 28.284841198, 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.153721963 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmsnap_latency": { 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmxattr_latency": { 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setattr_latency": { 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 7, 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.800891711, 2024-03-16T10:26:49.979 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.114413101 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setdirlayout_latency": { 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setfilelock_latency": { 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setlayout_latency": { 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setxattr_latency": { 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "req_symlink_latency": { 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2, 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.002240163, 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001120081 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "req_unlink_latency": { 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 27840, 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 53.573640305, 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001924340 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.981 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_revoke_eviction": 0, 2024-03-16T10:26:49.982 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_acquisition_throttle": 0, 2024-03-16T10:26:49.982 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getvxattr_latency": { 2024-03-16T10:26:49.982 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.982 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.982 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.982 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:49.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.982 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_sessions": { 2024-03-16T10:26:49.982 INFO:teuthology.orchestra.run.smithi192.stdout: "session_count": 1, 2024-03-16T10:26:49.982 INFO:teuthology.orchestra.run.smithi192.stdout: "session_add": 1, 2024-03-16T10:26:49.982 INFO:teuthology.orchestra.run.smithi192.stdout: "session_remove": 0, 2024-03-16T10:26:49.982 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_open": 1, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_stale": 0, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "total_load": 38490, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "average_load": 38490, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "avg_session_uptime": 494, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "mdthresh_evicted": 0 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "mempool": { 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_bytes": 0, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_items": 0, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_items": 0, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_items": 0, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T10:26:49.983 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_items": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_bytes": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_items": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_bytes": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_items": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_items": 0, 2024-03-16T10:26:49.984 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_bytes": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_items": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_bytes": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_items": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_bytes": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_items": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_items": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_items": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_bytes": 307929, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_items": 134, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_bytes": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_items": 0, 2024-03-16T10:26:49.985 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_bytes": 0, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_items": 0, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_bytes": 0, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_items": 0, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_bytes": 0, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_items": 0, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_bytes": 7344, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_items": 155, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_items": 0, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_bytes": 0, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_items": 0, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_bytes": 1430110, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_items": 19684, 2024-03-16T10:26:49.986 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_bytes": 0, 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_items": 0, 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_bytes": 0, 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_items": 0 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "objecter": { 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "op_active": 0, 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "op_laggy": 0, 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send": 164619, 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send_bytes": 628308864, 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "op_resend": 28, 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "op_reply": 164619, 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "op_latency": { 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 164619, 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 1302.702143031, 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.007913437 2024-03-16T10:26:49.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "op_inflight": 0, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "oplen_avg": { 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 164619, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 202456 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "op": 164619, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "op_r": 91235, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "op_w": 73384, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "op_rmw": 0, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "op_pg": 0, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_stat": 776, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_create": 16516, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_read": 1173, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_write": 4749, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writefull": 536, 2024-03-16T10:26:49.988 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writesame": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_append": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_zero": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_truncate": 164, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_delete": 48647, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_mapext": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_sparse_read": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_clonerange": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_getxattr": 90059, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_setxattr": 33032, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_cmpxattr": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_rmxattr": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_resetxattrs": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_call": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_watch": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_notify": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T10:26:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls_filter": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_other": 7, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_active": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_send": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_resend": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_ping": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_active": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_send": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_resend": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_active": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_send": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_resend": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_active": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_send": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_resend": 0, 2024-03-16T10:26:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: "command_active": 0, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "command_send": 0, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "command_resend": 0, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "map_epoch": 88, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "map_full": 0, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "map_inc": 3, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_sessions": 12, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_open": 12, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_close": 0, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_laggy": 0, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_wr": 5439, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_rd": 5, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_del": 1353 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "oft": { 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_objs": 1, 2024-03-16T10:26:49.991 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_kv_pairs": 0, 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_updates": 0, 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_removes": 0 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "osd-slow-ops": { 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_ops_count": 0 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "purge_queue": { 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops": 0, 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops_high_water": 14, 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing": 0, 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_high_water": 6, 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed_ops": 48274, 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed": 47026, 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_item_in_journal": 0 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T10:26:49.992 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 511593, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 119731738, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 511593, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 511593, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 119731738, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.993 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_bytes": { 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 164619, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 633059629, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 164619, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 633059629, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.994 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_ops": { 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 1024, 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 164619, 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 164619, 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 164619, 2024-03-16T10:26:49.995 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 164619, 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle": { 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 47026, 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 4750874, 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 47026, 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-16T10:26:49.996 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 1173, 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 4750874, 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle-0x556e5ce1a500": { 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 255197, 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 571737766, 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:49.997 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 255197, 2024-03-16T10:26:49.998 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-16T10:26:49.998 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-16T10:26:49.998 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 3576, 2024-03-16T10:26:49.998 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 571737766, 2024-03-16T10:26:49.998 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:49.998 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:49.998 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:49.998 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:49.998 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:49.998 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:49.998 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-16T10:26:50.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:49 smithi077 ceph-mon[32400]: pgmap v643: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 op/s 2024-03-16T10:26:50.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:49 smithi193 ceph-mon[36311]: pgmap v643: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 op/s 2024-03-16T10:26:50.342 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:49 smithi192 ceph-mon[36354]: pgmap v643: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 op/s 2024-03-16T10:26:50.873 INFO:tasks.check_counter:Daemon mds.h mds.exported=14 2024-03-16T10:26:50.873 INFO:tasks.check_counter:Daemon mds.h mds.imported=80 2024-03-16T10:26:50.873 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=4427 2024-03-16T10:26:50.873 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=16 2024-03-16T10:26:50.873 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=164 2024-03-16T10:26:50.873 DEBUG:tasks.check_counter:Getting stats from k 2024-03-16T10:26:50.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-16T10:26:51.134 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T10:26:52.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:51 smithi077 ceph-mon[32400]: pgmap v644: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 12 op/s 2024-03-16T10:26:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:51 smithi193 ceph-mon[36311]: pgmap v644: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 12 op/s 2024-03-16T10:26:52.280 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:51 smithi192 ceph-mon[36354]: pgmap v644: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 12 op/s 2024-03-16T10:26:52.901 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-16T10:26:52.901 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T10:26:52.901 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 1493, 2024-03-16T10:26:52.901 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 1493, 2024-03-16T10:26:52.901 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 4287639, 2024-03-16T10:26:52.901 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 473951, 2024-03-16T10:26:52.901 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 6, 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 2, 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.101558854, 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.032891269, 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.029005070, 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.031393769, 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1492, 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.035461869, 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000023768 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.902 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 626, 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 777, 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 154562, 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 540466, 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 5, 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 4, 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.048902138, 2024-03-16T10:26:52.903 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.016445221, 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.010301634, 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.017589323, 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 775, 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.020091898, 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000025925 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T10:26:52.904 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 341648 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 241, 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 210, 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 937132, 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 125102, 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 3, 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 2, 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.028687345, 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.008654265, 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.021551697, 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.003740014, 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:52.905 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 209, 2024-03-16T10:26:52.906 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.005955743, 2024-03-16T10:26:52.906 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000028496 2024-03-16T10:26:52.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.906 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:52.906 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.906 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.906 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.906 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 936864, 2024-03-16T10:26:52.906 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 124784 2024-03-16T10:26:52.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.906 INFO:teuthology.orchestra.run.smithi192.stdout: "cct": { 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "total_workers": 1, 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "unhealthy_workers": 0 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-MDSRank": { 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 3554, 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.078393006, 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000022057 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-PurgeQueue": { 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.907 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "mds": { 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "request": 0, 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "reply": 0, 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "reply_latency": { 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_reply": 0, 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "forward": 0, 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_complete": 0, 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_keys": 0, 2024-03-16T10:26:52.908 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_commit": 0, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_split": 0, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_merge": 0, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes": 555, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_top": 0, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_bottom": 555, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pin_tail": 0, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pinned": 555, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_expired": 0, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_with_caps": 0, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "caps": 0, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "subtrees": 1, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse": 0, 2024-03-16T10:26:52.909 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_hit": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_forward": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_discover": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_dir_fetch": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_remote_ino": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_lock": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "load_cent": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "q": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "exported": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "exported_inodes": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "imported": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "imported_inodes": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_dir_fetch": 0, 2024-03-16T10:26:52.910 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_backtrace_fetch": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_peer_discover": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rfiles": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rbytes": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rsnaps": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_set_tag": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_inotable_repaired": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_inodes": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_file_inodes": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_inode_file_caps": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_grant": 0, 2024-03-16T10:26:52.911 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps": 0, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps_dirty": 0, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_cap_release": 0, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "process_request_cap_release": 0 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_cache": { 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update": 0, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update_receipt": 0, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_try_discover": 0, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_send_discover": 0, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_handle_discover": 0, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays": 0, 2024-03-16T10:26:52.912 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_delayed": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_enqueuing": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_created": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_enqueued": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_reintegrated": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_migrated": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_processing": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_enqueued": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_prioritized": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_started": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_completed": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_quiesce_path": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_quiesce_inode": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_exportdir": 0, 2024-03-16T10:26:52.913 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_flush": 0, 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragmentdir": 0, 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragstats": 0, 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_inodestats": 0 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_log": { 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "evlrg": 0, 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "evadd": 0, 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "evex": 0, 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "evtrm": 0, 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "ev": 2703, 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "evexg": 0, 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "evexd": 0, 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "segadd": 0, 2024-03-16T10:26:52.914 INFO:teuthology.orchestra.run.smithi192.stdout: "segex": 0, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "segtrm": 0, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "seg": 4, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "segmjr": 2, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "segexg": 0, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "segexd": 0, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "expos": 4194304, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "wrpos": 8038024, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "rdpos": 8036886, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "jlat": { 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "replayed": 2703 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.915 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_mem": { 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "ino": 6, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "ino+": 577, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "ino-": 571, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "dir": 4, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "dir+": 49, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "dir-": 45, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "dn": 555, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "dn+": 1505, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "dn-": 950, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "cap": 0, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "cap+": 0, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "cap-": 0, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "rss": 50584, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "heap": 215420 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_server": { 2024-03-16T10:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_client_request": 0, 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_server_request": 0, 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_request": 0, 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_session": 0, 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_peer_request": 0, 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "req_create_latency": { 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getattr_latency": { 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getfilelock_latency": { 2024-03-16T10:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: "req_link_latency": { 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookup_latency": { 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.918 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookuphash_latency": { 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupino_latency": { 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupname_latency": { 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.919 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupparent_latency": { 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupsnap_latency": { 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lssnap_latency": { 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.920 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mkdir_latency": { 2024-03-16T10:26:52.921 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.921 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.921 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.921 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mknod_latency": { 2024-03-16T10:26:52.921 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.921 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mksnap_latency": { 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: "req_open_latency": { 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: "req_readdir_latency": { 2024-03-16T10:26:52.922 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rename_latency": { 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: "req_renamesnap_latency": { 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: "req_snapdiff_latency": { 2024-03-16T10:26:52.923 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmdir_latency": { 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmsnap_latency": { 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmxattr_latency": { 2024-03-16T10:26:52.924 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setattr_latency": { 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setdirlayout_latency": { 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setfilelock_latency": { 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.925 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setlayout_latency": { 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setxattr_latency": { 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: "req_symlink_latency": { 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.926 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: "req_unlink_latency": { 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_revoke_eviction": 0, 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_acquisition_throttle": 0, 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getvxattr_latency": { 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:52.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_sessions": { 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "session_count": 1, 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "session_add": 2, 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "session_remove": 1, 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_open": 1, 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_stale": 0, 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "total_load": 0, 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "average_load": 0, 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "avg_session_uptime": 1898, 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "mdthresh_evicted": 0 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "mempool": { 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_bytes": 0, 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_items": 0, 2024-03-16T10:26:52.928 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_items": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_items": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_items": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_bytes": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_items": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_bytes": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_items": 0, 2024-03-16T10:26:52.929 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_items": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_bytes": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_items": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_bytes": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_items": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_bytes": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_items": 0, 2024-03-16T10:26:52.930 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_items": 0, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_items": 0, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_bytes": 189400, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_items": 51, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_bytes": 0, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_items": 0, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_bytes": 0, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_items": 0, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_bytes": 0, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_items": 0, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_bytes": 0, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_items": 0, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_bytes": 7344, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_items": 155, 2024-03-16T10:26:52.931 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_items": 0, 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_bytes": 0, 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_items": 0, 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_bytes": 409688, 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_items": 2271, 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_bytes": 0, 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_items": 0, 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_bytes": 0, 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_items": 0 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "objecter": { 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "op_active": 0, 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "op_laggy": 0, 2024-03-16T10:26:52.932 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send": 2115, 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send_bytes": 21, 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: "op_resend": 3, 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: "op_reply": 2115, 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: "op_latency": { 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2115, 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 1.593969441, 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000753649 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: "op_inflight": 0, 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: "oplen_avg": { 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2115, 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 2116 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.933 INFO:teuthology.orchestra.run.smithi192.stdout: "op": 2115, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "op_r": 2115, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "op_w": 0, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "op_rmw": 0, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "op_pg": 0, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_stat": 1325, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_create": 0, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_read": 788, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_write": 0, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writefull": 0, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writesame": 0, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_append": 0, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_zero": 0, 2024-03-16T10:26:52.934 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_truncate": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_delete": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_mapext": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_sparse_read": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_clonerange": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_getxattr": 1, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_setxattr": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_cmpxattr": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_rmxattr": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_resetxattrs": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_call": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_watch": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_notify": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls": 0, 2024-03-16T10:26:52.935 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls_filter": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_other": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_active": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_send": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_resend": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_ping": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_active": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_send": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_resend": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_active": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_send": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_resend": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_active": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_send": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_resend": 0, 2024-03-16T10:26:52.936 INFO:teuthology.orchestra.run.smithi192.stdout: "command_active": 0, 2024-03-16T10:26:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: "command_send": 0, 2024-03-16T10:26:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: "command_resend": 0, 2024-03-16T10:26:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: "map_epoch": 88, 2024-03-16T10:26:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: "map_full": 0, 2024-03-16T10:26:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: "map_inc": 3, 2024-03-16T10:26:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_sessions": 5, 2024-03-16T10:26:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_open": 5, 2024-03-16T10:26:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_close": 0, 2024-03-16T10:26:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_laggy": 0, 2024-03-16T10:26:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_wr": 0, 2024-03-16T10:26:52.937 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_rd": 2, 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_del": 0 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: "oft": { 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_objs": 0, 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_kv_pairs": 0, 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_updates": 0, 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_removes": 0 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: "osd-slow-ops": { 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_ops_count": 0 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: "purge_queue": { 2024-03-16T10:26:52.938 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops": 0, 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing": 0, 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_high_water": 0, 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed_ops": 0, 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed": 0, 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_item_in_journal": 0 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 2356, 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 5152036, 2024-03-16T10:26:52.939 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 2356, 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2356, 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 5152036, 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_bytes": { 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:52.940 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 2115, 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 3843725, 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2115, 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 3843725, 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_ops": { 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:52.941 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 1024, 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 2115, 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 2115, 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2115, 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 2115, 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.942 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle": { 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 0, 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 0, 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:52.943 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle-0x55daa187c6e0": { 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 0, 2024-03-16T10:26:52.944 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 0, 2024-03-16T10:26:52.945 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-16T10:26:52.945 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-16T10:26:52.945 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-16T10:26:52.945 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-16T10:26:52.945 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:52.945 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-16T10:26:52.945 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-16T10:26:53.682 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-16T10:26:53.683 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-16T10:26:53.683 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-03-16T10:26:53.683 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-03-16T10:26:53.683 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-03-16T10:26:53.683 DEBUG:tasks.check_counter:Getting stats from c 2024-03-16T10:26:53.684 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-16T10:26:53.937 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:26:54.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:53 smithi193 ceph-mon[36311]: pgmap v645: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 12 op/s 2024-03-16T10:26:54.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:53 smithi077 ceph-mon[32400]: pgmap v645: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 12 op/s 2024-03-16T10:26:54.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:53 smithi192 ceph-mon[36354]: pgmap v645: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 12 op/s 2024-03-16T10:26:55.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:55 smithi193 ceph-mon[36311]: pgmap v646: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T10:26:55.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:55 smithi192 ceph-mon[36354]: pgmap v646: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T10:26:55.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:55 smithi077 ceph-mon[32400]: pgmap v646: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T10:26:55.618 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-16T10:26:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T10:26:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 14688, 2024-03-16T10:26:55.618 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 11327, 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 7383401, 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 6967980, 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 12, 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 6, 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.694555021, 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.272469914, 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.542023601, 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.019036598, 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 11323, 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.295790492, 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000026122 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.619 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 23924, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.003251176, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000135 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 948672, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 146704 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1843, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1902, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 618313, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 1231560, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 9, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 6, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.117509244, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.046491266, 2024-03-16T10:26:55.620 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.056491145, 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.016493597, 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1904, 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.065765356, 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000034540 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2206, 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000338040, 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000153 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 344160 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T10:26:55.621 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 11811, 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 14243, 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 3545384, 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 7340085, 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 33, 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 6, 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.477199628, 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.229704860, 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.355749430, 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.012578374, 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 14241, 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.599302214, 2024-03-16T10:26:55.622 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000042082 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 12520, 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.002996127, 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000239 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 928, 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 736 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-03-16T10:26:55.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1123, 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 1.327021559, 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001181675 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1162, 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.076734698, 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000066036 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:55.624 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 7633, 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 5929, 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 5929, 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 52.177782292, 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.008800435 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 1704, 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 29, 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 2, 2024-03-16T10:26:55.625 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 25, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 6, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 9, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 10, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 13, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 518, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 4, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 16634, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 10884, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 1704, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 258, 2024-03-16T10:26:55.626 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 138, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 18, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 73, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 4, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 391, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 803, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 736, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 15642193, 2024-03-16T10:26:55.627 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 0, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 803, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 803, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 15, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 415, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 1046, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T10:26:55.628 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 940, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 522, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 34, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 2198 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 248, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 3914, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 2199, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 2474, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 204, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 552, 2024-03-16T10:26:55.629 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 552, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 18, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 2, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-03-16T10:26:55.630 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 2703, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 2703, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 4, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 4, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 2, 2024-03-16T10:26:55.631 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194304, 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 8038024, 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 0, 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 180, 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 2.935367796, 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.016307598 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 0 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 30, 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 829, 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 799, 2024-03-16T10:26:55.632 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 29, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 1988, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 1959, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 25, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 1945, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 1920, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 718, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 718, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 62152, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 223612 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 9458, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 124, 2024-03-16T10:26:55.633 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 7633, 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 11, 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 1526, 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 391, 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 3.044039512, 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.007785267 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-03-16T10:26:55.634 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 4413, 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 44.139226463, 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.010002090 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.635 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-03-16T10:26:55.636 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-03-16T10:26:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-03-16T10:26:55.637 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 258, 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.204270731, 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000791747 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 18, 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.024641991, 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001368999 2024-03-16T10:26:55.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 391, 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 3.503110584, 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.008959362 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.641 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.642 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 458, 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 1.262493011, 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.002756534 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-03-16T10:26:55.644 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 1, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 2, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 1, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 1, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 1089, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 1089, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 496, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-03-16T10:26:55.645 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-03-16T10:26:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-03-16T10:26:55.647 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 291684, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 118, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 440, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 5, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 7344, 2024-03-16T10:26:55.648 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 155, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 122237, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 476, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-03-16T10:26:55.649 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 1827, 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 4293351, 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 31, 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 1827, 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1827, 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 8.968863411, 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.004909065 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1827, 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 1979 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 1827, 2024-03-16T10:26:55.650 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 850, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 977, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 0, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 10, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 47, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 227, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 54, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 2, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 564, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-03-16T10:26:55.651 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 803, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 20, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-03-16T10:26:55.652 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 88, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-03-16T10:26:55.653 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 3, 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 12, 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 12, 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 207, 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 0, 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 45 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 1, 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.654 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 6, 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 6, 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 552, 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 552, 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-16T10:26:55.655 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 28338, 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 9154582, 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 28338, 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 28338, 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 9154582, 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-03-16T10:26:55.656 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 1827, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 4349103, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 1827, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 4349103, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.657 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 1827, 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 1827, 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 1827, 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 1827, 2024-03-16T10:26:55.658 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 552, 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 55752, 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 552, 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-16T10:26:55.659 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 47, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 55752, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x555d77026500": { 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 2703, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 3843720, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 2703, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-16T10:26:55.660 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-16T10:26:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 180, 2024-03-16T10:26:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 3843720, 2024-03-16T10:26:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:26:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:55.661 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:55.661 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-16T10:26:56.452 INFO:tasks.check_counter:Daemon mds.c mds.exported=18 2024-03-16T10:26:56.452 INFO:tasks.check_counter:Daemon mds.c mds.imported=4 2024-03-16T10:26:56.452 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=248 2024-03-16T10:26:56.452 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=3914 2024-03-16T10:26:56.452 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=2 2024-03-16T10:26:56.452 DEBUG:tasks.check_counter:Getting stats from f 2024-03-16T10:26:56.452 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-16T10:26:56.784 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:26:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:57 smithi193 ceph-mon[36311]: pgmap v647: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 11 op/s 2024-03-16T10:26:58.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:57 smithi077 ceph-mon[32400]: pgmap v647: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 11 op/s 2024-03-16T10:26:58.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:57 smithi192 ceph-mon[36354]: pgmap v647: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 11 op/s 2024-03-16T10:26:58.479 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-16T10:26:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T10:26:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 201, 2024-03-16T10:26:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 201, 2024-03-16T10:26:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 69615, 2024-03-16T10:26:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 63104, 2024-03-16T10:26:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 6, 2024-03-16T10:26:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 2, 2024-03-16T10:26:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.016189313, 2024-03-16T10:26:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.004459889, 2024-03-16T10:26:58.479 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.004793199, 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.004881087, 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 200, 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.007478966, 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000037394 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.480 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 356, 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 397, 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 1164065, 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 270412, 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 5, 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.022922585, 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.007465788, 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.006302487, 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.007433341, 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 395, 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.018559062, 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000046984 2024-03-16T10:26:58.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 157456 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 153, 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 142, 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 2521841, 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 59745, 2024-03-16T10:26:58.482 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 3, 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 2, 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.013605266, 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.003958889, 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.007572965, 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.002435426, 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 141, 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.004365449, 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000030960 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 239360, 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 34304 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1052, 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.030046622, 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000028561 2024-03-16T10:26:58.484 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.485 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 592, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 407, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 185, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-03-16T10:26:58.486 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 10, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 2, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-03-16T10:26:58.487 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 4, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 234, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-03-16T10:26:58.488 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-03-16T10:26:58.489 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-03-16T10:26:58.490 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 2028, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-03-16T10:26:58.491 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 2, 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 11, 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 13628367, 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 13655081, 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 13651323, 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 2028 2024-03-16T10:26:58.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 11, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 606, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 595, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 10, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 45, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 35, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 592, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 1226, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 634, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 48144, 2024-03-16T10:26:58.493 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 215420 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.494 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.495 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.496 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-03-16T10:26:58.498 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.498 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.498 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.498 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.498 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-03-16T10:26:58.498 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-03-16T10:26:58.499 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.500 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.501 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.503 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-03-16T10:26:58.504 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 1, 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 1, 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 0, 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 0, 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 1900, 2024-03-16T10:26:58.505 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T10:26:58.506 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-03-16T10:26:58.507 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 189239, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 48, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-03-16T10:26:58.508 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 7344, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 155, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 454533, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 2504, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-03-16T10:26:58.509 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 632, 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 26, 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 4, 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 632, 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 632, 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.274995795, 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000435119 2024-03-16T10:26:58.510 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 632, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 633 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 632, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 632, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 388, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 241, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-03-16T10:26:58.511 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 2, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T10:26:58.512 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-03-16T10:26:58.513 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 0, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 0, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 5, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 5, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.514 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 0, 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-03-16T10:26:58.515 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 706, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 3685205, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 706, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 706, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 3685205, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.516 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 632, 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 3333949, 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 632, 2024-03-16T10:26:58.517 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 3333949, 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:58.518 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 632, 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 632, 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 632, 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 632, 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-16T10:26:58.519 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x55899bea66e0": { 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-16T10:26:58.520 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:26:58.521 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-16T10:26:59.299 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-16T10:26:59.299 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-16T10:26:59.299 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-16T10:26:59.299 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-16T10:26:59.299 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-16T10:26:59.300 DEBUG:tasks.check_counter:Getting stats from i 2024-03-16T10:26:59.300 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-16T10:26:59.553 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:26:59.666 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:26:59 smithi192 ceph-mon[36354]: pgmap v648: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 op/s 2024-03-16T10:26:59.680 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:26:59 smithi193 ceph-mon[36311]: pgmap v648: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 op/s 2024-03-16T10:26:59.744 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:26:59 smithi077 ceph-mon[32400]: pgmap v648: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 op/s 2024-03-16T10:27:01.240 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 2479, 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 2392, 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 774575, 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 5774582, 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 18, 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 5, 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.173588230, 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.064140354, 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.071363806, 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.037622418, 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:27:01.241 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2390, 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.079413724, 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000033227 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2209, 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000326481, 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000147 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T10:27:01.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 13315, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 10027, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 6088962, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 2804329, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 14, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 7, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.618130038, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.238373849, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.488987099, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.011924952, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 10025, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.252252553, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000025162 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 21854, 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.002713502, 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000124 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 346896 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 13433, 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 15973, 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 4886250, 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 9092178, 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 21, 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 10, 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.551725795, 2024-03-16T10:27:01.244 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.258928260, 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.411741994, 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.016753013, 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 15973, 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.482127134, 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000030183 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 14196, 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.003005204, 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000211 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 957728, 2024-03-16T10:27:01.245 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 133808 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1193, 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 1.705566665, 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001429645 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:01.246 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1200, 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.078778052, 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000065648 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 9308, 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 7481, 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 7481, 2024-03-16T10:27:01.247 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 131.077205400, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.017521348 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 1827, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 29, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 2, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 25, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 6, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 9, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 10, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 13, 2024-03-16T10:27:01.248 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 388, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 1, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 1, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 4, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 19349, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 14809, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 1827, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 199, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 82, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 17, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 7, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 3, 2024-03-16T10:27:01.249 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 348, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 1284, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 736, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 15642193, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 0, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 1284, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T10:27:01.250 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 1284, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 13, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 602, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 1210, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 1146, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 537, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 33, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 2311 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 210, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 3200, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 1837, 2024-03-16T10:27:01.251 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 2110, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 195, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 598, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 598, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-03-16T10:27:01.252 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 17, 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 2, 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 3156, 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 3156, 2024-03-16T10:27:01.253 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 5, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 5, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 2, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194304, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 8767572, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 0, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 147, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.873412103, 2024-03-16T10:27:01.254 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.005941578 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 0 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 30, 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 822, 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 792, 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 29, 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 1620, 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 1591, 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 25, 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 1803, 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 1778, 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 1, 2024-03-16T10:27:01.255 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 1076, 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 1075, 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 63320, 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 223612 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 11024, 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 28, 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 9308, 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 12, 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 1354, 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 434, 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 5.108599870, 2024-03-16T10:27:01.256 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.011770967 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 429, 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 1.480712371, 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.003451543 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 5442, 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 119.297435935, 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.021921616 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1, 2024-03-16T10:27:01.258 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000099277, 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000099277 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.259 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.260 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.260 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.260 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-03-16T10:27:01.260 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.260 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.260 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.261 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 331, 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.258232280, 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000780157 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 17, 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.020150947, 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001185349 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 435, 2024-03-16T10:27:01.262 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 4.090131148, 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.009402600 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.263 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.265 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.266 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 392, 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.821843572, 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.002096539 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 1, 2024-03-16T10:27:01.267 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 2, 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 1, 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 1, 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 1612, 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 1612, 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 587, 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-03-16T10:27:01.268 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T10:27:01.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-03-16T10:27:01.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 286321, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 140, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 968, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 11, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 7344, 2024-03-16T10:27:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 155, 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 122789, 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 489, 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-03-16T10:27:01.272 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 2314, 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 5062107, 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 29, 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 2314, 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2314, 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 6.303903610, 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.002724245 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2314, 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 2483 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.273 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 2314, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 1320, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 994, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 0, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 10, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 36, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 183, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 51, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 2, 2024-03-16T10:27:01.274 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 611, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 1284, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 20, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-03-16T10:27:01.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-03-16T10:27:01.276 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 88, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 3, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 12, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 12, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 239, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 0, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 47 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 1, 2024-03-16T10:27:01.277 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 6, 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 6, 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 598, 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 598, 2024-03-16T10:27:01.278 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 29223, 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 9301421, 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 29223, 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 29223, 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 9301421, 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:27:01.279 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 2314, 2024-03-16T10:27:01.280 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 5122505, 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 2314, 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 5122505, 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-16T10:27:01.281 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 2314, 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 2314, 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 2314, 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 2314, 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-03-16T10:27:01.282 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 598, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 60398, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 598, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 36, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 60398, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:01.283 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x555b1b802500": { 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 3156, 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 4573268, 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 3156, 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 147, 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 4573268, 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:27:01.284 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:01.285 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:01.285 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:01.285 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:01.285 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:01.285 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-16T10:27:02.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:01 smithi193 ceph-mon[36311]: pgmap v649: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 12 op/s 2024-03-16T10:27:02.061 INFO:tasks.check_counter:Daemon mds.i mds.exported=17 2024-03-16T10:27:02.062 INFO:tasks.check_counter:Daemon mds.i mds.imported=3 2024-03-16T10:27:02.062 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=210 2024-03-16T10:27:02.062 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=3200 2024-03-16T10:27:02.062 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=2 2024-03-16T10:27:02.062 DEBUG:tasks.check_counter:Getting stats from l 2024-03-16T10:27:02.062 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-16T10:27:02.244 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:01 smithi077 ceph-mon[32400]: pgmap v649: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 12 op/s 2024-03-16T10:27:02.313 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T10:27:02.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:01 smithi192 ceph-mon[36354]: pgmap v649: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 12 op/s 2024-03-16T10:27:03.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:02 smithi193 ceph-mon[36311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:27:03.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:02 smithi193 ceph-mon[36311]: pgmap v650: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 11 op/s 2024-03-16T10:27:03.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:02 smithi077 ceph-mon[32400]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:27:03.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:02 smithi077 ceph-mon[32400]: pgmap v650: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 11 op/s 2024-03-16T10:27:03.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:02 smithi192 ceph-mon[36354]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T10:27:03.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:02 smithi192 ceph-mon[36354]: pgmap v650: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 11 op/s 2024-03-16T10:27:03.978 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-16T10:27:03.979 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T10:27:03.979 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 870, 2024-03-16T10:27:03.979 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 838, 2024-03-16T10:27:03.979 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 161813790, 2024-03-16T10:27:03.979 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 325778, 2024-03-16T10:27:03.979 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 7, 2024-03-16T10:27:03.979 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 3, 2024-03-16T10:27:03.979 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.144399951, 2024-03-16T10:27:03.979 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.020498518, 2024-03-16T10:27:03.979 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.109166668, 2024-03-16T10:27:03.979 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.015656872, 2024-03-16T10:27:03.979 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 837, 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.019116923, 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000022839 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 944432, 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 126704 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T10:27:03.980 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 835, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 989, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 247687647, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 610625, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 7, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 6, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.148821176, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.020659056, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.107164767, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.015979448, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 987, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.022800088, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000023100 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.981 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 345408 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1495, 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1495, 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 164038695, 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 474644, 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 6, 2024-03-16T10:27:03.982 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 5, 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.172961622, 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.027265907, 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.105821845, 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.031693172, 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1494, 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.037637066, 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000025192 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.983 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 4682, 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.239685071, 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000051192 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:03.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.985 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 47306, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 10939, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 36367, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 31678, 2024-03-16T10:27:03.986 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 1, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-03-16T10:27:03.987 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T10:27:03.988 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-03-16T10:27:03.989 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-03-16T10:27:03.990 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 255197, 2024-03-16T10:27:03.991 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 137, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 22, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 363507248, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 575932070, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 575931471, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.992 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 255197 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 17, 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 47736, 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 47719, 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 18, 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 1609, 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 1591, 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 47306, 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 103994, 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 56688, 2024-03-16T10:27:03.993 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 136628, 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 215420 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.994 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.995 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-03-16T10:27:03.996 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.997 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-03-16T10:27:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-03-16T10:27:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-03-16T10:27:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-03-16T10:27:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:03.999 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.000 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.001 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 1, 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 1, 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 0, 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 1, 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 1904, 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-03-16T10:27:04.006 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-03-16T10:27:04.007 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-03-16T10:27:04.008 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 239579, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 82, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-03-16T10:27:04.009 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 7344, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 155, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 33366195, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 189901, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-03-16T10:27:04.010 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 2951, 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 21, 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 3, 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 2951, 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2951, 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 8.428871207, 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.002856276 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2951, 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 2952 2024-03-16T10:27:04.011 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 2951, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 2951, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 1729, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 1220, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-03-16T10:27:04.012 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 1, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-03-16T10:27:04.013 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-03-16T10:27:04.014 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 88, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 3, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 11, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 11, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-03-16T10:27:04.015 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 0, 2024-03-16T10:27:04.016 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 3196, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 573237034, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 3196, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3196, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 573237034, 2024-03-16T10:27:04.017 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-16T10:27:04.018 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 2951, 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 571779390, 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 2951, 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 571779390, 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:27:04.019 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 2951, 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 2951, 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 2951, 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 2951, 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-03-16T10:27:04.020 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.021 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x55a03b4b06e0": { 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-16T10:27:04.022 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-16T10:27:04.023 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-16T10:27:04.023 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-16T10:27:04.023 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:04.023 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-16T10:27:04.023 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-16T10:27:04.930 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-16T10:27:04.931 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-16T10:27:04.931 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-03-16T10:27:04.931 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-03-16T10:27:04.931 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-03-16T10:27:04.931 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-16T10:27:04.941 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-16T10:27:04.941 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:27:04.941 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T10:27:04.960 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-03-16T10:27:04.960 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T10:27:05.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:27:05.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.182+0000 7f91e0d77640 1 -- 172.21.15.77:0/3607902378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91dc0ffaf0 msgr2=0x7f91dc0fff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.182+0000 7f91e0d77640 1 --2- 172.21.15.77:0/3607902378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91dc0ffaf0 0x7f91dc0fff10 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009930 tx=0x7f91cc02bc30 comp rx=0 tx=0).stop 2024-03-16T10:27:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.182+0000 7f91e0d77640 1 -- 172.21.15.77:0/3607902378 shutdown_connections 2024-03-16T10:27:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.182+0000 7f91e0d77640 1 --2- 172.21.15.77:0/3607902378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc103480 0x7f91dc109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.182+0000 7f91e0d77640 1 --2- 172.21.15.77:0/3607902378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91dc102ac0 0x7f91dc102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.182+0000 7f91e0d77640 1 --2- 172.21.15.77:0/3607902378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91dc0ffaf0 0x7f91dc0fff10 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.182+0000 7f91e0d77640 1 -- 172.21.15.77:0/3607902378 >> 172.21.15.77:0/3607902378 conn(0x7f91dc0fb420 msgr2=0x7f91dc0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.182+0000 7f91e0d77640 1 -- 172.21.15.77:0/3607902378 shutdown_connections 2024-03-16T10:27:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.182+0000 7f91e0d77640 1 -- 172.21.15.77:0/3607902378 wait complete. 2024-03-16T10:27:05.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.183+0000 7f91e0d77640 1 Processor -- start 2024-03-16T10:27:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.183+0000 7f91e0d77640 1 -- start start 2024-03-16T10:27:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.183+0000 7f91e0d77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91dc0ffaf0 0x7f91dc197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.183+0000 7f91e0d77640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91dc102ac0 0x7f91dc198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.183+0000 7f91e0d77640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc103480 0x7f91dc19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.183+0000 7f91e0d77640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc10d4c0 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.183+0000 7f91e0d77640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc10d340 con 0x7f91dc103480 2024-03-16T10:27:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.183+0000 7f91e0d77640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc10d640 con 0x7f91dc102ac0 2024-03-16T10:27:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91da59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91dc0ffaf0 0x7f91dc197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91da59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91dc0ffaf0 0x7f91dc197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53410/0 (socket says 172.21.15.77:53410) 2024-03-16T10:27:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91da59c640 1 -- 172.21.15.77:0/570599124 learned_addr learned my addr 172.21.15.77:0/570599124 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:27:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91d9d9b640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91dc102ac0 0x7f91dc198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91dad9d640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc103480 0x7f91dc19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91da59c640 1 -- 172.21.15.77:0/570599124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91dc102ac0 msgr2=0x7f91dc198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91da59c640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91dc102ac0 0x7f91dc198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91da59c640 1 -- 172.21.15.77:0/570599124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc103480 msgr2=0x7f91dc19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91da59c640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc103480 0x7f91dc19f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91da59c640 1 -- 172.21.15.77:0/570599124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91dc19f810 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91dad9d640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc103480 0x7f91dc19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91d9d9b640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91dc102ac0 0x7f91dc198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91da59c640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91dc0ffaf0 0x7f91dc197bf0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f91cc02c7b0 tx=0x7f91cc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91cb7fe640 1 -- 172.21.15.77:0/570599124 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc041070 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91cb7fe640 1 -- 172.21.15.77:0/570599124 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91cc004450 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91e0d77640 1 -- 172.21.15.77:0/570599124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91dc19faa0 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91cb7fe640 1 -- 172.21.15.77:0/570599124 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc003e50 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.184+0000 7f91e0d77640 1 -- 172.21.15.77:0/570599124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91dc19ff90 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.185+0000 7f91e0d77640 1 -- 172.21.15.77:0/570599124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91dc108c10 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.186+0000 7f91cb7fe640 1 -- 172.21.15.77:0/570599124 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f91cc044070 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.187+0000 7f91cb7fe640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f91c40a29a0 0x7f91c40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:05.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.187+0000 7f91d9d9b640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f91c40a29a0 0x7f91c40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:05.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.187+0000 7f91cb7fe640 1 -- 172.21.15.77:0/570599124 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f91cc0f0490 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.187+0000 7f91d9d9b640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f91c40a29a0 0x7f91c40a4e60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f91dc199100 tx=0x7f91d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:05.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.195+0000 7f91cb7fe640 1 -- 172.21.15.77:0/570599124 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f91cc0bd5c0 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.377+0000 7f91e0d77640 1 -- 172.21.15.77:0/570599124 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91dc0623e0 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.379+0000 7f91cb7fe640 1 -- 172.21.15.77:0/570599124 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22496 (secure 0 0 0) 0x7f91cc0bcec0 con 0x7f91dc0ffaf0 2024-03-16T10:27:05.380 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:27:05.380 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:23:44.816264+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:27:05.382 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.381+0000 7f91e0d77640 1 -- 172.21.15.77:0/570599124 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f91c40a29a0 msgr2=0x7f91c40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.381+0000 7f91e0d77640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f91c40a29a0 0x7f91c40a4e60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f91dc199100 tx=0x7f91d0005f90 comp rx=0 tx=0).stop 2024-03-16T10:27:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.381+0000 7f91e0d77640 1 -- 172.21.15.77:0/570599124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91dc0ffaf0 msgr2=0x7f91dc197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.381+0000 7f91e0d77640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91dc0ffaf0 0x7f91dc197bf0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f91cc02c7b0 tx=0x7f91cc02c650 comp rx=0 tx=0).stop 2024-03-16T10:27:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.381+0000 7f91e0d77640 1 -- 172.21.15.77:0/570599124 shutdown_connections 2024-03-16T10:27:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.381+0000 7f91e0d77640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc103480 0x7f91dc19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.381+0000 7f91e0d77640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f91c40a29a0 0x7f91c40a4e60 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.381+0000 7f91e0d77640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91dc102ac0 0x7f91dc198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.381+0000 7f91e0d77640 1 --2- 172.21.15.77:0/570599124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91dc0ffaf0 0x7f91dc197bf0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.381+0000 7f91e0d77640 1 -- 172.21.15.77:0/570599124 >> 172.21.15.77:0/570599124 conn(0x7f91dc0fb420 msgr2=0x7f91dc105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.381+0000 7f91e0d77640 1 -- 172.21.15.77:0/570599124 shutdown_connections 2024-03-16T10:27:05.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.382+0000 7f91e0d77640 1 -- 172.21.15.77:0/570599124 wait complete. 2024-03-16T10:27:05.394 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T10:10:22.965004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T10:15:00.264039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T10:15:01.536067+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T10:27:05.394 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T10:27:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.507+0000 7f379aab2640 1 -- 172.21.15.77:0/4237948182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37941006e0 msgr2=0x7f3794100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.507+0000 7f379aab2640 1 --2- 172.21.15.77:0/4237948182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37941006e0 0x7f3794100b00 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f378c009ca0 tx=0x7f378c02c0f0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.507+0000 7f379aab2640 1 -- 172.21.15.77:0/4237948182 shutdown_connections 2024-03-16T10:27:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.507+0000 7f379aab2640 1 --2- 172.21.15.77:0/4237948182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3794102290 0x7f3794108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.507+0000 7f379aab2640 1 --2- 172.21.15.77:0/4237948182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37941018d0 0x7f3794101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.507+0000 7f379aab2640 1 --2- 172.21.15.77:0/4237948182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f37941006e0 0x7f3794100b00 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.507+0000 7f379aab2640 1 -- 172.21.15.77:0/4237948182 >> 172.21.15.77:0/4237948182 conn(0x7f37940fbc70 msgr2=0x7f37940fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.507+0000 7f379aab2640 1 -- 172.21.15.77:0/4237948182 shutdown_connections 2024-03-16T10:27:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.507+0000 7f379aab2640 1 -- 172.21.15.77:0/4237948182 wait complete. 2024-03-16T10:27:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f379aab2640 1 Processor -- start 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f379aab2640 1 -- start start 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f379aab2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37941006e0 0x7f3794199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f379aab2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37941018d0 0x7f379419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f379aab2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3794102290 0x7f37941a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f379aab2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f379410c360 con 0x7f3794102290 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f379aab2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f379410c1e0 con 0x7f37941006e0 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f379aab2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f379410c4e0 con 0x7f37941018d0 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f379904f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3794102290 0x7f37941a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f379904f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3794102290 0x7f37941a1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53418/0 (socket says 172.21.15.77:53418) 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f3793fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37941018d0 0x7f379419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f379904f640 1 -- 172.21.15.77:0/2661052975 learned_addr learned my addr 172.21.15.77:0/2661052975 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.508+0000 7f379884e640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37941006e0 0x7f3794199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.509+0000 7f3793fff640 1 -- 172.21.15.77:0/2661052975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37941006e0 msgr2=0x7f3794199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.509+0000 7f3793fff640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37941006e0 0x7f3794199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.509+0000 7f3793fff640 1 -- 172.21.15.77:0/2661052975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3794102290 msgr2=0x7f37941a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.509+0000 7f3793fff640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3794102290 0x7f37941a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.509+0000 7f3793fff640 1 -- 172.21.15.77:0/2661052975 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37941a1970 con 0x7f37941018d0 2024-03-16T10:27:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.509+0000 7f379904f640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3794102290 0x7f37941a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:27:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.509+0000 7f3793fff640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37941018d0 0x7f379419a290 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f378800e9f0 tx=0x7f378800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.509+0000 7f3791ffb640 1 -- 172.21.15.77:0/2661052975 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378800ccc0 con 0x7f37941018d0 2024-03-16T10:27:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.509+0000 7f379aab2640 1 -- 172.21.15.77:0/2661052975 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37941a1c60 con 0x7f37941018d0 2024-03-16T10:27:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.509+0000 7f3791ffb640 1 -- 172.21.15.77:0/2661052975 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3788010910 con 0x7f37941018d0 2024-03-16T10:27:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.509+0000 7f3791ffb640 1 -- 172.21.15.77:0/2661052975 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788018960 con 0x7f37941018d0 2024-03-16T10:27:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.509+0000 7f379aab2640 1 -- 172.21.15.77:0/2661052975 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37941a21b0 con 0x7f37941018d0 2024-03-16T10:27:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.510+0000 7f379aab2640 1 -- 172.21.15.77:0/2661052975 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37941a2380 con 0x7f37941018d0 2024-03-16T10:27:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.512+0000 7f3791ffb640 1 -- 172.21.15.77:0/2661052975 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f3788010ac0 con 0x7f37941018d0 2024-03-16T10:27:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.512+0000 7f3791ffb640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f377c0a2800 0x7f377c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.513+0000 7f379884e640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f377c0a2800 0x7f377c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.513+0000 7f3791ffb640 1 -- 172.21.15.77:0/2661052975 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f37880cede0 con 0x7f37941018d0 2024-03-16T10:27:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.513+0000 7f379884e640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f377c0a2800 0x7f377c0a4cc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f378c02c700 tx=0x7f378c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:05.516 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.515+0000 7f3791ffb640 1 -- 172.21.15.77:0/2661052975 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f378809b820 con 0x7f37941018d0 2024-03-16T10:27:05.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.693+0000 7f379aab2640 1 -- 172.21.15.77:0/2661052975 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f37941a2670 con 0x7f37941018d0 2024-03-16T10:27:05.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.693+0000 7f3791ffb640 1 -- 172.21.15.77:0/2661052975 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f378809f8c0 con 0x7f37941018d0 2024-03-16T10:27:05.694 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-03-16T10:27:05.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.695+0000 7f379aab2640 1 -- 172.21.15.77:0/2661052975 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f377c0a2800 msgr2=0x7f377c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.695+0000 7f379aab2640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f377c0a2800 0x7f377c0a4cc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f378c02c700 tx=0x7f378c032590 comp rx=0 tx=0).stop 2024-03-16T10:27:05.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.696+0000 7f379aab2640 1 -- 172.21.15.77:0/2661052975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37941018d0 msgr2=0x7f379419a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.696+0000 7f379aab2640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37941018d0 0x7f379419a290 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f378800e9f0 tx=0x7f378800eeb0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.696+0000 7f379aab2640 1 -- 172.21.15.77:0/2661052975 shutdown_connections 2024-03-16T10:27:05.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.696+0000 7f379aab2640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3794102290 0x7f37941a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.696+0000 7f379aab2640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f377c0a2800 0x7f377c0a4cc0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.696+0000 7f379aab2640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37941018d0 0x7f379419a290 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.696+0000 7f379aab2640 1 --2- 172.21.15.77:0/2661052975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37941006e0 0x7f3794199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.696+0000 7f379aab2640 1 -- 172.21.15.77:0/2661052975 >> 172.21.15.77:0/2661052975 conn(0x7f37940fbc70 msgr2=0x7f37940fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:05.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.696+0000 7f379aab2640 1 -- 172.21.15.77:0/2661052975 shutdown_connections 2024-03-16T10:27:05.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.696+0000 7f379aab2640 1 -- 172.21.15.77:0/2661052975 wait complete. 2024-03-16T10:27:05.708 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T10:27:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.811+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1132772947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bb8102290 msgr2=0x7f1bb8108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.811+0000 7f1bbfb09640 1 --2- 172.21.15.77:0/1132772947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bb8102290 0x7f1bb8108b20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009ca0 tx=0x7f1bb002c0f0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.811+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1132772947 shutdown_connections 2024-03-16T10:27:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.811+0000 7f1bbfb09640 1 --2- 172.21.15.77:0/1132772947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bb8102290 0x7f1bb8108b20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.811+0000 7f1bbfb09640 1 --2- 172.21.15.77:0/1132772947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb81018d0 0x7f1bb8101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.811+0000 7f1bbfb09640 1 --2- 172.21.15.77:0/1132772947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bb81006e0 0x7f1bb8100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1132772947 >> 172.21.15.77:0/1132772947 conn(0x7f1bb80fbc70 msgr2=0x7f1bb80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1132772947 shutdown_connections 2024-03-16T10:27:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1132772947 wait complete. 2024-03-16T10:27:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbfb09640 1 Processor -- start 2024-03-16T10:27:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbfb09640 1 -- start start 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbfb09640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bb81006e0 0x7f1bb80694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbfb09640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bb81018d0 0x7f1bb819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbfb09640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8102290 0x7f1bb819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbfb09640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb810c340 con 0x7f1bb8102290 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbfb09640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb810c1c0 con 0x7f1bb81018d0 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbfb09640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb810c4c0 con 0x7f1bb81006e0 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbe0a6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8102290 0x7f1bb819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbe0a6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8102290 0x7f1bb819ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53438/0 (socket says 172.21.15.77:53438) 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbe0a6640 1 -- 172.21.15.77:0/1699313268 learned_addr learned my addr 172.21.15.77:0/1699313268 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbd8a5640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bb81006e0 0x7f1bb80694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.812+0000 7f1bbd0a4640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bb81018d0 0x7f1bb819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.813+0000 7f1bbe0a6640 1 -- 172.21.15.77:0/1699313268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bb81006e0 msgr2=0x7f1bb80694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.813+0000 7f1bbe0a6640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bb81006e0 0x7f1bb80694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.813+0000 7f1bbe0a6640 1 -- 172.21.15.77:0/1699313268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bb81018d0 msgr2=0x7f1bb819c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.813+0000 7f1bbe0a6640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bb81018d0 0x7f1bb819c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.813+0000 7f1bbe0a6640 1 -- 172.21.15.77:0/1699313268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb819f730 con 0x7f1bb8102290 2024-03-16T10:27:05.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.813+0000 7f1bbe0a6640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8102290 0x7f1bb819ee20 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0037040 tx=0x7f1bb0032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:05.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.813+0000 7f1baeffd640 1 -- 172.21.15.77:0/1699313268 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0041020 con 0x7f1bb8102290 2024-03-16T10:27:05.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.813+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1699313268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb819f960 con 0x7f1bb8102290 2024-03-16T10:27:05.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.813+0000 7f1baeffd640 1 -- 172.21.15.77:0/1699313268 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bb0034cd0 con 0x7f1bb8102290 2024-03-16T10:27:05.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.813+0000 7f1baeffd640 1 -- 172.21.15.77:0/1699313268 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0039040 con 0x7f1bb8102290 2024-03-16T10:27:05.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.813+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1699313268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb819fe50 con 0x7f1bb8102290 2024-03-16T10:27:05.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.814+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1699313268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb8107a00 con 0x7f1bb8102290 2024-03-16T10:27:05.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.815+0000 7f1baeffd640 1 -- 172.21.15.77:0/1699313268 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f1bb0044020 con 0x7f1bb8102290 2024-03-16T10:27:05.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.816+0000 7f1baeffd640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1ba40a2800 0x7f1ba40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:05.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.816+0000 7f1baeffd640 1 -- 172.21.15.77:0/1699313268 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f1bb00f0c30 con 0x7f1bb8102290 2024-03-16T10:27:05.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.816+0000 7f1bbd8a5640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1ba40a2800 0x7f1ba40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:05.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.816+0000 7f1bbd8a5640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1ba40a2800 0x7f1ba40a4cc0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8101730 tx=0x7f1ba8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:05.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:05.819+0000 7f1baeffd640 1 -- 172.21.15.77:0/1699313268 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1bb004b020 con 0x7f1bb8102290 2024-03-16T10:27:05.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:05 smithi077 ceph-mon[32400]: pgmap v651: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-16T10:27:05.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:05 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/570599124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:27:05.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:05 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2661052975' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:27:06.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.002+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1699313268 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1bb802d340 con 0x7f1bb8102290 2024-03-16T10:27:06.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.003+0000 7f1baeffd640 1 -- 172.21.15.77:0/1699313268 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f1bb00bddf0 con 0x7f1bb8102290 2024-03-16T10:27:06.004 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-03-16T10:27:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.005+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1699313268 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1ba40a2800 msgr2=0x7f1ba40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.005+0000 7f1bbfb09640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1ba40a2800 0x7f1ba40a4cc0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8101730 tx=0x7f1ba8005f90 comp rx=0 tx=0).stop 2024-03-16T10:27:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.005+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1699313268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8102290 msgr2=0x7f1bb819ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.005+0000 7f1bbfb09640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8102290 0x7f1bb819ee20 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0037040 tx=0x7f1bb0032f40 comp rx=0 tx=0).stop 2024-03-16T10:27:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.005+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1699313268 shutdown_connections 2024-03-16T10:27:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.005+0000 7f1bbfb09640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1bb8102290 0x7f1bb819ee20 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.005+0000 7f1bbfb09640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bb81018d0 0x7f1bb819c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.005+0000 7f1bbfb09640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f1ba40a2800 0x7f1ba40a4cc0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.005+0000 7f1bbfb09640 1 --2- 172.21.15.77:0/1699313268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bb81006e0 0x7f1bb80694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.005+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1699313268 >> 172.21.15.77:0/1699313268 conn(0x7f1bb80fbc70 msgr2=0x7f1bb8104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:06.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.005+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1699313268 shutdown_connections 2024-03-16T10:27:06.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.005+0000 7f1bbfb09640 1 -- 172.21.15.77:0/1699313268 wait complete. 2024-03-16T10:27:06.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T10:27:06.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.123+0000 7fb861fab640 1 -- 172.21.15.77:0/1105354260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb85c1018d0 msgr2=0x7fb85c101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:06.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.123+0000 7fb861fab640 1 --2- 172.21.15.77:0/1105354260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb85c1018d0 0x7fb85c101d50 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb850009900 tx=0x7fb85002bc30 comp rx=0 tx=0).stop 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.123+0000 7fb861fab640 1 -- 172.21.15.77:0/1105354260 shutdown_connections 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.123+0000 7fb861fab640 1 --2- 172.21.15.77:0/1105354260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb85c102290 0x7fb85c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.123+0000 7fb861fab640 1 --2- 172.21.15.77:0/1105354260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb85c1018d0 0x7fb85c101d50 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.123+0000 7fb861fab640 1 --2- 172.21.15.77:0/1105354260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb85c1006e0 0x7fb85c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.123+0000 7fb861fab640 1 -- 172.21.15.77:0/1105354260 >> 172.21.15.77:0/1105354260 conn(0x7fb85c0fbc70 msgr2=0x7fb85c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.123+0000 7fb861fab640 1 -- 172.21.15.77:0/1105354260 shutdown_connections 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.123+0000 7fb861fab640 1 -- 172.21.15.77:0/1105354260 wait complete. 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.124+0000 7fb861fab640 1 Processor -- start 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.124+0000 7fb861fab640 1 -- start start 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.124+0000 7fb861fab640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb85c1006e0 0x7fb85c197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.124+0000 7fb861fab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb85c1018d0 0x7fb85c198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.124+0000 7fb861fab640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb85c102290 0x7fb85c19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:06.127 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.124+0000 7fb861fab640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb85c10c340 con 0x7fb85c1018d0 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.124+0000 7fb861fab640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb85c10c1c0 con 0x7fb85c102290 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.124+0000 7fb861fab640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb85c10c4c0 con 0x7fb85c1006e0 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.124+0000 7fb85affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb85c1018d0 0x7fb85c198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb85affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb85c1018d0 0x7fb85c198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53460/0 (socket says 172.21.15.77:53460) 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb85affd640 1 -- 172.21.15.77:0/2648535826 learned_addr learned my addr 172.21.15.77:0/2648535826 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb85bfff640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb85c102290 0x7fb85c19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb85b7fe640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb85c1006e0 0x7fb85c197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb85bfff640 1 -- 172.21.15.77:0/2648535826 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb85c1006e0 msgr2=0x7fb85c197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb85bfff640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb85c1006e0 0x7fb85c197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb85bfff640 1 -- 172.21.15.77:0/2648535826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb85c1018d0 msgr2=0x7fb85c198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb85bfff640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb85c1018d0 0x7fb85c198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb85bfff640 1 -- 172.21.15.77:0/2648535826 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb85c19f750 con 0x7fb85c102290 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb85affd640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb85c1018d0 0x7fb85c198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:27:06.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb85bfff640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb85c102290 0x7fb85c19f0e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb854009fd0 tx=0x7fb85400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:06.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb858ff9640 1 -- 172.21.15.77:0/2648535826 <== mon.2 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85400ce20 con 0x7fb85c102290 2024-03-16T10:27:06.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb858ff9640 1 -- 172.21.15.77:0/2648535826 <== mon.2 v2:172.21.15.192:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb854005020 con 0x7fb85c102290 2024-03-16T10:27:06.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb861fab640 1 -- 172.21.15.77:0/2648535826 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb85c19fa40 con 0x7fb85c102290 2024-03-16T10:27:06.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb858ff9640 1 -- 172.21.15.77:0/2648535826 <== mon.2 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb854010720 con 0x7fb85c102290 2024-03-16T10:27:06.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.125+0000 7fb861fab640 1 -- 172.21.15.77:0/2648535826 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb85c19ff10 con 0x7fb85c102290 2024-03-16T10:27:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.126+0000 7fb861fab640 1 -- 172.21.15.77:0/2648535826 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb85c1a0200 con 0x7fb85c102290 2024-03-16T10:27:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.128+0000 7fb858ff9640 1 -- 172.21.15.77:0/2648535826 <== mon.2 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fb854005540 con 0x7fb85c102290 2024-03-16T10:27:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.128+0000 7fb858ff9640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb8440a2800 0x7fb8440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.128+0000 7fb85b7fe640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb8440a2800 0x7fb8440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.129+0000 7fb858ff9640 1 -- 172.21.15.77:0/2648535826 <== mon.2 v2:172.21.15.192:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7fb8540cf0b0 con 0x7fb85c102290 2024-03-16T10:27:06.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.129+0000 7fb85b7fe640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb8440a2800 0x7fb8440a4cc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb85c101730 tx=0x7fb84c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:06.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.133+0000 7fb858ff9640 1 -- 172.21.15.77:0/2648535826 <== mon.2 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb85409c1e0 con 0x7fb85c102290 2024-03-16T10:27:06.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:05 smithi193 ceph-mon[36311]: pgmap v651: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-16T10:27:06.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:05 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/570599124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:27:06.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:05 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2661052975' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:27:06.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.313+0000 7fb861fab640 1 -- 172.21.15.77:0/2648535826 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb85c1a0470 con 0x7fb85c102290 2024-03-16T10:27:06.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.314+0000 7fb858ff9640 1 -- 172.21.15.77:0/2648535826 <== mon.2 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fb85409bae0 con 0x7fb85c102290 2024-03-16T10:27:06.315 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2024-03-16T10:27:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.315+0000 7fb861fab640 1 -- 172.21.15.77:0/2648535826 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb8440a2800 msgr2=0x7fb8440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.315+0000 7fb861fab640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb8440a2800 0x7fb8440a4cc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb85c101730 tx=0x7fb84c005f90 comp rx=0 tx=0).stop 2024-03-16T10:27:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.315+0000 7fb861fab640 1 -- 172.21.15.77:0/2648535826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb85c102290 msgr2=0x7fb85c19f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.315+0000 7fb861fab640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb85c102290 0x7fb85c19f0e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb854009fd0 tx=0x7fb85400ef10 comp rx=0 tx=0).stop 2024-03-16T10:27:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.316+0000 7fb861fab640 1 -- 172.21.15.77:0/2648535826 shutdown_connections 2024-03-16T10:27:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.316+0000 7fb861fab640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb85c102290 0x7fb85c19f0e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.316+0000 7fb861fab640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb85c1018d0 0x7fb85c198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.316+0000 7fb861fab640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fb8440a2800 0x7fb8440a4cc0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.316+0000 7fb861fab640 1 --2- 172.21.15.77:0/2648535826 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb85c1006e0 0x7fb85c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.316+0000 7fb861fab640 1 -- 172.21.15.77:0/2648535826 >> 172.21.15.77:0/2648535826 conn(0x7fb85c0fbc70 msgr2=0x7fb85c0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.316+0000 7fb861fab640 1 -- 172.21.15.77:0/2648535826 shutdown_connections 2024-03-16T10:27:06.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.316+0000 7fb861fab640 1 -- 172.21.15.77:0/2648535826 wait complete. 2024-03-16T10:27:06.328 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T10:27:06.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:05 smithi192 ceph-mon[36354]: pgmap v651: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-16T10:27:06.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:05 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/570599124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:27:06.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:05 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2661052975' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 -- 172.21.15.77:0/865531615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c9c101800 msgr2=0x7f4c9c101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 --2- 172.21.15.77:0/865531615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c9c101800 0x7f4c9c101c20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009900 tx=0x7f4c8c02bc30 comp rx=0 tx=0).stop 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 -- 172.21.15.77:0/865531615 shutdown_connections 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 --2- 172.21.15.77:0/865531615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c9c1033b0 0x7f4c9c109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 --2- 172.21.15.77:0/865531615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c9c1029f0 0x7f4c9c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 --2- 172.21.15.77:0/865531615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c9c101800 0x7f4c9c101c20 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 -- 172.21.15.77:0/865531615 >> 172.21.15.77:0/865531615 conn(0x7f4c9c0fcd50 msgr2=0x7f4c9c0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 -- 172.21.15.77:0/865531615 shutdown_connections 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 -- 172.21.15.77:0/865531615 wait complete. 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 Processor -- start 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 -- start start 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c9c101800 0x7f4c9c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:06.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c9c1029f0 0x7f4c9c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c9c1033b0 0x7f4c9c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9c10d480 con 0x7f4c9c1033b0 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9c10d300 con 0x7f4c9c1029f0 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca2f77640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9c10d600 con 0x7f4c9c101800 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.434+0000 7f4ca1514640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c9c1033b0 0x7f4c9c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4ca1514640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c9c1033b0 0x7f4c9c19f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53490/0 (socket says 172.21.15.77:53490) 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4ca1514640 1 -- 172.21.15.77:0/1293074528 learned_addr learned my addr 172.21.15.77:0/1293074528 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4ca0d13640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c9c101800 0x7f4c9c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4c9bfff640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c9c1029f0 0x7f4c9c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4ca1514640 1 -- 172.21.15.77:0/1293074528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c9c101800 msgr2=0x7f4c9c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4ca1514640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c9c101800 0x7f4c9c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4ca1514640 1 -- 172.21.15.77:0/1293074528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c9c1029f0 msgr2=0x7f4c9c198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4ca1514640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c9c1029f0 0x7f4c9c198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4ca1514640 1 -- 172.21.15.77:0/1293074528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9c19f740 con 0x7f4c9c1033b0 2024-03-16T10:27:06.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4ca1514640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c9c1033b0 0x7f4c9c19f0d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4c9400eb10 tx=0x7f4c9400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:06.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4c99ffb640 1 -- 172.21.15.77:0/1293074528 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94004b20 con 0x7f4c9c1033b0 2024-03-16T10:27:06.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4c99ffb640 1 -- 172.21.15.77:0/1293074528 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c94004cc0 con 0x7f4c9c1033b0 2024-03-16T10:27:06.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4ca2f77640 1 -- 172.21.15.77:0/1293074528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c9c19fa30 con 0x7f4c9c1033b0 2024-03-16T10:27:06.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4c99ffb640 1 -- 172.21.15.77:0/1293074528 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94010760 con 0x7f4c9c1033b0 2024-03-16T10:27:06.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.435+0000 7f4ca2f77640 1 -- 172.21.15.77:0/1293074528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c9c19ff80 con 0x7f4c9c1033b0 2024-03-16T10:27:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.436+0000 7f4ca2f77640 1 -- 172.21.15.77:0/1293074528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c9c108b20 con 0x7f4c9c1033b0 2024-03-16T10:27:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.437+0000 7f4c99ffb640 1 -- 172.21.15.77:0/1293074528 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f4c94005020 con 0x7f4c9c1033b0 2024-03-16T10:27:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.438+0000 7f4c99ffb640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4c840a2800 0x7f4c840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.438+0000 7f4c99ffb640 1 -- 172.21.15.77:0/1293074528 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f4c940cf0b0 con 0x7f4c9c1033b0 2024-03-16T10:27:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.438+0000 7f4ca0d13640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4c840a2800 0x7f4c840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:06.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.438+0000 7f4ca0d13640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4c840a2800 0x7f4c840a4cc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c000c00 tx=0x7f4c8c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:06.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.442+0000 7f4c99ffb640 1 -- 172.21.15.77:0/1293074528 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4c9409c1e0 con 0x7f4c9c1033b0 2024-03-16T10:27:06.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.677+0000 7f4ca2f77640 1 -- 172.21.15.77:0/1293074528 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4c9c02d1b0 con 0x7f4c9c1033b0 2024-03-16T10:27:06.678 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.677+0000 7f4c99ffb640 1 -- 172.21.15.77:0/1293074528 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+590 (secure 0 0 0) 0x7f4c9409bae0 con 0x7f4c9c1033b0 2024-03-16T10:27:06.678 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/233716866 2024-03-17T10:07:58.634847+0000 2024-03-16T10:27:06.679 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/3665134846 2024-03-17T10:07:58.634847+0000 2024-03-16T10:27:06.679 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/3504149086 2024-03-17T10:07:32.924837+0000 2024-03-16T10:27:06.679 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/3120915723 2024-03-17T10:07:32.924837+0000 2024-03-16T10:27:06.679 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/465332686 2024-03-17T10:07:58.634847+0000 2024-03-16T10:27:06.679 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6801/1353424963 2024-03-17T10:07:58.634847+0000 2024-03-16T10:27:06.679 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6800/1353424963 2024-03-17T10:07:58.634847+0000 2024-03-16T10:27:06.679 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6801/1093017736 2024-03-17T10:07:32.924837+0000 2024-03-16T10:27:06.679 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:6800/1093017736 2024-03-17T10:07:32.924837+0000 2024-03-16T10:27:06.679 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77:0/1991017150 2024-03-17T10:07:32.924837+0000 2024-03-16T10:27:06.680 INFO:teuthology.orchestra.run.smithi077.stderr:listed 10 entries 2024-03-16T10:27:06.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.679+0000 7f4ca2f77640 1 -- 172.21.15.77:0/1293074528 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4c840a2800 msgr2=0x7f4c840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:06.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.679+0000 7f4ca2f77640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4c840a2800 0x7f4c840a4cc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c000c00 tx=0x7f4c8c005fd0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.679+0000 7f4ca2f77640 1 -- 172.21.15.77:0/1293074528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c9c1033b0 msgr2=0x7f4c9c19f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:06.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.679+0000 7f4ca2f77640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c9c1033b0 0x7f4c9c19f0d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4c9400eb10 tx=0x7f4c9400c2d0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.679+0000 7f4ca2f77640 1 -- 172.21.15.77:0/1293074528 shutdown_connections 2024-03-16T10:27:06.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.679+0000 7f4ca2f77640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c9c1033b0 0x7f4c9c19f0d0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.679+0000 7f4ca2f77640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c9c1029f0 0x7f4c9c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.679+0000 7f4ca2f77640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f4c840a2800 0x7f4c840a4cc0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.679+0000 7f4ca2f77640 1 --2- 172.21.15.77:0/1293074528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c9c101800 0x7f4c9c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:06.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.679+0000 7f4ca2f77640 1 -- 172.21.15.77:0/1293074528 >> 172.21.15.77:0/1293074528 conn(0x7f4c9c0fcd50 msgr2=0x7f4c9c105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:06.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.679+0000 7f4ca2f77640 1 -- 172.21.15.77:0/1293074528 shutdown_connections 2024-03-16T10:27:06.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:06.679+0000 7f4ca2f77640 1 -- 172.21.15.77:0/1293074528 wait complete. 2024-03-16T10:27:06.695 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-16T10:27:06.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:27:06.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-16T10:27:06.746 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T10:27:06.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:27:06.746 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-16T10:27:06.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:27:06.782 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-03-16T10:27:06.853 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-16T10:27:06.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:27:06.853 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T10:27:06.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T10:27:06.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-03-16T10:27:07.047 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-16T10:27:07.065 INFO:tasks.cephadm:Teardown begin 2024-03-16T10:27:07.065 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-16T10:27:07.066 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:27:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.239+0000 7f9472790640 1 -- 172.21.15.77:0/475256413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f946c100700 msgr2=0x7f946c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.239+0000 7f9472790640 1 --2- 172.21.15.77:0/475256413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f946c100700 0x7f946c100b20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f945c009900 tx=0x7f945c02bc40 comp rx=0 tx=0).stop 2024-03-16T10:27:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.239+0000 7f9472790640 1 -- 172.21.15.77:0/475256413 shutdown_connections 2024-03-16T10:27:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.239+0000 7f9472790640 1 --2- 172.21.15.77:0/475256413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f946c1022b0 0x7f946c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.239+0000 7f9472790640 1 --2- 172.21.15.77:0/475256413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c1018f0 0x7f946c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.239+0000 7f9472790640 1 --2- 172.21.15.77:0/475256413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f946c100700 0x7f946c100b20 secure :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f945c009900 tx=0x7f945c02bc40 comp rx=0 tx=0).stop 2024-03-16T10:27:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.239+0000 7f9472790640 1 -- 172.21.15.77:0/475256413 >> 172.21.15.77:0/475256413 conn(0x7f946c0fbc70 msgr2=0x7f946c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.239+0000 7f9472790640 1 -- 172.21.15.77:0/475256413 shutdown_connections 2024-03-16T10:27:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.240+0000 7f9472790640 1 -- 172.21.15.77:0/475256413 wait complete. 2024-03-16T10:27:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.241+0000 7f9472790640 1 Processor -- start 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.241+0000 7f9472790640 1 -- start start 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.241+0000 7f9472790640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c1018f0 0x7f946c1111f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.241+0000 7f9472790640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f946c1022b0 0x7f946c111730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.241+0000 7f9472790640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f946c10f830 0x7f946c10fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.241+0000 7f9472790640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f946c109f00 con 0x7f946c1018f0 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.241+0000 7f9472790640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f946c109d80 con 0x7f946c1022b0 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.241+0000 7f9472790640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f946c10a080 con 0x7f946c10f830 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.242+0000 7f9470d2d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f946c10f830 0x7f946c10fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.242+0000 7f9470d2d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f946c10f830 0x7f946c10fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.77:46546/0 (socket says 172.21.15.77:46546) 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.242+0000 7f9470d2d640 1 -- 172.21.15.77:0/675416805 learned_addr learned my addr 172.21.15.77:0/675416805 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.243+0000 7f946bfff640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c1018f0 0x7f946c1111f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.243+0000 7f946b7fe640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f946c1022b0 0x7f946c111730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.243+0000 7f946bfff640 1 -- 172.21.15.77:0/675416805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f946c10f830 msgr2=0x7f946c10fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.243+0000 7f946bfff640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f946c10f830 0x7f946c10fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.243+0000 7f946bfff640 1 -- 172.21.15.77:0/675416805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f946c1022b0 msgr2=0x7f946c111730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.243+0000 7f946bfff640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f946c1022b0 0x7f946c111730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.243+0000 7f946bfff640 1 -- 172.21.15.77:0/675416805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946c110550 con 0x7f946c1018f0 2024-03-16T10:27:07.248 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:06 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1699313268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:27:07.248 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:06 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2648535826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:27:07.248 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:06 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1293074528' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T10:27:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.246+0000 7f946bfff640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c1018f0 0x7f946c1111f0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f946c100240 tx=0x7f945c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.247+0000 7f94697fa640 1 -- 172.21.15.77:0/675416805 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c041070 con 0x7f946c1018f0 2024-03-16T10:27:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.247+0000 7f94697fa640 1 -- 172.21.15.77:0/675416805 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f945c0044d0 con 0x7f946c1018f0 2024-03-16T10:27:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.247+0000 7f94697fa640 1 -- 172.21.15.77:0/675416805 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c003e60 con 0x7f946c1018f0 2024-03-16T10:27:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.247+0000 7f9472790640 1 -- 172.21.15.77:0/675416805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f946c071230 con 0x7f946c1018f0 2024-03-16T10:27:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.248+0000 7f9472790640 1 -- 172.21.15.77:0/675416805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f946c071640 con 0x7f946c1018f0 2024-03-16T10:27:07.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.249+0000 7f9472790640 1 -- 172.21.15.77:0/675416805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f946c0685d0 con 0x7f946c1018f0 2024-03-16T10:27:07.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:06 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1699313268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:27:07.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:06 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2648535826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:27:07.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:06 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1293074528' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T10:27:07.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.253+0000 7f94697fa640 1 -- 172.21.15.77:0/675416805 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f945c044070 con 0x7f946c1018f0 2024-03-16T10:27:07.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.254+0000 7f94697fa640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94540a29b0 0x7f94540a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:07.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.254+0000 7f94697fa640 1 -- 172.21.15.77:0/675416805 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f945c0f0560 con 0x7f946c1018f0 2024-03-16T10:27:07.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.255+0000 7f946b7fe640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94540a29b0 0x7f94540a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:07.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.257+0000 7f946b7fe640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94540a29b0 0x7f94540a4e70 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f946c110f60 tx=0x7f9460005cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:07.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.260+0000 7f94697fa640 1 -- 172.21.15.77:0/675416805 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f945c0bd670 con 0x7f946c1018f0 2024-03-16T10:27:07.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:06 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1699313268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T10:27:07.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:06 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2648535826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T10:27:07.418 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:06 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1293074528' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T10:27:07.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.487+0000 7f9472790640 1 -- 172.21.15.77:0/675416805 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f946c107a70 con 0x7f946c1018f0 2024-03-16T10:27:07.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.488+0000 7f94697fa640 1 -- 172.21.15.77:0/675416805 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+11940 (secure 0 0 0) 0x7f945c0bcf70 con 0x7f946c1018f0 2024-03-16T10:27:07.489 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:27:07.489 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":50,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:27:01.713623+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26800":{"gid":26800,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/946505511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":946505511},{"type":"v1","addr":"172.21.15.193:6835","nonce":946505511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[14658,24455,24482,24454,24475]},"id":1}]} 2024-03-16T10:27:07.489 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 50 2024-03-16T10:27:07.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.490+0000 7f9472790640 1 -- 172.21.15.77:0/675416805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94540a29b0 msgr2=0x7f94540a4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.490+0000 7f9472790640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94540a29b0 0x7f94540a4e70 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f946c110f60 tx=0x7f9460005cc0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.490+0000 7f9472790640 1 -- 172.21.15.77:0/675416805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c1018f0 msgr2=0x7f946c1111f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.490+0000 7f9472790640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c1018f0 0x7f946c1111f0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f946c100240 tx=0x7f945c005d50 comp rx=0 tx=0).stop 2024-03-16T10:27:07.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.490+0000 7f9472790640 1 -- 172.21.15.77:0/675416805 shutdown_connections 2024-03-16T10:27:07.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.490+0000 7f9472790640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f94540a29b0 0x7f94540a4e70 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.490+0000 7f9472790640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f946c10f830 0x7f946c10fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.490+0000 7f9472790640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f946c1022b0 0x7f946c111730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.490+0000 7f9472790640 1 --2- 172.21.15.77:0/675416805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c1018f0 0x7f946c1111f0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.490+0000 7f9472790640 1 -- 172.21.15.77:0/675416805 >> 172.21.15.77:0/675416805 conn(0x7f946c0fbc70 msgr2=0x7f946c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:07.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.490+0000 7f9472790640 1 -- 172.21.15.77:0/675416805 shutdown_connections 2024-03-16T10:27:07.492 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.490+0000 7f9472790640 1 -- 172.21.15.77:0/675416805 wait complete. 2024-03-16T10:27:07.502 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T10:27:07.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.604+0000 7f7045fa9640 1 -- 172.21.15.77:0/523340582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7040101800 msgr2=0x7f7040101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.604+0000 7f7045fa9640 1 --2- 172.21.15.77:0/523340582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7040101800 0x7f7040101c20 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f7030009900 tx=0x7f703002bc30 comp rx=0 tx=0).stop 2024-03-16T10:27:07.607 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.604+0000 7f7045fa9640 1 -- 172.21.15.77:0/523340582 shutdown_connections 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.604+0000 7f7045fa9640 1 --2- 172.21.15.77:0/523340582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70401033b0 0x7f7040109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.604+0000 7f7045fa9640 1 --2- 172.21.15.77:0/523340582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70401029f0 0x7f7040102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.604+0000 7f7045fa9640 1 --2- 172.21.15.77:0/523340582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7040101800 0x7f7040101c20 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.604+0000 7f7045fa9640 1 -- 172.21.15.77:0/523340582 >> 172.21.15.77:0/523340582 conn(0x7f70400fcd90 msgr2=0x7f70400ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.604+0000 7f7045fa9640 1 -- 172.21.15.77:0/523340582 shutdown_connections 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.604+0000 7f7045fa9640 1 -- 172.21.15.77:0/523340582 wait complete. 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f7045fa9640 1 Processor -- start 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f7045fa9640 1 -- start start 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f7045fa9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7040101800 0x7f7040075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f7045fa9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70401029f0 0x7f7040073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f7045fa9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70401033b0 0x7f7040074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f7045fa9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f704010d480 con 0x7f70401029f0 2024-03-16T10:27:07.608 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f7045fa9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f704010d300 con 0x7f70401033b0 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f7045fa9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f704010d600 con 0x7f7040101800 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f703effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70401029f0 0x7f7040073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f703effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70401029f0 0x7f7040073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53530/0 (socket says 172.21.15.77:53530) 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f703effd640 1 -- 172.21.15.77:0/3368619226 learned_addr learned my addr 172.21.15.77:0/3368619226 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f703f7fe640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7040101800 0x7f7040075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.605+0000 7f703ffff640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70401033b0 0x7f7040074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.606+0000 7f703effd640 1 -- 172.21.15.77:0/3368619226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7040101800 msgr2=0x7f7040075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.606+0000 7f703effd640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7040101800 0x7f7040075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.606+0000 7f703effd640 1 -- 172.21.15.77:0/3368619226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70401033b0 msgr2=0x7f7040074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.606+0000 7f703effd640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70401033b0 0x7f7040074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.606+0000 7f703effd640 1 -- 172.21.15.77:0/3368619226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70400749a0 con 0x7f70401029f0 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.606+0000 7f703effd640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70401029f0 0x7f7040073bd0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f703400e9f0 tx=0x7f703400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.606+0000 7f703cff9640 1 -- 172.21.15.77:0/3368619226 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703400cd80 con 0x7f70401029f0 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.606+0000 7f703cff9640 1 -- 172.21.15.77:0/3368619226 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7034010910 con 0x7f70401029f0 2024-03-16T10:27:07.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.606+0000 7f7045fa9640 1 -- 172.21.15.77:0/3368619226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70401a1c10 con 0x7f70401029f0 2024-03-16T10:27:07.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.606+0000 7f703cff9640 1 -- 172.21.15.77:0/3368619226 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034018990 con 0x7f70401029f0 2024-03-16T10:27:07.610 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.606+0000 7f7045fa9640 1 -- 172.21.15.77:0/3368619226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70401a21b0 con 0x7f70401029f0 2024-03-16T10:27:07.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.607+0000 7f7045fa9640 1 -- 172.21.15.77:0/3368619226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7040108b20 con 0x7f70401029f0 2024-03-16T10:27:07.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.608+0000 7f703cff9640 1 -- 172.21.15.77:0/3368619226 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f7034010ac0 con 0x7f70401029f0 2024-03-16T10:27:07.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.609+0000 7f703cff9640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f70280a2880 0x7f70280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:07.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.609+0000 7f703f7fe640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f70280a2880 0x7f70280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:07.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.609+0000 7f703cff9640 1 -- 172.21.15.77:0/3368619226 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f70340cf050 con 0x7f70401029f0 2024-03-16T10:27:07.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.610+0000 7f703f7fe640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f70280a2880 0x7f70280a4d40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7030000c00 tx=0x7f7030005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:07.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.612+0000 7f703cff9640 1 -- 172.21.15.77:0/3368619226 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f703409bb30 con 0x7f70401029f0 2024-03-16T10:27:07.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.832+0000 7f7045fa9640 1 -- 172.21.15.77:0/3368619226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f704002d1b0 con 0x7f70401029f0 2024-03-16T10:27:07.834 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.833+0000 7f703cff9640 1 -- 172.21.15.77:0/3368619226 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+11940 (secure 0 0 0) 0x7f703409fbd0 con 0x7f70401029f0 2024-03-16T10:27:07.834 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:27:07.834 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2545278002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2545278002},{"type":"v1","addr":"172.21.15.77:6835","nonce":2545278002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/1581869083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":1581869083},{"type":"v1","addr":"172.21.15.77:6837","nonce":1581869083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":50,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T10:15:02.344673+0000","modified":"2024-03-16T10:27:01.713623+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24482,"mds_1":24475,"mds_2":24455,"mds_3":24454,"mds_4":14658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14646":{"gid":14646,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3943375297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3943375297},{"type":"v1","addr":"172.21.15.77:6839","nonce":3943375297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14658":{"gid":14658,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.77:6841/857863693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6840","nonce":857863693},{"type":"v1","addr":"172.21.15.77:6841","nonce":857863693}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3214007033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3214007033},{"type":"v1","addr":"172.21.15.192:6833","nonce":3214007033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1430295144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1430295144},{"type":"v1","addr":"172.21.15.193:6833","nonce":1430295144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3288720506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3288720506},{"type":"v1","addr":"172.21.15.192:6835","nonce":3288720506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/3018320989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3018320989},{"type":"v1","addr":"172.21.15.193:6837","nonce":3018320989}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6837/440836568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":440836568},{"type":"v1","addr":"172.21.15.192:6837","nonce":440836568}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/3991082280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3991082280},{"type":"v1","addr":"172.21.15.193:6839","nonce":3991082280}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.192:6839/3223311954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3223311954},{"type":"v1","addr":"172.21.15.192:6839","nonce":3223311954}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26800":{"gid":26800,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/946505511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":946505511},{"type":"v1","addr":"172.21.15.193:6835","nonce":946505511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24482,"qdb_cluster":[14658,24455,24482,24454,24475]},"id":1}]} 2024-03-16T10:27:07.834 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 50 2024-03-16T10:27:07.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.834+0000 7f7045fa9640 1 -- 172.21.15.77:0/3368619226 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f70280a2880 msgr2=0x7f70280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.834+0000 7f7045fa9640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f70280a2880 0x7f70280a4d40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7030000c00 tx=0x7f7030005fd0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.835+0000 7f7045fa9640 1 -- 172.21.15.77:0/3368619226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70401029f0 msgr2=0x7f7040073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.835+0000 7f7045fa9640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70401029f0 0x7f7040073bd0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f703400e9f0 tx=0x7f703400eeb0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.835+0000 7f7045fa9640 1 -- 172.21.15.77:0/3368619226 shutdown_connections 2024-03-16T10:27:07.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.835+0000 7f7045fa9640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70401033b0 0x7f7040074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.835+0000 7f7045fa9640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70401029f0 0x7f7040073bd0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.836 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.835+0000 7f7045fa9640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f70280a2880 0x7f70280a4d40 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.835+0000 7f7045fa9640 1 --2- 172.21.15.77:0/3368619226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7040101800 0x7f7040075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.835+0000 7f7045fa9640 1 -- 172.21.15.77:0/3368619226 >> 172.21.15.77:0/3368619226 conn(0x7f70400fcd90 msgr2=0x7f7040105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:07.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.835+0000 7f7045fa9640 1 -- 172.21.15.77:0/3368619226 shutdown_connections 2024-03-16T10:27:07.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.835+0000 7f7045fa9640 1 -- 172.21.15.77:0/3368619226 wait complete. 2024-03-16T10:27:07.847 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T10:27:07.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.951+0000 7f2568d52640 1 -- 172.21.15.77:0/1069125721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25641018f0 msgr2=0x7f2564101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.951+0000 7f2568d52640 1 --2- 172.21.15.77:0/1069125721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25641018f0 0x7f2564101d70 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f2558009900 tx=0x7f255802bc30 comp rx=0 tx=0).stop 2024-03-16T10:27:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.951+0000 7f2568d52640 1 -- 172.21.15.77:0/1069125721 shutdown_connections 2024-03-16T10:27:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.951+0000 7f2568d52640 1 --2- 172.21.15.77:0/1069125721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25641022b0 0x7f2564108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.951+0000 7f2568d52640 1 --2- 172.21.15.77:0/1069125721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25641018f0 0x7f2564101d70 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.951+0000 7f2568d52640 1 --2- 172.21.15.77:0/1069125721 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2564100700 0x7f2564100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.951+0000 7f2568d52640 1 -- 172.21.15.77:0/1069125721 >> 172.21.15.77:0/1069125721 conn(0x7f25640fbc70 msgr2=0x7f25640fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.951+0000 7f2568d52640 1 -- 172.21.15.77:0/1069125721 shutdown_connections 2024-03-16T10:27:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.951+0000 7f2568d52640 1 -- 172.21.15.77:0/1069125721 wait complete. 2024-03-16T10:27:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.951+0000 7f2568d52640 1 Processor -- start 2024-03-16T10:27:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2568d52640 1 -- start start 2024-03-16T10:27:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2568d52640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2564100700 0x7f2564104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2568d52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25641018f0 0x7f2564102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2568d52640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25641022b0 0x7f2564103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2568d52640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f256410c380 con 0x7f25641018f0 2024-03-16T10:27:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2568d52640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f256410c200 con 0x7f25641022b0 2024-03-16T10:27:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2568d52640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f256410c500 con 0x7f2564100700 2024-03-16T10:27:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2561d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25641018f0 0x7f2564102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f256259c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2564100700 0x7f2564104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2561d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25641018f0 0x7f2564102cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53542/0 (socket says 172.21.15.77:53542) 2024-03-16T10:27:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2561d9b640 1 -- 172.21.15.77:0/3471163015 learned_addr learned my addr 172.21.15.77:0/3471163015 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:27:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2562d9d640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25641022b0 0x7f2564103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:07.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2561d9b640 1 -- 172.21.15.77:0/3471163015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2564100700 msgr2=0x7f2564104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2561d9b640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2564100700 0x7f2564104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2561d9b640 1 -- 172.21.15.77:0/3471163015 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25641022b0 msgr2=0x7f2564103210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:07.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2561d9b640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25641022b0 0x7f2564103210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:07.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2561d9b640 1 -- 172.21.15.77:0/3471163015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2564103aa0 con 0x7f25641018f0 2024-03-16T10:27:07.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f256259c640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2564100700 0x7f2564104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T10:27:07.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2561d9b640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25641018f0 0x7f2564102cd0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2558009230 tx=0x7f255802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:07.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f25537fe640 1 -- 172.21.15.77:0/3471163015 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558041070 con 0x7f25641018f0 2024-03-16T10:27:07.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f25537fe640 1 -- 172.21.15.77:0/3471163015 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2558004450 con 0x7f25641018f0 2024-03-16T10:27:07.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f2568d52640 1 -- 172.21.15.77:0/3471163015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25641a1c30 con 0x7f25641018f0 2024-03-16T10:27:07.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.952+0000 7f25537fe640 1 -- 172.21.15.77:0/3471163015 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558003db0 con 0x7f25641018f0 2024-03-16T10:27:07.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.953+0000 7f2568d52640 1 -- 172.21.15.77:0/3471163015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25641a2180 con 0x7f25641018f0 2024-03-16T10:27:07.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.953+0000 7f2568d52640 1 -- 172.21.15.77:0/3471163015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25641a2350 con 0x7f25641018f0 2024-03-16T10:27:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.955+0000 7f25537fe640 1 -- 172.21.15.77:0/3471163015 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f2558044070 con 0x7f25641018f0 2024-03-16T10:27:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.955+0000 7f25537fe640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f254c0a2800 0x7f254c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.955+0000 7f256259c640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f254c0a2800 0x7f254c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.955+0000 7f25537fe640 1 -- 172.21.15.77:0/3471163015 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f25580f08d0 con 0x7f25641018f0 2024-03-16T10:27:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.956+0000 7f256259c640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f254c0a2800 0x7f254c0a4cc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2564101750 tx=0x7f2554005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:07.959+0000 7f25537fe640 1 -- 172.21.15.77:0/3471163015 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f25580bda00 con 0x7f25641018f0 2024-03-16T10:27:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.141+0000 7f2568d52640 1 -- 172.21.15.77:0/3471163015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25641a2520 con 0x7f25641018f0 2024-03-16T10:27:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.143+0000 7f25537fe640 1 -- 172.21.15.77:0/3471163015 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22496 (secure 0 0 0) 0x7f25580bd300 con 0x7f25641018f0 2024-03-16T10:27:08.144 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-03-16T10:27:08.145 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":88,"fsid":"c6ee6c60-e37c-11ee-95c9-87774f69a715","created":"2024-03-16T10:06:58.028804+0000","modified":"2024-03-16T10:23:44.816264+0000","last_up_change":"2024-03-16T10:14:09.244059+0000","last_in_change":"2024-03-16T10:13:49.151363+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T10:10:22.965004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T10:15:00.264039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T10:15:01.536067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b40ed61e-36f0-43e0-8f47-e80bc6001587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6803","nonce":2169364613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6805","nonce":2169364613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6809","nonce":2169364613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2169364613},{"type":"v1","addr":"172.21.15.77:6807","nonce":2169364613}]},"public_addr":"172.21.15.77:6803/2169364613","cluster_addr":"172.21.15.77:6805/2169364613","heartbeat_back_addr":"172.21.15.77:6809/2169364613","heartbeat_front_addr":"172.21.15.77:6807/2169364613","state":["exists","up"]},{"osd":1,"uuid":"f7d45429-9695-4b62-a44f-edc170f07838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6801","nonce":4276469545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6803","nonce":4276469545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6807","nonce":4276469545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4276469545},{"type":"v1","addr":"172.21.15.192:6805","nonce":4276469545}]},"public_addr":"172.21.15.192:6801/4276469545","cluster_addr":"172.21.15.192:6803/4276469545","heartbeat_back_addr":"172.21.15.192:6807/4276469545","heartbeat_front_addr":"172.21.15.192:6805/4276469545","state":["exists","up"]},{"osd":2,"uuid":"e84f6bf0-8054-4e5d-84eb-af5a7cb52cbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6801","nonce":2172182947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6803","nonce":2172182947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6807","nonce":2172182947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2172182947},{"type":"v1","addr":"172.21.15.193:6805","nonce":2172182947}]},"public_addr":"172.21.15.193:6801/2172182947","cluster_addr":"172.21.15.193:6803/2172182947","heartbeat_back_addr":"172.21.15.193:6807/2172182947","heartbeat_front_addr":"172.21.15.193:6805/2172182947","state":["exists","up"]},{"osd":3,"uuid":"d140b161-7795-4be8-bbb4-8a010ea7707e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6811","nonce":2887114423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6813","nonce":2887114423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6817","nonce":2887114423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2887114423},{"type":"v1","addr":"172.21.15.77:6815","nonce":2887114423}]},"public_addr":"172.21.15.77:6811/2887114423","cluster_addr":"172.21.15.77:6813/2887114423","heartbeat_back_addr":"172.21.15.77:6817/2887114423","heartbeat_front_addr":"172.21.15.77:6815/2887114423","state":["exists","up"]},{"osd":4,"uuid":"5101c752-e69f-4cff-af41-8818a614d0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6809","nonce":3948604951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6811","nonce":3948604951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6815","nonce":3948604951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3948604951},{"type":"v1","addr":"172.21.15.192:6813","nonce":3948604951}]},"public_addr":"172.21.15.192:6809/3948604951","cluster_addr":"172.21.15.192:6811/3948604951","heartbeat_back_addr":"172.21.15.192:6815/3948604951","heartbeat_front_addr":"172.21.15.192:6813/3948604951","state":["exists","up"]},{"osd":5,"uuid":"e66d1765-4379-49e9-9fa8-531107546970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6809","nonce":83797054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6811","nonce":83797054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6815","nonce":83797054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83797054},{"type":"v1","addr":"172.21.15.193:6813","nonce":83797054}]},"public_addr":"172.21.15.193:6809/83797054","cluster_addr":"172.21.15.193:6811/83797054","heartbeat_back_addr":"172.21.15.193:6815/83797054","heartbeat_front_addr":"172.21.15.193:6813/83797054","state":["exists","up"]},{"osd":6,"uuid":"e3ff0ae8-233d-4840-8cd1-a0050cd4aa43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6819","nonce":668716736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6821","nonce":668716736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6825","nonce":668716736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":668716736},{"type":"v1","addr":"172.21.15.77:6823","nonce":668716736}]},"public_addr":"172.21.15.77:6819/668716736","cluster_addr":"172.21.15.77:6821/668716736","heartbeat_back_addr":"172.21.15.77:6825/668716736","heartbeat_front_addr":"172.21.15.77:6823/668716736","state":["exists","up"]},{"osd":7,"uuid":"9c422662-cf7a-478c-ae70-309d21cbb34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6817","nonce":384726243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6819","nonce":384726243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6823","nonce":384726243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":384726243},{"type":"v1","addr":"172.21.15.192:6821","nonce":384726243}]},"public_addr":"172.21.15.192:6817/384726243","cluster_addr":"172.21.15.192:6819/384726243","heartbeat_back_addr":"172.21.15.192:6823/384726243","heartbeat_front_addr":"172.21.15.192:6821/384726243","state":["exists","up"]},{"osd":8,"uuid":"09fc6b3a-3c26-42b2-a975-51c1c4617605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6817","nonce":3436040269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6819","nonce":3436040269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6823","nonce":3436040269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3436040269},{"type":"v1","addr":"172.21.15.193:6821","nonce":3436040269}]},"public_addr":"172.21.15.193:6817/3436040269","cluster_addr":"172.21.15.193:6819/3436040269","heartbeat_back_addr":"172.21.15.193:6823/3436040269","heartbeat_front_addr":"172.21.15.193:6821/3436040269","state":["exists","up"]},{"osd":9,"uuid":"fe5dbc58-ce7a-48ac-816f-df30c3560a15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6827","nonce":2903071086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6829","nonce":2903071086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6833","nonce":2903071086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":2903071086},{"type":"v1","addr":"172.21.15.77:6831","nonce":2903071086}]},"public_addr":"172.21.15.77:6827/2903071086","cluster_addr":"172.21.15.77:6829/2903071086","heartbeat_back_addr":"172.21.15.77:6833/2903071086","heartbeat_front_addr":"172.21.15.77:6831/2903071086","state":["exists","up"]},{"osd":10,"uuid":"c77ee11a-014b-40fa-bb49-297f4144eec5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6825","nonce":946112377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6827","nonce":946112377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6831","nonce":946112377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":946112377},{"type":"v1","addr":"172.21.15.192:6829","nonce":946112377}]},"public_addr":"172.21.15.192:6825/946112377","cluster_addr":"172.21.15.192:6827/946112377","heartbeat_back_addr":"172.21.15.192:6831/946112377","heartbeat_front_addr":"172.21.15.192:6829/946112377","state":["exists","up"]},{"osd":11,"uuid":"fafc7e50-31b9-4bb9-9e17-aee4f80a2e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6825","nonce":2760387113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6827","nonce":2760387113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6831","nonce":2760387113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2760387113},{"type":"v1","addr":"172.21.15.193:6829","nonce":2760387113}]},"public_addr":"172.21.15.193:6825/2760387113","cluster_addr":"172.21.15.193:6827/2760387113","heartbeat_back_addr":"172.21.15.193:6831/2760387113","heartbeat_front_addr":"172.21.15.193:6829/2760387113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:32.179327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:09:56.251265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:20.021179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:10:44.941867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:09.534174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:34.471841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:11:59.015505+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:24.899196+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:12:49.728425+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:15.946980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:13:41.931395+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T10:14:07.116746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/233716866":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3665134846":"2024-03-17T10:07:58.634847+0000","172.21.15.77:0/3504149086":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/3120915723":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/465332686":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6800/1353424963":"2024-03-17T10:07:58.634847+0000","172.21.15.77:6801/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:6800/1093017736":"2024-03-17T10:07:32.924837+0000","172.21.15.77:0/1991017150":"2024-03-17T10:07:32.924837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T10:27:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.145+0000 7f2568d52640 1 -- 172.21.15.77:0/3471163015 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f254c0a2800 msgr2=0x7f254c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.145+0000 7f2568d52640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f254c0a2800 0x7f254c0a4cc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2564101750 tx=0x7f2554005f90 comp rx=0 tx=0).stop 2024-03-16T10:27:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.145+0000 7f2568d52640 1 -- 172.21.15.77:0/3471163015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25641018f0 msgr2=0x7f2564102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.145+0000 7f2568d52640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25641018f0 0x7f2564102cd0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2558009230 tx=0x7f255802c650 comp rx=0 tx=0).stop 2024-03-16T10:27:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.145+0000 7f2568d52640 1 -- 172.21.15.77:0/3471163015 shutdown_connections 2024-03-16T10:27:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.145+0000 7f2568d52640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25641022b0 0x7f2564103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.145+0000 7f2568d52640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25641018f0 0x7f2564102cd0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.145+0000 7f2568d52640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f254c0a2800 0x7f254c0a4cc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.145+0000 7f2568d52640 1 --2- 172.21.15.77:0/3471163015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2564100700 0x7f2564104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.145+0000 7f2568d52640 1 -- 172.21.15.77:0/3471163015 >> 172.21.15.77:0/3471163015 conn(0x7f25640fbc70 msgr2=0x7f25640fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:08.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.145+0000 7f2568d52640 1 -- 172.21.15.77:0/3471163015 shutdown_connections 2024-03-16T10:27:08.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.145+0000 7f2568d52640 1 -- 172.21.15.77:0/3471163015 wait complete. 2024-03-16T10:27:08.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:07 smithi077 ceph-mon[32400]: pgmap v652: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-16T10:27:08.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:07 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/675416805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:27:08.156 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:07 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3368619226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:27:08.158 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-16T10:27:08.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:07 smithi193 ceph-mon[36311]: pgmap v652: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-16T10:27:08.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:07 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/675416805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:27:08.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:07 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3368619226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:27:08.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.260+0000 7f9764ec6640 1 -- 172.21.15.77:0/3457545171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97601017f0 msgr2=0x7f9760101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.260+0000 7f9764ec6640 1 --2- 172.21.15.77:0/3457545171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97601017f0 0x7f9760101c10 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f9758009900 tx=0x7f975802bc30 comp rx=0 tx=0).stop 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.260+0000 7f9764ec6640 1 -- 172.21.15.77:0/3457545171 shutdown_connections 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.260+0000 7f9764ec6640 1 --2- 172.21.15.77:0/3457545171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97601033a0 0x7f9760109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.260+0000 7f9764ec6640 1 --2- 172.21.15.77:0/3457545171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97601029e0 0x7f9760102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.260+0000 7f9764ec6640 1 --2- 172.21.15.77:0/3457545171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97601017f0 0x7f9760101c10 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.260+0000 7f9764ec6640 1 -- 172.21.15.77:0/3457545171 >> 172.21.15.77:0/3457545171 conn(0x7f97600fcd40 msgr2=0x7f97600ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.260+0000 7f9764ec6640 1 -- 172.21.15.77:0/3457545171 shutdown_connections 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.260+0000 7f9764ec6640 1 -- 172.21.15.77:0/3457545171 wait complete. 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f9764ec6640 1 Processor -- start 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f9764ec6640 1 -- start start 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f9764ec6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97601017f0 0x7f9760197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f9764ec6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97601029e0 0x7f9760198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:08.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f9764ec6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97601033a0 0x7f976019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:08.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f9764ec6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f976010d4c0 con 0x7f97601017f0 2024-03-16T10:27:08.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f9764ec6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f976010d340 con 0x7f97601029e0 2024-03-16T10:27:08.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f9764ec6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f976010d640 con 0x7f97601033a0 2024-03-16T10:27:08.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f975e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97601017f0 0x7f9760197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:08.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f975e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97601017f0 0x7f9760197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53566/0 (socket says 172.21.15.77:53566) 2024-03-16T10:27:08.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f975e59c640 1 -- 172.21.15.77:0/1916615579 learned_addr learned my addr 172.21.15.77:0/1916615579 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:27:08.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f975ed9d640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97601033a0 0x7f976019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:08.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f975dd9b640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97601029e0 0x7f9760198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:08.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f975e59c640 1 -- 172.21.15.77:0/1916615579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97601033a0 msgr2=0x7f976019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:08.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f975e59c640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97601033a0 0x7f976019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f975e59c640 1 -- 172.21.15.77:0/1916615579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97601029e0 msgr2=0x7f9760198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:08.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f975e59c640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97601029e0 0x7f9760198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f975e59c640 1 -- 172.21.15.77:0/1916615579 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976019f760 con 0x7f97601017f0 2024-03-16T10:27:08.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.261+0000 7f975e59c640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97601017f0 0x7f9760197b40 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f9758000c00 tx=0x7f975802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:08.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.262+0000 7f974f7fe640 1 -- 172.21.15.77:0/1916615579 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758043020 con 0x7f97601017f0 2024-03-16T10:27:08.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.262+0000 7f974f7fe640 1 -- 172.21.15.77:0/1916615579 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97580054e0 con 0x7f97601017f0 2024-03-16T10:27:08.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.262+0000 7f9764ec6640 1 -- 172.21.15.77:0/1916615579 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f976019f9f0 con 0x7f97601017f0 2024-03-16T10:27:08.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.262+0000 7f974f7fe640 1 -- 172.21.15.77:0/1916615579 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758003c40 con 0x7f97601017f0 2024-03-16T10:27:08.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.262+0000 7f9764ec6640 1 -- 172.21.15.77:0/1916615579 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f976019ff40 con 0x7f97601017f0 2024-03-16T10:27:08.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.263+0000 7f9764ec6640 1 -- 172.21.15.77:0/1916615579 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9760067d40 con 0x7f97601017f0 2024-03-16T10:27:08.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.264+0000 7f974f7fe640 1 -- 172.21.15.77:0/1916615579 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f9758046020 con 0x7f97601017f0 2024-03-16T10:27:08.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.264+0000 7f974f7fe640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f97480a2800 0x7f97480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:08.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.264+0000 7f975dd9b640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f97480a2800 0x7f97480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:08.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.264+0000 7f974f7fe640 1 -- 172.21.15.77:0/1916615579 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8161+0+0 (secure 0 0 0) 0x7f97580f0dd0 con 0x7f97601017f0 2024-03-16T10:27:08.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.265+0000 7f975dd9b640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f97480a2800 0x7f97480a4cc0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9760199050 tx=0x7f9754005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:08.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.272+0000 7f974f7fe640 1 -- 172.21.15.77:0/1916615579 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f975804d020 con 0x7f97601017f0 2024-03-16T10:27:08.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:07 smithi192 ceph-mon[36354]: pgmap v652: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-16T10:27:08.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:07 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/675416805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:27:08.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:07 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3368619226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T10:27:08.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.493+0000 7f9764ec6640 1 -- 172.21.15.77:0/1916615579 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f97600623e0 con 0x7f97601017f0 2024-03-16T10:27:08.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.496+0000 7f974f7fe640 1 -- 172.21.15.77:0/1916615579 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v51) ==== 161+0+0 (secure 0 0 0) 0x7f97580bdf00 con 0x7f97601017f0 2024-03-16T10:27:08.497 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-16T10:27:08.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.499+0000 7f9764ec6640 1 -- 172.21.15.77:0/1916615579 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f97480a2800 msgr2=0x7f97480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:08.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.499+0000 7f9764ec6640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f97480a2800 0x7f97480a4cc0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9760199050 tx=0x7f9754005f90 comp rx=0 tx=0).stop 2024-03-16T10:27:08.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.499+0000 7f9764ec6640 1 -- 172.21.15.77:0/1916615579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97601017f0 msgr2=0x7f9760197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:08.500 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.499+0000 7f9764ec6640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97601017f0 0x7f9760197b40 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f9758000c00 tx=0x7f975802c650 comp rx=0 tx=0).stop 2024-03-16T10:27:08.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.499+0000 7f9764ec6640 1 -- 172.21.15.77:0/1916615579 shutdown_connections 2024-03-16T10:27:08.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.499+0000 7f9764ec6640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f97480a2800 0x7f97480a4cc0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.499+0000 7f9764ec6640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97601033a0 0x7f976019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.499+0000 7f9764ec6640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97601029e0 0x7f9760198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.499+0000 7f9764ec6640 1 --2- 172.21.15.77:0/1916615579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97601017f0 0x7f9760197b40 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.499+0000 7f9764ec6640 1 -- 172.21.15.77:0/1916615579 >> 172.21.15.77:0/1916615579 conn(0x7f97600fcd40 msgr2=0x7f9760105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:08.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.499+0000 7f9764ec6640 1 -- 172.21.15.77:0/1916615579 shutdown_connections 2024-03-16T10:27:08.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.499+0000 7f9764ec6640 1 -- 172.21.15.77:0/1916615579 wait complete. 2024-03-16T10:27:08.513 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-16T10:27:08.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 -- 172.21.15.77:0/3775833712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c1022a0 msgr2=0x7f845c108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:08.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 --2- 172.21.15.77:0/3775833712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c1022a0 0x7f845c108b30 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f8454009ca0 tx=0x7f845402c0f0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 -- 172.21.15.77:0/3775833712 shutdown_connections 2024-03-16T10:27:08.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 --2- 172.21.15.77:0/3775833712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c1022a0 0x7f845c108b30 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 --2- 172.21.15.77:0/3775833712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f845c1018e0 0x7f845c101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 --2- 172.21.15.77:0/3775833712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f845c1006f0 0x7f845c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 -- 172.21.15.77:0/3775833712 >> 172.21.15.77:0/3775833712 conn(0x7f845c0fbc80 msgr2=0x7f845c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:08.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 -- 172.21.15.77:0/3775833712 shutdown_connections 2024-03-16T10:27:08.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 -- 172.21.15.77:0/3775833712 wait complete. 2024-03-16T10:27:08.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 Processor -- start 2024-03-16T10:27:08.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 -- start start 2024-03-16T10:27:08.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c1006f0 0x7f845c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:08.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f845c1018e0 0x7f845c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f845c1022a0 0x7f845c19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f845c10d490 con 0x7f845c1006f0 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f845c10d310 con 0x7f845c1022a0 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.620+0000 7f84640c6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f845c10d610 con 0x7f845c1018e0 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8461e62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c1006f0 0x7f845c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8461e62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c1006f0 0x7f845c197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53618/0 (socket says 172.21.15.77:53618) 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8461e62640 1 -- 172.21.15.77:0/453623414 learned_addr learned my addr 172.21.15.77:0/453623414 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8461661640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f845c1018e0 0x7f845c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8462663640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f845c1022a0 0x7f845c19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8461e62640 1 -- 172.21.15.77:0/453623414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f845c1018e0 msgr2=0x7f845c198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8461e62640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f845c1018e0 0x7f845c198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8461e62640 1 -- 172.21.15.77:0/453623414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f845c1022a0 msgr2=0x7f845c19f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:08.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8461e62640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f845c1022a0 0x7f845c19f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8461e62640 1 -- 172.21.15.77:0/453623414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845c19f730 con 0x7f845c1006f0 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8461e62640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c1006f0 0x7f845c197b10 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f845800c970 tx=0x7f845800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8452ffd640 1 -- 172.21.15.77:0/453623414 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8458004220 con 0x7f845c1006f0 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f84640c6640 1 -- 172.21.15.77:0/453623414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f845c19fa20 con 0x7f845c1006f0 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8452ffd640 1 -- 172.21.15.77:0/453623414 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84580043c0 con 0x7f845c1006f0 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f8452ffd640 1 -- 172.21.15.77:0/453623414 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845800f450 con 0x7f845c1006f0 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.621+0000 7f84640c6640 1 -- 172.21.15.77:0/453623414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f845c19ff70 con 0x7f845c1006f0 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.622+0000 7f84640c6640 1 -- 172.21.15.77:0/453623414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f845c107a10 con 0x7f845c1006f0 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.623+0000 7f8452ffd640 1 -- 172.21.15.77:0/453623414 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f84580079b0 con 0x7f845c1006f0 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.624+0000 7f8452ffd640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84480a2800 0x7f84480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.624+0000 7f8461661640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84480a2800 0x7f84480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.624+0000 7f8452ffd640 1 -- 172.21.15.77:0/453623414 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8591+0+0 (secure 0 0 0) 0x7f84580ce890 con 0x7f845c1006f0 2024-03-16T10:27:08.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.624+0000 7f8461661640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84480a2800 0x7f84480a4cc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f845c199020 tx=0x7f844c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:08.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.628+0000 7f8452ffd640 1 -- 172.21.15.77:0/453623414 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8458014030 con 0x7f845c1006f0 2024-03-16T10:27:08.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:08.840+0000 7f84640c6640 1 -- 172.21.15.77:0/453623414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f845c02d430 con 0x7f845c1006f0 2024-03-16T10:27:08.917 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-e[66486]: ignoring --setuser ceph since I am not root 2024-03-16T10:27:08.917 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-e[66486]: ignoring --setgroup ceph since I am not root 2024-03-16T10:27:08.917 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-mds[66490]: -- 172.21.15.192:0/1135987341 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e8f97ada40 con 0x55e8f9780800 2024-03-16T10:27:08.917 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-e[66486]: starting mds.e at 2024-03-16T10:27:08.917 INFO:journalctl@ceph.mds.k.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-k[66166]: ignoring --setuser ceph since I am not root 2024-03-16T10:27:08.918 INFO:journalctl@ceph.mds.k.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-k[66166]: ignoring --setgroup ceph since I am not root 2024-03-16T10:27:08.918 INFO:journalctl@ceph.mds.k.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-k[66166]: starting mds.k at 2024-03-16T10:27:08.918 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b[67101]: ignoring --setuser ceph since I am not root 2024-03-16T10:27:08.918 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b[67101]: ignoring --setgroup ceph since I am not root 2024-03-16T10:27:08.918 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-mds[67105]: -- 172.21.15.192:0/47294697 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a67386ba40 con 0x55a67383f000 2024-03-16T10:27:08.918 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b[67101]: starting mds.b at 2024-03-16T10:27:08.919 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-h[66794]: ignoring --setuser ceph since I am not root 2024-03-16T10:27:08.919 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-h[66794]: ignoring --setgroup ceph since I am not root 2024-03-16T10:27:08.919 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-mds[66798]: -- 172.21.15.192:0/1909284984 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c5351a7a40 con 0x55c53517a800 2024-03-16T10:27:08.919 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-h[66794]: starting mds.h at 2024-03-16T10:27:08.962 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a[32396]: 2024-03-16T10:27:08.494+0000 7ff0364c1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T10:27:08.963 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a[72289]: ignoring --setuser ceph since I am not root 2024-03-16T10:27:08.963 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a[72289]: ignoring --setgroup ceph since I am not root 2024-03-16T10:27:08.963 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-mds[72293]: -- 172.21.15.77:0/527958766 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55569cdd1a40 con 0x55569cda5000 2024-03-16T10:27:08.963 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a[72289]: starting mds.a at 2024-03-16T10:27:08.963 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-g[71925]: ignoring --setuser ceph since I am not root 2024-03-16T10:27:08.963 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-g[71925]: ignoring --setgroup ceph since I am not root 2024-03-16T10:27:08.963 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-mds[71929]: -- 172.21.15.77:0/1046779411 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ed088c9a40 con 0x55ed0889d000 2024-03-16T10:27:08.963 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-g[71925]: starting mds.g at 2024-03-16T10:27:08.964 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-c[67182]: ignoring --setuser ceph since I am not root 2024-03-16T10:27:08.964 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-c[67182]: ignoring --setgroup ceph since I am not root 2024-03-16T10:27:08.964 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mds[67186]: -- 172.21.15.193:0/1862366883 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5624c4efda40 con 0x5624c4ed1000 2024-03-16T10:27:08.964 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-c[67182]: starting mds.c at 2024-03-16T10:27:08.964 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-i[67505]: ignoring --setuser ceph since I am not root 2024-03-16T10:27:08.964 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-i[67505]: ignoring --setgroup ceph since I am not root 2024-03-16T10:27:08.965 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mds[67509]: -- 172.21.15.193:0/980893140 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56414457da40 con 0x564144551000 2024-03-16T10:27:08.965 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-i[67505]: starting mds.i at 2024-03-16T10:27:08.965 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-l[66560]: ignoring --setuser ceph since I am not root 2024-03-16T10:27:08.965 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-l[66560]: ignoring --setgroup ceph since I am not root 2024-03-16T10:27:08.965 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mds[66564]: -- 172.21.15.193:0/1072333828 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a2e8ec7a40 con 0x55a2e8e9b000 2024-03-16T10:27:08.965 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-l[66560]: starting mds.l at 2024-03-16T10:27:08.965 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f[70835]: ignoring --setuser ceph since I am not root 2024-03-16T10:27:08.965 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f[70835]: ignoring --setgroup ceph since I am not root 2024-03-16T10:27:08.966 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mds[70839]: -- 172.21.15.193:0/1442001372 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c9f2c39a40 con 0x55c9f2c0c800 2024-03-16T10:27:08.966 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f[70835]: starting mds.f at 2024-03-16T10:27:09.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/3471163015' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:27:09.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1916615579' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T10:27:09.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-mon[32400]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T10:27:09.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-mon[32400]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T10:27:09.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1916615579' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T10:27:09.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-mon[32400]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T10:27:09.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-mon[32400]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-16T10:27:09.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/453623414' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:27:09.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:08 smithi077 ceph-mon[32400]: pgmap v654: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 716 B/s wr, 12 op/s 2024-03-16T10:27:09.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/3471163015' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:27:09.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1916615579' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T10:27:09.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mon[36311]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T10:27:09.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mon[36311]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T10:27:09.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1916615579' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T10:27:09.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mon[36311]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T10:27:09.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mon[36311]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-16T10:27:09.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/453623414' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:27:09.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:08 smithi193 ceph-mon[36311]: pgmap v654: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 716 B/s wr, 12 op/s 2024-03-16T10:27:09.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/3471163015' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T10:27:09.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1916615579' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T10:27:09.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-mon[36354]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T10:27:09.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-mon[36354]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T10:27:09.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1916615579' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T10:27:09.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-mon[36354]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T10:27:09.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-mon[36354]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-16T10:27:09.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/453623414' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T10:27:09.417 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:08 smithi192 ceph-mon[36354]: pgmap v654: 97 pgs: 97 active+clean; 220 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 716 B/s wr, 12 op/s 2024-03-16T10:27:09.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.502+0000 7f8452ffd640 1 -- 172.21.15.77:0/453623414 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v52) ==== 104+0+0 (secure 0 0 0) 0x7f845809b180 con 0x7f845c1006f0 2024-03-16T10:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.505+0000 7f84640c6640 1 -- 172.21.15.77:0/453623414 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84480a2800 msgr2=0x7f84480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.505+0000 7f84640c6640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84480a2800 0x7f84480a4cc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f845c199020 tx=0x7f844c005f90 comp rx=0 tx=0).stop 2024-03-16T10:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.505+0000 7f84640c6640 1 -- 172.21.15.77:0/453623414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c1006f0 msgr2=0x7f845c197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.505+0000 7f84640c6640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c1006f0 0x7f845c197b10 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f845800c970 tx=0x7f845800ce30 comp rx=0 tx=0).stop 2024-03-16T10:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.505+0000 7f84640c6640 1 -- 172.21.15.77:0/453623414 shutdown_connections 2024-03-16T10:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.505+0000 7f84640c6640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f845c1022a0 0x7f845c19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.505+0000 7f84640c6640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f84480a2800 0x7f84480a4cc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.505+0000 7f84640c6640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f845c1018e0 0x7f845c198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:09.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.505+0000 7f84640c6640 1 --2- 172.21.15.77:0/453623414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c1006f0 0x7f845c197b10 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:09.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.505+0000 7f84640c6640 1 -- 172.21.15.77:0/453623414 >> 172.21.15.77:0/453623414 conn(0x7f845c0fbc80 msgr2=0x7f845c104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:09.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.505+0000 7f84640c6640 1 -- 172.21.15.77:0/453623414 shutdown_connections 2024-03-16T10:27:09.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.505+0000 7f84640c6640 1 -- 172.21.15.77:0/453623414 wait complete. 2024-03-16T10:27:09.522 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-16T10:27:09.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.632+0000 7fe202697640 1 -- 172.21.15.77:0/247105851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1fc0ffe60 msgr2=0x7fe1fc100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:09.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.632+0000 7fe202697640 1 --2- 172.21.15.77:0/247105851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1fc0ffe60 0x7fe1fc100280 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009900 tx=0x7fe1ec02bc30 comp rx=0 tx=0).stop 2024-03-16T10:27:09.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.632+0000 7fe202697640 1 -- 172.21.15.77:0/247105851 shutdown_connections 2024-03-16T10:27:09.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.632+0000 7fe202697640 1 --2- 172.21.15.77:0/247105851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1fc101a00 0x7fe1fc108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:09.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.632+0000 7fe202697640 1 --2- 172.21.15.77:0/247105851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1fc101040 0x7fe1fc1014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:09.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.632+0000 7fe202697640 1 --2- 172.21.15.77:0/247105851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1fc0ffe60 0x7fe1fc100280 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:09.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.632+0000 7fe202697640 1 -- 172.21.15.77:0/247105851 >> 172.21.15.77:0/247105851 conn(0x7fe1fc0fb420 msgr2=0x7fe1fc0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.632+0000 7fe202697640 1 -- 172.21.15.77:0/247105851 shutdown_connections 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.632+0000 7fe202697640 1 -- 172.21.15.77:0/247105851 wait complete. 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.632+0000 7fe202697640 1 Processor -- start 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.632+0000 7fe202697640 1 -- start start 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe202697640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1fc0ffe60 0x7fe1fc197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe202697640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1fc101040 0x7fe1fc1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe202697640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1fc101a00 0x7fe1fc19f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe202697640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc10b320 con 0x7fe1fc101040 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe202697640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc10b1a0 con 0x7fe1fc101a00 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe202697640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc10b4a0 con 0x7fe1fc0ffe60 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe1fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1fc101040 0x7fe1fc1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe1fb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1fc101040 0x7fe1fc1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53636/0 (socket says 172.21.15.77:53636) 2024-03-16T10:27:09.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe1fb7fe640 1 -- 172.21.15.77:0/2718766632 learned_addr learned my addr 172.21.15.77:0/2718766632 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe1fbfff640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1fc0ffe60 0x7fe1fc197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe1fb7fe640 1 -- 172.21.15.77:0/2718766632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1fc0ffe60 msgr2=0x7fe1fc197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe1fb7fe640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1fc0ffe60 0x7fe1fc197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe1fb7fe640 1 -- 172.21.15.77:0/2718766632 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1fc101a00 msgr2=0x7fe1fc19f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe200c34640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1fc101a00 0x7fe1fc19f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe1fb7fe640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1fc101a00 0x7fe1fc19f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe1fb7fe640 1 -- 172.21.15.77:0/2718766632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1fc19f7c0 con 0x7fe1fc101040 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.633+0000 7fe1fb7fe640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1fc101040 0x7fe1fc1980e0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000b4a0 tx=0x7fe1f000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.634+0000 7fe1f97fa640 1 -- 172.21.15.77:0/2718766632 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0014070 con 0x7fe1fc101040 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.634+0000 7fe1f97fa640 1 -- 172.21.15.77:0/2718766632 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1f0010af0 con 0x7fe1fc101040 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.634+0000 7fe202697640 1 -- 172.21.15.77:0/2718766632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1fc19fab0 con 0x7fe1fc101040 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.634+0000 7fe1f97fa640 1 -- 172.21.15.77:0/2718766632 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0017d10 con 0x7fe1fc101040 2024-03-16T10:27:09.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.634+0000 7fe202697640 1 -- 172.21.15.77:0/2718766632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1fc1a0000 con 0x7fe1fc101040 2024-03-16T10:27:09.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.634+0000 7fe202697640 1 -- 172.21.15.77:0/2718766632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1fc107280 con 0x7fe1fc101040 2024-03-16T10:27:09.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.636+0000 7fe1f97fa640 1 -- 172.21.15.77:0/2718766632 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7fe1f0017070 con 0x7fe1fc101040 2024-03-16T10:27:09.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.636+0000 7fe1f97fa640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe1e40a27f0 0x7fe1e40a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:09.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.636+0000 7fe1fbfff640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe1e40a27f0 0x7fe1e40a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:09.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.636+0000 7fe1f97fa640 1 -- 172.21.15.77:0/2718766632 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8591+0+0 (secure 0 0 0) 0x7fe1f00cdef0 con 0x7fe1fc101040 2024-03-16T10:27:09.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.637+0000 7fe1fbfff640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe1e40a27f0 0x7fe1e40a4cb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec000c00 tx=0x7fe1ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:09.641 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.641+0000 7fe1f97fa640 1 -- 172.21.15.77:0/2718766632 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe1f009ae30 con 0x7fe1fc101040 2024-03-16T10:27:09.815 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:09.814+0000 7fe202697640 1 -- 172.21.15.77:0/2718766632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe1fc02d1b0 con 0x7fe1fc101040 2024-03-16T10:27:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T10:27:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T10:27:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: Cluster is now healthy 2024-03-16T10:27:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/453623414' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T10:27:10.245 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.192:6834/4217286214,v1:172.21.15.192:6835/4217286214] up:boot 2024-03-16T10:27:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.77:6838/1202008633,v1:172.21.15.77:6839/1202008633] up:boot 2024-03-16T10:27:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.193:6834/865091424,v1:172.21.15.193:6835/865091424] up:boot 2024-03-16T10:27:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.193:6836/2926840270,v1:172.21.15.193:6837/2926840270] up:boot 2024-03-16T10:27:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.77:6840/4103084580,v1:172.21.15.77:6841/4103084580] up:boot 2024-03-16T10:27:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.192:6832/717756389,v1:172.21.15.192:6833/717756389] up:boot 2024-03-16T10:27:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.193:6838/3784098191,v1:172.21.15.193:6839/3784098191] up:boot 2024-03-16T10:27:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.193:6832/1731141210,v1:172.21.15.193:6833/1731141210] up:boot 2024-03-16T10:27:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.192:6836/4019123184,v1:172.21.15.192:6838/4019123184] up:boot 2024-03-16T10:27:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: mds.? [v2:172.21.15.192:6837/2643916452,v1:172.21.15.192:6839/2643916452] up:boot 2024-03-16T10:27:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: fsmap 12 up:standby 2024-03-16T10:27:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T10:27:10.246 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T10:27:10.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T10:27:10.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T10:27:10.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T10:27:10.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T10:27:10.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T10:27:10.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T10:27:10.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T10:27:10.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T10:27:10.247 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:09 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2718766632' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T10:27:10.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: Cluster is now healthy 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/453623414' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.192:6834/4217286214,v1:172.21.15.192:6835/4217286214] up:boot 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.77:6838/1202008633,v1:172.21.15.77:6839/1202008633] up:boot 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.193:6834/865091424,v1:172.21.15.193:6835/865091424] up:boot 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.193:6836/2926840270,v1:172.21.15.193:6837/2926840270] up:boot 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.77:6840/4103084580,v1:172.21.15.77:6841/4103084580] up:boot 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.192:6832/717756389,v1:172.21.15.192:6833/717756389] up:boot 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.193:6838/3784098191,v1:172.21.15.193:6839/3784098191] up:boot 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.193:6832/1731141210,v1:172.21.15.193:6833/1731141210] up:boot 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.192:6836/4019123184,v1:172.21.15.192:6838/4019123184] up:boot 2024-03-16T10:27:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: mds.? [v2:172.21.15.192:6837/2643916452,v1:172.21.15.192:6839/2643916452] up:boot 2024-03-16T10:27:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: fsmap 12 up:standby 2024-03-16T10:27:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T10:27:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T10:27:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T10:27:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T10:27:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T10:27:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T10:27:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T10:27:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T10:27:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T10:27:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T10:27:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:09 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2718766632' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T10:27:10.319 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: Cluster is now healthy 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/453623414' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.192:6834/4217286214,v1:172.21.15.192:6835/4217286214] up:boot 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.77:6838/1202008633,v1:172.21.15.77:6839/1202008633] up:boot 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.193:6834/865091424,v1:172.21.15.193:6835/865091424] up:boot 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.193:6836/2926840270,v1:172.21.15.193:6837/2926840270] up:boot 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.77:6840/4103084580,v1:172.21.15.77:6841/4103084580] up:boot 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.192:6832/717756389,v1:172.21.15.192:6833/717756389] up:boot 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.193:6838/3784098191,v1:172.21.15.193:6839/3784098191] up:boot 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.193:6832/1731141210,v1:172.21.15.193:6833/1731141210] up:boot 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.192:6836/4019123184,v1:172.21.15.192:6838/4019123184] up:boot 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: mds.? [v2:172.21.15.192:6837/2643916452,v1:172.21.15.192:6839/2643916452] up:boot 2024-03-16T10:27:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: fsmap 12 up:standby 2024-03-16T10:27:10.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T10:27:10.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T10:27:10.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T10:27:10.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T10:27:10.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T10:27:10.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T10:27:10.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T10:27:10.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T10:27:10.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T10:27:10.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: from='mgr.14152 172.21.15.77:0/2450491629' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T10:27:10.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:09 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2718766632' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T10:27:10.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.503+0000 7fe1f97fa640 1 -- 172.21.15.77:0/2718766632 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v90) ==== 181+0+0 (secure 0 0 0) 0x7fe1f009a730 con 0x7fe1fc101040 2024-03-16T10:27:10.504 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2024-03-16T10:27:10.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.506+0000 7fe202697640 1 -- 172.21.15.77:0/2718766632 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe1e40a27f0 msgr2=0x7fe1e40a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:10.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.506+0000 7fe202697640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe1e40a27f0 0x7fe1e40a4cb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec000c00 tx=0x7fe1ec005fd0 comp rx=0 tx=0).stop 2024-03-16T10:27:10.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.506+0000 7fe202697640 1 -- 172.21.15.77:0/2718766632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1fc101040 msgr2=0x7fe1fc1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:10.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.506+0000 7fe202697640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1fc101040 0x7fe1fc1980e0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000b4a0 tx=0x7fe1f000b960 comp rx=0 tx=0).stop 2024-03-16T10:27:10.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.506+0000 7fe202697640 1 -- 172.21.15.77:0/2718766632 shutdown_connections 2024-03-16T10:27:10.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.506+0000 7fe202697640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1fc101a00 0x7fe1fc19f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:10.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.506+0000 7fe202697640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1fc101040 0x7fe1fc1980e0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:10.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.506+0000 7fe202697640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7fe1e40a27f0 0x7fe1e40a4cb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:10.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.506+0000 7fe202697640 1 --2- 172.21.15.77:0/2718766632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1fc0ffe60 0x7fe1fc197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:10.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.506+0000 7fe202697640 1 -- 172.21.15.77:0/2718766632 >> 172.21.15.77:0/2718766632 conn(0x7fe1fc0fb420 msgr2=0x7fe1fc104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:10.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.507+0000 7fe202697640 1 -- 172.21.15.77:0/2718766632 shutdown_connections 2024-03-16T10:27:10.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.507+0000 7fe202697640 1 -- 172.21.15.77:0/2718766632 wait complete. 2024-03-16T10:27:10.522 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-16T10:27:10.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.628+0000 7f817a8b2640 1 -- 172.21.15.77:0/3080565465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8174102a00 msgr2=0x7f8174102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:10.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.628+0000 7f817a8b2640 1 --2- 172.21.15.77:0/3080565465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8174102a00 0x7f8174102e80 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f8168009900 tx=0x7f816802bc30 comp rx=0 tx=0).stop 2024-03-16T10:27:10.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.628+0000 7f817a8b2640 1 -- 172.21.15.77:0/3080565465 shutdown_connections 2024-03-16T10:27:10.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.628+0000 7f817a8b2640 1 --2- 172.21.15.77:0/3080565465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81741033c0 0x7f8174109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:10.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.628+0000 7f817a8b2640 1 --2- 172.21.15.77:0/3080565465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8174102a00 0x7f8174102e80 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:10.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.628+0000 7f817a8b2640 1 --2- 172.21.15.77:0/3080565465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8174101810 0x7f8174101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:10.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.628+0000 7f817a8b2640 1 -- 172.21.15.77:0/3080565465 >> 172.21.15.77:0/3080565465 conn(0x7f81740fcd80 msgr2=0x7f81740ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:10.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.628+0000 7f817a8b2640 1 -- 172.21.15.77:0/3080565465 shutdown_connections 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.628+0000 7f817a8b2640 1 -- 172.21.15.77:0/3080565465 wait complete. 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.628+0000 7f817a8b2640 1 Processor -- start 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.628+0000 7f817a8b2640 1 -- start start 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f817a8b2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8174101810 0x7f8174199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f817a8b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8174102a00 0x7f817419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f817a8b2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81741033c0 0x7f81741a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f817a8b2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f817410d490 con 0x7f8174102a00 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f817a8b2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f817410d310 con 0x7f81741033c0 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f817a8b2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f817410d610 con 0x7f8174101810 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f81737fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8174102a00 0x7f817419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f81737fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8174102a00 0x7f817419a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53650/0 (socket says 172.21.15.77:53650) 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f81737fe640 1 -- 172.21.15.77:0/1999540655 learned_addr learned my addr 172.21.15.77:0/1999540655 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f8178e4f640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81741033c0 0x7f81741a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f8173fff640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8174101810 0x7f8174199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f81737fe640 1 -- 172.21.15.77:0/1999540655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8174101810 msgr2=0x7f8174199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f81737fe640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8174101810 0x7f8174199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f81737fe640 1 -- 172.21.15.77:0/1999540655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81741033c0 msgr2=0x7f81741a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f81737fe640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81741033c0 0x7f81741a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f81737fe640 1 -- 172.21.15.77:0/1999540655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81741a1970 con 0x7f8174102a00 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f8173fff640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8174101810 0x7f8174199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f81737fe640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8174102a00 0x7f817419a290 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f8168009230 tx=0x7f8168005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f81717fa640 1 -- 172.21.15.77:0/1999540655 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168041070 con 0x7f8174102a00 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f81717fa640 1 -- 172.21.15.77:0/1999540655 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81680054e0 con 0x7f8174102a00 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f817a8b2640 1 -- 172.21.15.77:0/1999540655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81741a1c00 con 0x7f8174102a00 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f81717fa640 1 -- 172.21.15.77:0/1999540655 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168034440 con 0x7f8174102a00 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.629+0000 7f817a8b2640 1 -- 172.21.15.77:0/1999540655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81741a2140 con 0x7f8174102a00 2024-03-16T10:27:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.630+0000 7f817a8b2640 1 -- 172.21.15.77:0/1999540655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81741a2310 con 0x7f8174102a00 2024-03-16T10:27:10.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.631+0000 7f81717fa640 1 -- 172.21.15.77:0/1999540655 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 137290+0+0 (secure 0 0 0) 0x7f8168044070 con 0x7f8174102a00 2024-03-16T10:27:10.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.632+0000 7f81717fa640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f815c0a2880 0x7f815c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T10:27:10.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.632+0000 7f8173fff640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f815c0a2880 0x7f815c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T10:27:10.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.632+0000 7f81717fa640 1 -- 172.21.15.77:0/1999540655 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8095+0+0 (secure 0 0 0) 0x7f81680f0600 con 0x7f8174102a00 2024-03-16T10:27:10.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.632+0000 7f8173fff640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f815c0a2880 0x7f815c0a4d40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8164005d00 tx=0x7f8164005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T10:27:10.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.636+0000 7f81717fa640 1 -- 172.21.15.77:0/1999540655 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f81680bd780 con 0x7f8174102a00 2024-03-16T10:27:10.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:10.812+0000 7f817a8b2640 1 -- 172.21.15.77:0/1999540655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f81741a24e0 con 0x7f8174102a00 2024-03-16T10:27:11.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.508+0000 7f81717fa640 1 -- 172.21.15.77:0/1999540655 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v91) ==== 169+0+0 (secure 0 0 0) 0x7f81680bd080 con 0x7f8174102a00 2024-03-16T10:27:11.510 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2024-03-16T10:27:11.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.512+0000 7f817a8b2640 1 -- 172.21.15.77:0/1999540655 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f815c0a2880 msgr2=0x7f815c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:11.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.512+0000 7f817a8b2640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f815c0a2880 0x7f815c0a4d40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8164005d00 tx=0x7f8164005c90 comp rx=0 tx=0).stop 2024-03-16T10:27:11.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.512+0000 7f817a8b2640 1 -- 172.21.15.77:0/1999540655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8174102a00 msgr2=0x7f817419a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T10:27:11.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.512+0000 7f817a8b2640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8174102a00 0x7f817419a290 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f8168009230 tx=0x7f8168005dd0 comp rx=0 tx=0).stop 2024-03-16T10:27:11.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.512+0000 7f817a8b2640 1 -- 172.21.15.77:0/1999540655 shutdown_connections 2024-03-16T10:27:11.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.512+0000 7f817a8b2640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81741033c0 0x7f81741a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:11.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.512+0000 7f817a8b2640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8174102a00 0x7f817419a290 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:11.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.512+0000 7f817a8b2640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.77:6800/1427985582,v1:172.21.15.77:6801/1427985582] conn(0x7f815c0a2880 0x7f815c0a4d40 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:11.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.512+0000 7f817a8b2640 1 --2- 172.21.15.77:0/1999540655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8174101810 0x7f8174199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T10:27:11.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.512+0000 7f817a8b2640 1 -- 172.21.15.77:0/1999540655 >> 172.21.15.77:0/1999540655 conn(0x7f81740fcd80 msgr2=0x7f8174073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T10:27:11.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.512+0000 7f817a8b2640 1 -- 172.21.15.77:0/1999540655 shutdown_connections 2024-03-16T10:27:11.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-03-16T10:27:11.512+0000 7f817a8b2640 1 -- 172.21.15.77:0/1999540655 wait complete. 2024-03-16T10:27:11.530 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T10:27:11.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T10:27:11.590 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T10:27:11.622 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-16T10:27:11.622 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T10:27:11.637 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T10:27:11.652 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T10:27:11.677 INFO:tasks.cephadm:Stopping all daemons... 2024-03-16T10:27:11.677 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-16T10:27:11.677 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.a 2024-03-16T10:27:11.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:11 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/2718766632' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T10:27:11.916 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:11 smithi192 ceph-mon[36354]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T10:27:11.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:11 smithi192 ceph-mon[36354]: from='client.? 172.21.15.77:0/1999540655' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T10:27:11.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:11 smithi192 ceph-mon[36354]: pgmap v656: 65 pgs: 65 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:27:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:11 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/2718766632' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T10:27:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:11 smithi077 ceph-mon[32400]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T10:27:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:11 smithi077 ceph-mon[32400]: from='client.? 172.21.15.77:0/1999540655' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T10:27:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:11 smithi077 ceph-mon[32400]: pgmap v656: 65 pgs: 65 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:27:11.995 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:11 smithi077 systemd[1]: Stopping Ceph mon.a for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:27:12.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:11 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/2718766632' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T10:27:12.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:11 smithi193 ceph-mon[36311]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T10:27:12.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:11 smithi193 ceph-mon[36311]: from='client.? 172.21.15.77:0/1999540655' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T10:27:12.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:11 smithi193 ceph-mon[36311]: pgmap v656: 65 pgs: 65 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T10:27:12.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:11 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a[32396]: 2024-03-16T10:27:11.996+0000 7ff03bccc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:27:12.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:11 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a[32396]: 2024-03-16T10:27:11.996+0000 7ff03bccc700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-16T10:27:12.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:12 smithi077 podman[98320]: 2024-03-16 10:27:12.242419005 +0000 UTC m=+0.378561143 container died e223b065e454dab2fb1a8ac03938cd8ff6e5bf2079ab742e294be811e58ba028 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-03-16T10:27:12.891 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:12 smithi077 podman[98320]: 2024-03-16 10:27:12.673088253 +0000 UTC m=+0.809230390 container cleanup e223b065e454dab2fb1a8ac03938cd8ff6e5bf2079ab742e294be811e58ba028 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:27:12.891 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:12 smithi077 bash[98320]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a 2024-03-16T10:27:13.141 INFO:journalctl@ceph.mon.a.smithi077.stdout:Mar 16 10:27:12 smithi077 podman[98334]: 2024-03-16 10:27:12.891615921 +0000 UTC m=+0.652248143 container remove e223b065e454dab2fb1a8ac03938cd8ff6e5bf2079ab742e294be811e58ba028 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-16T10:27:13.145 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.a.service' 2024-03-16T10:27:13.188 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:27:13.188 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-16T10:27:13.189 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-16T10:27:13.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.b 2024-03-16T10:27:13.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:13 smithi192 systemd[1]: Stopping Ceph mon.b for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:27:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-b[36347]: 2024-03-16T10:27:13.518+0000 7fb0d1623700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:27:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-b[36347]: 2024-03-16T10:27:13.518+0000 7fb0d1623700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-16T10:27:13.917 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:13 smithi192 podman[71089]: 2024-03-16 10:27:13.703105904 +0000 UTC m=+0.315195233 container died 632d7ca9fce96d32df18434062e77000e0659db114ac4218fa6d718db3df47a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-16T10:27:14.297 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:14 smithi192 podman[71089]: 2024-03-16 10:27:14.078242512 +0000 UTC m=+0.690331840 container cleanup 632d7ca9fce96d32df18434062e77000e0659db114ac4218fa6d718db3df47a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-16T10:27:14.297 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 16 10:27:14 smithi192 bash[71089]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-b 2024-03-16T10:27:14.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.b.service' 2024-03-16T10:27:14.570 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:27:14.570 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-16T10:27:14.571 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-16T10:27:14.571 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.c 2024-03-16T10:27:14.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:14 smithi193 systemd[1]: Stopping Ceph mon.c for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:27:15.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:14 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-c[36307]: 2024-03-16T10:27:14.897+0000 7f0e43e51700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:27:15.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:14 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-c[36307]: 2024-03-16T10:27:14.897+0000 7f0e43e51700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-16T10:27:15.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:15 smithi193 podman[72169]: 2024-03-16 10:27:15.091700424 +0000 UTC m=+0.325203767 container died aff55e0c4197a64dca758112ef56969a1dfb5084573623e16bf173e53d51b123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-16T10:27:15.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:15 smithi193 podman[72169]: 2024-03-16 10:27:15.464723576 +0000 UTC m=+0.698226947 container cleanup aff55e0c4197a64dca758112ef56969a1dfb5084573623e16bf173e53d51b123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-16T10:27:15.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 16 10:27:15 smithi193 bash[72169]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mon-c 2024-03-16T10:27:15.886 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mon.c.service' 2024-03-16T10:27:15.939 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:27:15.939 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-16T10:27:15.939 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-16T10:27:15.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.x 2024-03-16T10:27:16.245 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:27:15 smithi077 systemd[1]: Stopping Ceph mgr.x for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:27:16.571 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:27:16 smithi077 podman[98443]: 2024-03-16 10:27:16.276340289 +0000 UTC m=+0.143222849 container died 955d6952e8d970895cba8348aa7085b503f86234dfe725e2476f9ad76a1e1042 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-16T10:27:16.863 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:27:16 smithi077 podman[98443]: 2024-03-16 10:27:16.67878055 +0000 UTC m=+0.545663111 container cleanup 955d6952e8d970895cba8348aa7085b503f86234dfe725e2476f9ad76a1e1042 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux ) 2024-03-16T10:27:16.863 INFO:journalctl@ceph.mgr.x.smithi077.stdout:Mar 16 10:27:16 smithi077 bash[98443]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-x 2024-03-16T10:27:16.980 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-16T10:27:17.018 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:27:17.019 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-16T10:27:17.019 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-16T10:27:17.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.y 2024-03-16T10:27:17.413 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:27:17 smithi192 systemd[1]: Stopping Ceph mgr.y for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:27:17.666 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:27:17 smithi192 podman[71213]: 2024-03-16 10:27:17.412869009 +0000 UTC m=+0.189784136 container died 48b8155497ba5445f552cda164d4ff31640bbe63c3bb99d11bf575238c77fbcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:27:18.078 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:27:17 smithi192 podman[71213]: 2024-03-16 10:27:17.818299288 +0000 UTC m=+0.595214408 container cleanup 48b8155497ba5445f552cda164d4ff31640bbe63c3bb99d11bf575238c77fbcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T10:27:18.078 INFO:journalctl@ceph.mgr.y.smithi192.stdout:Mar 16 10:27:17 smithi192 bash[71213]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-y 2024-03-16T10:27:18.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-16T10:27:18.214 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:27:18.215 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-16T10:27:18.215 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-16T10:27:18.215 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.z 2024-03-16T10:27:18.503 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:27:18 smithi193 systemd[1]: Stopping Ceph mgr.z for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:27:18.870 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:27:18 smithi193 podman[72292]: 2024-03-16 10:27:18.565683998 +0000 UTC m=+0.153014420 container died 2fdf6e064b0be34be34ed9180e57fd87c4755b710d8ddb0ae1c559905b5f1395 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:27:19.213 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:27:18 smithi193 podman[72292]: 2024-03-16 10:27:18.970253352 +0000 UTC m=+0.557583783 container cleanup 2fdf6e064b0be34be34ed9180e57fd87c4755b710d8ddb0ae1c559905b5f1395 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-16T10:27:19.213 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 16 10:27:18 smithi193 bash[72292]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mgr-z 2024-03-16T10:27:19.341 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-16T10:27:19.378 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:27:19.378 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-16T10:27:19.378 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-16T10:27:19.378 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.0 2024-03-16T10:27:19.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:19 smithi077 systemd[1]: Stopping Ceph osd.0 for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:27:20.245 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:19 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0[44735]: 2024-03-16T10:27:19.807+0000 7f56867c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:27:20.245 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:19 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0[44735]: 2024-03-16T10:27:19.807+0000 7f56867c5700 -1 osd.0 91 *** Got signal Terminated *** 2024-03-16T10:27:20.245 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:19 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0[44735]: 2024-03-16T10:27:19.807+0000 7f56867c5700 -1 osd.0 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T10:27:25.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:25 smithi077 podman[98565]: 2024-03-16 10:27:25.258349892 +0000 UTC m=+5.689953061 container died 4cb67a40b04117414f0903cd1cb4247bc0a036a211914666126c74fcc613331d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-16T10:27:25.843 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:25 smithi077 podman[98565]: 2024-03-16 10:27:25.625146181 +0000 UTC m=+6.056749376 container cleanup 4cb67a40b04117414f0903cd1cb4247bc0a036a211914666126c74fcc613331d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-16T10:27:25.843 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:25 smithi077 bash[98565]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0 2024-03-16T10:27:26.241 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:25 smithi077 podman[98583]: 2024-03-16 10:27:25.843253238 +0000 UTC m=+0.578989971 container remove 4cb67a40b04117414f0903cd1cb4247bc0a036a211914666126c74fcc613331d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T10:27:26.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:26 smithi077 podman[98675]: 2024-03-16 10:27:26.144403203 +0000 UTC m=+0.018756880 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:27:26.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:26 smithi077 podman[98675]: 2024-03-16 10:27:26.252659672 +0000 UTC m=+0.127013379 container create 1248ae09c1033c63b65d247873364696f5818068b95f5a42537503c28aa3d6df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-16T10:27:26.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:26 smithi077 podman[98675]: 2024-03-16 10:27:26.496863368 +0000 UTC m=+0.371217033 container init 1248ae09c1033c63b65d247873364696f5818068b95f5a42537503c28aa3d6df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T10:27:26.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:26 smithi077 podman[98675]: 2024-03-16 10:27:26.500007468 +0000 UTC m=+0.374361140 container start 1248ae09c1033c63b65d247873364696f5818068b95f5a42537503c28aa3d6df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True) 2024-03-16T10:27:26.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:26 smithi077 podman[98675]: 2024-03-16 10:27:26.605729211 +0000 UTC m=+0.480082881 container attach 1248ae09c1033c63b65d247873364696f5818068b95f5a42537503c28aa3d6df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:27:27.167 INFO:journalctl@ceph.osd.0.smithi077.stdout:Mar 16 10:27:26 smithi077 podman[98675]: 2024-03-16 10:27:26.794219197 +0000 UTC m=+0.668572861 container died 1248ae09c1033c63b65d247873364696f5818068b95f5a42537503c28aa3d6df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-16T10:27:27.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.0.service' 2024-03-16T10:27:27.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:27:27.563 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-16T10:27:27.563 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-16T10:27:27.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.1 2024-03-16T10:27:27.898 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:27 smithi192 systemd[1]: Stopping Ceph osd.1 for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:27:28.166 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:27 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1[40959]: 2024-03-16T10:27:27.897+0000 7fec8c14b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:27:28.166 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:27 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1[40959]: 2024-03-16T10:27:27.897+0000 7fec8c14b700 -1 osd.1 91 *** Got signal Terminated *** 2024-03-16T10:27:28.167 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:27 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1[40959]: 2024-03-16T10:27:27.897+0000 7fec8c14b700 -1 osd.1 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T10:27:33.667 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:33 smithi192 podman[71338]: 2024-03-16 10:27:33.412833162 +0000 UTC m=+5.653045110 container died d20a9a5694b6677f3eadb22d549524084b9618f13969153b19dc0e19bf424a8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:27:34.091 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:33 smithi192 podman[71338]: 2024-03-16 10:27:33.814572107 +0000 UTC m=+6.054784064 container cleanup d20a9a5694b6677f3eadb22d549524084b9618f13969153b19dc0e19bf424a8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-16T10:27:34.091 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:33 smithi192 bash[71338]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1 2024-03-16T10:27:34.394 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:34 smithi192 podman[71355]: 2024-03-16 10:27:34.091618731 +0000 UTC m=+0.676124280 container remove d20a9a5694b6677f3eadb22d549524084b9618f13969153b19dc0e19bf424a8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-16T10:27:34.667 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:34 smithi192 podman[71450]: 2024-03-16 10:27:34.296712564 +0000 UTC m=+0.017618792 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:27:34.667 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:34 smithi192 podman[71450]: 2024-03-16 10:27:34.417494539 +0000 UTC m=+0.138400756 container create e0ae233446f97fd49048a5871e066cc68009fc79eb4371c9c35945c08f8d45cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-16T10:27:35.108 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:34 smithi192 podman[71450]: 2024-03-16 10:27:34.746439206 +0000 UTC m=+0.467345430 container init e0ae233446f97fd49048a5871e066cc68009fc79eb4371c9c35945c08f8d45cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:27:35.108 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:34 smithi192 podman[71450]: 2024-03-16 10:27:34.749863409 +0000 UTC m=+0.470769630 container start e0ae233446f97fd49048a5871e066cc68009fc79eb4371c9c35945c08f8d45cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-16T10:27:35.108 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:34 smithi192 podman[71450]: 2024-03-16 10:27:34.879140703 +0000 UTC m=+0.600046929 container attach e0ae233446f97fd49048a5871e066cc68009fc79eb4371c9c35945c08f8d45cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T10:27:35.373 INFO:journalctl@ceph.osd.1.smithi192.stdout:Mar 16 10:27:35 smithi192 podman[71450]: 2024-03-16 10:27:35.10836845 +0000 UTC m=+0.829274672 container died e0ae233446f97fd49048a5871e066cc68009fc79eb4371c9c35945c08f8d45cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-16T10:27:35.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.1.service' 2024-03-16T10:27:35.767 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:27:35.768 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-16T10:27:35.768 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-16T10:27:35.768 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.2 2024-03-16T10:27:36.067 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:35 smithi193 systemd[1]: Stopping Ceph osd.2 for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:27:36.502 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:36 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2[41492]: 2024-03-16T10:27:36.064+0000 7f8a13bb5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:27:36.503 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:36 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2[41492]: 2024-03-16T10:27:36.064+0000 7f8a13bb5700 -1 osd.2 91 *** Got signal Terminated *** 2024-03-16T10:27:36.503 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:36 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2[41492]: 2024-03-16T10:27:36.064+0000 7f8a13bb5700 -1 osd.2 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T10:27:41.725 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:41 smithi193 podman[72428]: 2024-03-16 10:27:41.417891914 +0000 UTC m=+5.462104658 container died b1fcc1d9d7c0fd0fa01747d5b53d70659b240c2eff43d85bed6833fc3bafb24b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-16T10:27:42.003 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:41 smithi193 podman[72428]: 2024-03-16 10:27:41.841162433 +0000 UTC m=+5.885375186 container cleanup b1fcc1d9d7c0fd0fa01747d5b53d70659b240c2eff43d85bed6833fc3bafb24b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-16T10:27:42.003 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:41 smithi193 bash[72428]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2 2024-03-16T10:27:42.498 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:42 smithi193 podman[72445]: 2024-03-16 10:27:42.067871732 +0000 UTC m=+0.647687129 container remove b1fcc1d9d7c0fd0fa01747d5b53d70659b240c2eff43d85bed6833fc3bafb24b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:27:42.752 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:42 smithi193 podman[72533]: 2024-03-16 10:27:42.401507771 +0000 UTC m=+0.017766413 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:27:42.753 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:42 smithi193 podman[72533]: 2024-03-16 10:27:42.51036392 +0000 UTC m=+0.126622587 container create a6596024e38b64151111e4d2c415e8c17ddf9d8c561f2c4a3b633b391b3423ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-16T10:27:43.060 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:42 smithi193 podman[72533]: 2024-03-16 10:27:42.763099024 +0000 UTC m=+0.379357647 container init a6596024e38b64151111e4d2c415e8c17ddf9d8c561f2c4a3b633b391b3423ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-03-16T10:27:43.060 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:42 smithi193 podman[72533]: 2024-03-16 10:27:42.766172094 +0000 UTC m=+0.382430717 container start a6596024e38b64151111e4d2c415e8c17ddf9d8c561f2c4a3b633b391b3423ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T10:27:43.060 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:42 smithi193 podman[72533]: 2024-03-16 10:27:42.86327708 +0000 UTC m=+0.479535723 container attach a6596024e38b64151111e4d2c415e8c17ddf9d8c561f2c4a3b633b391b3423ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-16T10:27:43.324 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 16 10:27:43 smithi193 podman[72533]: 2024-03-16 10:27:43.060254907 +0000 UTC m=+0.676513542 container died a6596024e38b64151111e4d2c415e8c17ddf9d8c561f2c4a3b633b391b3423ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-03-16T10:27:43.716 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.2.service' 2024-03-16T10:27:43.757 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:27:43.757 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-16T10:27:43.757 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-16T10:27:43.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.3 2024-03-16T10:27:44.072 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:43 smithi077 systemd[1]: Stopping Ceph osd.3 for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:27:44.495 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:44 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3[51690]: 2024-03-16T10:27:44.070+0000 7ff256247700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:27:44.495 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:44 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3[51690]: 2024-03-16T10:27:44.070+0000 7ff256247700 -1 osd.3 91 *** Got signal Terminated *** 2024-03-16T10:27:44.495 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:44 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3[51690]: 2024-03-16T10:27:44.070+0000 7ff256247700 -1 osd.3 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T10:27:46.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:46 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:46.669+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:47.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:47 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:47.487+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:47.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:47 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:47.709+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:48.735 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:48 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3[51690]: 2024-03-16T10:27:48.344+0000 7ff25283b700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.613630+0000 front 2024-03-16T10:27:23.613681+0000 (oldest deadline 2024-03-16T10:27:48.313766+0000) 2024-03-16T10:27:48.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:48 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:48.490+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:48.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:48 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:48.735+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:49.714 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:49 smithi077 podman[98795]: 2024-03-16 10:27:49.427855251 +0000 UTC m=+5.480028483 container died 3fd1d99c2fd882180ef0a162c2a8bf1aed8ce505f467284fa3eedc49f9e9afa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-16T10:27:49.714 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:49 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:49.358+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:27:49.753 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:49 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:49.460+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:27:49.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:49 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:49.460+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:49.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:49 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:49.563+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:27:49.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:49 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:49.764+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:49.995 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:49 smithi077 podman[98795]: 2024-03-16 10:27:49.821367724 +0000 UTC m=+5.873540957 container cleanup 3fd1d99c2fd882180ef0a162c2a8bf1aed8ce505f467284fa3eedc49f9e9afa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:27:49.995 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:49 smithi077 bash[98795]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3 2024-03-16T10:27:50.378 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:50 smithi077 podman[98811]: 2024-03-16 10:27:50.073045186 +0000 UTC m=+0.644226626 container remove 3fd1d99c2fd882180ef0a162c2a8bf1aed8ce505f467284fa3eedc49f9e9afa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-16T10:27:50.668 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:50 smithi077 podman[98906]: 2024-03-16 10:27:50.289366916 +0000 UTC m=+0.017662136 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:27:50.668 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:50 smithi077 podman[98906]: 2024-03-16 10:27:50.398894325 +0000 UTC m=+0.127189543 container create 579fb2b7f09cb139f916dbf2a103ad7f2b55209168920dd751a8eb9059e8c668 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-16T10:27:50.668 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:50 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:50.377+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:27:50.752 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:50 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:50.432+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:27:50.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:50 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:50.471+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:50.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:50 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:50.608+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:27:50.956 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:50 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:50.782+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:50.957 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:50 smithi077 podman[98906]: 2024-03-16 10:27:50.668102502 +0000 UTC m=+0.396397728 container init 579fb2b7f09cb139f916dbf2a103ad7f2b55209168920dd751a8eb9059e8c668 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-16T10:27:50.957 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:50 smithi077 podman[98906]: 2024-03-16 10:27:50.671354495 +0000 UTC m=+0.399649710 container start 579fb2b7f09cb139f916dbf2a103ad7f2b55209168920dd751a8eb9059e8c668 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-16T10:27:50.957 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:50 smithi077 podman[98906]: 2024-03-16 10:27:50.776667962 +0000 UTC m=+0.504963181 container attach 579fb2b7f09cb139f916dbf2a103ad7f2b55209168920dd751a8eb9059e8c668 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD) 2024-03-16T10:27:51.245 INFO:journalctl@ceph.osd.3.smithi077.stdout:Mar 16 10:27:50 smithi077 podman[98906]: 2024-03-16 10:27:50.956741136 +0000 UTC m=+0.685036352 container died 579fb2b7f09cb139f916dbf2a103ad7f2b55209168920dd751a8eb9059e8c668 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:27:51.572 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:51 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:51.350+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:27:51.591 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.3.service' 2024-03-16T10:27:51.631 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:27:51.632 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-16T10:27:51.632 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-16T10:27:51.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.4 2024-03-16T10:27:51.752 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:51 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:51.475+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:27:51.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:51 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:51.456+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:51.841 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:51 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:51.742+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:51.916 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:51 smithi192 systemd[1]: Stopping Ceph osd.4 for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:27:51.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:51 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:51.559+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:27:52.417 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:51 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4[47783]: 2024-03-16T10:27:51.953+0000 7f8d1ec56700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:27:52.417 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:51 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4[47783]: 2024-03-16T10:27:51.953+0000 7f8d1ec56700 -1 osd.4 91 *** Got signal Terminated *** 2024-03-16T10:27:52.417 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:51 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4[47783]: 2024-03-16T10:27:51.953+0000 7f8d1ec56700 -1 osd.4 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T10:27:52.705 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:52 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:52.384+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:27:52.753 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:52 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:52.448+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:27:52.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:52 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:52.407+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:52.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:52 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:52.561+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:27:52.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:52 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:52.704+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:53.657 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:53 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:53.656+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:53.658 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:53 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:53.396+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:27:53.752 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:53 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:53.471+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:27:53.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:53 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:53.371+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:53.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:53 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:53.527+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:27:54.745 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:54 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:54.666+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:54.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:54 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:54.429+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:27:54.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:54 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:54.429+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:27:54.752 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:54 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:54.487+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:27:54.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:54 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:54.352+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:54.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:54 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:54.352+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:27:54.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:54 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:54.564+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:27:54.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:27:54 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:27:54.708+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:27:55.708 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:55 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:55.428+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:27:55.708 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:55 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:55.428+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:27:55.753 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:55 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:55.460+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:27:55.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:55 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:55.323+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:55.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:55 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:55.323+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:27:55.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:55 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:55.608+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:27:55.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:27:55 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:27:55.720+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:27:55.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:55 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:55.706+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:56.654 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:56 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:56.474+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:27:56.654 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:56 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:56.285+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:56.654 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:56 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:56.285+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:27:56.695 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:56 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:56.430+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:27:56.696 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:56 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:56.430+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:27:56.776 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:27:56 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:27:56.685+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:27:56.776 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:56 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:56.584+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:27:56.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:56 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:56.695+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:57.166 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:56 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4[47783]: 2024-03-16T10:27:56.775+0000 7f8d1b24a700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.016593+0000 front 2024-03-16T10:27:32.016629+0000 (oldest deadline 2024-03-16T10:27:56.116742+0000) 2024-03-16T10:27:57.667 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:57 smithi192 podman[71570]: 2024-03-16 10:27:57.370412264 +0000 UTC m=+5.535189479 container died dcd4be9c88ccf075e520aaa4033bb84aec4926870bfe418d0cb340f71e15995d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T10:27:57.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:57 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:57.560+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:27:57.728 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:57 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:57.470+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:27:57.728 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:57 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:57.470+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:27:57.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:57 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:57.326+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:57.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:57 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:57.326+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:27:57.753 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:57 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:57.436+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:27:57.753 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:57 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:57.436+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.928186+0000 front 2024-03-16T10:27:28.928057+0000 (oldest deadline 2024-03-16T10:27:57.128566+0000) 2024-03-16T10:27:57.989 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:57 smithi192 podman[71570]: 2024-03-16 10:27:57.795299241 +0000 UTC m=+5.960076436 container cleanup dcd4be9c88ccf075e520aaa4033bb84aec4926870bfe418d0cb340f71e15995d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-16T10:27:57.989 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:57 smithi192 bash[71570]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4 2024-03-16T10:27:57.989 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:27:57 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:27:57.735+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:27:57.994 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:57 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:57.727+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:57.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:57 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:57.727+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:27:58.252 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:27:57 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:27:57.767+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:27:58.411 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:57 smithi192 podman[71587]: 2024-03-16 10:27:57.988748009 +0000 UTC m=+0.613886401 container remove dcd4be9c88ccf075e520aaa4033bb84aec4926870bfe418d0cb340f71e15995d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-03-16T10:27:58.667 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:58 smithi192 podman[71677]: 2024-03-16 10:27:58.314490409 +0000 UTC m=+0.017835940 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:27:58.667 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:58 smithi192 podman[71677]: 2024-03-16 10:27:58.431207936 +0000 UTC m=+0.134553451 container create 980355255b439dfe70d4831783d8efa4b2caa3cca448219abb61a5d98c048436 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-16T10:27:58.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:58 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:58.596+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:27:58.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:58 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:58.596+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:27:58.741 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:58 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:58.351+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:58.741 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:58 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:58.351+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:27:58.741 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:58 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:58.413+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:27:58.742 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:58 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:58.413+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.928186+0000 front 2024-03-16T10:27:28.928057+0000 (oldest deadline 2024-03-16T10:27:57.128566+0000) 2024-03-16T10:27:58.745 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:58 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:58.683+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:58.745 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:58 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:58.683+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:27:58.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:58 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:58.472+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:27:58.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:58 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:58.472+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:27:58.990 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:27:58 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:27:58.705+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:27:58.990 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:58 smithi192 podman[71677]: 2024-03-16 10:27:58.700698799 +0000 UTC m=+0.404044321 container init 980355255b439dfe70d4831783d8efa4b2caa3cca448219abb61a5d98c048436 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:27:58.990 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:58 smithi192 podman[71677]: 2024-03-16 10:27:58.703853388 +0000 UTC m=+0.407198905 container start 980355255b439dfe70d4831783d8efa4b2caa3cca448219abb61a5d98c048436 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True) 2024-03-16T10:27:58.990 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:58 smithi192 podman[71677]: 2024-03-16 10:27:58.800716624 +0000 UTC m=+0.504062159 container attach 980355255b439dfe70d4831783d8efa4b2caa3cca448219abb61a5d98c048436 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:27:59.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:27:58 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:27:58.740+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:27:59.371 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 16 10:27:58 smithi192 podman[71677]: 2024-03-16 10:27:58.990693905 +0000 UTC m=+0.694039426 container died 980355255b439dfe70d4831783d8efa4b2caa3cca448219abb61a5d98c048436 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-16T10:27:59.666 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:59 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:59.599+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:27:59.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:27:59 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:27:59.599+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:27:59.709 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:59 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:59.463+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:27:59.709 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:59 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:27:59.463+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.928186+0000 front 2024-03-16T10:27:28.928057+0000 (oldest deadline 2024-03-16T10:27:57.128566+0000) 2024-03-16T10:27:59.710 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:59 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:59.306+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:27:59.710 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:27:59 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:27:59.306+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:27:59.710 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:27:59 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:27:59.708+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:27:59.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.4.service' 2024-03-16T10:27:59.745 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:59 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:59.688+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:27:59.745 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:27:59 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:27:59.688+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:27:59.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:59 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:59.444+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:27:59.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:27:59 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:27:59.444+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:27:59.790 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:27:59.791 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-16T10:27:59.791 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-16T10:27:59.791 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.5 2024-03-16T10:28:00.002 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:27:59 smithi193 systemd[1]: Stopping Ceph osd.5 for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:28:00.166 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:27:59 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:27:59.701+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:00.502 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:00 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:00.340+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:00.503 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:00 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:00.340+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:00.503 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:00 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:00.102+0000 7fd9e1281700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:28:00.503 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:00 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:00.102+0000 7fd9e1281700 -1 osd.5 91 *** Got signal Terminated *** 2024-03-16T10:28:00.503 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:00 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:00.102+0000 7fd9e1281700 -1 osd.5 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T10:28:00.701 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:00 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:00.440+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:00.701 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:00 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:00.440+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:00.916 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:00 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:00.660+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:00.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:00 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:00.573+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:00.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:00 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:00.573+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:00.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:00 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:00.700+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:00.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:00 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:00.700+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:01.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:00 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:00.668+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:01.003 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:00 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:00.503+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:28:01.003 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:00 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:00.503+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.928186+0000 front 2024-03-16T10:27:28.928057+0000 (oldest deadline 2024-03-16T10:27:57.128566+0000) 2024-03-16T10:28:01.503 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:01 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:01.472+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:28:01.503 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:01 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:01.472+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.928186+0000 front 2024-03-16T10:27:28.928057+0000 (oldest deadline 2024-03-16T10:27:57.128566+0000) 2024-03-16T10:28:01.504 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:01 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:01.352+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:01.504 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:01 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:01.352+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:01.734 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:01 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:01.468+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:01.735 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:01 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:01.468+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:01.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:01 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:01.706+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:01.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:01 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:01.603+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:01.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:01 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:01.603+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:01.994 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:01 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:01.734+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:01.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:01 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:01.734+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:02.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:01 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:01.673+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:02.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:01 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:01.673+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:02.712 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:02 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:02.451+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:28:02.712 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:02 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:02.451+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.928186+0000 front 2024-03-16T10:27:28.928057+0000 (oldest deadline 2024-03-16T10:27:57.128566+0000) 2024-03-16T10:28:02.712 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:02 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:02.390+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:02.712 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:02 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:02.390+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:02.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:02 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:02.445+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:02.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:02 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:02.445+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:02.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:02 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:02.445+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:02.745 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:02 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:02.686+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:02.745 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:02 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:02.686+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:02.916 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:02 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:02.671+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:02.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:02 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:02.572+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:02.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:02 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:02.572+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:02.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:02 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:02.572+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:03.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:02 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:02.710+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:03.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:02 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:02.710+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:03.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:03 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:03.481+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:03.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:03 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:03.481+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:03.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:03 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:03.481+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:03.745 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:03 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:03.676+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:03.745 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:03 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:03.676+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:03.745 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:03 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:03.676+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.425690+0000 front 2024-03-16T10:27:40.425833+0000 (oldest deadline 2024-03-16T10:28:02.725701+0000) 2024-03-16T10:28:03.752 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:03 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:03.448+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:28:03.753 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:03 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:03.448+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.928186+0000 front 2024-03-16T10:27:28.928057+0000 (oldest deadline 2024-03-16T10:27:57.128566+0000) 2024-03-16T10:28:03.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:03 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:03.363+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:03.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:03 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:03.363+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:03.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:03 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:03.659+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:03.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:03 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:03.659+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:03.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:03 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:03.620+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:03.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:03 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:03.620+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:03.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:03 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:03.620+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:04.252 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:03 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:03.757+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:04.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:03 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:03.757+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:04.753 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:04 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:04.402+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:19.526626+0000 front 2024-03-16T10:27:19.526609+0000 (oldest deadline 2024-03-16T10:27:48.927349+0000) 2024-03-16T10:28:04.753 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:04 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5[48316]: 2024-03-16T10:28:04.402+0000 7fd9d986d700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.928186+0000 front 2024-03-16T10:27:28.928057+0000 (oldest deadline 2024-03-16T10:27:57.128566+0000) 2024-03-16T10:28:04.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:04 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:04.413+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:04.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:04 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:04.413+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:04.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:04 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:04.413+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:04.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:04 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:04.639+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:04.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:04 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:04.639+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:04.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:04 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:04.627+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:04.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:04 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:04.627+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:04.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:04 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:04.627+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:04.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:04 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:04.515+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:04.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:04 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:04.515+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:04.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:04 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:04.515+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:04.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:04 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:04.717+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:04.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:04 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:04.717+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:04.996 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:04 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:04.717+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.425690+0000 front 2024-03-16T10:27:40.425833+0000 (oldest deadline 2024-03-16T10:28:02.725701+0000) 2024-03-16T10:28:05.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:04 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:04.790+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:05.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:04 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:04.790+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:05.753 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:05 smithi193 podman[72663]: 2024-03-16 10:28:05.560391782 +0000 UTC m=+5.574831340 container died b633f7e1d978c4e805bca16ec8f50b496fd8e9db2efa2c1626515e5732f5319b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5) 2024-03-16T10:28:05.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:05 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:05.420+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:05.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:05 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:05.420+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:05.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:05 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:05.420+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:05.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:05 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:05.627+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:05.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:05 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:05.627+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:05.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:05 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:05.582+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:05.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:05 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:05.582+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:05.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:05 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:05.582+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:05.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:05 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:05.533+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:05.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:05 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:05.533+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:05.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:05 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:05.533+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:05.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:05 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:05.688+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:05.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:05 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:05.688+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:05.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:05 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:05.688+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.425690+0000 front 2024-03-16T10:27:40.425833+0000 (oldest deadline 2024-03-16T10:28:02.725701+0000) 2024-03-16T10:28:06.205 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:05 smithi193 podman[72663]: 2024-03-16 10:28:05.970092573 +0000 UTC m=+5.984532129 container cleanup b633f7e1d978c4e805bca16ec8f50b496fd8e9db2efa2c1626515e5732f5319b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-16T10:28:06.205 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:05 smithi193 bash[72663]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5 2024-03-16T10:28:06.205 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:05 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:05.792+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:06.206 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:05 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:05.792+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:06.503 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:06 smithi193 podman[72682]: 2024-03-16 10:28:06.20472262 +0000 UTC m=+0.638109129 container remove b633f7e1d978c4e805bca16ec8f50b496fd8e9db2efa2c1626515e5732f5319b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:28:06.503 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:06 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:06.406+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:06.503 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:06 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:06.406+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:06.503 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:06 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:06.406+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:06.770 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:06 smithi193 podman[72776]: 2024-03-16 10:28:06.412407767 +0000 UTC m=+0.017609741 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:28:06.770 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:06 smithi193 podman[72776]: 2024-03-16 10:28:06.514827765 +0000 UTC m=+0.120029777 container create af0d2ce3ccad88cf53de78519c9afa0ac2c55b244b706a92907a5cf37b6ac7cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:28:06.770 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:06 smithi193 podman[72776]: 2024-03-16 10:28:06.769883458 +0000 UTC m=+0.375085439 container init af0d2ce3ccad88cf53de78519c9afa0ac2c55b244b706a92907a5cf37b6ac7cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-03-16T10:28:06.916 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:06 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:06.675+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:06.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:06 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:06.675+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:06.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:06 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:06.573+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:06.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:06 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:06.573+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:06.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:06 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:06.573+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:06.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:06 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:06.502+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:06.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:06 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:06.502+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:06.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:06 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:06.502+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:06.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:06 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:06.697+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:06.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:06 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:06.697+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:06.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:06 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:06.697+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.425690+0000 front 2024-03-16T10:27:40.425833+0000 (oldest deadline 2024-03-16T10:28:02.725701+0000) 2024-03-16T10:28:07.061 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:06 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:06.799+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:07.061 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:06 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:06.799+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:07.061 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:06 smithi193 podman[72776]: 2024-03-16 10:28:06.773407384 +0000 UTC m=+0.378609361 container start af0d2ce3ccad88cf53de78519c9afa0ac2c55b244b706a92907a5cf37b6ac7cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-03-16T10:28:07.061 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:06 smithi193 podman[72776]: 2024-03-16 10:28:06.87830904 +0000 UTC m=+0.483511040 container attach af0d2ce3ccad88cf53de78519c9afa0ac2c55b244b706a92907a5cf37b6ac7cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True) 2024-03-16T10:28:07.358 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 16 10:28:07 smithi193 podman[72776]: 2024-03-16 10:28:07.061106625 +0000 UTC m=+0.666308605 container died af0d2ce3ccad88cf53de78519c9afa0ac2c55b244b706a92907a5cf37b6ac7cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:28:07.657 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:07 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:07.451+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:07.657 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:07 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:07.451+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:07.657 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:07 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:07.451+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:07.678 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.5.service' 2024-03-16T10:28:07.719 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:28:07.720 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-16T10:28:07.720 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-16T10:28:07.720 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.6 2024-03-16T10:28:07.916 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:07 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:07.675+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:07.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:07 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:07.675+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:07.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:07 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:07.585+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:07.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:07 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:07.585+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:07.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:07 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:07.585+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:07.921 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:07 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:07.775+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:07.921 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:07 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:07.775+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:07.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:07.736+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:07.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:07.736+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:07.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:07.736+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.425690+0000 front 2024-03-16T10:27:40.425833+0000 (oldest deadline 2024-03-16T10:28:02.725701+0000) 2024-03-16T10:28:07.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:07 smithi077 systemd[1]: Stopping Ceph osd.6 for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:28:07.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:07.535+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:07.996 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:07.535+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:07.996 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:07 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:07.535+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:08.495 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:08.033+0000 7fc90260c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:28:08.495 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:08.033+0000 7fc90260c700 -1 osd.6 91 *** Got signal Terminated *** 2024-03-16T10:28:08.495 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:08.033+0000 7fc90260c700 -1 osd.6 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T10:28:08.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:08.460+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:08.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:08.460+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:08.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:08.460+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:08.916 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:08.659+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:08.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:08.659+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:08.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:08.612+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:08.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:08.612+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:08.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:08 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:08.612+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:08.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:08.584+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:08.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:08.584+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:08.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:08.584+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:08.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:08.687+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:08.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:08.687+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:08.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:08 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:08.687+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.425690+0000 front 2024-03-16T10:27:40.425833+0000 (oldest deadline 2024-03-16T10:28:02.725701+0000) 2024-03-16T10:28:09.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:08.766+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:09.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:08 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:08.766+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:09.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:09 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:09.435+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:09.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:09 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:09.435+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:09.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:09 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:09.435+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:09.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:09 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:09.645+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:09.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:09 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:09.645+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:09.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:09 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:09.645+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.867501+0000 front 2024-03-16T10:27:44.867596+0000 (oldest deadline 2024-03-16T10:28:09.567651+0000) 2024-03-16T10:28:09.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:09 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:09.647+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:09.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:09 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:09.647+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:09.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:09 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:09.647+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:09.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:09.586+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:09.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:09.586+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:09.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:09.586+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:09.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:09.732+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:09.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:09.732+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:09.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:09.732+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.425690+0000 front 2024-03-16T10:27:40.425833+0000 (oldest deadline 2024-03-16T10:28:02.725701+0000) 2024-03-16T10:28:10.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:09 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:09.742+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:10.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:09 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:09.742+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:10.748 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:10 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:10.412+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:10.748 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:10 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:10.412+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:10.748 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:10 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:10.412+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:10.916 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:10 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:10.632+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:10.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:10 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:10.632+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:10.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:10 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:10.632+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.867501+0000 front 2024-03-16T10:27:44.867596+0000 (oldest deadline 2024-03-16T10:28:09.567651+0000) 2024-03-16T10:28:10.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:10 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:10.660+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:10.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:10 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:10.660+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:10.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:10 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:10.660+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:10.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:10 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:10.695+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:10.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:10 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:10.695+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:10.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:10 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:10.695+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.425690+0000 front 2024-03-16T10:27:40.425833+0000 (oldest deadline 2024-03-16T10:28:02.725701+0000) 2024-03-16T10:28:10.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:10 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:10.575+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:10.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:10 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:10.575+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:10.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:10 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:10.575+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:11.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:10 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:10.747+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:11.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:10 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:10.747+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:11.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:11 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:11.390+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:11.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:11 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:11.390+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:11.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:11 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:11.390+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:11.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:11 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:11.390+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:11.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:11 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:11.607+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:11.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:11 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:11.607+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:11.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:11 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:11.607+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.867501+0000 front 2024-03-16T10:27:44.867596+0000 (oldest deadline 2024-03-16T10:28:09.567651+0000) 2024-03-16T10:28:11.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:11 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:11.623+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:11.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:11 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:11.623+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:11.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:11 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:11.623+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:11.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:11 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:11.565+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:11.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:11 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:11.565+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:11.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:11 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:11.565+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:11.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:11 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:11.684+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:11.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:11 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:11.684+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:11.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:11 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:11.684+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.425690+0000 front 2024-03-16T10:27:40.425833+0000 (oldest deadline 2024-03-16T10:28:02.725701+0000) 2024-03-16T10:28:11.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:11 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:11.684+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:49.027416+0000 front 2024-03-16T10:27:49.027525+0000 (oldest deadline 2024-03-16T10:28:11.327601+0000) 2024-03-16T10:28:12.252 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:11 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:11.761+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:12.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:11 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:11.761+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:12.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:11 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:11.761+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:12.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:12 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:12.429+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:12.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:12 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:12.429+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:12.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:12 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:12.429+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:12.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:12 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:12.429+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:12.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:12 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:12.657+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:12.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:12 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:12.657+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:12.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:12 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:12.657+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.867501+0000 front 2024-03-16T10:27:44.867596+0000 (oldest deadline 2024-03-16T10:28:09.567651+0000) 2024-03-16T10:28:12.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:12 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:12.586+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:12.918 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:12 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:12.586+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:12.918 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:12 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:12.586+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:12.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:12 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:12.546+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:12.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:12 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:12.546+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:12.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:12 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:12.546+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:12.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:12 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:12.649+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:22.521753+0000 front 2024-03-16T10:27:22.521866+0000 (oldest deadline 2024-03-16T10:27:46.021891+0000) 2024-03-16T10:28:12.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:12 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:12.649+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.824578+0000 front 2024-03-16T10:27:32.824560+0000 (oldest deadline 2024-03-16T10:27:56.924440+0000) 2024-03-16T10:28:12.996 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:12 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:12.649+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.425690+0000 front 2024-03-16T10:27:40.425833+0000 (oldest deadline 2024-03-16T10:28:02.725701+0000) 2024-03-16T10:28:12.996 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:12 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6[58694]: 2024-03-16T10:28:12.649+0000 7fc8fabf8700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:49.027416+0000 front 2024-03-16T10:27:49.027525+0000 (oldest deadline 2024-03-16T10:28:11.327601+0000) 2024-03-16T10:28:13.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:12 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:12.788+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:13.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:12 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:12.788+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:13.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:12 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:12.788+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:13.668 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:13 smithi077 podman[99022]: 2024-03-16 10:28:13.388101632 +0000 UTC m=+5.476945503 container died a6120bab28c92e7e0f09199dbabe126792b9b5b048781dc7c9b5d5ad77f30f8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-16T10:28:13.668 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:13 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:13.591+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:13.669 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:13 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:13.591+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:13.669 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:13 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:13.591+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:13.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:13 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:13.463+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:13.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:13 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:13.463+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:13.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:13 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:13.463+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:13.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:13 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:13.463+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:13.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:13.611+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:13.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:13.611+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:13.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:13.611+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.867501+0000 front 2024-03-16T10:27:44.867596+0000 (oldest deadline 2024-03-16T10:28:09.567651+0000) 2024-03-16T10:28:13.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:13.589+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:13.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:13.589+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:13.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:13.589+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:13.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:13 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:13.589+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:13.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:13 smithi077 podman[99022]: 2024-03-16 10:28:13.759443342 +0000 UTC m=+5.848287221 container cleanup a6120bab28c92e7e0f09199dbabe126792b9b5b048781dc7c9b5d5ad77f30f8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-16T10:28:13.995 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:13 smithi077 bash[99022]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6 2024-03-16T10:28:14.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:13 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:13.781+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:14.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:13 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:13.781+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:14.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:13 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:13.781+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:14.375 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:14 smithi077 podman[99039]: 2024-03-16 10:28:14.053131551 +0000 UTC m=+0.658871803 container remove a6120bab28c92e7e0f09199dbabe126792b9b5b048781dc7c9b5d5ad77f30f8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-16T10:28:14.627 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:14 smithi077 podman[99129]: 2024-03-16 10:28:14.277944673 +0000 UTC m=+0.018515245 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:28:14.627 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:14 smithi077 podman[99129]: 2024-03-16 10:28:14.395635556 +0000 UTC m=+0.136206109 container create ee6be76f76ff803924ae1e9c6a501dff9beb686e899a720ed098ae6837e20861 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-16T10:28:14.627 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:14 smithi077 podman[99129]: 2024-03-16 10:28:14.623556907 +0000 UTC m=+0.364127465 container init ee6be76f76ff803924ae1e9c6a501dff9beb686e899a720ed098ae6837e20861 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-16T10:28:14.627 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:14 smithi077 podman[99129]: 2024-03-16 10:28:14.627091331 +0000 UTC m=+0.367661886 container start ee6be76f76ff803924ae1e9c6a501dff9beb686e899a720ed098ae6837e20861 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-16T10:28:14.627 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:14 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:14.553+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:14.628 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:14 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:14.553+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:14.628 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:14 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:14.553+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:14.628 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:14 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:14.553+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:14.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:14 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:14.458+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:14.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:14 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:14.458+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:14.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:14 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:14.458+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:14.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:14 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:14.458+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:14.916 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:14 smithi077 podman[99129]: 2024-03-16 10:28:14.723171794 +0000 UTC m=+0.463742352 container attach ee6be76f76ff803924ae1e9c6a501dff9beb686e899a720ed098ae6837e20861 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T10:28:14.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:14 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:14.594+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:14.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:14 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:14.594+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:14.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:14 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:14.594+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.867501+0000 front 2024-03-16T10:27:44.867596+0000 (oldest deadline 2024-03-16T10:28:09.567651+0000) 2024-03-16T10:28:14.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:14 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:14.555+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:14.918 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:14 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:14.555+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:14.918 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:14 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:14.555+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:14.918 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:14 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:14.555+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:15.245 INFO:journalctl@ceph.osd.6.smithi077.stdout:Mar 16 10:28:14 smithi077 podman[99129]: 2024-03-16 10:28:14.916525485 +0000 UTC m=+0.657096041 container died ee6be76f76ff803924ae1e9c6a501dff9beb686e899a720ed098ae6837e20861 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-16T10:28:15.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:14 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:14.781+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:15.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:14 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:14.781+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:15.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:14 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:14.781+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:15.541 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.6.service' 2024-03-16T10:28:15.587 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:28:15.587 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-16T10:28:15.587 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-16T10:28:15.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.7 2024-03-16T10:28:15.739 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:15 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:15.447+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:15.739 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:15 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:15.447+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:15.739 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:15 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:15.447+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:15.739 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:15 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:15.447+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:15.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:15.598+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:15.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:15.598+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:15.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:15.598+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:15.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:15.598+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:15.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:15.625+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:15.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:15.625+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:15.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:15.625+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.867501+0000 front 2024-03-16T10:27:44.867596+0000 (oldest deadline 2024-03-16T10:28:09.567651+0000) 2024-03-16T10:28:15.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:15 smithi192 systemd[1]: Stopping Ceph osd.7 for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:28:15.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:15 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:15.602+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:15.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:15 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:15.602+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:15.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:15 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:15.602+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:15.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:15 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:15.602+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:16.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:15 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:15.738+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:16.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:15 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:15.738+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:16.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:15 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:15.738+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:16.416 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:15.916+0000 7f42236ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:28:16.416 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:15.916+0000 7f42236ce700 -1 osd.7 91 *** Got signal Terminated *** 2024-03-16T10:28:16.417 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:15 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:15.916+0000 7f42236ce700 -1 osd.7 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T10:28:16.675 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:16 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:16.555+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:16.675 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:16 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:16.555+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:16.676 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:16 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:16.555+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:16.676 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:16 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:16.555+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:16.691 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:16 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:16.409+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:16.691 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:16 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:16.409+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:16.691 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:16 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:16.409+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:16.691 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:16 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:16.409+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:16.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:16 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:16.606+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:16.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:16 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:16.606+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:16.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:16 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:16.606+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:16.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:16 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:16.606+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:17.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:16 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:16.691+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:17.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:16 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:16.691+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:17.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:16 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:16.691+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:17.167 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:16 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:16.675+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:17.167 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:16 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:16.675+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:17.167 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:16 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:16.675+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.867501+0000 front 2024-03-16T10:27:44.867596+0000 (oldest deadline 2024-03-16T10:28:09.567651+0000) 2024-03-16T10:28:17.694 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:17 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:17.510+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:17.694 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:17 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:17.510+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:17.694 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:17 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:17.510+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:17.694 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:17 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:17.510+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:17.732 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:17 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:17.427+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:17.732 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:17 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:17.427+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:17.732 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:17 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:17.427+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:17.733 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:17 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:17.427+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:17.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:17 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:17.588+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:17.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:17 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:17.588+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:17.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:17 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:17.588+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:17.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:17 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:17.588+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:18.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:17 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:17.732+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:18.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:17 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:17.732+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:18.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:17 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:17.732+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:18.166 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:17 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:17.704+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:18.167 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:17 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:17.704+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:18.167 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:17 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:17.704+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.867501+0000 front 2024-03-16T10:27:44.867596+0000 (oldest deadline 2024-03-16T10:28:09.567651+0000) 2024-03-16T10:28:18.710 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:18 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:18.440+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:18.710 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:18 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:18.440+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:18.710 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:18 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:18.440+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:18.710 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:18 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:18.440+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:18.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:18 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:18.732+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:18.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:18 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:18.732+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:18.917 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:18 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:18.732+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.867501+0000 front 2024-03-16T10:27:44.867596+0000 (oldest deadline 2024-03-16T10:28:09.567651+0000) 2024-03-16T10:28:18.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:18 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:18.492+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:18.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:18 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:18.492+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:18.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:18 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:18.492+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:18.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:18 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:18.492+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:18.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:18 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:18.541+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:18.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:18 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:18.541+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:18.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:18 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:18.541+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:18.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:18 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:18.541+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:19.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:18 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:18.709+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:19.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:18 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:18.709+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:19.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:18 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:18.709+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:19.685 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:19 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:19.393+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:19.685 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:19 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:19.393+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:19.685 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:19 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:19.393+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:19.685 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:19 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:19.393+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:19.783 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:19 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:19.446+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:19.783 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:19 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:19.446+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:19.783 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:19 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:19.446+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:19.783 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:19 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:19.446+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:19.783 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:19 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:19.446+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:19.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:19 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:19.569+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:19.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:19 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:19.569+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:19.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:19 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:19.569+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:19.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:19 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:19.569+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:20.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:19 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:19.684+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:20.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:19 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:19.684+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:20.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:19 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:19.684+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:20.167 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:19 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:19.782+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:20.167 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:19 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:19.782+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:20.167 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:19 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:19.782+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.867501+0000 front 2024-03-16T10:27:44.867596+0000 (oldest deadline 2024-03-16T10:28:09.567651+0000) 2024-03-16T10:28:20.727 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:20 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:20.376+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:20.727 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:20 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:20.376+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:20.727 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:20 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:20.376+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:20.728 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:20 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:20.376+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:20.728 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:20 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:20.376+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:20.783 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:20 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:20.418+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:20.783 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:20 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:20.418+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:20.783 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:20 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:20.418+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:20.783 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:20 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:20.418+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:20.783 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:20 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:20.418+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:20.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:20 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:20.573+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:20.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:20 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:20.573+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:20.996 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:20 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:20.573+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:20.996 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:20 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:20.573+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:20.996 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:20 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:20.573+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:21.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:20 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:20.727+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:21.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:20 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:20.727+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:21.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:20 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:20.727+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:21.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:20 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:20.727+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.935075+0000 front 2024-03-16T10:27:54.934899+0000 (oldest deadline 2024-03-16T10:28:20.135426+0000) 2024-03-16T10:28:21.167 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:20 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:20.783+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:27.865500+0000 front 2024-03-16T10:27:27.865425+0000 (oldest deadline 2024-03-16T10:27:53.765594+0000) 2024-03-16T10:28:21.167 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:20 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:20.783+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.866603+0000 front 2024-03-16T10:27:37.866632+0000 (oldest deadline 2024-03-16T10:28:03.166566+0000) 2024-03-16T10:28:21.167 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:20 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:20.783+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.867501+0000 front 2024-03-16T10:27:44.867596+0000 (oldest deadline 2024-03-16T10:28:09.567651+0000) 2024-03-16T10:28:21.167 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:20 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7[54776]: 2024-03-16T10:28:20.783+0000 7f421bcba700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.369185+0000 front 2024-03-16T10:27:55.369205+0000 (oldest deadline 2024-03-16T10:28:20.669277+0000) 2024-03-16T10:28:21.720 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:21 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:21.356+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:21.720 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:21 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:21.356+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:21.720 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:21 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:21.356+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:21.720 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:21 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:21.356+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:21.720 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:21 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:21.356+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:21.751 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:21 smithi192 podman[71783]: 2024-03-16 10:28:21.455268117 +0000 UTC m=+5.669462758 container died 1300011d8b354639d8bfad2a8152dbc3e9be3524dacdfbfe8ac702119702690f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:28:21.752 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:21 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:21.441+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:21.752 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:21 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:21.441+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:21.752 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:21 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:21.441+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:21.752 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:21 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:21.441+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:21.752 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:21 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:21.441+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:21.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:21 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:21.608+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:21.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:21 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:21.608+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:21.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:21 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:21.608+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:21.996 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:21 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:21.608+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:21.996 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:21 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:21.608+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:22.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:21 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:21.719+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:22.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:21 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:21.719+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:22.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:21 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:21.719+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:22.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:21 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:21.719+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.935075+0000 front 2024-03-16T10:27:54.934899+0000 (oldest deadline 2024-03-16T10:28:20.135426+0000) 2024-03-16T10:28:22.060 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:21 smithi192 podman[71783]: 2024-03-16 10:28:21.866697734 +0000 UTC m=+6.080892370 container cleanup 1300011d8b354639d8bfad2a8152dbc3e9be3524dacdfbfe8ac702119702690f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-16T10:28:22.060 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:21 smithi192 bash[71783]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7 2024-03-16T10:28:22.417 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:22 smithi192 podman[71812]: 2024-03-16 10:28:22.05979625 +0000 UTC m=+0.604053157 container remove 1300011d8b354639d8bfad2a8152dbc3e9be3524dacdfbfe8ac702119702690f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-16T10:28:22.703 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:22 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:22.405+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:22.703 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:22 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:22.405+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:22.704 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:22 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:22.405+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:22.704 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:22 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:22.405+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:22.704 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:22 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:22.405+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:22.713 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:22 smithi192 podman[71897]: 2024-03-16 10:28:22.354659093 +0000 UTC m=+0.017467099 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:28:22.713 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:22 smithi192 podman[71897]: 2024-03-16 10:28:22.47734151 +0000 UTC m=+0.140149502 container create 7fc652af947bb99fb568887f658aea22a1a5dc77b566070696ceb82c4bb1c1ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-16T10:28:22.713 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:22 smithi192 podman[71897]: 2024-03-16 10:28:22.713380259 +0000 UTC m=+0.376188262 container init 7fc652af947bb99fb568887f658aea22a1a5dc77b566070696ceb82c4bb1c1ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T10:28:22.713 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:22 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:22.444+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:22.713 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:22 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:22.444+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:22.713 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:22 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:22.444+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:22.713 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:22 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:22.444+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:22.714 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:22 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:22.444+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:22.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:22 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:22.635+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:22.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:22 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:22.635+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:22.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:22 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:22.635+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:22.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:22 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:22.635+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:22.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:22 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:22.635+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:23.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:22 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:22.703+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:23.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:22 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:22.703+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:23.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:22 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:22.703+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:23.002 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:22 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:22.703+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.935075+0000 front 2024-03-16T10:27:54.934899+0000 (oldest deadline 2024-03-16T10:28:20.135426+0000) 2024-03-16T10:28:23.013 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:22 smithi192 podman[71897]: 2024-03-16 10:28:22.716619929 +0000 UTC m=+0.379427923 container start 7fc652af947bb99fb568887f658aea22a1a5dc77b566070696ceb82c4bb1c1ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-03-16T10:28:23.013 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:22 smithi192 podman[71897]: 2024-03-16 10:28:22.821670054 +0000 UTC m=+0.484478061 container attach 7fc652af947bb99fb568887f658aea22a1a5dc77b566070696ceb82c4bb1c1ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:28:23.308 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 16 10:28:23 smithi192 podman[71897]: 2024-03-16 10:28:23.0139119 +0000 UTC m=+0.676719900 container died 7fc652af947bb99fb568887f658aea22a1a5dc77b566070696ceb82c4bb1c1ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:28:23.609 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:23 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:23.477+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:23.609 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:23 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:23.477+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:23.609 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:23 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:23.477+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:23.609 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:23 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:23.477+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:23.609 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:23 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:23.477+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:23.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.7.service' 2024-03-16T10:28:23.673 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:28:23.673 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-16T10:28:23.674 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-16T10:28:23.674 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.8 2024-03-16T10:28:23.703 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:23 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:23.436+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:23.704 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:23 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:23.436+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:23.704 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:23 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:23.436+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:23.704 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:23 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:23.436+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:23.704 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:23 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:23.436+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:23.704 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:23 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:23.675+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:23.704 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:23 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:23.675+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:23.704 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:23 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:23.675+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:23.704 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:23 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:23.675+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.935075+0000 front 2024-03-16T10:27:54.934899+0000 (oldest deadline 2024-03-16T10:28:20.135426+0000) 2024-03-16T10:28:23.985 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:23 smithi193 systemd[1]: Stopping Ceph osd.8 for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:28:23.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:23 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:23.630+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:23.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:23 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:23.630+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:23.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:23 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:23.630+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:23.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:23 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:23.630+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:23.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:23 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:23.630+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:24.252 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:23 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:23.984+0000 7fde3059c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:28:24.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:23 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:23.984+0000 7fde3059c700 -1 osd.8 91 *** Got signal Terminated *** 2024-03-16T10:28:24.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:23 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:23.984+0000 7fde3059c700 -1 osd.8 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T10:28:24.752 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:24 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:24.658+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:24.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:24 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:24.658+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:24.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:24 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:24.658+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:24.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:24 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:24.658+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.935075+0000 front 2024-03-16T10:27:54.934899+0000 (oldest deadline 2024-03-16T10:28:20.135426+0000) 2024-03-16T10:28:24.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:24 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:24.486+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:24.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:24 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:24.486+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:24.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:24 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:24.486+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:24.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:24 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:24.486+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:24.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:24 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:24.486+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:24.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:24 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:24.428+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:24.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:24 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:24.428+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:24.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:24 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:24.428+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:24.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:24 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:24.428+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:24.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:24 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:24.428+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:24.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:24 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:24.671+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:24.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:24 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:24.671+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:24.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:24 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:24.671+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:24.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:24 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:24.671+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:24.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:24 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:24.671+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:25.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:25 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:25.394+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:25.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:25 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:25.394+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:25.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:25 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:25.394+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:25.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:25 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:25.394+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:25.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:25 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:25.394+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:25.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:25 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:25.676+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:25.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:25 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:25.676+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:25.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:25 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:25.676+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:25.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:25 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:25.676+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.935075+0000 front 2024-03-16T10:27:54.934899+0000 (oldest deadline 2024-03-16T10:28:20.135426+0000) 2024-03-16T10:28:25.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:25 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:25.498+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:25.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:25 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:25.498+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:25.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:25 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:25.498+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:25.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:25 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:25.498+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:25.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:25 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:25.498+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:25.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:25 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:25.650+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:25.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:25 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:25.650+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:25.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:25 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:25.650+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:25.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:25 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:25.650+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:25.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:25 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:25.650+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:26.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:26 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:26.357+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:26.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:26 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:26.357+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:26.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:26 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:26.357+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:26.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:26 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:26.357+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:26.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:26 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:26.357+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:26.912 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:26 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:26.619+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:26.912 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:26 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:26.619+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:26.912 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:26 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:26.619+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:26.912 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:26 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:26.619+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:26.912 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:26 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:26.619+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:27.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:26 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:26.633+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:27.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:26 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:26.633+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:27.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:26 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:26.633+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:27.003 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:26 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:26.633+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.935075+0000 front 2024-03-16T10:27:54.934899+0000 (oldest deadline 2024-03-16T10:28:20.135426+0000) 2024-03-16T10:28:27.003 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:26 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:26.544+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:27.003 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:26 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:26.544+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:27.003 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:26 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:26.544+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:27.003 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:26 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:26.544+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:27.003 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:26 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:26.544+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:27.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:27 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:27.375+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:27.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:27 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:27.375+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:27.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:27 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:27.375+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:27.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:27 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:27.375+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:27.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:27 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:27.375+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:27.934 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:27 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:27.514+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:27.934 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:27 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:27.514+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:27.934 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:27 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:27.514+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:27.934 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:27 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:27.514+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:27.934 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:27 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:27.514+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:27.935 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:27 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:27.514+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:27.935 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:27 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:27.658+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:27.935 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:27 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:27.658+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:27.935 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:27 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:27.658+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:27.935 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:27 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:27.658+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.935075+0000 front 2024-03-16T10:27:54.934899+0000 (oldest deadline 2024-03-16T10:28:20.135426+0000) 2024-03-16T10:28:27.935 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:27 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:27.658+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:03.536820+0000 front 2024-03-16T10:28:03.536894+0000 (oldest deadline 2024-03-16T10:28:27.036845+0000) 2024-03-16T10:28:27.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:27 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:27.582+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:27.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:27 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:27.582+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:27.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:27 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:27.582+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:27.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:27 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:27.582+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:27.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:27 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:27.582+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:27.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:27 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:27.582+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:03.601135+0000 front 2024-03-16T10:28:03.600966+0000 (oldest deadline 2024-03-16T10:28:27.100970+0000) 2024-03-16T10:28:28.666 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:28 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:28.406+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:28.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:28 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:28.406+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:28.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:28 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:28.406+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:28.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:28 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:28.406+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:28.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:28 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:28.406+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:28.752 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:28 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:28.632+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:32.830713+0000 front 2024-03-16T10:27:32.830594+0000 (oldest deadline 2024-03-16T10:27:57.530794+0000) 2024-03-16T10:28:28.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:28 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:28.632+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.531350+0000 front 2024-03-16T10:27:37.531460+0000 (oldest deadline 2024-03-16T10:28:01.631543+0000) 2024-03-16T10:28:28.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:28 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:28.632+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:46.733593+0000 front 2024-03-16T10:27:46.733698+0000 (oldest deadline 2024-03-16T10:28:11.433599+0000) 2024-03-16T10:28:28.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:28 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:28.632+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.935075+0000 front 2024-03-16T10:27:54.934899+0000 (oldest deadline 2024-03-16T10:28:20.135426+0000) 2024-03-16T10:28:28.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:28 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8[55187]: 2024-03-16T10:28:28.632+0000 7fde28b88700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:03.536820+0000 front 2024-03-16T10:28:03.536894+0000 (oldest deadline 2024-03-16T10:28:27.036845+0000) 2024-03-16T10:28:28.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:28 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:28.492+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:28.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:28 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:28.492+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:28.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:28 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:28.492+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:28.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:28 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:28.492+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:28.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:28 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:28.492+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:28.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:28 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:28.492+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:28.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:28 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:28.558+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:28.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:28 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:28.558+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:28.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:28 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:28.558+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:28.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:28 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:28.558+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:28.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:28 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:28.558+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:28.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:28 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:28.558+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:03.601135+0000 front 2024-03-16T10:28:03.600966+0000 (oldest deadline 2024-03-16T10:28:27.100970+0000) 2024-03-16T10:28:29.844 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:29 smithi193 podman[72882]: 2024-03-16 10:28:29.514867061 +0000 UTC m=+5.653503407 container died fcde8b8494444c4f88320cba06e36beaba8a7922f427ff6bc3d0441e8f717ef5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-16T10:28:29.844 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:29 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:29.510+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:29.844 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:29 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:29.510+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:29.844 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:29 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:29.510+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:29.844 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:29 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:29.510+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:29.844 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:29 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:29.510+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:29.844 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:29 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:29.510+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:29.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:29 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:29.438+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:29.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:29 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:29.438+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:29.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:29 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:29.438+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:29.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:29 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:29.438+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:29.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:29 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:29.438+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:29.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:29 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:29.565+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:29.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:29 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:29.565+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:29.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:29 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:29.565+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:29.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:29 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:29.565+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:29.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:29 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:29.565+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:29.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:29 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:29.565+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:03.601135+0000 front 2024-03-16T10:28:03.600966+0000 (oldest deadline 2024-03-16T10:28:27.100970+0000) 2024-03-16T10:28:30.101 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:29 smithi193 podman[72882]: 2024-03-16 10:28:29.942541267 +0000 UTC m=+6.081177625 container cleanup fcde8b8494444c4f88320cba06e36beaba8a7922f427ff6bc3d0441e8f717ef5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-16T10:28:30.102 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:29 smithi193 bash[72882]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8 2024-03-16T10:28:30.502 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:30 smithi193 podman[72912]: 2024-03-16 10:28:30.10192298 +0000 UTC m=+0.590219576 container remove fcde8b8494444c4f88320cba06e36beaba8a7922f427ff6bc3d0441e8f717ef5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-16T10:28:30.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:30 smithi193 podman[73003]: 2024-03-16 10:28:30.405537969 +0000 UTC m=+0.017777438 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:28:30.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:30 smithi193 podman[73003]: 2024-03-16 10:28:30.502499654 +0000 UTC m=+0.114739129 container create afd8cdb0b4be808e8f41f8fea591c9210b2d230e20874abfa293ea8aa5852a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-16T10:28:30.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:30 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:30.535+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:30.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:30 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:30.535+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:30.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:30 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:30.535+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:30.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:30 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:30.535+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:30.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:30 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:30.535+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:30.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:30 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:30.535+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:30.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:30 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:30.419+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:30.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:30 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:30.419+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:30.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:30 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:30.419+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:30.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:30 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:30.419+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:30.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:30 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:30.419+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:30.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:30 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:30.419+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:30.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:30 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:30.536+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:30.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:30 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:30.536+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:30.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:30 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:30.536+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:30.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:30 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:30.536+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:30.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:30 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:30.536+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:30.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:30 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:30.536+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:03.601135+0000 front 2024-03-16T10:28:03.600966+0000 (oldest deadline 2024-03-16T10:28:27.100970+0000) 2024-03-16T10:28:31.048 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:30 smithi193 podman[73003]: 2024-03-16 10:28:30.755731376 +0000 UTC m=+0.367970846 container init afd8cdb0b4be808e8f41f8fea591c9210b2d230e20874abfa293ea8aa5852a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-16T10:28:31.048 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:30 smithi193 podman[73003]: 2024-03-16 10:28:30.758761806 +0000 UTC m=+0.371001270 container start afd8cdb0b4be808e8f41f8fea591c9210b2d230e20874abfa293ea8aa5852a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-16T10:28:31.048 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:30 smithi193 podman[73003]: 2024-03-16 10:28:30.839143144 +0000 UTC m=+0.451382613 container attach afd8cdb0b4be808e8f41f8fea591c9210b2d230e20874abfa293ea8aa5852a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-16T10:28:31.342 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 16 10:28:31 smithi193 podman[73003]: 2024-03-16 10:28:31.048241926 +0000 UTC m=+0.660481395 container died afd8cdb0b4be808e8f41f8fea591c9210b2d230e20874abfa293ea8aa5852a6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-16T10:28:31.593 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:31 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:31.500+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:31.594 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:31 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:31.500+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:31.594 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:31 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:31.500+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:31.594 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:31 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:31.500+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:31.594 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:31 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:31.500+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:31.594 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:31 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:31.500+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:31.608 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.8.service' 2024-03-16T10:28:31.666 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:31 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:31.373+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:31.666 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:31 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:31.373+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:31.666 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:31 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:31.373+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:31.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:31 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:31.373+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:31.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:31 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:31.373+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:31.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:31 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:31.373+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:31.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:28:31.690 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-16T10:28:31.690 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-16T10:28:31.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.9 2024-03-16T10:28:31.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:31 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:31.511+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:31.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:31 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:31.511+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:31.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:31 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:31.511+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:31.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:31 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:31.511+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:31.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:31 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:31.511+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:31.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:31 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:31.511+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:03.601135+0000 front 2024-03-16T10:28:03.600966+0000 (oldest deadline 2024-03-16T10:28:27.100970+0000) 2024-03-16T10:28:31.996 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:31 smithi077 systemd[1]: Stopping Ceph osd.9 for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:28:32.473 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:32.012+0000 7f79c1b8a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:28:32.473 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:32.012+0000 7f79c1b8a700 -1 osd.9 91 *** Got signal Terminated *** 2024-03-16T10:28:32.473 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:32.012+0000 7f79c1b8a700 -1 osd.9 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T10:28:32.473 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:32.472+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:32.473 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:32.472+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:32.473 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:32.472+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:32.474 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:32.472+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:32.474 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:32.472+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:32.666 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:32 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:32.376+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:32.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:32 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:32.376+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:32.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:32 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:32.376+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:32.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:32 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:32.376+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:32.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:32 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:32.376+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:32.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:32 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:32.376+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:32.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:32 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:32.472+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:03.601135+0000 front 2024-03-16T10:28:03.600966+0000 (oldest deadline 2024-03-16T10:28:27.100970+0000) 2024-03-16T10:28:32.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:32 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:32.482+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:32.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:32 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:32.482+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:32.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:32 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:32.482+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:32.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:32 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:32.482+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:32.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:32 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:32.482+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:32.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:32 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:32.482+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:33.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:33 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:33.384+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:33.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:33 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:33.384+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:33.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:33 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:33.384+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:33.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:33 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:33.384+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:33.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:33 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:33.384+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:33.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:33 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:33.384+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:33.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:33 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:33.479+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:33.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:33 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:33.479+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:33.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:33 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:33.479+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:33.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:33 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:33.479+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:33.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:33 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:33.479+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:33.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:33 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:33.479+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:33.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:33 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:33.518+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:33.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:33 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:33.518+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:33.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:33 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:33.518+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:33.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:33 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:33.518+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:33.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:33 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:33.518+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:33.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:33 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:33.518+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:03.601135+0000 front 2024-03-16T10:28:03.600966+0000 (oldest deadline 2024-03-16T10:28:27.100970+0000) 2024-03-16T10:28:34.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:34 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:34.492+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:34.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:34 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:34.492+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:34.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:34 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:34.492+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:34.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:34 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:34.492+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:34.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:34 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:34.492+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:34.745 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:34 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:34.492+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:03.601135+0000 front 2024-03-16T10:28:03.600966+0000 (oldest deadline 2024-03-16T10:28:27.100970+0000) 2024-03-16T10:28:34.752 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:34 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:34.436+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:34.752 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:34 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:34.436+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:34.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:34 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:34.436+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:34.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:34 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:34.436+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:34.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:34 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:34.436+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:34.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:34 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:34.437+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:34.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:34 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:34.417+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:34.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:34 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:34.417+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:34.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:34 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:34.417+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:34.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:34 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:34.417+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:34.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:34 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:34.417+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:34.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:34 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:34.417+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:35.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:35 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:35.376+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:35.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:35 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:35.376+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:35.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:35 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:35.376+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:35.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:35 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:35.377+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:35.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:35 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:35.377+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:35.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:35 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:35.377+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:35.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:35 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:35.418+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:35.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:35 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:35.418+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:35.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:35 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:35.418+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:35.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:35 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:35.418+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:35.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:35 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:35.418+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:35.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:35 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:35.418+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:35.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:35 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:35.533+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:35.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:35 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:35.533+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:35.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:35 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:35.533+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:35.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:35 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:35.533+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:35.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:35 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:35.533+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:35.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:35 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:35.533+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:03.601135+0000 front 2024-03-16T10:28:03.600966+0000 (oldest deadline 2024-03-16T10:28:27.100970+0000) 2024-03-16T10:28:36.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:36 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:36.414+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:36.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:36 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:36.414+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:36.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:36 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:36.414+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:36.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:36 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:36.414+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:36.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:36 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:36.414+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:36.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:36 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:36.414+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:36.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:36 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:36.435+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:36.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:36 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:36.435+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:36.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:36 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:36.435+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:36.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:36 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:36.435+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:36.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:36 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:36.435+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:36.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:36 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:36.435+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:36.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:36 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:36.547+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:24.593668+0000 front 2024-03-16T10:27:24.593934+0000 (oldest deadline 2024-03-16T10:27:48.594414+0000) 2024-03-16T10:28:36.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:36 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:36.548+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:28.594820+0000 front 2024-03-16T10:27:28.594754+0000 (oldest deadline 2024-03-16T10:27:53.894800+0000) 2024-03-16T10:28:36.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:36 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:36.548+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:37.395901+0000 front 2024-03-16T10:27:37.395912+0000 (oldest deadline 2024-03-16T10:28:01.496005+0000) 2024-03-16T10:28:36.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:36 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:36.548+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:43.197139+0000 front 2024-03-16T10:27:43.197113+0000 (oldest deadline 2024-03-16T10:28:14.398010+0000) 2024-03-16T10:28:36.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:36 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:36.548+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:56.099500+0000 front 2024-03-16T10:27:56.099403+0000 (oldest deadline 2024-03-16T10:28:19.599297+0000) 2024-03-16T10:28:36.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:36 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9[65720]: 2024-03-16T10:28:36.548+0000 7f79be17e700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:03.601135+0000 front 2024-03-16T10:28:03.600966+0000 (oldest deadline 2024-03-16T10:28:27.100970+0000) 2024-03-16T10:28:37.663 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:37 smithi077 podman[99248]: 2024-03-16 10:28:37.408246331 +0000 UTC m=+5.516726693 container died a1b7b6f74a8e50072f50ec6c452c60e692937fb4c9df25b9871186b6906756e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-16T10:28:37.666 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:37 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:37.409+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:37.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:37 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:37.409+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:37.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:37 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:37.409+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:37.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:37 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:37.409+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:37.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:37 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:37.409+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:37.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:37 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:37.409+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:37.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:37 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:37.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:37.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:37 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:37.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:37.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:37 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:37.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:37.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:37 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:37.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:37.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:37 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:37.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:37.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:37 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:37.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:37.930 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:37 smithi077 podman[99248]: 2024-03-16 10:28:37.746127149 +0000 UTC m=+5.854607510 container cleanup a1b7b6f74a8e50072f50ec6c452c60e692937fb4c9df25b9871186b6906756e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-16T10:28:37.931 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:37 smithi077 bash[99248]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9 2024-03-16T10:28:38.245 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:37 smithi077 podman[99269]: 2024-03-16 10:28:37.93049544 +0000 UTC m=+0.514923236 container remove a1b7b6f74a8e50072f50ec6c452c60e692937fb4c9df25b9871186b6906756e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-16T10:28:38.666 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:38 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:38.408+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:38.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:38 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:38.408+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:38.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:38 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:38.408+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:38.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:38 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:38.408+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:38.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:38 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:38.408+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:38.667 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:38 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:38.408+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:38.667 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:38 smithi077 podman[99359]: 2024-03-16 10:28:38.347980026 +0000 UTC m=+0.100693957 container create 303da5822d59c275293d2f8078f58223780df3b67e482a013c445956bc844548 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-16T10:28:38.667 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:38 smithi077 podman[99359]: 2024-03-16 10:28:38.265344829 +0000 UTC m=+0.018058771 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:28:38.668 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:38 smithi077 podman[99359]: 2024-03-16 10:28:38.583853366 +0000 UTC m=+0.336567301 container init 303da5822d59c275293d2f8078f58223780df3b67e482a013c445956bc844548 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-16T10:28:38.668 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:38 smithi077 podman[99359]: 2024-03-16 10:28:38.586805933 +0000 UTC m=+0.339519865 container start 303da5822d59c275293d2f8078f58223780df3b67e482a013c445956bc844548 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:28:38.668 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:38 smithi077 podman[99359]: 2024-03-16 10:28:38.667176129 +0000 UTC m=+0.419890068 container attach 303da5822d59c275293d2f8078f58223780df3b67e482a013c445956bc844548 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:28:38.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:38 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:38.444+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:38.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:38 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:38.444+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:38.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:38 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:38.444+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:38.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:38 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:38.444+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:38.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:38 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:38.444+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:38.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:38 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:38.444+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:38.995 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:38 smithi077 podman[99359]: 2024-03-16 10:28:38.876518101 +0000 UTC m=+0.629232033 container died 303da5822d59c275293d2f8078f58223780df3b67e482a013c445956bc844548 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:28:39.444 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.9.service' 2024-03-16T10:28:39.472 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:39 smithi077 podman[99384]: 2024-03-16 10:28:39.4204434 +0000 UTC m=+0.539288091 container remove 303da5822d59c275293d2f8078f58223780df3b67e482a013c445956bc844548 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-16T10:28:39.472 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:39 smithi077 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.9.service: Deactivated successfully. 2024-03-16T10:28:39.472 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:39 smithi077 systemd[1]: Stopped Ceph osd.9 for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:28:39.472 INFO:journalctl@ceph.osd.9.smithi077.stdout:Mar 16 10:28:39 smithi077 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.9.service: Consumed 42.890s CPU time. 2024-03-16T10:28:39.488 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:28:39.488 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-16T10:28:39.488 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-16T10:28:39.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.10 2024-03-16T10:28:39.752 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:39 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:39.413+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:39.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:39 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:39.413+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:39.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:39 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:39.413+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:39.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:39 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:39.413+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:39.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:39 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:39.413+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:39.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:39 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:39.413+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:39.803 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:39 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:39.424+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:39.803 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:39 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:39.424+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:39.803 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:39 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:39.424+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:39.803 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:39 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:39.424+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:39.803 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:39 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:39.424+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:39.803 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:39 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:39.424+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:39.803 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:39 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:39.424+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T10:28:10.345826+0000 front 2024-03-16T10:28:10.345790+0000 (oldest deadline 2024-03-16T10:28:38.545896+0000) 2024-03-16T10:28:39.803 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:39 smithi192 systemd[1]: Stopping Ceph osd.10 for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:28:39.803 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:39 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:39.802+0000 7f1bdcfd0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:28:39.804 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:39 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:39.802+0000 7f1bdcfd0700 -1 osd.10 91 *** Got signal Terminated *** 2024-03-16T10:28:39.804 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:39 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:39.802+0000 7f1bdcfd0700 -1 osd.10 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T10:28:40.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:40 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:40.428+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:40.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:40 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:40.428+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:40.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:40 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:40.428+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:40.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:40 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:40.428+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:40.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:40 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:40.428+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:40.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:40 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:40.428+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:40.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:40 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:40.464+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:40.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:40 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:40.464+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:40.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:40 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:40.464+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:40.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:40 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:40.464+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:40.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:40 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:40.464+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:40.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:40 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:40.464+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:40.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:40 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:40.464+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T10:28:10.345826+0000 front 2024-03-16T10:28:10.345790+0000 (oldest deadline 2024-03-16T10:28:38.545896+0000) 2024-03-16T10:28:41.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:41 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:41.411+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:41.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:41 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:41.411+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:41.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:41 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:41.411+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:41.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:41 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:41.411+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:41.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:41 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:41.411+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:41.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:41 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:41.411+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:41.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:41 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:41.496+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:41.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:41 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:41.496+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:41.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:41 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:41.496+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:41.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:41 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:41.496+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:41.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:41 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:41.496+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:41.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:41 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:41.496+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:41.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:41 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:41.496+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T10:28:10.345826+0000 front 2024-03-16T10:28:10.345790+0000 (oldest deadline 2024-03-16T10:28:38.545896+0000) 2024-03-16T10:28:42.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:42 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:42.375+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:42.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:42 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:42.375+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:42.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:42 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:42.375+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:42.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:42 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:42.375+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:42.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:42 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:42.375+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:42.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:42 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:42.375+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:42.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:42 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:42.375+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6820 osd.7 since back 2024-03-16T10:28:16.167142+0000 front 2024-03-16T10:28:16.167086+0000 (oldest deadline 2024-03-16T10:28:41.466840+0000) 2024-03-16T10:28:42.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:42 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:42.455+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:42.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:42 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:42.455+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:42.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:42 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:42.455+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:42.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:42 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:42.455+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:42.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:42 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:42.455+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:42.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:42 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:42.455+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:42.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:42 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:42.455+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T10:28:10.345826+0000 front 2024-03-16T10:28:10.345790+0000 (oldest deadline 2024-03-16T10:28:38.545896+0000) 2024-03-16T10:28:43.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:43 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:43.377+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:43.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:43 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:43.377+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:43.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:43 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:43.377+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:43.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:43 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:43.377+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:43.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:43 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:43.377+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:43.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:43 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:43.377+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:43.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:43 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:43.377+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6820 osd.7 since back 2024-03-16T10:28:16.167142+0000 front 2024-03-16T10:28:16.167086+0000 (oldest deadline 2024-03-16T10:28:41.466840+0000) 2024-03-16T10:28:43.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:43 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:43.470+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:43.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:43 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:43.470+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:43.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:43 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:43.470+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:43.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:43 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:43.470+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:43.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:43 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:43.470+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:43.918 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:43 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:43.470+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:43.918 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:43 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:43.470+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T10:28:10.345826+0000 front 2024-03-16T10:28:10.345790+0000 (oldest deadline 2024-03-16T10:28:38.545896+0000) 2024-03-16T10:28:44.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:44 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:44.331+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:44.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:44 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:44.331+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:44.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:44 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:44.331+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:44.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:44 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:44.331+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:44.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:44 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:44.331+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:44.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:44 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:44.331+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:44.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:44 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:44.331+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6820 osd.7 since back 2024-03-16T10:28:16.167142+0000 front 2024-03-16T10:28:16.167086+0000 (oldest deadline 2024-03-16T10:28:41.466840+0000) 2024-03-16T10:28:44.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:44 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:44.476+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.138451+0000 front 2024-03-16T10:27:23.138430+0000 (oldest deadline 2024-03-16T10:27:49.038493+0000) 2024-03-16T10:28:44.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:44 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:44.476+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:30.739494+0000 front 2024-03-16T10:27:30.739529+0000 (oldest deadline 2024-03-16T10:27:57.740265+0000) 2024-03-16T10:28:44.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:44 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:44.476+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:39.441259+0000 front 2024-03-16T10:27:39.441439+0000 (oldest deadline 2024-03-16T10:28:01.741229+0000) 2024-03-16T10:28:44.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:44 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:44.476+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:44.542818+0000 front 2024-03-16T10:27:44.542930+0000 (oldest deadline 2024-03-16T10:28:13.343204+0000) 2024-03-16T10:28:44.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:44 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:44.476+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:55.044161+0000 front 2024-03-16T10:27:55.044213+0000 (oldest deadline 2024-03-16T10:28:19.144306+0000) 2024-03-16T10:28:44.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:44 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:44.476+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:28:04.445517+0000 front 2024-03-16T10:28:04.445482+0000 (oldest deadline 2024-03-16T10:28:30.345189+0000) 2024-03-16T10:28:44.917 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:44 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10[61687]: 2024-03-16T10:28:44.476+0000 7f1bd95c4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.77:6822 osd.6 since back 2024-03-16T10:28:10.345826+0000 front 2024-03-16T10:28:10.345790+0000 (oldest deadline 2024-03-16T10:28:38.545896+0000) 2024-03-16T10:28:45.470 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:45 smithi192 podman[72001]: 2024-03-16 10:28:45.194076409 +0000 UTC m=+5.508441768 container died baa1f177535e11047f45b319ec5b3357ceaf3b17ca32136130619517d452ae4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-16T10:28:45.752 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:45 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:45.342+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:45.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:45 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:45.342+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:45.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:45 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:45.342+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:45.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:45 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:45.342+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:45.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:45 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:45.342+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:45.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:45 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:45.342+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:45.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:45 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:45.342+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6820 osd.7 since back 2024-03-16T10:28:16.167142+0000 front 2024-03-16T10:28:16.167086+0000 (oldest deadline 2024-03-16T10:28:41.466840+0000) 2024-03-16T10:28:45.853 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:45 smithi192 podman[72001]: 2024-03-16 10:28:45.568825382 +0000 UTC m=+5.883190741 container cleanup baa1f177535e11047f45b319ec5b3357ceaf3b17ca32136130619517d452ae4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:28:45.853 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:45 smithi192 bash[72001]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10 2024-03-16T10:28:46.166 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:45 smithi192 podman[72019]: 2024-03-16 10:28:45.853319266 +0000 UTC m=+0.653783891 container remove baa1f177535e11047f45b319ec5b3357ceaf3b17ca32136130619517d452ae4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True) 2024-03-16T10:28:46.440 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:46 smithi192 podman[72109]: 2024-03-16 10:28:46.187204059 +0000 UTC m=+0.113571127 container create 839ba8d8a347ad81dbcef495b2debf21f20c5fde9c8466045183f1fe4d1f3ef3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-16T10:28:46.441 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:46 smithi192 podman[72109]: 2024-03-16 10:28:46.091299467 +0000 UTC m=+0.017666560 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:28:46.441 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:46 smithi192 podman[72109]: 2024-03-16 10:28:46.382018947 +0000 UTC m=+0.308386040 container init 839ba8d8a347ad81dbcef495b2debf21f20c5fde9c8466045183f1fe4d1f3ef3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:28:46.441 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:46 smithi192 podman[72109]: 2024-03-16 10:28:46.385228831 +0000 UTC m=+0.311595912 container start 839ba8d8a347ad81dbcef495b2debf21f20c5fde9c8466045183f1fe4d1f3ef3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-16T10:28:46.441 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:46 smithi192 podman[72109]: 2024-03-16 10:28:46.439872845 +0000 UTC m=+0.366239921 container attach 839ba8d8a347ad81dbcef495b2debf21f20c5fde9c8466045183f1fe4d1f3ef3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-16T10:28:46.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:46 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:46.328+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:46.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:46 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:46.328+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:46.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:46 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:46.328+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:46.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:46 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:46.328+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:46.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:46 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:46.328+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:46.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:46 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:46.328+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:46.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:46 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:46.328+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6820 osd.7 since back 2024-03-16T10:28:16.167142+0000 front 2024-03-16T10:28:16.167086+0000 (oldest deadline 2024-03-16T10:28:41.466840+0000) 2024-03-16T10:28:46.916 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:46 smithi192 podman[72109]: 2024-03-16 10:28:46.673067193 +0000 UTC m=+0.599434269 container died 839ba8d8a347ad81dbcef495b2debf21f20c5fde9c8466045183f1fe4d1f3ef3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-16T10:28:47.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.10.service' 2024-03-16T10:28:47.251 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:47 smithi192 podman[72135]: 2024-03-16 10:28:47.20199739 +0000 UTC m=+0.523744688 container remove 839ba8d8a347ad81dbcef495b2debf21f20c5fde9c8466045183f1fe4d1f3ef3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-16T10:28:47.252 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:47 smithi192 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.10.service: Deactivated successfully. 2024-03-16T10:28:47.252 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:47 smithi192 systemd[1]: Stopped Ceph osd.10 for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:28:47.252 INFO:journalctl@ceph.osd.10.smithi192.stdout:Mar 16 10:28:47 smithi192 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.10.service: Consumed 1min 578ms CPU time. 2024-03-16T10:28:47.268 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:28:47.268 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-16T10:28:47.268 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-16T10:28:47.269 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.11 2024-03-16T10:28:47.630 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:47 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:47.288+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:47.630 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:47 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:47.288+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:47.630 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:47 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:47.288+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:47.630 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:47 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:47.288+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:47.630 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:47 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:47.288+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:47.630 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:47 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:47.288+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:47.630 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:47 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:47.288+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6820 osd.7 since back 2024-03-16T10:28:16.167142+0000 front 2024-03-16T10:28:16.167086+0000 (oldest deadline 2024-03-16T10:28:41.466840+0000) 2024-03-16T10:28:47.631 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:47 smithi193 systemd[1]: Stopping Ceph osd.11 for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:28:47.631 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:47 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:47.628+0000 7f1acd1c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:28:47.631 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:47 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:47.628+0000 7f1acd1c1700 -1 osd.11 91 *** Got signal Terminated *** 2024-03-16T10:28:47.631 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:47 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:47.628+0000 7f1acd1c1700 -1 osd.11 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T10:28:48.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:48 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:48.335+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:48.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:48 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:48.335+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:48.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:48 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:48.335+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:48.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:48 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:48.335+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:48.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:48 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:48.335+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:48.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:48 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:48.335+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:48.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:48 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:48.335+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6820 osd.7 since back 2024-03-16T10:28:16.167142+0000 front 2024-03-16T10:28:16.167086+0000 (oldest deadline 2024-03-16T10:28:41.466840+0000) 2024-03-16T10:28:49.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:49 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:49.384+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:49.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:49 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:49.384+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:49.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:49 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:49.384+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:49.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:49 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:49.384+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:49.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:49 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:49.384+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:49.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:49 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:49.384+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:49.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:49 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:49.384+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6820 osd.7 since back 2024-03-16T10:28:16.167142+0000 front 2024-03-16T10:28:16.167086+0000 (oldest deadline 2024-03-16T10:28:41.466840+0000) 2024-03-16T10:28:50.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:50 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:50.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:50.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:50 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:50.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:50.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:50 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:50.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:50.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:50 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:50.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:50.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:50 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:50.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:50.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:50 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:50.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:50.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:50 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:50.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6820 osd.7 since back 2024-03-16T10:28:16.167142+0000 front 2024-03-16T10:28:16.167086+0000 (oldest deadline 2024-03-16T10:28:41.466840+0000) 2024-03-16T10:28:51.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:51 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:51.415+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:51.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:51 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:51.415+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:51.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:51 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:51.415+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:51.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:51 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:51.415+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:51.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:51 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:51.415+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:51.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:51 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:51.415+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:51.754 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:51 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:51.415+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6820 osd.7 since back 2024-03-16T10:28:16.167142+0000 front 2024-03-16T10:28:16.167086+0000 (oldest deadline 2024-03-16T10:28:41.466840+0000) 2024-03-16T10:28:52.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:52 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:52.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-03-16T10:27:23.358444+0000 front 2024-03-16T10:27:23.358460+0000 (oldest deadline 2024-03-16T10:27:46.858456+0000) 2024-03-16T10:28:52.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:52 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:52.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6804 osd.1 since back 2024-03-16T10:27:29.060049+0000 front 2024-03-16T10:27:29.060033+0000 (oldest deadline 2024-03-16T10:27:53.760167+0000) 2024-03-16T10:28:52.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:52 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:52.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-16T10:27:40.761603+0000 front 2024-03-16T10:27:40.761768+0000 (oldest deadline 2024-03-16T10:28:03.661671+0000) 2024-03-16T10:28:52.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:52 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:52.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.77:6814 osd.3 since back 2024-03-16T10:27:48.862984+0000 front 2024-03-16T10:27:48.862900+0000 (oldest deadline 2024-03-16T10:28:11.162806+0000) 2024-03-16T10:28:52.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:52 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:52.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6812 osd.4 since back 2024-03-16T10:27:54.063944+0000 front 2024-03-16T10:27:54.064005+0000 (oldest deadline 2024-03-16T10:28:19.963755+0000) 2024-03-16T10:28:52.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:52 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:52.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-16T10:27:59.964133+0000 front 2024-03-16T10:27:59.964120+0000 (oldest deadline 2024-03-16T10:28:26.964645+0000) 2024-03-16T10:28:52.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:52 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11[62270]: 2024-03-16T10:28:52.397+0000 7f1ac57ad700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.192:6820 osd.7 since back 2024-03-16T10:28:16.167142+0000 front 2024-03-16T10:28:16.167086+0000 (oldest deadline 2024-03-16T10:28:41.466840+0000) 2024-03-16T10:28:53.253 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:52 smithi193 podman[73106]: 2024-03-16 10:28:52.964698553 +0000 UTC m=+5.505516179 container died a1c1422524f7cdeabcf70d14973fd91392eaa6a1ab51d60378732f380a9975ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:28:53.680 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:53 smithi193 podman[73106]: 2024-03-16 10:28:53.454694067 +0000 UTC m=+5.995511694 container cleanup a1c1422524f7cdeabcf70d14973fd91392eaa6a1ab51d60378732f380a9975ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:28:53.681 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:53 smithi193 bash[73106]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11 2024-03-16T10:28:54.002 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:53 smithi193 podman[73125]: 2024-03-16 10:28:53.680748031 +0000 UTC m=+0.709549007 container remove a1c1422524f7cdeabcf70d14973fd91392eaa6a1ab51d60378732f380a9975ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:28:54.325 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:54 smithi193 podman[73216]: 2024-03-16 10:28:54.006450414 +0000 UTC m=+0.113346551 container create 84c4195844cd9402356ae0602da42adba01627460f62f873527afa12d0aebf18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-16T10:28:54.325 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:54 smithi193 podman[73216]: 2024-03-16 10:28:53.911292399 +0000 UTC m=+0.018188505 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T10:28:54.325 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:54 smithi193 podman[73216]: 2024-03-16 10:28:54.250677795 +0000 UTC m=+0.357573903 container init 84c4195844cd9402356ae0602da42adba01627460f62f873527afa12d0aebf18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:28:54.326 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:54 smithi193 podman[73216]: 2024-03-16 10:28:54.253752847 +0000 UTC m=+0.360648933 container start 84c4195844cd9402356ae0602da42adba01627460f62f873527afa12d0aebf18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-16T10:28:54.326 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:54 smithi193 podman[73216]: 2024-03-16 10:28:54.325616546 +0000 UTC m=+0.432512644 container attach 84c4195844cd9402356ae0602da42adba01627460f62f873527afa12d0aebf18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-16T10:28:54.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 16 10:28:54 smithi193 podman[73216]: 2024-03-16 10:28:54.539412056 +0000 UTC m=+0.646308151 container died 84c4195844cd9402356ae0602da42adba01627460f62f873527afa12d0aebf18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-osd-11-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-16T10:28:55.164 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@osd.11.service' 2024-03-16T10:28:55.203 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:28:55.204 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-16T10:28:55.204 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-16T10:28:55.204 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.a 2024-03-16T10:28:55.495 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:28:55 smithi077 systemd[1]: Stopping Ceph mds.a for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:28:55.995 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:28:55 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a[72289]: 2024-03-16T10:28:55.630+0000 7f51aee41700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:28:55.995 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:28:55 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a[72289]: 2024-03-16T10:28:55.630+0000 7f51aee41700 -1 mds.a *** got signal Terminated *** 2024-03-16T10:28:59.890 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:28:59 smithi077 podman[99476]: 2024-03-16 10:28:59.638359368 +0000 UTC m=+4.239988189 container died 2c643f6991bd5e3d168773ab9669ba48f8c506a4f8d822c71bcbffefb34e754a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-16T10:29:00.157 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:28:59 smithi077 podman[99476]: 2024-03-16 10:28:59.972507573 +0000 UTC m=+4.574136380 container cleanup 2c643f6991bd5e3d168773ab9669ba48f8c506a4f8d822c71bcbffefb34e754a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-16T10:29:00.157 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:28:59 smithi077 bash[99476]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a 2024-03-16T10:29:00.401 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.a.service' 2024-03-16T10:29:00.425 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:29:00 smithi077 podman[99502]: 2024-03-16 10:29:00.157077214 +0000 UTC m=+0.513873555 container remove 2c643f6991bd5e3d168773ab9669ba48f8c506a4f8d822c71bcbffefb34e754a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-16T10:29:00.425 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:29:00 smithi077 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.a.service: Deactivated successfully. 2024-03-16T10:29:00.426 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:29:00 smithi077 systemd[1]: Stopped Ceph mds.a for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:29:00.426 INFO:journalctl@ceph.mds.a.smithi077.stdout:Mar 16 10:29:00 smithi077 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.a.service: Consumed 4min 5.968s CPU time. 2024-03-16T10:29:00.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:29:00.440 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-16T10:29:00.440 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-16T10:29:00.440 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.d 2024-03-16T10:29:00.745 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:29:00 smithi077 systemd[1]: Stopping Ceph mds.d for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:29:01.074 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:29:00 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-d[71330]: 2024-03-16T10:29:00.750+0000 7fb6a74df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:29:01.074 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:29:00 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-d[71330]: 2024-03-16T10:29:00.750+0000 7fb6a74df700 -1 mds.d *** got signal Terminated *** 2024-03-16T10:29:05.110 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:29:04 smithi077 podman[99609]: 2024-03-16 10:29:04.761126926 +0000 UTC m=+4.103419120 container died ffb63e7b892a7e0ddd4667a48efff0c3897bb082d80aac64b1af52060afebd3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-16T10:29:05.377 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:29:05 smithi077 podman[99609]: 2024-03-16 10:29:05.192800969 +0000 UTC m=+4.535093160 container cleanup ffb63e7b892a7e0ddd4667a48efff0c3897bb082d80aac64b1af52060afebd3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-d, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-16T10:29:05.377 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:29:05 smithi077 bash[99609]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-d 2024-03-16T10:29:05.613 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.d.service' 2024-03-16T10:29:05.636 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:29:05 smithi077 podman[99625]: 2024-03-16 10:29:05.377261196 +0000 UTC m=+0.614773753 container remove ffb63e7b892a7e0ddd4667a48efff0c3897bb082d80aac64b1af52060afebd3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-16T10:29:05.637 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:29:05 smithi077 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.d.service: Deactivated successfully. 2024-03-16T10:29:05.637 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:29:05 smithi077 systemd[1]: Stopped Ceph mds.d for c6ee6c60-e37c-11ee-95c9-87774f69a715. 2024-03-16T10:29:05.637 INFO:journalctl@ceph.mds.d.smithi077.stdout:Mar 16 10:29:05 smithi077 systemd[1]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.d.service: Consumed 1.002s CPU time. 2024-03-16T10:29:05.649 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:29:05.650 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-16T10:29:05.650 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-16T10:29:05.650 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.g 2024-03-16T10:29:05.981 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:29:05 smithi077 systemd[1]: Stopping Ceph mds.g for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:29:06.245 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:29:05 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-g[71925]: 2024-03-16T10:29:05.979+0000 7f32499c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:29:06.245 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:29:05 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-g[71925]: 2024-03-16T10:29:05.979+0000 7f32499c3700 -1 mds.g *** got signal Terminated *** 2024-03-16T10:29:08.831 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:29:08 smithi077 podman[99731]: 2024-03-16 10:29:08.556654153 +0000 UTC m=+2.683755969 container died 367a20f0727ea66891733d90982f79474afb1613a3e8aa56541cc864cc7deb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-16T10:29:09.181 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:29:08 smithi077 podman[99731]: 2024-03-16 10:29:08.930240511 +0000 UTC m=+3.057342282 container cleanup 367a20f0727ea66891733d90982f79474afb1613a3e8aa56541cc864cc7deb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-g, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-16T10:29:09.181 INFO:journalctl@ceph.mds.g.smithi077.stdout:Mar 16 10:29:08 smithi077 bash[99731]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-g 2024-03-16T10:29:09.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.g.service' 2024-03-16T10:29:09.373 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:29:09.374 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-16T10:29:09.374 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-16T10:29:09.374 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.j 2024-03-16T10:29:09.745 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:29:09 smithi077 systemd[1]: Stopping Ceph mds.j for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:29:09.745 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:29:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-j[70985]: 2024-03-16T10:29:09.675+0000 7f415335f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:29:09.745 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:29:09 smithi077 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-j[70985]: 2024-03-16T10:29:09.675+0000 7f415335f700 -1 mds.j *** got signal Terminated *** 2024-03-16T10:29:13.995 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:29:13 smithi077 podman[99852]: 2024-03-16 10:29:13.681052846 +0000 UTC m=+4.093843965 container died 08b26e4e30121d1eefeb76cd8066227a6491c13922bec06f77d1ea9fcce460e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-j, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-16T10:29:14.318 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:29:14 smithi077 podman[99852]: 2024-03-16 10:29:14.100753732 +0000 UTC m=+4.513544849 container cleanup 08b26e4e30121d1eefeb76cd8066227a6491c13922bec06f77d1ea9fcce460e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-j, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-16T10:29:14.318 INFO:journalctl@ceph.mds.j.smithi077.stdout:Mar 16 10:29:14 smithi077 bash[99852]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-j 2024-03-16T10:29:14.523 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.j.service' 2024-03-16T10:29:14.556 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:29:14.556 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-16T10:29:14.557 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-16T10:29:14.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.b 2024-03-16T10:29:14.862 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:29:14 smithi192 systemd[1]: Stopping Ceph mds.b for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:29:15.166 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:29:14 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b[67101]: 2024-03-16T10:29:14.860+0000 7f370821d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:29:15.166 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:29:14 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b[67101]: 2024-03-16T10:29:14.860+0000 7f370821d700 -1 mds.b *** got signal Terminated *** 2024-03-16T10:29:16.887 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:29:16 smithi192 podman[72225]: 2024-03-16 10:29:16.581279519 +0000 UTC m=+1.828038219 container died ca4655a331f9ccf6f9ede3fb3a655e2f9f98ee23e0ef71aa95308657ddc1ca6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T10:29:17.167 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:29:16 smithi192 podman[72225]: 2024-03-16 10:29:16.986536838 +0000 UTC m=+2.233295542 container cleanup ca4655a331f9ccf6f9ede3fb3a655e2f9f98ee23e0ef71aa95308657ddc1ca6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-16T10:29:17.167 INFO:journalctl@ceph.mds.b.smithi192.stdout:Mar 16 10:29:16 smithi192 bash[72225]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-b 2024-03-16T10:29:17.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.b.service' 2024-03-16T10:29:17.408 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:29:17.408 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-16T10:29:17.408 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-16T10:29:17.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.e 2024-03-16T10:29:17.732 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:29:17 smithi192 systemd[1]: Stopping Ceph mds.e for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:29:18.166 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:29:17 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-e[66486]: 2024-03-16T10:29:17.732+0000 7f5313a85700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:29:18.166 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:29:17 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-e[66486]: 2024-03-16T10:29:17.732+0000 7f5313a85700 -1 mds.e *** got signal Terminated *** 2024-03-16T10:29:20.867 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:29:20 smithi192 podman[72344]: 2024-03-16 10:29:20.560537661 +0000 UTC m=+2.933534346 container died 05840130053b69d8d94798244f3b2530fa02c4df3058103085f08a2689556fce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-e, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-16T10:29:21.166 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:29:20 smithi192 podman[72344]: 2024-03-16 10:29:20.950222117 +0000 UTC m=+3.323218793 container cleanup 05840130053b69d8d94798244f3b2530fa02c4df3058103085f08a2689556fce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-16T10:29:21.167 INFO:journalctl@ceph.mds.e.smithi192.stdout:Mar 16 10:29:20 smithi192 bash[72344]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-e 2024-03-16T10:29:21.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.e.service' 2024-03-16T10:29:21.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:29:21.351 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-16T10:29:21.351 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-16T10:29:21.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.h 2024-03-16T10:29:21.665 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:29:21 smithi192 systemd[1]: Stopping Ceph mds.h for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:29:21.916 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:29:21 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-h[66794]: 2024-03-16T10:29:21.663+0000 7fdda4693700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:29:21.917 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:29:21 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-h[66794]: 2024-03-16T10:29:21.663+0000 7fdda4693700 -1 mds.h *** got signal Terminated *** 2024-03-16T10:29:24.916 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:29:24 smithi192 podman[72478]: 2024-03-16 10:29:24.564834255 +0000 UTC m=+2.998539333 container died cf66993ac3c766aee293cd2c6bf97bb533f648edff0eb828faaa538b741a20e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-h, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-16T10:29:25.290 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:29:25 smithi192 podman[72478]: 2024-03-16 10:29:25.022440188 +0000 UTC m=+3.456145255 container cleanup cf66993ac3c766aee293cd2c6bf97bb533f648edff0eb828faaa538b741a20e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T10:29:25.290 INFO:journalctl@ceph.mds.h.smithi192.stdout:Mar 16 10:29:25 smithi192 bash[72478]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-h 2024-03-16T10:29:25.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.h.service' 2024-03-16T10:29:25.436 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:29:25.436 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-16T10:29:25.436 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-16T10:29:25.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.k 2024-03-16T10:29:25.666 INFO:journalctl@ceph.mds.k.smithi192.stdout:Mar 16 10:29:25 smithi192 systemd[1]: Stopping Ceph mds.k for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:29:26.167 INFO:journalctl@ceph.mds.k.smithi192.stdout:Mar 16 10:29:25 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-k[66166]: 2024-03-16T10:29:25.759+0000 7fb1de70c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:29:26.167 INFO:journalctl@ceph.mds.k.smithi192.stdout:Mar 16 10:29:25 smithi192 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-k[66166]: 2024-03-16T10:29:25.759+0000 7fb1de70c700 -1 mds.k *** got signal Terminated *** 2024-03-16T10:29:28.865 INFO:journalctl@ceph.mds.k.smithi192.stdout:Mar 16 10:29:28 smithi192 podman[72598]: 2024-03-16 10:29:28.585656668 +0000 UTC m=+2.925051802 container died 841d624d452acf9b150a2ff341f04e565c10b63e4a9b8ee3f2b614aa7e2ce0bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-k, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:29:29.134 INFO:journalctl@ceph.mds.k.smithi192.stdout:Mar 16 10:29:28 smithi192 podman[72598]: 2024-03-16 10:29:28.968067287 +0000 UTC m=+3.307462409 container cleanup 841d624d452acf9b150a2ff341f04e565c10b63e4a9b8ee3f2b614aa7e2ce0bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-k, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307) 2024-03-16T10:29:29.135 INFO:journalctl@ceph.mds.k.smithi192.stdout:Mar 16 10:29:28 smithi192 bash[72598]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-k 2024-03-16T10:29:29.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.k.service' 2024-03-16T10:29:29.379 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:29:29.380 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-16T10:29:29.380 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-16T10:29:29.380 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.c 2024-03-16T10:29:29.720 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:29:29 smithi193 systemd[1]: Stopping Ceph mds.c for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:29:29.976 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:29:29 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-c[67182]: 2024-03-16T10:29:29.719+0000 7f6e5732c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:29:29.976 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:29:29 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-c[67182]: 2024-03-16T10:29:29.719+0000 7f6e5732c700 -1 mds.c *** got signal Terminated *** 2024-03-16T10:29:32.912 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:29:32 smithi193 podman[73321]: 2024-03-16 10:29:32.57067649 +0000 UTC m=+3.000676178 container died b02aaf5c0f0890b18d7e43862355b6180771997e89e128a78eb555065eb613a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T10:29:33.211 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:29:33 smithi193 podman[73321]: 2024-03-16 10:29:33.00196483 +0000 UTC m=+3.431964509 container cleanup b02aaf5c0f0890b18d7e43862355b6180771997e89e128a78eb555065eb613a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-c, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:29:33.211 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 16 10:29:33 smithi193 bash[73321]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-c 2024-03-16T10:29:33.427 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.c.service' 2024-03-16T10:29:33.465 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:29:33.465 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-16T10:29:33.465 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-16T10:29:33.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.f 2024-03-16T10:29:33.752 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:29:33 smithi193 systemd[1]: Stopping Ceph mds.f for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:29:34.253 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:29:33 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f[70835]: 2024-03-16T10:29:33.789+0000 7f21deab1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:29:34.253 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:29:33 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f[70835]: 2024-03-16T10:29:33.789+0000 7f21deab1700 -1 mds.f *** got signal Terminated *** 2024-03-16T10:29:36.873 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:29:36 smithi193 podman[73454]: 2024-03-16 10:29:36.576031055 +0000 UTC m=+2.895745404 container died f7a8a18435c45a99939187ad87731ffb5a1d2ce589d33cdecc5e79f4820a80cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-16T10:29:37.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:29:36 smithi193 podman[73454]: 2024-03-16 10:29:36.972412839 +0000 UTC m=+3.292127181 container cleanup f7a8a18435c45a99939187ad87731ffb5a1d2ce589d33cdecc5e79f4820a80cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:29:37.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 16 10:29:36 smithi193 bash[73454]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-f 2024-03-16T10:29:37.315 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.f.service' 2024-03-16T10:29:37.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:29:37.350 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-16T10:29:37.351 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-16T10:29:37.351 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.i 2024-03-16T10:29:37.502 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:29:37 smithi193 systemd[1]: Stopping Ceph mds.i for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:29:38.003 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:29:37 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-i[67505]: 2024-03-16T10:29:37.667+0000 7ff41132f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:29:38.003 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:29:37 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-i[67505]: 2024-03-16T10:29:37.667+0000 7ff41132f700 -1 mds.i *** got signal Terminated *** 2024-03-16T10:29:40.753 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:29:40 smithi193 podman[73575]: 2024-03-16 10:29:40.569333813 +0000 UTC m=+3.004279620 container died bb895664766a526a7a14f42a960733a0fd09f83cdc6f4580579443c20ad4a1c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-16T10:29:41.227 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:29:40 smithi193 podman[73575]: 2024-03-16 10:29:40.94228262 +0000 UTC m=+3.377228425 container cleanup bb895664766a526a7a14f42a960733a0fd09f83cdc6f4580579443c20ad4a1c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-i, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:29:41.227 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 16 10:29:40 smithi193 bash[73575]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-i 2024-03-16T10:29:41.356 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.i.service' 2024-03-16T10:29:41.390 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:29:41.391 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-16T10:29:41.391 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-16T10:29:41.391 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.l 2024-03-16T10:29:41.502 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:29:41 smithi193 systemd[1]: Stopping Ceph mds.l for c6ee6c60-e37c-11ee-95c9-87774f69a715... 2024-03-16T10:29:42.003 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:29:41 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-l[66560]: 2024-03-16T10:29:41.704+0000 7f44d8373700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T10:29:42.003 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:29:41 smithi193 ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-l[66560]: 2024-03-16T10:29:41.704+0000 7f44d8373700 -1 mds.l *** got signal Terminated *** 2024-03-16T10:29:44.896 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:29:44 smithi193 podman[73695]: 2024-03-16 10:29:44.570635062 +0000 UTC m=+2.968786073 container died c1f47e11a6a321e9e87ae0132b98f347aaa6d218618dbc78cd6bdc8825aa743d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-l, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T10:29:45.188 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:29:44 smithi193 podman[73695]: 2024-03-16 10:29:44.970809541 +0000 UTC m=+3.368960539 container cleanup c1f47e11a6a321e9e87ae0132b98f347aaa6d218618dbc78cd6bdc8825aa743d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T10:29:45.188 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 16 10:29:44 smithi193 bash[73695]: ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715-mds-l 2024-03-16T10:29:45.312 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c6ee6c60-e37c-11ee-95c9-87774f69a715@mds.l.service' 2024-03-16T10:29:45.345 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T10:29:45.345 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-16T10:29:45.345 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T10:29:45.545 INFO:teuthology.orchestra.run.smithi077.stdout:Deleting cluster with fsid: c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:29:48.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T10:29:48.191 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:29:50.778 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T10:29:50.969 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:29:53.359 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T10:29:53.393 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T10:29:53.428 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T10:29:53.456 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-16T10:29:53.459 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362/remote/smithi077/crash 2024-03-16T10:29:53.459 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/crash -- . 2024-03-16T10:29:53.489 INFO:teuthology.orchestra.run.smithi077.stderr:tar: /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T10:29:53.489 INFO:teuthology.orchestra.run.smithi077.stderr:tar: Error is not recoverable: exiting now 2024-03-16T10:29:53.491 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362/remote/smithi192/crash 2024-03-16T10:29:53.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/crash -- . 2024-03-16T10:29:53.528 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T10:29:53.529 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2024-03-16T10:29:53.530 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362/remote/smithi193/crash 2024-03-16T10:29:53.531 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/crash -- . 2024-03-16T10:29:53.573 INFO:teuthology.orchestra.run.smithi193.stderr:tar: /var/lib/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T10:29:53.574 INFO:teuthology.orchestra.run.smithi193.stderr:tar: Error is not recoverable: exiting now 2024-03-16T10:29:53.575 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-16T10:29:53.575 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T10:29:53.612 INFO:teuthology.orchestra.run.smithi077.stdout:2024-03-16T10:08:56.601034+0000 mon.a (mon.0) 208 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T10:29:53.613 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-16T10:29:53.613 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[SEC\]' /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T10:29:53.688 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]' /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T10:29:53.761 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[WRN\]' /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T10:29:53.833 INFO:teuthology.orchestra.run.smithi077.stdout:2024-03-16T10:08:56.601034+0000 mon.a (mon.0) 208 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T10:29:53.833 INFO:tasks.cephadm:Compressing logs... 2024-03-16T10:29:53.833 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T10:29:53.876 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T10:29:53.879 DEBUG:teuthology.orchestra.run.smithi193:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T10:29:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/cephadm.log 2024-03-16T10:29:53.906 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-16T10:29:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-16T10:29:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-16T10:29:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T10:29:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-16T10:29:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-16T10:29:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-16T10:29:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-16T10:29:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-16T10:29:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mgr.x.log: 90.2% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T10:29:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T10:29:53.909 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.0.log: 94.7% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-16T10:29:53.910 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log 2024-03-16T10:29:53.910 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.cephadm.log: 82.9% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T10:29:53.910 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T10:29:53.910 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T10:29:53.913 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-16T10:29:53.917 INFO:teuthology.orchestra.run.smithi077.stderr: 86.7% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T10:29:53.917 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-16T10:29:53.917 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-16T10:29:53.919 INFO:teuthology.orchestra.run.smithi193.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T10:29:53.919 INFO:teuthology.orchestra.run.smithi193.stderr:: No such file or directory 2024-03-16T10:29:53.919 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-16T10:29:53.919 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T10:29:53.919 INFO:teuthology.orchestra.run.smithi193.stderr:gzip/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.l.log: -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.f.log: 90.4% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.cephadm.log: 82.5% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log: 86.7% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr: 92.7% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-16T10:29:53.920 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-16T10:29:53.921 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T10:29:53.922 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T10:29:53.922 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-16T10:29:53.922 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T10:29:53.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-16T10:29:53.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-16T10:29:53.925 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.audit.log: /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-16T10:29:53.925 INFO:teuthology.orchestra.run.smithi192.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T10:29:53.925 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-16T10:29:53.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-16T10:29:53.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-16T10:29:53.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-16T10:29:53.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-16T10:29:53.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T10:29:53.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log 2024-03-16T10:29:53.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T10:29:53.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-16T10:29:53.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-16T10:29:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.9.log: 91.0% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T10:29:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-16T10:29:53.929 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-16T10:29:53.930 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.d.log: /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-16T10:29:53.957 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T10:29:54.055 INFO:teuthology.orchestra.run.smithi192.stderr: 82.5% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T10:29:54.057 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T10:29:54.057 INFO:teuthology.orchestra.run.smithi192.stderr: 86.7% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T10:29:54.161 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-16T10:29:54.184 INFO:teuthology.orchestra.run.smithi077.stderr: 94.9% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-16T10:29:54.190 INFO:teuthology.orchestra.run.smithi192.stderr: 91.0% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T10:29:54.410 INFO:teuthology.orchestra.run.smithi193.stderr: 94.6% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-16T10:29:54.431 INFO:teuthology.orchestra.run.smithi192.stderr: 94.8% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-16T10:29:54.543 INFO:teuthology.orchestra.run.smithi193.stderr: 91.8% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-16T10:29:54.616 INFO:teuthology.orchestra.run.smithi192.stderr: 94.9% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-16T10:29:55.085 INFO:teuthology.orchestra.run.smithi077.stderr: 90.4% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-16T10:29:55.160 INFO:teuthology.orchestra.run.smithi192.stderr: 91.8% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-16T10:29:58.264 INFO:teuthology.orchestra.run.smithi077.stderr: 94.4% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-16T10:29:59.283 INFO:teuthology.orchestra.run.smithi193.stderr: 94.5% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-16T10:29:59.911 INFO:teuthology.orchestra.run.smithi193.stderr: 94.5% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-16T10:30:00.609 INFO:teuthology.orchestra.run.smithi077.stderr: 91.7% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-16T10:30:03.716 INFO:teuthology.orchestra.run.smithi077.stderr: 91.8% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-16T10:30:03.784 INFO:teuthology.orchestra.run.smithi193.stderr: 91.9% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-16T10:30:04.988 INFO:teuthology.orchestra.run.smithi193.stderr: 91.6% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-16T10:30:05.491 INFO:teuthology.orchestra.run.smithi077.stderr: 91.7% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-16T10:30:06.246 INFO:teuthology.orchestra.run.smithi077.stderr: 91.6% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-16T10:30:06.248 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-03-16T10:30:06.248 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m12.360s 2024-03-16T10:30:06.248 INFO:teuthology.orchestra.run.smithi077.stderr:user 0m45.028s 2024-03-16T10:30:06.248 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m1.445s 2024-03-16T10:30:07.426 INFO:teuthology.orchestra.run.smithi193.stderr: 91.8% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-16T10:30:08.209 INFO:teuthology.orchestra.run.smithi193.stderr: 91.7% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-16T10:30:10.343 INFO:teuthology.orchestra.run.smithi193.stderr: 94.8% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-16T10:30:10.345 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-16T10:30:10.345 INFO:teuthology.orchestra.run.smithi193.stderr:real 0m16.452s 2024-03-16T10:30:10.345 INFO:teuthology.orchestra.run.smithi193.stderr:user 1m14.834s 2024-03-16T10:30:10.346 INFO:teuthology.orchestra.run.smithi193.stderr:sys 0m2.258s 2024-03-16T10:30:11.804 INFO:teuthology.orchestra.run.smithi192.stderr: 91.8% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-16T10:30:16.674 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-16T10:30:18.529 INFO:teuthology.orchestra.run.smithi192.stderr: 94.4% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-16T10:30:20.936 INFO:teuthology.orchestra.run.smithi192.stderr: 91.9% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-16T10:30:23.336 INFO:teuthology.orchestra.run.smithi192.stderr: 91.8% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-16T10:33:55.184 INFO:teuthology.orchestra.run.smithi192.stderr: 93.0% -- replaced with /var/log/ceph/c6ee6c60-e37c-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-16T10:33:55.186 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-16T10:33:55.186 INFO:teuthology.orchestra.run.smithi192.stderr:real 4m1.292s 2024-03-16T10:33:55.186 INFO:teuthology.orchestra.run.smithi192.stderr:user 4m19.974s 2024-03-16T10:33:55.186 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m8.700s 2024-03-16T10:33:55.186 INFO:tasks.cephadm:Archiving logs... 2024-03-16T10:33:55.187 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362/remote/smithi077/log 2024-03-16T10:33:55.188 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T10:34:02.257 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362/remote/smithi192/log 2024-03-16T10:34:02.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T10:34:45.079 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362/remote/smithi193/log 2024-03-16T10:34:45.081 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T10:34:54.723 INFO:tasks.cephadm:Removing cluster... 2024-03-16T10:34:54.723 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 --force 2024-03-16T10:34:54.916 INFO:teuthology.orchestra.run.smithi077.stdout:Deleting cluster with fsid: c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:34:55.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 --force 2024-03-16T10:34:55.499 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:34:56.067 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c6ee6c60-e37c-11ee-95c9-87774f69a715 --force 2024-03-16T10:34:56.259 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: c6ee6c60-e37c-11ee-95c9-87774f69a715 2024-03-16T10:34:56.588 INFO:tasks.cephadm:Removing cephadm ... 2024-03-16T10:34:56.588 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T10:34:56.608 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T10:34:56.625 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T10:34:56.641 INFO:tasks.cephadm:Teardown complete 2024-03-16T10:34:56.641 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-16T10:34:56.687 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-16T10:34:56.687 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T10:34:56.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T10:34:56.692 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T10:34:59.994 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-16T10:35:00.005 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-16T10:35:00.005 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T10:35:00.009 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T10:35:00.013 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T10:35:00.023 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-03-16T10:35:00.026 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-03-16T10:35:00.028 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T10:35:00.028 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-03-16T10:35:00.028 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 35 +1228us[+1268us] +/- 86ms 2024-03-16T10:35:00.028 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -904us[ -863us] +/- 73ms 2024-03-16T10:35:00.028 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -89us[ -49us] +/- 71ms 2024-03-16T10:35:00.028 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:35:00.030 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-03-16T10:35:00.034 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T10:35:00.034 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-16T10:35:00.035 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +1226us[+1226us] +/- 86ms 2024-03-16T10:35:00.035 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -907us[ -893us] +/- 73ms 2024-03-16T10:35:00.035 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 -75us[ -75us] +/- 71ms 2024-03-16T10:35:00.035 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:35:00.134 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T10:35:00.134 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-03-16T10:35:00.134 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +1215us[+1215us] +/- 86ms 2024-03-16T10:35:00.134 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -914us[ -882us] +/- 73ms 2024-03-16T10:35:00.135 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -113us[ -113us] +/- 71ms 2024-03-16T10:35:00.135 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T10:35:00.146 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-16T10:35:00.157 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-16T10:35:00.158 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-16T10:35:00.169 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T10:35:00.256 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T10:35:00.364 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T10:35:00.441 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T10:35:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:35:00.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T10:35:00.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:35:00.509 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T10:35:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:35:00.541 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-16T10:35:00.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-16T10:35:00.563 INFO:teuthology.task.internal:Duration was 2132.937330 seconds 2024-03-16T10:35:00.564 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-16T10:35:00.578 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-16T10:35:00.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T10:35:00.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T10:35:00.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T10:35:00.625 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T10:35:00.630 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T10:35:00.649 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T10:35:00.965 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-16T10:35:00.966 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-03-16T10:35:00.966 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T10:35:00.986 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-03-16T10:35:00.987 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T10:35:01.019 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-03-16T10:35:01.019 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T10:35:01.040 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-16T10:35:01.041 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T10:35:01.043 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T10:35:01.062 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T10:35:01.108 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-16T10:35:01.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T10:35:01.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T10:35:01.136 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T10:35:01.508 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-16T10:35:01.520 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-16T10:35:01.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T10:35:01.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T10:35:01.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T10:35:01.612 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-16T10:35:01.624 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T10:35:01.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T10:35:01.629 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T10:35:01.649 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-03-16T10:35:01.651 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-03-16T10:35:01.678 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-03-16T10:35:01.703 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T10:35:01.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:35:01.718 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T10:35:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:35:01.742 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T10:35:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T10:35:01.757 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-16T10:35:01.768 INFO:teuthology.task.internal:Transferring archived files... 2024-03-16T10:35:01.770 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362/remote/smithi077 2024-03-16T10:35:01.770 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T10:35:01.814 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362/remote/smithi192 2024-03-16T10:35:01.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T10:35:01.851 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362/remote/smithi193 2024-03-16T10:35:01.851 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T10:35:01.889 INFO:teuthology.task.internal:Removing archive directory... 2024-03-16T10:35:01.889 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T10:35:01.892 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T10:35:01.894 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T10:35:01.937 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-16T10:35:01.953 INFO:teuthology.task.internal:Not uploading archives. 2024-03-16T10:35:01.953 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-16T10:35:01.964 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-16T10:35:01.965 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T10:35:01.967 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T10:35:01.969 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T10:35:01.979 INFO:teuthology.orchestra.run.smithi077.stdout: 265940 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 10:35 /home/ubuntu/cephtest 2024-03-16T10:35:01.981 INFO:teuthology.orchestra.run.smithi192.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 10:35 /home/ubuntu/cephtest 2024-03-16T10:35:01.990 INFO:teuthology.orchestra.run.smithi193.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 10:35 /home/ubuntu/cephtest 2024-03-16T10:35:01.991 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-16T10:35:02.003 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-16T10:35:02.028 INFO:teuthology.nuke:Checking targets against current locks 2024-03-16T10:35:02.053 DEBUG:teuthology.nuke:shortname: smithi077 2024-03-16T10:35:02.053 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T10:35:02.077 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 09:51:16.111295', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ2FZxf5/LJXZGsV8pmiuXF3Hjd41v7mum45mz3oKFyNtE6oX42CH5CTeJSv3SOumW7IPJxQmor8Zu3lbPhAr+o='} 2024-03-16T10:35:02.081 DEBUG:teuthology.nuke:shortname: smithi192 2024-03-16T10:35:02.081 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T10:35:02.103 DEBUG:teuthology.nuke:shortname: smithi193 2024-03-16T10:35:02.103 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T10:35:02.109 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 09:51:16.112595', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAzFybMcil7BL7NFNbBAVHUUnrirtRtw+GwH9Tp8nJ1BcaKKkr6HkNgN0NTbwykXMLYWg03eBExHTXYoqm/Mk9Q='} 2024-03-16T10:35:02.137 INFO:teuthology.orchestra.console.smithi077:Power off 2024-03-16T10:35:02.137 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T10:35:02.160 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-03-16T10:35:02.160 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T10:35:02.178 INFO:teuthology.orchestra.console.smithi192:Power off 2024-03-16T10:35:02.178 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T10:35:02.189 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-03-16T10:35:02.202 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2024-03-16T10:35:02.203 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T10:35:02.225 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-16T10:35:06.190 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T10:35:06.226 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T10:35:06.300 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-03-16T10:35:06.337 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-16T10:35:07.045 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602362', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 09:51:16.113766', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHFlL6gCLB5Xa1zmqlMg5/mtIGvkcW8EeXZc+KA/vmSb70RX6OxRF1/PDtnZVk0mTww4mAdtjekKyUzNmXgndPg='} 2024-03-16T10:35:07.095 INFO:teuthology.orchestra.console.smithi193:Power off 2024-03-16T10:35:07.095 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T10:35:07.119 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2024-03-16T10:35:07.120 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T10:35:07.142 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-16T10:35:10.301 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T10:35:10.338 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T10:35:10.412 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-03-16T10:35:10.449 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2024-03-16T10:35:10.549 INFO:teuthology.orchestra.console.smithi192:Power off completed 2024-03-16T10:35:11.143 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T10:35:11.254 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-16T10:35:14.414 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T10:35:14.526 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-03-16T10:35:14.626 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-03-16T10:35:15.255 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T10:35:15.364 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2024-03-16T10:35:15.465 INFO:teuthology.orchestra.console.smithi193:Power off completed 2024-03-16T10:35:15.572 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} duration: 2132.937330007553 failure_reason: '"2024-03-16T10:08:56.601034+0000 mon.a (mon.0) 208 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-16T10:35:15.573 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T10:35:15.661 INFO:teuthology.run:FAIL